diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e3b46ecf..4718bd4d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -99,7 +99,7 @@ jobs: sudo apt-get install -y graphviz - name: Build docs run: | - make -C docs html + make -C docs html llms touch docs/build/html/.nojekyll - name: Deploy docs to gh-pages uses: peaceiris/actions-gh-pages@v4 diff --git a/AGENTS.md b/AGENTS.md index dd862621..0082366a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,7 @@ # pgraph-python — Agent Instructions +> **Audience:** agents working on pgraph's source code. If you're helping someone *use* pgraph, read https://petercorke.github.io/pgraph-python/llms.txt instead (or `llms-full.txt` for the complete documentation). + Part of the RVC ecosystem. **Read [rvc-ecosystem/AGENTS.md](https://github.com/petercorke/rvc-ecosystem/blob/main/AGENTS.md) first** — it defines shared conventions: repo ownership, math invariants, dependency boundaries, git/PR workflow, code standards, tech-debt tracking. This file only adds what's specific to this repo. | | | @@ -8,10 +10,19 @@ Part of the RVC ecosystem. **Read [rvc-ecosystem/AGENTS.md](https://github.com/p | Nickname | pgraph-python | | Owner | Peter Corke (`petercorke`) | | Default branch | `main` | -| Contribution model | Branch → PR; direct push to `main` at Peter's discretion | +| Contribution model | Branch → PR; `main` is protected: CI must pass and the branch must be up to date (`gh pr update-branch`) | ## Notes specific to this repo -- Minimal graph/path-planning utility package — no internal ecosystem dependencies. -- Used by `robotics-toolbox-python` (path/roadmap planning) and `machinevision-toolbox-python`. -- Pure Python — builds with Hatch (`hatchling`) directly. +- Minimal graph/path-planning package. Pure Python, builds with Hatch (`hatchling`); Python 3.9+. Depends on `numpy`, `matplotlib` and `spatialmath-python`; `ansitable` and `scipy` arrive through spatialmath. **Do not add scipy to `pyproject.toml`**: it is imported lazily, only inside `Graph.build_kdtree()`. +- Used, unpinned (`"pgraph-python"`), by `robotics-toolbox-python` (`PRMPlanner`, `RRTPlanner`, `LatticePlanner`, `PoseGraph`, which subclass `UVertex`/`DVertex`) and `machinevision-toolbox-python` (`BundleAdjust`). A breaking change needs a deprecation cycle with a working fallback (`warnings.warn(..., DeprecationWarning, stacklevel=2)`); run their pgraph-using tests before a release. +- **Layout.** `src/pgraph/PGraph.py` holds `Graph`, `UGraph`, `DGraph`, `Vertex`, `Edge` and `SE2Metric`; `exporters.py` the `export()` formats. There is one `Vertex` class: a vertex has no direction, the graph decides how an edge is recorded (`UGraph`/`DGraph` implement the hooks `_attach`, `_detach`, `_incoming` and `_endpoints_match`). `UVertex`/`DVertex` are deprecated aliases, by module `__getattr__`. +- **Records that must agree.** A vertex has `_edgelist` (leaving, or incident if undirected) and `_inlist` (entering, directed only); the graph has `_vertexlist`, `_vertexdict` (names) and `_edges` (an insertion-ordered dict). `g.validate()` checks them all; use it in tests of anything that edits the structure. +- **Two staleness flags, consumed independently.** `_connectivitychange` (components, reset by `_graphcolor`) and `_spatialchange` (the k-d tree). Set both wherever a vertex is added or removed; only connectivity cares about edges. Never share one flag. +- **`Vertex.__setattr__` is hot.** It exists to tell the graph that `coord` was assigned. Anything run per vertex (`Vertex.__init__`, `add_vertex`, components) writes `vertex.__dict__` directly to skip it; keep doing that, and declare new attributes at class level so mypy knows them. +- **Pickle and `copy` use a flat description** (`__getstate__`/`__setstate__`: vertices, edges by index, replayed through `_attach`); `subgraph` reuses it. Derived or transient state on a graph (the k-d tree) must be excluded there. A bare vertex or edge refuses to be pickled or copied. +- **Tests** are `unittest` classes in `tests/`, run with `pytest`; write expected values by hand or from an oracle in the test, never from the code under test. Import names explicitly (a star import is a Codacy `F405`); route `pickle` through one helper marked `# nosec B301 nosemgrep`. Codecov gates patch coverage, so every changed line needs a test. `mypy src` is a CI check. +- **Performance claims are measured.** `examples/benchmark.py` (`--md`, `--directed`) produces the README tables; regenerate them after a performance change and compare with `v0.6.5` (`git archive v0.6.5 src`), not with a development build. +- **Docs** are Sphinx (`make -C docs html`; `-n` for nitpicky) with `sphinx-pyrunblock` `.. runblock:: pycon` examples that execute at build. `docs/llms_intro.md` is the hand-written part of `llms.txt`: update it when the API changes; `docs/llms_txt.py` builds `llms.txt` and `llms-full.txt` from it and the toctrees (`make html llms`), and CI publishes them with the site. +- **Release.** Version is in `pyproject.toml`, history in `CHANGELOG.md`. Publishing is `.github/workflows/publish.yml` on a GitHub Release (trusted publishing, environment `pypi`): it refuses unless the tag equals the version and the commit is on `main`. Don't tag or upload by hand. +- `examples/london.json` is OpenStreetMap data (© OpenStreetMap contributors, ODbL); keep the attribution with it. diff --git a/README.md b/README.md index 436c9e07..d0ff4260 100644 --- a/README.md +++ b/README.md @@ -91,7 +91,7 @@ Map data © OpenStreetMap contributors, available under the [Open Database Licen ## Using pgraph -Full details are given in the [online documentation](https://petercorke.github.io/pgraph-python). +Full details are given in the [online documentation](https://petercorke.github.io/pgraph-python). For an AI assistant there is a short summary, [llms.txt](https://petercorke.github.io/pgraph-python/llms.txt), and the complete documentation as one file, [llms-full.txt](https://petercorke.github.io/pgraph-python/llms-full.txt). ### Properties and methods of the graph diff --git a/docs/Makefile b/docs/Makefile index d0c3cbf1..8abe31bd 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -12,7 +12,11 @@ BUILDDIR = build help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) -.PHONY: help Makefile +# llms.txt and llms-full.txt for AI assistants, written into build/html; run after html +llms: + SPHINXBUILD="$(SPHINXBUILD)" python llms_txt.py + +.PHONY: help Makefile llms # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). diff --git a/docs/llms_intro.md b/docs/llms_intro.md new file mode 100644 index 00000000..8969ac8b --- /dev/null +++ b/docs/llms_intro.md @@ -0,0 +1,43 @@ +pgraph is a pure-Python package of graph classes: directed and undirected graphs of vertices connected by edges, with edge costs and path planning (breadth-first, uniform-cost and A*). Vertices can be *embedded*, given a coordinate in a space of any dimension; edge costs then come from a distance metric (L2, L1, SE2 or your own), and the graph can be queried spatially, plotted and exported. Self-loops and parallel edges are allowed. It requires Python 3.9 or later and depends on NumPy, Matplotlib and spatialmath-python. Install with `pip install pgraph-python`; import it as `pgraph`. + +## Quick start + +```python +from pgraph import UGraph + +g = UGraph(dim=2) # DGraph for directed; Graph(directed=flag) chooses +a = g.add_vertex([0, 0], name="a") # returns the Vertex +b = g.add_vertex([3, 4], name="b") +c = g.add_vertex([6, 0], name="c") +g.add_edge(a, b) # cost defaults to the metric distance, 5.0 +g.add_edge("b", "c", cost=7) # vertices or their names; or give the cost + +path, length, _ = g.path_Astar("a", "c") # None if there is no path +print([v.name for v in path], length) # ['a', 'b', 'c'] 12.0 +g.plot() +g.highlight_path(path) +``` + +## Key points + +- **Classes.** `UGraph` (undirected) and `DGraph` (directed) share the base class `Graph`, which is concrete: `Graph(directed=True)` is a `DGraph`. There is one `Vertex` class for both and one `Edge` class. `UVertex` and `DVertex` are deprecated aliases of `Vertex`. Constructor arguments are keyword-only: `UGraph(dim=2, metric="L2", heuristic=None, vertex_class=None, edge_class=None, verbose=False)`; `UGraph(2)` is a `TypeError`. +- **Vertices.** `g.add_vertex(coord=None, name=None)` returns the vertex. Default names are `"#0"`, `"#1"`, ..., names are unique, and `g["a"]` or `g[0]` finds a vertex. `for v in g` iterates in the order added; `g.n`, `g.ne` and `g.nc` are the numbers of vertices, edges and components. `g.remove_vertex(v)` and `g.remove_edge(e)` remove them, with their edges. +- **Edges.** `g.add_edge(v1, v2, cost=None)`; with no `cost` it is the metric distance between the coordinates, and `None` if the vertices have none (planning then raises `ValueError`). `g.edges()` and `v.edges()` return new lists in the order added. `e.v1` and `e.v2` are read-only. `v.adjacent()` is the neighbours, `v.successors()` and `v.predecessors()` are the directed ones, `v.edgeto(w)` the cheapest edge to `w` and `v.edges_to(w)` all of them. A self-loop of an undirected vertex is in `v.edges()` twice and counts 2 in its degree. +- **Directed graphs.** `degree` is in plus out (`in_degree`, `out_degree`); `nc` and `component` are weakly connected, ignoring direction. +- **Planning.** `path_BFS(S, G)` returns `(path, length)` with the fewest edges; `path_UCS(S, G)` and `path_Astar(S, G)` return `(path, length, parents)` of least cost. `S` and `G` are vertices or names, and all three return `None` when there is no path, so check before unpacking. +- **Metrics.** `"L2"` (default), `"L1"`, `"SE2"` for `(x, y, theta)` poses, `SE2Metric(weight)` to weight the angle, or a function of one difference vector. A* uses `heuristic` (default: the metric), which must be in the same units as the edge costs and must not exceed the true cost to go, or the path is not the shortest. +- **Spatial queries** on an embedded graph: `closest(point)` returns `(vertex, distance)`, `near(point, radius)`, `within(box)` with `box` a 2 x n array of lower and upper corner, `coordinates()` the `(dim, n)` matrix of coordinates, and `subgraph(vertices)`, which takes vertices, names or a boolean mask and returns a new graph. `build_kdtree()` is an optional index (it needs scipy) that `closest` and `near` then use. +- **Components and matrices.** `g.component(c)` is the vertices in component number `c`, `v.label` is the component of a vertex, `g.samecomponent(v1, v2)`. `adjacency()`, `incidence()`, `laplacian()`, `degree()` and `connectivity()` return dense NumPy arrays. +- **Output.** `g.plot()` (arrowheads for a directed graph, arcs for parallel edges, circles for loops), `highlight_path(path)`, and `g.export(file)` for GraphViz dot, Mermaid, GraphML, ELK JSON or an image (images need the GraphViz executables). +- **Subclassing.** `class Foo(Vertex)` and `UGraph(vertex_class=Foo, edge_class=Bar)` make the graph build them; a subclass `__init__` must pass `coord` and `name` (vertices) or `v1, v2, cost, data` (edges) to its parent. +- **Copying.** `g.copy()` and `pickle` work for a whole graph; a vertex or edge cannot be copied or pickled on its own, and a graph whose metric is a `lambda` cannot be pickled. + +## Common mistakes + +- `UGraph(6)` raises `TypeError`: write `UGraph(dim=6)`. +- Unpacking the result of a planner without checking for `None`. +- Coordinates in one unit and costs in another (metres against kilometres): the A* heuristic is then too large and the path found is not the shortest. +- `g.component(v)` takes a component *number*; use `v.label` for the component of a vertex. +- Assigning `e.v1 = ...` fails, they are read-only; `add_vertex` raises `ValueError` for a name already used or a vertex that is already in a graph. +- After `build_kdtree()`, change a coordinate by assigning (`v.coord = x`, `v.coord += x`), not in place (`v.coord[0] = x`), which the index cannot see. +- `g.edges()` is a copy: changing the list does not change the graph. diff --git a/docs/llms_txt.py b/docs/llms_txt.py new file mode 100644 index 00000000..5a28a5d7 --- /dev/null +++ b/docs/llms_txt.py @@ -0,0 +1,170 @@ +"""Write llms.txt and llms-full.txt for the documentation site. + +``llms.txt`` (see https://llmstxt.org) is a short Markdown file that tells AI assistants +what the package is, how to use it, and where its documentation lives. ``llms-full.txt`` +is the whole documentation as plain text, so an assistant can read it in a single fetch. + +The package is small, so ``llms.txt`` is mostly prose: the hand-written ``llms_intro.md`` +(what it does, a quick start, key points, common mistakes) followed by a generated list +of links. ``llms-full.txt`` is this Sphinx site rendered by Sphinx's ``text`` builder, so +the autodoc API reference is included, in the order of the toctrees, which are followed +from ``index.rst`` through the nested pages. Run after the HTML build:: + + make html llms # from docs/ + +The files are written into ``build/html`` so they are published with the site. +""" + +from __future__ import annotations + +import os +import re +import shutil + +# Used only to re-invoke `sphinx-build`, resolved via shutil.which() below; this is a +# local/CI docs-build script, not code that runs against untrusted input. +import subprocess # nosec B404 +import sys +from importlib.metadata import metadata +from pathlib import Path + +DOCS = Path(__file__).resolve().parent +SOURCE = DOCS / "source" +BUILD = DOCS / "build" +TEXT = BUILD / "text" +HTML = BUILD / "html" +INTRO = DOCS / "llms_intro.md" + +# One-line descriptions of the pages listed in the toctrees +PAGE_NOTES = { + "index": "overview", + "intro": "tutorial: building a graph, vertices, edges, plotting, path planning, directed graphs", + "api": "the API, in three parts: the common methods, vertices and edges, and the policy page", + "common": "the methods that UGraph and DGraph share, by task", + "construction": "UGraph, DGraph, Graph, constructors from a dict or an adjacency matrix, distance metrics", + "modification": "adding and removing vertices and edges", + "analysis": "counts, adjacency, degree, components, coordinates, spatial queries, subgraphs", + "representations": "adjacency, incidence, degree and Laplacian matrices", + "visualization": "plot(), its options, highlighting vertices, edges and paths", + "export": "writing a graph as GraphViz dot, Mermaid, GraphML, ELK or an image", + "planning": "path_BFS, path_UCS and path_Astar", + "edges_vertices": "the data structures: Vertex and Edge, and the edge lists of a vertex", + "policy": "object lifecycle: what may be done with a vertex or edge outside a graph", +} + +# Links that belong in llms.txt but are not pages of this site +EXTRA_LINKS = [ + ("Source code", "https://github.com/petercorke/pgraph-python", "GitHub repository"), + ("PyPI", "https://pypi.org/project/pgraph-python/", "install with pip install pgraph-python"), + ("Changelog", "https://github.com/petercorke/pgraph-python/blob/main/CHANGELOG.md", "release history and how to migrate from 0.6"), + ("Examples", "https://github.com/petercorke/pgraph-python/tree/main/examples", "Queensland and central London route-planning notebooks, and a benchmark"), + ("RVC ecosystem", "https://github.com/petercorke/rvc-ecosystem", "how pgraph relates to the Robotics and Machine Vision Toolboxes"), + ("Contributing", "https://github.com/petercorke/pgraph-python/blob/main/AGENTS.md", "repo conventions and workflow, for agents working on pgraph's source code"), +] + + +def toctree_entries(document: Path) -> list[str]: + """Document names listed in the toctrees of one document, in order. + + :param document: an ``.rst`` file + :return: document names such as ``"common"`` + """ + entries, in_toctree = [], False + for line in document.read_text().splitlines(): + if line.startswith(".. toctree::"): + in_toctree = True + elif in_toctree and line.strip() and not line.startswith((" ", "\t")): + in_toctree = False # the directive's indented body has ended + elif in_toctree and line.strip() and not line.strip().startswith(":"): + entries.append(line.strip()) + return entries + + +def toctree_pages(index: Path) -> list[str]: + """All the documents below a master document, depth first, in toctree order. + + The pages of this site are nested: ``index`` lists ``api``, which lists + ``common``, which lists the pages of the common methods. + + :param index: the master document, ``index.rst`` + :return: document names such as ``"construction"``, each once + """ + pages: list[str] = [] + + def walk(document: Path) -> None: + for name in toctree_entries(document): + if name not in pages: + pages.append(name) + child = document.parent / f"{name}.rst" + if child.exists(): + walk(child) + + walk(index) + return pages + + +def title(text: str) -> str: + """The page title: the first non-blank line that isn't an overline of punctuation.""" + for line in text.splitlines(): + if line.strip() and not re.fullmatch(r"[*=\-~#^]+", line.strip()): + return line.strip() + return "" + + +def main() -> None: + # Read the name, summary and documentation URL from the installed package's metadata + # (built from pyproject.toml). Reading pyproject.toml directly would need tomllib, + # which is Python 3.11+, and the Pages workflow builds the docs on 3.9. + meta = metadata("pgraph-python") + urls = dict(u.split(", ", 1) for u in meta.get_all("Project-URL")) + base = urls["Documentation"].rstrip("/") + "/" + name, summary = meta["Name"], meta["Summary"] + + # SPHINXBUILD matches the override convention of docs/Makefile's + # `SPHINXBUILD ?= sphinx-build` -- resolved via shutil.which() to a full path. + sphinxbuild_name = os.environ.get("SPHINXBUILD", "sphinx-build") + sphinxbuild = shutil.which(sphinxbuild_name) + if sphinxbuild is None: + raise SystemExit(f"sphinx-build not found: {sphinxbuild_name!r}") + subprocess.run( # nosec B603 -- sphinxbuild resolved via shutil.which() above # nosemgrep: python.lang.security.audit.dangerous-subprocess-use-audit.dangerous-subprocess-use-audit + [sphinxbuild, "-q", "-b", "text", str(SOURCE), str(TEXT)], # nosemgrep: python.lang.security.audit.dangerous-subprocess-use-tainted-env-args.dangerous-subprocess-use-tainted-env-args + check=True, + ) + pages = ["index"] + toctree_pages(SOURCE / "index.rst") + texts = {p: (TEXT / f"{p}.txt").read_text() for p in pages} + + # llms.txt: prose introduction, then links + index = [ + f"# {name}", + "", + f"> {summary}.", + "", + INTRO.read_text().strip(), + "", + "## Documentation", + "", + f"The whole documentation, including the API reference, is available as a single text file: [llms-full.txt]({base}llms-full.txt).", + "", + *(f"- [{title(texts[p])}]({base}{p}.html): {PAGE_NOTES.get(p, '')}" for p in pages), + "", + "## Optional", + "", + *(f"- [{label}]({url}): {note}" for label, url, note in EXTRA_LINKS), + "", + ] + HTML.mkdir(parents=True, exist_ok=True) + (HTML / "llms.txt").write_text("\n".join(index)) + + # llms-full.txt: the prose introduction, then every page + def block(url: str, body: str) -> str: + return f"\n{'=' * 72}\nPage: {url}\n{'=' * 72}\n\n{body.strip()}\n" + + parts = [f"# {name}: {summary}\n\nDocumentation: {base}\n\n{INTRO.read_text().strip()}\n"] + parts += [block(f"{base}{p}.html", texts[p]) for p in pages] + full = re.sub(r"\n{3,}", "\n\n", "".join(parts)) # the text builder leaves runs of blank lines + (HTML / "llms-full.txt").write_text(full) + print(f"llms.txt: {len(pages)} pages; llms-full.txt: {len(full) // 1000} kB", file=sys.stderr) + + +if __name__ == "__main__": + main()