Docs: creates markdown reference

[lukegalbraithrussell]

Summary

This PR replaces the html reference with a md-based on. Tighter integration with docusaurus, and allows easier agent access as the md files are accessible. Also will now be trackable in Google Analaytics

Testing

Category

• slack_bolt.App and/or its core components
• slack_bolt.async_app.AsyncApp and/or its core components
• Adapters in slack_bolt.adapter
• Document pages under /docs
• Others

Requirements

Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to those rules.

• I've read and understood the Contributing Guidelines and have done my best effort to follow them.
• I've read and agree to the Code of Conduct.
• I've run ./scripts/install_all_and_run_tests.sh after making the changes.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.54%. Comparing base (69bd90f) to head (ff4a9f2).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1565   +/-   ##
=======================================
  Coverage   91.54%   91.54%           
=======================================
  Files         228      228           
  Lines        7285     7285           
=======================================
  Hits         6669     6669           
  Misses        616      616           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[WilliamBergamin]

Thanks for working on this 💯

Left some comments/questions before we can move forward with this 🙏

[lukegalbraithrussell]

@WilliamBergamin Using a new library that's a bit more consistent. Still a lot of custom code, but that's just the nature of the beast given our unique docusaurus setup - it's mostly on that docusaurus side, not the python repo side

[WilliamBergamin]

I like like the direction here. Moving the reference to Markdown is a clear win over the pdoc3 HTML blob:

• the pages are agent-readable
• they can be tracked in GA
• they slot into Docusaurus natively.

griffe is also the right parser choice (it resolves our __all__ re-exports and parses the Google-style docstrings correctly).

My main concern is scripts/generate_api_docs.py as a long-term maintenance surface: ~640 lines of hand-rolled code is a lot to keep working as slack_bolt, griffe, and Docusaurus all evolve. I dug into how the other tools ship their reference and I think we can remove a big chunk of the hand rolled code.

node-slack-sdk is the closest working model. It ships a committed Markdown reference that docs.slack.dev consumes, and it does two things we're currently doing the hard way:

1. No hand-built sidebar.json. It uses a Docusaurus autogenerated sidebar (dirName + a generated-index link); deno-slack-sdk and slack-cli do the same. Worth noting: the docs site imports exactly one _sidebar.json per tool today and has no mechanism to import a second reference/sidebar.json, so the artifact we generate can't actually be wired in as-is. Switching to an autogenerated Reference category in our existing docs/english/_sidebar.json removes ~90 lines and the whole "how does the site pick this up" question.
2. Signatures in fenced code blocks. MDX leaves < and { literal inside code fences, so the escaping only really needs to cover prose docstring text, not signatures. That shrinks the escaping/hazard surface a lot.

What I'd suggest (keep griffe, cut the rest):

• Drop the sidebar.json generation; add an autogenerated Reference category to docs/english/_sidebar.json (node/deno/slack-cli pattern).
• Move the imperative signature/section rendering into a small set of Jinja2 templates (the mkdocstrings/quartodoc approach). Same output, roughly half the code, and future tweaks become template edits.
• Keep a minimal prose-level MDX escaper + the hazard gate, and wire the hazard gate into CI.
• Pin griffe in a requirements file (it's currently pip install -U griffe, so output can shift under us), and add a CI job that regenerates and fails on git diff drift, since the generated tree is committed and nothing guards it right now.

Happy to pair on the template refactor if useful.

[WilliamBergamin]

Thanks for working on this 🙏 but I'm starting to think we might want to break this out into 3 PRs

Splitting this PR into three stacked PRs (less custom code)

This PR migrates the API reference from pdoc3 HTML to a griffe-generated Markdown tree under docs/english/reference/, wired into Docusaurus via an autogenerated category in docs/english/_sidebar.json. The direction and griffe as the parser are good 🚀 the concern is that scripts/generate_api_docs.py (573 lines) is a hand-rolled maintenance surface, and the generated tree is committed but unguarded.

Much of the script is defensive logic that only exists to absorb docstring formatting problems at generation time:

• _reflow_indented_code: rewrites indented code blocks into fenced blocks. Dead code today (every docstring example is already fenced, so it never fires).
• _check_mdx_hazards: fails generation on MDX/JSX/ESM hazards (<, import, export at column 0), but only when someone manually regenerates — never on a PR.
• _escape_mdx: escapes </{ in prose so Docusaurus (which parses .md as MDX) doesn't read prose as JSX/JS. This is legitimate rendering and stays.

The generator can't shrink while it's the only thing guarding these invariants. The fix: move the guarantee into a test that runs on every PR, clean up the docstrings that feed it, and then the generator becomes a plain griffe→Jinja2 renderer.

Proposed decomposition into three stacked PRs, each independently valuable:

PR 1 (ruff)  →  PR 2 (docstring test + fixes)  →  PR 3 (= this PR, slimmed generator + tree)
   off main            stacks on PR 1                 rebase this branch after 1 & 2 land

---

PR 1: Adopt Ruff as linter + formatter (replaces Black + Flake8)

Goal: Ruff can lint and format docstrings (docstring code examples + a curated D set). Doing it first means all mechanical docstring reformatting happens once, before the tree is regenerated in PR 3.

Changes

- `pyproject.toml`: add `[tool.ruff]` (`line-length = 125`, `target-version = "py37"`), `[tool.ruff.lint]` at parity with `.flake8` (keep `E`/`W`/`F`, `ignore = ["E402", "F841", "F821"]`; `W503` is a formatter concern in Ruff, drop it), enable a curated docstring set via `[tool.ruff.lint.pydocstyle] convention = "google"` while ignoring noisy missing-docstring rules (`D100`–`D107`) and the manual-only ones (`D205` missing-blank-after-summary ×90, `D417` undocumented-param ×7 — these need real prose work, deferred to PR 2's docstring pass). Add `[tool.ruff.format] docstring-code-format = true`. Remove `[tool.black]`. - Delete `.flake8` (config now lives in `pyproject.toml`). - `requirements/dev_tools.txt`: replace `black` + `flake8` with a `ruff` pin (keep `mypy`). - `scripts/format.sh`: `black slack_bolt/ tests/` → `ruff format slack_bolt/ tests/`. - `scripts/lint.sh`: `flake8 slack_bolt/ && flake8 examples/` → `ruff check slack_bolt/ examples/`. - CI (`.github/workflows/ci-build.yml`): add a `ruff format --check` step to the existing `lint` job (CI enforces no formatter today — this closes that gap). - Apply the one-time mechanical result of `ruff format` + `ruff check --fix` (incl. the auto-fixable D-rules) across the repo. Large but mostly cosmetic; does **not** fence unfenced code — that's PR 2.

Verification: ./scripts/format.sh --no-install clean, ./scripts/lint.sh --no-install green, ./scripts/run_mypy.sh --no-install unchanged, full test suite passes.

---

PR 2: Docstring rendering test (griffe, latest-Python only) + source fixes

Goal: encode the invariants _check_mdx_hazards enforces — as a test that runs on every PR — and fix the docstrings that violate them. This is what lets PR 3's generator drop its defensive logic.

New tests/docstring/test_rendering.py: load the package with griffe exactly as the generator does — griffe.load("slack_bolt", search_paths=[REPO_ROOT], docstring_parser=griffe.Parser.google) — walk every documented object's docstring, and fail if any contains, outside a ``` fence:

• a line at column 0 starting with < (JSX) or import /export (ESM) — the _check_mdx_hazards invariant, checked at the source instead of post-render;
• a bare indented code block (blank line + ≥4-space indent) — the precondition that keeps _reflow_indented_code unnecessary.

Why griffe, not stdlib ast: it enumerates the same objects the generator documents (respecting __all__ re-export inlining) and parses docstrings the same way, so the test's scope == the generator's scope. The fence-aware hazard scanner can be lifted almost verbatim out of _check_mdx_hazards, so no logic is duplicated or left behind in the generator.

Latest-Python-only gating (required — griffe 2.x needs a modern Python and can't run on 3.7/3.8):

• Add griffe (pinned) to a new requirements/docs.txt (shared by the test and, in PR 3, the generator).
• Add a dedicated CI job docstring mirroring lint/typecheck: setup-python ${{ env.LATEST_SUPPORTED_PY }} (3.14) → pip install . -r requirements/docs.txt → pytest tests/docstring/.
• The 3.7–3.14 unittest matrix runs specific paths, never a blanket tests/, so it won't pick this up. Guard the module with pytest.importorskip("griffe") + a sys.version_info skip so the codecov job and local run_tests.sh (which default-collect tests/) skip it cleanly instead of erroring.

Source fixes (bundled here): fence any remaining unfenced code examples and fix any column-0 MDX hazards in slack_bolt/** docstrings so the new test is green — these are the docstring edits currently spread across this branch. Re-run ruff format after fencing.

Verification: pytest tests/docstring/ green on 3.14; skipped (not errored) where griffe is absent.

---

PR 3: Slim generator (griffe → Jinja2) + the migration (this PR)

Goal: rebase this branch onto main (after PR 1 + PR 2 land) and reduce it to the Markdown migration with the simplest generator, now that PR 2 guards the invariants.

Generator rewrite (scripts/generate_api_docs.py):

• Keep griffe as the loader/parser and the page-model building (_load_package, _iter_modules, _documented_members, _inlined_export_target, _doc_id/_doc_route, _write_pages).
• Move the imperative Markdown rendering (_render_object, _render_docstring, signature builders, _render_body) into Jinja2 templates under scripts/templates/ (mkdocstrings/quartodoc-style).
• Delete _reflow_indented_code (dead) and _check_mdx_hazards (now the PR 2 test).
• Keep MDX escaping as a small Jinja escape_mdx filter applied to prose only; signatures stay inside fenced blocks (MDX leaves </{ literal inside fences).
• Add jinja2 (pinned) to requirements/docs.txt; update scripts/generate_api_docs.sh to pip install -r requirements/docs.txt instead of the current unpinned pip install -U griffe (pins griffe).

Migration payload (already on this branch): the regenerated docs/english/reference/ tree, the _sidebar.json autogenerated Reference category, docs/english/reference_redirects.json, and deletion of the old docs/reference/**/*.html.

Guard the committed tree (new CI job): a docs-drift job on 3.14 that installs requirements/docs.txt, runs python scripts/generate_api_docs.py, and git diff --exit-code -- docs/english/reference — fails if the committed tree drifts from source.

Not byte-identical: because this is a Jinja2 rewrite, the tree differs cosmetically from the current output. Correctness is established by the PR 2 test (inputs), the docs-drift job (output matches source), and a Docusaurus build check rather than a byte diff.

---

Notes

• Order: PR 1 → PR 2 → PR 3. PR 2 stacks on PR 1 (both touch docstrings; PR 1 formats, PR 2 fences). PR 3 needs PR 2's test merged so the slimmed generator is safe.
• requirements/docs.txt is introduced in PR 2 (griffe) and extended in PR 3 (jinja2) — one pinned home for doc tooling that's currently pip install -U'd ad hoc.
• The docstring test and generator span sync and async modules automatically (griffe loads the whole package); just ensure PR 2's docstring source fixes cover async_* counterparts.

[WilliamBergamin]

it feels like we are getting somewhere slowly 🥇

I'm still super uneasy with the amount of custom code required to do this, we have no idea how brittle this might be, I left a few comments maybe they will help with this

[WilliamBergamin]

In this file lest make the code as self explanatory as possibel and keep the comment and docstring to an absolute minimum, recently I've been finding that a lot of comments in normally very readable code tend to create bias in AI tools 🤔

I think we can get ride of most comments and docstring in this file

[WilliamBergamin]

This type of stuff is just super brittle 😅 we need to find a way to make it stable, we may need many unit tests to make sure the script works like we need it

Alternately if we can find a way to implement this using templates then it could make it more stable and may not need as many unit tests

[WilliamBergamin]

Maybe we can use something like https://github.com/pycqa/bandit to make this more reliable and not build this ourselves

[lukegalbraithrussell]

@WilliamBergamin I replaced some of the manual code with a library: https://markdown-it-py.readthedocs.io/en/latest/. I also added tests

[WilliamBergamin]

On the _check_mdx_hazards MDX-safety gate

I went looking into whether we could offload this to an existing package instead of hand-rolling the check, starting with bandit, which came to mind. Sharing what I found, since I think it points at a more robust approach:

• bandit won't work here. It's a static analyzer for Python source (it builds an AST and flags eval, hardcoded secrets, shell=True, etc.). It has no concept of Markdown, code fences, JSX, or ESM, so it can't validate generated .md at all.
• There's no Python package that validates MDX, either. MDX is a JavaScript-ecosystem format (Docusaurus compiles it with micromark + acorn); on PyPI, "mdx" almost always means a python-markdown extension, which is unrelated. So our regex gate is necessarily approximating rules that only the real MDX compiler defines, so it will drift from acorn's actual behavior over time.

Given that, rather than growing the heuristic, what do you think about validating with the real MDX compiler in a CI-only GitHub Actions job? @mdx-js/mdx's compile() is ground truth: it uses the same acorn parser the docs.slack.dev build does, so "does it throw?" is exactly "will Docusaurus choke on it?". Rough shape:

• A new mdx-validate job in ci-build.yml (Node stays entirely in CI, never in local dev or the Python test suite): actions/setup-node, then npm ci, then run a small .mjs that globs docs/english/reference/**/*.md, compile()s each with { format: "mdx" }, and exits non-zero on the first failure.
• Validate the committed tree. The existing docs-reference drift job already guarantees it matches generation, so this stays Node-only and decoupled.
• One gotcha: every generated file leads with YAML frontmatter, so the compile needs remark-frontmatter (plus remark-gfm/remark-directive to mirror Docusaurus v3 defaults) to avoid false positives.

If we go this route and drop the Python heuristic entirely, the generation script sheds a nice chunk of code: the whole "Safety gate" section comes out:

• _check_mdx_hazards() and its call in main()
• _code_line_numbers() (only used by the gate)
• the _MDX_ESM_RE regex (only used by the gate)

Everything else stays: _MD_PARSER and the markdown_it import are still used by _reflow_indented_code, import re is still used by the other prose regexes, and the markdown-it-py pin in requirements/docs.txt stays (used directly, and transitive via griffe2md -> mdformat anyway).

Alternatively we keep _check_mdx_hazards as a fast, Node-free pre-check at generation time and treat the CI compile as the authoritative gate. Both are reasonable. Happy to open a follow-up PR with the job if this seems worthwhile. Non-blocking.

[WilliamBergamin]

I don't think we need this comment

Suggested change

	# griffe2md -- renders griffe objects to Markdown (pulls in griffelib + jinja2 + mdformat)
	# griffe2md

[WilliamBergamin]

Same here

Suggested change

	# griffelib -- griffe's PyPI distribution (import name `griffe`); pinned to match griffe2md's rendering
	# griffelib

[WilliamBergamin]

Same here

Suggested change

	# markdown-it-py -- CommonMark tokenizer used by generate_api_docs.py to locate code
	# blocks during the MDX-hazard check (also a transitive dep via griffe2md -> mdformat)
	# markdown-it-py