Skip to content

feat(content): convert Office documents to Markdown locally - #181

Merged
chaliy merged 2 commits into
mainfrom
claude/project-thread-ezx9rz
Oct 12, 2026
Merged

chaliy merged 2 commits into
mainfrom
claude/project-thread-ezx9rz

Conversation

@chaliy

@chaliy chaliy commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Requested by Михайло · project thread

What changed

Before: fetching a Word, PowerPoint or Excel file returned "Binary content is not supported". ODS files came back as garbled zip bytes, because their media type was not on the binary list.

After: these documents come back as Markdown, converted locally with no network call and no account:

  • DOCX: headings (Heading 1-6, Title), nested lists, and tables.
  • PPTX: one ## Slide N: title section per slide, in presentation order.
  • XLSX, XLSM, XLS, ODS: one ## sheet Markdown table per sheet, with the first row as header. Dates are shown as ISO, and output is capped at 1,000 rows by 50 columns with a sheet_truncated warning.

The processor matches on media type, and on the file extension when a file is served as application/octet-stream or application/zip. The document title comes from docProps/core.xml.

Markdown requests now also send Accept: text/markdown, text/html;q=0.9, … with Markdown first. Sites that serve Markdown on request (Cloudflare's Markdown for Agents) then return it instead of HTML.

Why

Agents regularly hit Office files on the web, and today they get nothing back. This is the local alternative to Cloudflare's hosted toMarkdown converter: documents never leave the process, and it works the same in every embedding.

Before / After

fetchkit fetch https://go.microsoft.com/fwlink/?LinkID=521962 (an XLSX served as application/octet-stream):

extraction_method: "office_spreadsheet"
---
## Sheet1

| Segment | Country | Product | Discount Band | Units Sold | … | Date | Month Number | Month Name | Year |
| --- | --- | --- | --- | --- | … | --- | --- | --- | --- |
| Government | Canada | Carretera | None | 1618.5 | … | 2014-01-01 | 1 | January | 2014 |

calibre's demo.docx produces # Demonstration of DOCX support in calibre, followed by its paragraphs and ## sections. The python-pptx test.pptx produces ## Slide 1: Presentation Title Text followed by Subtitle Text. Before this change, all three returned Binary content is not supported.

Risk

  • Medium: parsing untrusted archives. Zip containers are rejected if they have more than 10,000 entries or more than 200 MB of declared uncompressed content, before anything is inflated. Parts are read through a 64 MB cap, and parsing shares the PDF processor's blocking pool and two-document cap. This is recorded as TM-DOS-008 in the threat model.
  • Changing the Accept preference may make some servers return Markdown where they used to return HTML. That is the intended effect.
  • New dependencies: calamine (MIT), zip 8 (MIT) and quick-xml 0.41 (MIT). These are the versions calamine already uses, so no crate is duplicated.

Checklist

  • Unit tests are passed (cargo test --workspace; 9 new processor tests, plus integration tests for DOCX and the Markdown-first Accept header)
  • Smoke tests are passed (the real DOCX, XLSX and PPTX files above, via the CLI)
  • Documentation is updated (README, CHANGELOG, tool description)
  • Knowledge is up to date and not in conflict (fetchers, tool contract, threat model, log; check_okf.py passes)

Add OfficeProcessor for DOCX, PPTX, XLSX/XLSM, XLS and ODS. Word headings,
lists and tables, one section per slide in presentation order, one table
per sheet. Zip containers are checked against an entry and uncompressed
size budget before inflating (TM-DOS-008).

Markdown requests now put text/markdown first in Accept so sites that
serve Markdown on request return it instead of HTML.
@chaliy chaliy self-assigned this Oct 12, 2026
Clippy 1.99's double_must_use lint fires on code generated by
async-trait 0.1.89 for every async trait, failing Lint on main too.
0.1.92 no longer emits the redundant must_use.

chaliy commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

The Lint failure isn't caused by this PR's code. CI's stable toolchain is now Rust 1.99, and its new clippy::double_must_use lint flags code that async-trait 0.1.89 generates. That hits every async trait already in the crate (ContentProcessor, Fetcher, FileSaver, HttpTransport), so main fails the same way.

Fix in 1ab5437: bump async-trait to 0.1.92, which no longer generates the duplicate must_use. With that, cargo +stable clippy --workspace --all-targets -- -D warnings (1.99) is clean locally, and cargo test --workspace passes.


Generated by Claude Code

@chaliy
chaliy merged commit 2d02590 into main Oct 12, 2026
12 checks passed
@chaliy
chaliy deleted the claude/project-thread-ezx9rz branch October 12, 2026 17:46
chaliy added a commit that referenced this pull request Oct 12, 2026
…182)

<!-- claude-projects-attribution: {"github_login":"chaliy"} -->
_Requested by **Михайло** · [project
thread](https://claude.ai/code/project/chan_01G7CAdCSnaq5jtbh8B8KZfn?thread=cmsg_01G7CAdCSnaq5jtbh8B8KZfnV6z5srnRUUV63aDKwESstv)_

## What changed
PDF and Office conversion is now configurable and pluggable instead of
two fixed processors.

- One `DocumentProcessor` runs an ordered chain of `DocumentConverter`
backends, set per format. When a backend fails, doesn't support the
input, or returns nothing usable (for example a scanned PDF), the next
one runs.
- Built-in backends sit behind Cargo features:
  - `docs-pdf` (default on): `pdf-inspector`.
  - `docs-office` (default on): the local Office converter from #181.
  - `docs-cloudflare` (opt-in): Cloudflare Workers AI `toMarkdown`.
- `docs-http` (opt-in): a generic multipart `HttpConverter` for
self-hosted services such as Docling or MarkItDown.
- Remote backends upload documents to a third party. They run only with
an explicit `allow_remote(true)`, and never under
`ToolBuilder::hardened()`. Uploads go through a pluggable
`ConverterClient`, so a host can route them through its own egress.
Tokens are redacted from debug output and errors.
- Documents have their own download cap, `max_document_bytes` (25 MB
default), separate from `max_body_size`. Formats are confirmed by magic
bytes before any converter runs.
- The same config is available from every surface:
- Library: `ToolBuilder::document_conversion`, plus
`ToolBuilder::content_processor` to add custom processors ahead of the
built-ins.
- CLI and MCP: `--convert`, `--converter`, `--allow-remote-conversion`,
`--max-document-bytes`. The Cloudflare converter reads
`CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`.
  - Python: keyword arguments of the same names.
- For extension, the public traits are `DocumentConverter`,
`FormatDetector`, `ConverterClient` and `ContentProcessor`.
- When a document format is recognised but no converter handles it, the
error now names the format.

## Why
Design: [Fetchkit: PDF and Office document conversion
design](https://claude.ai/code/artifact/ac389330-4779-4b72-8cdb-49341e563d06).
Conversion should be configurable, feature-flagged and pluggable. It
should allow third-party services such as Cloudflare, and leave room for
markkit or other backends later.

## Before / After
Before: Office conversion was a fixed `OfficeProcessor` (from #181),
with no way to turn it off, choose a backend or add a remote one.

After, default build (unchanged output for supported formats):
```
$ fetchkit fetch https://calibre-ebook.com/downloads/demos/demo.docx
quality_score: 1.00
extraction_method: "office_docx"
---
# Demonstration of DOCX support in calibre
```
Restricting formats:
```
$ fetchkit fetch https://calibre-ebook.com/downloads/demos/demo.docx --convert pdf -o json
"error": "Binary content is not supported. ... DOCX conversion is not enabled in this fetchkit configuration; enable a document converter to convert it."
```
Remote converter without its feature:
```
$ fetchkit fetch …/demo.docx --converter cloudflare --allow-remote-conversion
Error: converter `cloudflare` requires the docs-cloudflare feature
```
The Cloudflare path is covered end to end with wiremock: the tool
downloads the DOCX, uploads it to a mock `toMarkdown`, and returns
`cloudflare_tomarkdown`. It has not been run against a real Cloudflare
account yet.

## Risk
- Medium.
- Unreleased API change: `PdfProcessor` and `OfficeProcessor` are
replaced by `DocumentProcessor` plus `PdfInspectorConverter` and
`OfficeConverter`.
- `pdf-inspector`, `calamine`, `zip` and `quick-xml` are now optional
dependencies. Default features keep them on.
- Default behaviour changes slightly in two ways:
  - Documents may now be up to 25 MB, where the old cap was 10 MB.
- `application/msword` and `application/epub+zip` are now treated as
binary.
- New threats are documented as TM-LEAK-006 (document disclosure to a
remote converter) and TM-LEAK-007 (converter credentials).

### Checklist
- [x] Unit tests are passed (`cargo test --workspace`; clippy with
default, no-default, `docs-all` and single-backend feature sets)
- [x] Smoke tests are passed (real DOCX and PDF through the release CLI)
- [x] Documentation is updated (README, CHANGELOG, docs/hardening.md)
- [x] Knowledge is up to date and not in conflict (fetchers, tool
contract, threat model, log; `check_okf.py` passes)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi

---
_Generated by [Claude
Code](https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi)_

Co-authored-by: Claude <noreply@anthropic.com>
chaliy added a commit that referenced this pull request Oct 12, 2026
#183)

## What changed
markkit (0.1.0, crates.io) becomes the default Office backend. The
in-crate `office` converter from #181 is deprecated and off by default.

- **New `docs-markkit` feature, on by default.** It adds
`MarkkitConverter` (name `markkit`, local) for DOCX, XLSX/XLSM, XLS and
ODS.
- Extraction methods are `markkit_docx`, `markkit_xlsx`, `markkit_xls`
and `markkit_ods`.
- Limits are tuned with
`MarkkitConverter::with_markkit(Markkit::builder()…)`. `Markkit` and
`MarkkitBuilder` are re-exported from `fetchkit::documents`.
- **Feature defaults:** `docs-pdf` + `docs-office` becomes `docs-pdf` +
`docs-markkit`. `docs-all` now includes markkit and excludes office.
- **`OfficeConverter` is `#[deprecated]`.** It still works behind
`docs-office`, and when that feature is enabled it runs after markkit as
a fallback.
- **Settings, CLI and Python:** `--converter markkit` /
`converters=["markkit"]` are accepted. When a feature is missing, the
error says so.
- markkit is built without its HTTP client, and the only reqwest in the
tree is fetchkit's own.

## Why
The repo owner decided that #181's in-crate Office parsing is not the
implementation to keep. Office parsing moves to markkit, which
fetchkit's design already planned as a pluggable backend. markkit has
zip-bomb, archive and spreadsheet-grid limits, and it skips hidden
sheets by default.

## Before / After
Before (default build): `fetchkit fetch …/demo.docx` gave
`extraction_method: "office_docx"`.

After:
```
$ fetchkit fetch https://calibre-ebook.com/downloads/demos/demo.docx
quality_score: 1.00
extraction_method: "markkit_docx"
---
# Demonstration of DOCX support in calibre
…
| ITEM | NEEDED |
| Books | 1 |
```
Local files: `.xlsx` gives `markkit_xlsx` (with a `hidden_sheet_skipped`
warning), `.xls` gives `markkit_xls`, `.ods` gives `markkit_ods`, and a
large sheet carries a `truncated_sheet` warning. Legacy `.doc` is still
rejected with "DOC conversion is not enabled".

## Risk
- Medium. Unreleased API change on top of #182.
- PPTX and XLSB are no longer converted by default, because markkit
doesn't support them yet. Enabling `docs-office` restores them.
- Spreadsheet titles are no longer extracted.
- markkit's default sheet caps are looser: 10,000 rows × 256 columns,
where `office` used 1,000 × 50. Output is still capped by
`max_body_size`.
- New transitive crates: `cfb`, `uuid`, `regex`, `anyhow`, `mime`,
`unicode-segmentation`.

### Checklist
- [x] Unit tests are passed: `cargo test --workspace`, plus feature sets
with no defaults, `docs-markkit` only, and `docs-all,docs-office`.
clippy `-D warnings` passes across the same sets.
- [x] Smoke tests are passed: the release CLI on a real DOCX, and local
XLSX/XLS/ODS files.
- [x] Documentation is updated: README, CHANGELOG (Added and
Deprecated), docs/hardening.md.
- [x] Knowledge is up to date and not in conflict: fetchers, tool
contract, TM-DOS-008, and the log. `check_okf.py` passes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi

---
_Generated by [Claude
Code](https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi)_

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant