Repository navigation
feat(content): convert Office documents to Markdown locally - #181
Merged
Merged
Conversation
Add OfficeProcessor for DOCX, PPTX, XLSX/XLSM, XLS and ODS. Word headings, lists and tables, one section per slide in presentation order, one table per sheet. Zip containers are checked against an entry and uncompressed size budget before inflating (TM-DOS-008). Markdown requests now put text/markdown first in Accept so sites that serve Markdown on request return it instead of HTML.
Clippy 1.99's double_must_use lint fires on code generated by async-trait 0.1.89 for every async trait, failing Lint on main too. 0.1.92 no longer emits the redundant must_use.
Contributor
Author
|
The Lint failure isn't caused by this PR's code. CI's stable toolchain is now Rust 1.99, and its new Fix in 1ab5437: bump Generated by Claude Code |
4 tasks done
chaliy
added a commit
that referenced
this pull request
Oct 12, 2026
…182) <!-- claude-projects-attribution: {"github_login":"chaliy"} --> _Requested by **Михайло** · [project thread](https://claude.ai/code/project/chan_01G7CAdCSnaq5jtbh8B8KZfn?thread=cmsg_01G7CAdCSnaq5jtbh8B8KZfnV6z5srnRUUV63aDKwESstv)_ ## What changed PDF and Office conversion is now configurable and pluggable instead of two fixed processors. - One `DocumentProcessor` runs an ordered chain of `DocumentConverter` backends, set per format. When a backend fails, doesn't support the input, or returns nothing usable (for example a scanned PDF), the next one runs. - Built-in backends sit behind Cargo features: - `docs-pdf` (default on): `pdf-inspector`. - `docs-office` (default on): the local Office converter from #181. - `docs-cloudflare` (opt-in): Cloudflare Workers AI `toMarkdown`. - `docs-http` (opt-in): a generic multipart `HttpConverter` for self-hosted services such as Docling or MarkItDown. - Remote backends upload documents to a third party. They run only with an explicit `allow_remote(true)`, and never under `ToolBuilder::hardened()`. Uploads go through a pluggable `ConverterClient`, so a host can route them through its own egress. Tokens are redacted from debug output and errors. - Documents have their own download cap, `max_document_bytes` (25 MB default), separate from `max_body_size`. Formats are confirmed by magic bytes before any converter runs. - The same config is available from every surface: - Library: `ToolBuilder::document_conversion`, plus `ToolBuilder::content_processor` to add custom processors ahead of the built-ins. - CLI and MCP: `--convert`, `--converter`, `--allow-remote-conversion`, `--max-document-bytes`. The Cloudflare converter reads `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`. - Python: keyword arguments of the same names. - For extension, the public traits are `DocumentConverter`, `FormatDetector`, `ConverterClient` and `ContentProcessor`. - When a document format is recognised but no converter handles it, the error now names the format. ## Why Design: [Fetchkit: PDF and Office document conversion design](https://claude.ai/code/artifact/ac389330-4779-4b72-8cdb-49341e563d06). Conversion should be configurable, feature-flagged and pluggable. It should allow third-party services such as Cloudflare, and leave room for markkit or other backends later. ## Before / After Before: Office conversion was a fixed `OfficeProcessor` (from #181), with no way to turn it off, choose a backend or add a remote one. After, default build (unchanged output for supported formats): ``` $ fetchkit fetch https://calibre-ebook.com/downloads/demos/demo.docx quality_score: 1.00 extraction_method: "office_docx" --- # Demonstration of DOCX support in calibre ``` Restricting formats: ``` $ fetchkit fetch https://calibre-ebook.com/downloads/demos/demo.docx --convert pdf -o json "error": "Binary content is not supported. ... DOCX conversion is not enabled in this fetchkit configuration; enable a document converter to convert it." ``` Remote converter without its feature: ``` $ fetchkit fetch …/demo.docx --converter cloudflare --allow-remote-conversion Error: converter `cloudflare` requires the docs-cloudflare feature ``` The Cloudflare path is covered end to end with wiremock: the tool downloads the DOCX, uploads it to a mock `toMarkdown`, and returns `cloudflare_tomarkdown`. It has not been run against a real Cloudflare account yet. ## Risk - Medium. - Unreleased API change: `PdfProcessor` and `OfficeProcessor` are replaced by `DocumentProcessor` plus `PdfInspectorConverter` and `OfficeConverter`. - `pdf-inspector`, `calamine`, `zip` and `quick-xml` are now optional dependencies. Default features keep them on. - Default behaviour changes slightly in two ways: - Documents may now be up to 25 MB, where the old cap was 10 MB. - `application/msword` and `application/epub+zip` are now treated as binary. - New threats are documented as TM-LEAK-006 (document disclosure to a remote converter) and TM-LEAK-007 (converter credentials). ### Checklist - [x] Unit tests are passed (`cargo test --workspace`; clippy with default, no-default, `docs-all` and single-backend feature sets) - [x] Smoke tests are passed (real DOCX and PDF through the release CLI) - [x] Documentation is updated (README, CHANGELOG, docs/hardening.md) - [x] Knowledge is up to date and not in conflict (fetchers, tool contract, threat model, log; `check_okf.py` passes) 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi --- _Generated by [Claude Code](https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi)_ Co-authored-by: Claude <noreply@anthropic.com>
4 tasks done
chaliy
added a commit
that referenced
this pull request
Oct 12, 2026
#183) ## What changed markkit (0.1.0, crates.io) becomes the default Office backend. The in-crate `office` converter from #181 is deprecated and off by default. - **New `docs-markkit` feature, on by default.** It adds `MarkkitConverter` (name `markkit`, local) for DOCX, XLSX/XLSM, XLS and ODS. - Extraction methods are `markkit_docx`, `markkit_xlsx`, `markkit_xls` and `markkit_ods`. - Limits are tuned with `MarkkitConverter::with_markkit(Markkit::builder()…)`. `Markkit` and `MarkkitBuilder` are re-exported from `fetchkit::documents`. - **Feature defaults:** `docs-pdf` + `docs-office` becomes `docs-pdf` + `docs-markkit`. `docs-all` now includes markkit and excludes office. - **`OfficeConverter` is `#[deprecated]`.** It still works behind `docs-office`, and when that feature is enabled it runs after markkit as a fallback. - **Settings, CLI and Python:** `--converter markkit` / `converters=["markkit"]` are accepted. When a feature is missing, the error says so. - markkit is built without its HTTP client, and the only reqwest in the tree is fetchkit's own. ## Why The repo owner decided that #181's in-crate Office parsing is not the implementation to keep. Office parsing moves to markkit, which fetchkit's design already planned as a pluggable backend. markkit has zip-bomb, archive and spreadsheet-grid limits, and it skips hidden sheets by default. ## Before / After Before (default build): `fetchkit fetch …/demo.docx` gave `extraction_method: "office_docx"`. After: ``` $ fetchkit fetch https://calibre-ebook.com/downloads/demos/demo.docx quality_score: 1.00 extraction_method: "markkit_docx" --- # Demonstration of DOCX support in calibre … | ITEM | NEEDED | | Books | 1 | ``` Local files: `.xlsx` gives `markkit_xlsx` (with a `hidden_sheet_skipped` warning), `.xls` gives `markkit_xls`, `.ods` gives `markkit_ods`, and a large sheet carries a `truncated_sheet` warning. Legacy `.doc` is still rejected with "DOC conversion is not enabled". ## Risk - Medium. Unreleased API change on top of #182. - PPTX and XLSB are no longer converted by default, because markkit doesn't support them yet. Enabling `docs-office` restores them. - Spreadsheet titles are no longer extracted. - markkit's default sheet caps are looser: 10,000 rows × 256 columns, where `office` used 1,000 × 50. Output is still capped by `max_body_size`. - New transitive crates: `cfb`, `uuid`, `regex`, `anyhow`, `mime`, `unicode-segmentation`. ### Checklist - [x] Unit tests are passed: `cargo test --workspace`, plus feature sets with no defaults, `docs-markkit` only, and `docs-all,docs-office`. clippy `-D warnings` passes across the same sets. - [x] Smoke tests are passed: the release CLI on a real DOCX, and local XLSX/XLS/ODS files. - [x] Documentation is updated: README, CHANGELOG (Added and Deprecated), docs/hardening.md. - [x] Knowledge is up to date and not in conflict: fetchers, tool contract, TM-DOS-008, and the log. `check_okf.py` passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi --- _Generated by [Claude Code](https://claude.ai/code/session_01Y2vd61x8kbPtmDxze9Q4Gi)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
Before: fetching a Word, PowerPoint or Excel file returned "Binary content is not supported". ODS files came back as garbled zip bytes, because their media type was not on the binary list.
After: these documents come back as Markdown, converted locally with no network call and no account:
## Slide N: titlesection per slide, in presentation order.## sheetMarkdown table per sheet, with the first row as header. Dates are shown as ISO, and output is capped at 1,000 rows by 50 columns with asheet_truncatedwarning.The processor matches on media type, and on the file extension when a file is served as
application/octet-streamorapplication/zip. The document title comes fromdocProps/core.xml.Markdown requests now also send
Accept: text/markdown, text/html;q=0.9, …with Markdown first. Sites that serve Markdown on request (Cloudflare's Markdown for Agents) then return it instead of HTML.Why
Agents regularly hit Office files on the web, and today they get nothing back. This is the local alternative to Cloudflare's hosted
toMarkdownconverter: documents never leave the process, and it works the same in every embedding.Before / After
fetchkit fetch https://go.microsoft.com/fwlink/?LinkID=521962(an XLSX served asapplication/octet-stream):calibre's
demo.docxproduces# Demonstration of DOCX support in calibre, followed by its paragraphs and##sections. The python-pptxtest.pptxproduces## Slide 1: Presentation Title Textfollowed bySubtitle Text. Before this change, all three returnedBinary content is not supported.Risk
calamine(MIT),zip8 (MIT) andquick-xml0.41 (MIT). These are the versions calamine already uses, so no crate is duplicated.Checklist
cargo test --workspace; 9 new processor tests, plus integration tests for DOCX and the Markdown-first Accept header)check_okf.pypasses)