Repository navigation
warm start for the docs build - #115
Merged
Merged
Conversation
The documentation build runs every example in src/microstructpy/examples
(docs/source/sphinx_gallery/plot_demos.py globs and runs them all), and
nothing is cached between Read the Docs builds. The periodic examples
added on this branch pushed that past the 15 minute limit of the Read
the Docs free plan.
basalt_circle is the single largest consumer at 216 s, 29% of the 741 s
the examples take in total, and more than any of the new examples. About
180 s of that is meshing, driven by mesh_max_edge_length on a domain of
diameter 10.
Measured on the same machine:
0.01 (before) 216.2 s
0.02 114.4 s
0.03 83.7 s
0.05 63.8 s
0.02 halves the example. Coarser values give much less back, since what
remains is positioning 1300 seeds, plotting them and verification. The
rendered figure is unchanged at publication size, with slightly coarser
triangles inside the large olivine grains.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A build on Read the Docs is stopped at fifteen minutes on the free plan.
The documentation build runs every example in src/microstructpy/examples
through docs/source/sphinx_gallery/plot_demos.py, which takes about
twenty minutes on that class of hardware, so the build cannot finish. The
examples produce the same figures every time, so this moves the work to
the documentation workflow, which has no such limit, and has Read the
Docs restore what it produced.
Sphinx-Gallery already skips a script whose hash matches the .md5 beside
the output of an earlier build. The difficulty is that plot_demos.py
finds the examples with glob and names none of them, so its hash does not
move when an example is edited or added, nor when the library that meshes
them changes. Caching on that hash alone would serve stale figures and
never report it.
docs/example_digest.py closes that gap. It digests the example inputs and
the library source and keeps the result in plot_demos.py as
EXAMPLES_DIGEST, so editing any of them changes the hash of the script
and the examples run again. The documentation check runs it with --check
and fails if the stored digest is out of date.
docs/example_cache.py packs and restores the Sphinx-Gallery output and
the PNG files the examples write, which the pages embed with figure::.
An archive carries the digest it was built from and restore refuses one
that does not match the working tree, so a build that cannot be warm runs
the examples and may time out rather than publish stale figures.
Measured locally, with the examples run on the same machine:
cold build 655.4 s
warm build 6.7 s
The documentation workflow publishes an archive from its html job to the
docs-cache release, and asks Read the Docs to build again so the archive
is picked up. That rebuild needs a READTHEDOCS_TOKEN secret. Without it
the step says so and does nothing, and the build has to be restarted by
hand once the archive is published.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The name of the archive was worked out in one job step and written to a file, and fetched in the next. If the first step failed, the second fetched an empty name, and the message about a missing archive hid it. Keep the name in a variable in a single step, and say which of the two things happened: no archive was published for this commit, or one was published and refused because its digest is not the digest of this commit. Both leave the examples to run, and the second should not happen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The matrix cancels the other two formats as soon as one fails, so a html build that stops partway through the examples takes latex and epub with it and there is nothing to compare it against. python_package.yaml already sets fail-fast: false for the same reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The file declared version="0.5". The XML specification allows only
'1.' followed by digits, production [26], so the declaration has never
been well formed. Older versions of Expat accepted it and newer ones
reject it, which is why this has only now started failing:
xml.parsers.expat.ExpatError: XML declaration not well-formed:
line 1, column 15
Column 15 is where the version value starts. The example is read by
xmltodict in cli.input2dict, so the failure stops the whole gallery and
takes the documentation build with it.
This has been in the file since before this branch, and it is not
reproducible everywhere: Expat 2.6.1 parses it without complaint, so a
developer can run the examples locally and see nothing wrong while the
same commit fails on a runner with a newer Expat. That is what made the
documentation check look flaky, failing in one format and passing in
another on the same commit.
Editing an example changes EXAMPLES_DIGEST, so the archive of figures is
invalidated and the examples run again, which is the intended behavior.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kip-hart
added this pull request to stack #116
October 8, 2026 04:27
The eight commits merged from develop change the library and
periodic_3D.xml, so the figures the examples produce are no longer the
ones in the published archive. The digest check caught it, which is what
it is for:
EXAMPLES_DIGEST is 314fadcaec7cb6e5, the examples digest to
2d39355b71ddd257.
The documentation workflow publishes an archive for the new digest and
asks Read the Docs to build again with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kip-hart
marked this pull request as ready for review
October 9, 2026 06:38
rimoli
approved these changes
Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Read the Docs stops a build at fifteen minutes on the free plan. The
documentation build runs every example in
src/microstructpy/examplestoproduce the figures the pages embed, which takes about twenty minutes on that
class of hardware, so the build cannot finish. Every build that gets as far as
running the examples is stopped at 901 seconds, including
3697c378, the headof #114 as I write this.
The examples produce the same figures every time. This moves that work to the
documentation workflow, which has no time limit, and has Read the Docs restore
what it produced.
How it works
Sphinx-Gallery already skips a script whose hash matches the
.md5storedbeside the output of an earlier build. The difficulty is that
docs/source/sphinx_gallery/plot_demos.pyfinds the examples withglobandnames none of them, so its hash does not move when an example is edited or
added, nor when the library that meshes them changes. Caching on that hash
alone would serve stale figures and never report it.
docs/example_digest.pycloses that gap. It digests the example inputs and thelibrary source and keeps the result in
plot_demos.pyasEXAMPLES_DIGEST.Editing an example changes the digest, which changes the hash of the script,
which makes Sphinx-Gallery run the examples again. The documentation check runs
it with
--checkand fails if the stored digest is out of date.docs/example_cache.pypacks and restores the Sphinx-Gallery output and thePNG files the examples write, which the pages embed with
figure::. An archivecarries the digest it was built from, and
restorerefuses one that does notmatch the working tree. A build that cannot be warm runs the examples and may
time out rather than publish stale figures.
The
htmljob publishes an archive to thedocs-cacherelease, named by thedigest, and asks Read the Docs to build again so the archive is picked up. A
post_installjob on Read the Docs fetches the archive for the digest of thecommit it is building, and restores it if the digests match.
Measurements
Locally, with the examples run on the same machine:
On Read the Docs, across the commits of this branch:
The timeouts are the intended behavior when no archive matches the commit. The
first build after a change to an example or to the library runs the examples,
fails, and goes green once the
htmljob has published an archive and askedfor a rebuild.
Two other fixes
mesh_max_edge_lengthinbasalt_circle.xmlgoes from 0.01 to 0.02. Thatexample was the largest single consumer of the build at 216 s, 29% of the 741 s
the examples take in total. About 180 s of it was meshing, driven by this
setting on a domain of diameter 10:
0.02 halves it, and coarser values give much less back. The rendered figure is
unchanged at publication size, with slightly coarser triangles inside the large
olivine grains.
elliptical_grains.xmldeclared<?xml version="0.5" ... ?>. The XMLspecification allows only
1.followed by digits, production [26], so thedeclaration has never been well formed. Expat 2.6.1 accepts it and newer
versions reject it, which is why this passed locally and failed on some runners,
and why the documentation check looked flaky: it failed in one format and passed
in another on the same commit. The file is older than this branch, and it was
the only one in the repository with an illegal version.
fail-fast: falseon the documentation matrix, so one failing format no longercancels the other two.
python_package.yamlalready sets it.Setting it up
The rebuild needs a
READTHEDOCS_TOKENsecret, from Settings and then APITokens on Read the Docs. Without it the step says so and does nothing, and a
build that has to wait for an archive must be restarted by hand.
Nothing is committed to the repository. The archives are release assets, 13.9 MB
each, so
docs/source/auto_examplesand the output directories of the examplesstay ignored.
Notes
The archive is only restored when its digest is the digest of the commit being
built, so an edited example is never served with the figures of an older one.
Three archives are published so far, one per digest.
docs/README.rstcovers building the documentation, how the cache works, andhow to publish one by hand.