Repository navigation
Allow reflection table experiment metadata over 64 KiB - #56
Merged
Merged
Conversation
… 64 KiB The old format caps an attribute at 64 KiB, so the identifiers vanished past about 4090 experiments. The newer format keeps small groups' links in insertion order, so the reader now asks for name order explicitly.
The writer ran with HDF5 errors silenced and never checked the results, so a failed attribute write produced a file without the attribute. The reader also leaked every identifier string it loaded.
jbeilstenedmands
approved these changes
Oct 12, 2026
jbeilstenedmands
left a comment
jbeilstenedmands
left a comment
Contributor
There was a problem hiding this comment.
Thanks, confirmed this now fixes the issue I saw in a real use case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writing a reflection table with more than about 4090 experiments
produces a file with no
identifiersattribute and no error. The filewas created with the default format bound,
which caps each attribute at 64 KiB, and the metadata writer
ran under
H5ErrorSilencerwithout checking its HDF5 calls,so the rejected attribute was skipped silently.
This PR writes the file in the HDF5 1.8 format, which moves large
attributes into dense storage
with no size cap. Attribute names and types are unchanged, and
reading the file needs HDF5 1.8 or newer. The newer format returns a
group's columns in insertion order rather than name order, so the
reader now asks for name order explicitly.
Changes
ReflectionTable::writecreates its file with an HDF5 1.8 lowerformat bound, via a new helper in
h5utils.hpp.write_experiment_metadatachecks every HDF5 call and throws onfailure.
read_experiment_metadatachecks its reads and frees the identifierstrings, which previously leaked on every load.