Skip to content

Omit blank fields from the model, as both writers already do - #242

Merged
samuelduchesne merged 6 commits into
mainfrom
fix/blank-fields-unset-183
Oct 26, 2026
Merged

samuelduchesne merged 6 commits into
mainfrom
fix/blank-fields-unset-183

Conversation

@samuelduchesne

@samuelduchesne samuelduchesne commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #183

Problem

For an object type with no extensible group, the IDF parser stored every blank field as "" (numeric fields too). write_idf and write_epjson both drop those blanks, so re-parsing idfkit's own IDF output did not equal the original. For example, Zone, Z1, 0, 0, 0, 0, 1, 1, , , , , ; had 12 keys in to_dict() before the write and 7 after.

Change

IDFParser._build_data_dict_cached now skips blank values instead of storing "", for the fixed fields of every object type. This matches what the extensible-group path already did after #182, and what idfkit-js does: packages/core/src/parse/idf.ts skips blank fields and drops trailing blank padding. The loop now zips the named fields, so it no longer walks the extensible tail.

IDFObject._set_field now treats None and "" as clearing a field, as assign_slot already does for an extensible slot. Writing a blank over an unset field does nothing. Without this, obj.f = obj.f on a parsed blank field stored None, bumped the mutation version and cleared the preserved source text. That made write_idf(preserve_formatting=True) reformat an object nobody had changed. On main, the stored "" matched and the assignment did nothing.

Effects:

  • Field access: a blank field now reads as None through attribute, item (obj["..."], obj[i]) and get() access. That is what a field cut off by an earlier semicolon has always returned. Under strict=True, an unset schema field still reads as None and never raises.
  • eppy compat: fieldvalues is built from field_order with data.get, so a blank slot now gives None, the same as a field never given. It used to give "".
  • Writers: no change. write_idf already writes a missing field as an empty slot, so an interior blank keeps its position (Zone,Gap Zone,,0,0,0,,2;). When an edited object is written with preserve_formatting=True, its blank commas are still kept, because of the existing keep_at_least rule.
  • An explicit 0 (0, 0.0, -0.0, 0.0E+0) is still a value.

The change is user-visible, since "" became None. So it comes with a CHANGELOG ### Fixed entry and a new "Blank fields are unset" section in the parsing-idf-epjson agent reference, with the bundle re-baked.

Not changed here: doc.add(..., field="") still stores "", because that path is in document.py. A strict parse still refuses a non-extensible object with extra trailing blank fields, as before.

Levels

[tool.idfkit.conformance] level moves from conformance-2026.19 (set by #236) to conformance-2026.20. That level differs from .19 only in removing the two Python round-trip entries this PR closes (idfkit/idfkit-conformance#47). idfkit.CONFORMANCE_LEVEL and the EPW sentinel table are regenerated from the tag; only their recorded level changes.

Tests

tests/test_idf_roundtrip.py::TestBlankFieldsAreUnset covers:

  • the truncated and blank-comma spellings giving equal models
  • interior blanks being absent, and zeros being kept
  • access returning None, in strict mode and through eppy fieldvalues
  • write_idf → re-parse giving an equal model, with the same key order
  • interior blanks being written in position
  • write_epjson equalling obj.data
  • the preserving writer keeping the blank commas of an edited object
  • re-assigning a blank field not counting as an edit
  • assigning None or "" clearing a field

make check passes. make test has one failure: test_weather_offline.py::test_a_read_only_cache_stops_the_nudge_without_the_env_var. It fails the same way on unmodified main in this environment, because the container runs as root and a chmod 0o500 cache directory is still writable for root. It is unrelated to this change.

Conformance

python3 runners/run.py --library ../idfkit at the conformance-2026.20 tag: PASS, exit 0. 74 cases, 226 assertions: 218 passed, 5 failed, all 5 of them outstanding exceptions (#186, #187). No stale entries and no unexpected failures.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LEegXpWqwRPaofngRx6L3h

A blank field of an object type with no repeating group was stored as
"", numeric fields included, while write_idf and write_epjson drop it.
Re-parsing idfkit's own IDF output therefore did not equal the original
document. The parser now skips a blank field, as it already did inside
repeating groups and as idfkit-js and ConvertInputFormat do, so it reads
as None like a field cut off by an earlier semicolon.

Closes #183

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEegXpWqwRPaofngRx6L3h
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEegXpWqwRPaofngRx6L3h

Copy link
Copy Markdown
Collaborator Author

corpus (python) is red because the fix works. At the pinned level (conformance-2026.16), the runner reports the two #183 allowlist entries as stale and exits 1:

stale exceptions: these cases now pass, so remove their entries (exit 1)
  numeric-zero-against-blank: python: round-trip now passes
  positional-trailing-unset-no-extensible: python: round-trip now passes

Nothing else changed: the 5 outstanding exceptions (#186, #187) are the same ones main reports.

This can't go green from this repo alone. As with #226 → conformance-2026.16, the unblock is:

  1. Remove the two python / round-trip entries for The document model keeps blank trailing fields that both writers strip, so the round trip is lossy #183 from known-divergence.toml in idfkit-conformance, and cut a new conformance-2026.N tag.
  2. Bump [tool.idfkit.conformance] level in pyproject.toml here to that tag. I'll push that as soon as the tag exists.

Moving to an existing later tag doesn't help: conformance-2026.19 still carries both entries.


Generated by Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEegXpWqwRPaofngRx6L3h
… fields

With a parsed blank field now absent from obj.data, assigning it its own
value (obj.f = obj.f) stored None, bumped the mutation version and cleared
the preserved source text, so a preserving write reformatted an object
nobody edited. _set_field now treats None and "" as clearing the field,
as assign_slot already does for an extensible slot, and a blank written
over an unset field is no change.

The parser's fixed-field loop zips the named fields so it no longer walks
the extensible tail, and the changelog entry is scoped to fixed fields in
every object type and written in the past tense.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEegXpWqwRPaofngRx6L3h
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEegXpWqwRPaofngRx6L3h
The level differs from conformance-2026.19 only by removing the two
python round-trip entries that this change closes (#183,
idfkit-conformance#47). idfkit.CONFORMANCE_LEVEL and the EPW sentinel
table are regenerated from the new tag; only their recorded level
changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LEegXpWqwRPaofngRx6L3h
@samuelduchesne
samuelduchesne merged commit 04ac857 into main Oct 26, 2026
27 checks passed
@samuelduchesne
samuelduchesne deleted the fix/blank-fields-unset-183 branch October 26, 2026 18:22
samuelduchesne pushed a commit that referenced this pull request Oct 28, 2026
Brings in the conformance-2026.20 pin from #242.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JkhUjaZe78iCNB1nvAkuzN
@samuelduchesne samuelduchesne mentioned this pull request Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The document model keeps blank trailing fields that both writers strip, so the round trip is lossy

2 participants