Skip to content

frequent_items_sketch: add reset(), decide emptiness by preamble longs - #529

Merged
leerho merged 2 commits into
masterfrom
fi-empty-preamble
Sep 24, 2026
Merged

leerho merged 2 commits into
masterfrom
fi-empty-preamble

Conversation

@leerho

@leerho leerho commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #527, which made is_empty() mean total_weight == 0. A sketch whose purges cleared every counter is non-empty and now serializes with the full preamble and no items. This PR makes the read side and the API consistent with that.

Changes

  • reset(), paralleling Java's reset(). Returns the sketch to the empty state: total_weight and offset are zeroed and the map restarts at LG_MIN_MAP_SIZE. The maximum map size, equality operator and allocator are retained. Adds a get_equal() accessor to reverse_purge_hash_map for this.
  • Emptiness on read is decided by preamble longs. Preamble longs must be 1 (empty) or 4 (non-empty). The flags byte is only cross-checked against it; either legacy empty bit (0x01 or 0x04) is still accepted, and a mismatch is rejected as corruption, as before.
  • Reject a non-empty image with non-positive total weight. No valid writer produces one, and it would otherwise load as an empty sketch and be rewritten in the 8-byte form. Written as !(total_weight > 0) so NaN is rejected for floating-point W.
  • Documented the flags byte: preamble longs decide emptiness, both empty bits are written only for historical compatibility, and no other flag bits are defined.

No change to the serialized format. Every image that deserialized before still deserializes, except the non-empty-with-zero-weight case above.

Tests

  • Purged-to-zero sketch: 32-byte image, preamble longs 4, flags 0, stream round trip (frequent_items_sketch: fix is_empty() after purge clears all counters #527 covered the byte path).
  • reset() after purges: empty, 8-byte image, max map size retained, and subsequent updates serialize identically to a newly constructed sketch.
  • Empty image accepted with flags 0x01, 0x04 and 0x05.
  • Corrupt images rejected: invalid preamble longs, empty flag disagreeing with preamble longs in either direction, zero total weight (bytes and stream).

Cross-language images

The [serialize_for_java] generator adds frequent_long_purged_cpp.sk and frequent_string_purged_cpp.sk: lg_max_map_size=8 and 193 distinct items, giving N=193, offset 1 and no retained items. Each is 32 bytes, preamble longs 4, flags 0. None of the existing images covers this form. apache/datasketches-java#770 reads these in checkCpp() and generates the Java counterparts; the C++ test that reads the Java images follows once that merges, since serde_compat.yml generates from Java main.

All fi_test cases pass, including [serde_compat] against the current Java-generated images.

Related

Java counterpart: apache/datasketches-java#770 (isEmpty() as streamWeight == 0, the same read-side checks, FrequentLongsSketch string format fixes, and the Java purged-to-zero images, which are byte-identical to the C++ ones).

🤖 Generated with Claude Code

- Add reset(), paralleling Java: returns the sketch to the empty state,
  keeping the maximum map size, equality operator and allocator.
- On deserialize, emptiness is determined by preamble longs (1 or 4).
  The flags byte is only cross-checked against it; either legacy empty
  bit is still accepted.
- Reject a non-empty image whose total weight is not positive. No valid
  writer produces it, and it would otherwise load as an empty sketch.
- Document that no flag bits other than the empty bits are defined.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho requested review from proost and tisonkun September 23, 2026 22:20
@coveralls

coveralls commented Sep 23, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 35929114126

Coverage increased (+0.05%) to 82.398%

Details

  • Coverage increased (+0.05%) from the base build.
  • Patch coverage: 23 of 23 lines across 2 files are fully covered (100%).
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 21560
Covered Lines: 17765
Line Coverage: 82.4%
Coverage Strength: 1363988.55 hits per line

💛 - Coveralls

…age tests

Longs and strings sketches with lg_max_map_size=8 and 193 distinct items:
non-empty (N=193, offset=1) with no retained items, serialized with the
full preamble.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leerho
leerho merged commit 70e462f into master Sep 24, 2026
32 checks passed
@leerho
leerho deleted the fi-empty-preamble branch September 24, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants