Frequent items: empty means zero stream weight, not zero active items

[leerho]

Java counterpart of apache/datasketches-cpp#527 and apache/datasketches-cpp#529.

Problem

isEmpty() in FrequentLongsSketch and FrequentItemsSketch was getNumActiveItems() == 0. A purge subtracts the median from every counter and drops those that reach zero, so with many low-frequency items it can remove every counter while streamWeight and offset stay positive. That state was reported as empty:

• toByteArray() wrote the 8-byte empty form, dropping stream weight and offset. After a round trip the sketch looked fresh and exact.
• merge() returned early and ignored the other sketch's stream weight and offset.

Reproducer: new FrequentLongsSketch(256) updated with 193 distinct items. Before this change it reports isEmpty() == true with stream length 193 and error 1; after a byte round trip, stream length 0 and error 0; merged into a sketch with one item, stream length 1 and error 0.

A non-empty sketch with no items is not equivalent to a fresh one: it says every item's count is at most offset, and its stream length counts.

Changes

• isEmpty() returns streamWeight == 0 in both classes. Updates only accept positive counts and merges only add, so only reset() returns a sketch to empty.
• toByteArray() handles zero active items in a non-empty sketch: full preamble with activeItems = 0 and no values or keys (32 bytes). This path was unreachable before, and the hash maps return null active arrays when empty.
• Reading images: PreLongs decides emptiness (1 empty, 4 non-empty). The empty flag (mask 0x05) is only cross-checked. A non-empty image whose stream weight is not positive is rejected as corrupt.
• FrequentLongsSketch string form: serializeToString() sets the flag from isEmpty(); getInstance(String) masks the flag with EMPTY_FLAG_MASK (was flags > 0), requires it to agree with the stream weight, and no longer rejects a non-empty sketch with no active items.
• PreambleUtil: documented emptiness and the flags byte (both legacy bits written, either accepted, no other bits defined), fixed the row label of the data start in the layout diagram, and removed the unused SER_DE_ID_SHORT.

No change to the binary format. Every image written by an earlier version still deserializes; purged-to-zero images written earlier were already stored as empty, and their stream weight and offset cannot be recovered.

Tests

• Purged-to-zero sketch, longs and items: not empty, 32-byte image with PreLongs 4 and flags 0, byte and string round trips, merge keeps stream weight and error.
• reset() after a purge returns to the empty 8-byte form.
• Empty image accepted with flags 0x01, 0x04 and 0x05.
• Corrupt images rejected: PreLongs 1 without the empty flag, PreLongs 4 with it, zero stream weight; string form with the flag disagreeing with the stream weight.
• LongsSketchTest.checkStringDeserEmptyCorrupt asserted the old rule (its input is a valid purged-to-zero sketch); it is now checkStringDeserNonEmptyNoItems.

Cross-language

• FrequentItemsSketchCrossLanguageTest generates frequent_long_purged_java.sk and frequent_string_purged_java.sk (lgMaxMapSize = 8, 193 distinct items) and checks them for Java, C++ and Go. The C++ images come from frequent_items_sketch: add reset(), decide emptiness by preamble longs datasketches-cpp#529; the Java and C++ images are byte-identical for both longs and strings.
• checkCpp() called stringsAscii and stringsUtf8 with GroupLanguage.JAVA, so it never read the C++ ascii and utf8 images. It now uses GroupLanguage.CPP.

Go reads these images too; checkGo() will expose the same bug in datasketches-go once it generates the purged images.

Unrelated failures

BloomFilterTest.basicDifferenceTest and TDigestDoubleSerializationTest.weightOverflowDoesNotChangeDigest fail on main without this change.

🤖 Generated with Claude Code