Skip to content

Fix CPU Huffman truncation of wide quantization outliers - #260

Open
VarShankar wants to merge 1 commit into
CODARcode:masterfrom
VarShankar:fix-cpu-huffman-int64-outliers
Open

VarShankar wants to merge 1 commit into
CODARcode:masterfrom
VarShankar:fix-cpu-huffman-int64-outliers

Conversation

@VarShankar

Copy link
Copy Markdown

Summary

The MGARD-CPU Huffman path accepts quantized coefficients as long int, but its outlier (miss) stream narrows every value to int. On LP64 systems, valid quantization indices outside the signed 32-bit range are therefore silently truncated, and decompression can violate the requested error tolerance by orders of magnitude.

This change:

  • keeps legacy int-width outlier streams when all misses fit in int;
  • stores the complete miss stream as long int when any miss requires the wider type;
  • decodes both legacy and wide streams using the recorded miss-byte count; and
  • rejects inconsistent miss-stream sizes.

Ordinary existing streams retain their previous representation. A new regression test covers both the legacy-width path and values outside the int range.

Validation

  • The focused reproducer fails on unpatched master and passes with this change.
  • The new Catch2 regression test passes.
  • All 310 configured MGARD-CPU and serial MGARD-X tests pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant