Skip to content

fix(core): range check number exponents as DynamoDB does - #619

Open
LeeroyHannigan wants to merge 3 commits into
mainfrom
fix/616-number-exponent-overflow
Open

LeeroyHannigan wants to merge 3 commits into
mainfrom
fix/616-number-exponent-overflow

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What

#394 (34aa2df, backup format v1) moved the three exponent sums in validate_and_normalize_number to saturating i64 arithmetic, which fixed the overflow #616 reports. Main 817b240 answers the first three rows of the table below as DynamoDB does. On abf8082 a debug build panicked on the first and third. Main still parses the exponent as an i32, so an exponent between the i32 and i64 limits is refused as cannot be converted to a numeric value, where DynamoDB answers Number overflow or Number underflow. This PR carries the i64 exponent, the zero check that goes with it, and DynamoDB's answers at the i64 edge.

In crates/core/src/validation/number.rs:

  • The exponent is parsed as an i64, as DynamoDB reads it, and the sums are done in i128, where none of them can overflow. This replaces feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394's saturating i64 sums and its len_i64 helper with digit_count and check_magnitude. 1E2147483648 is now Number overflow and 1E-2147483649 Number underflow. Main answers cannot be converted for both. 1E9223372036854775808 is still cannot be converted, as on DynamoDB.
  • A zero whose exponent does not fit in an i32 is range checked (0E2147483648 is Number overflow). Main refuses those zeros as cannot be converted, so the set of refused inputs does not change.
  • DynamoDB computes the exponent of a number's normalized form, the order of magnitude plus one, in 64 bits. A new helper, normalized_exponent_exceeds_i64, reports when that sum leaves the i64 range. For a zero, the sum leaves the range only when the exponent is exactly 9223372036854775807 with no fraction digit. The sum then wraps to a negative exponent, and the answer is Number underflow. A fraction digit lowers the exponent before the sum, so 0.0E9223372036854775807 stays Number overflow. For a nonzero value, the order of magnitude is now computed before the digit count. A sum outside the range then answers Number overflow, and the digit count does not run.
  • A zero whose exponent fits in an i32 is still accepted as 0. DynamoDB refuses 0E126 and 0.0E-130. But a zero refused here keeps its raw spelling in AttributeValue. Afterwards, only the PutItem item and UpdateItem's ExpressionAttributeValues and AttributeUpdates go through the number check. BatchWriteItem, TransactWriteItems, UpdateItem's key and the other operations would then store and look up 0E126 as a key separate from 0. BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618 covers that validation gap and the zero check together. The unit test zero_with_i32_exponent_stays_accepted_until_618 marks the place to change.

The PR keeps the test #394 added in number.rs, exponents_at_the_ends_of_i32_classify_the_magnitude.

The branch merges main at 887b6e2 (e80d018). Main had not changed number.rs or tests/rust/src/number_validation.rs since the branch was cut, and the merge has no conflicting file.

Measured

The DynamoDB column is PutItem with a non-key N attribute in us-east-1. Each value was sent twice, with the same answer both times. The four rows above the last one were measured on 2026-11-09, the last row on 2026-11-10, and the other rows on 2026-10-27. The ExtendDB columns are PutItem with the same attribute against dev-mode SQLite servers. Main 817b240 and the branch head of that day ran on 2026-10-30. For the last five rows, this branch ran at 39a41fb on 2026-11-10.

Value DynamoDB main 817b240 this branch
10E2147483647, 12E2147483647, 100E2147483647, -10E2147483647 Number overflow Number overflow Number overflow
1E+2147483647, 0.1E2147483647 Number overflow Number overflow Number overflow
1.5E-2147483648, -1.5E-2147483648, 0.01E-2147483648 Number underflow Number underflow Number underflow
1E2147483648, 1E4294967296, 1E9223372036854775807 Number overflow cannot be converted Number overflow
1E-2147483649, 1E-9223372036854775808 Number underflow cannot be converted Number underflow
1E9223372036854775808, 1E-9223372036854775809 cannot be converted cannot be converted cannot be converted
0E2147483648, 0E9223372036854775806 Number overflow cannot be converted Number overflow
0E-9223372036854775807, 0E-9223372036854775808 Number underflow cannot be converted Number underflow
0E126, 0.0E-130 Number overflow, Number underflow 0 0 (#618)
0E9223372036854775807 Number underflow cannot be converted Number underflow
0.0E9223372036854775807 Number overflow not run Number overflow
40 digits (1234567890123456789012345678901234567890) with E9223372036854775768 Number overflow not run Number overflow
The same 40 digits with E9223372036854775767 the message for more than 38 digits not run the message for more than 38 digits
The same 40 digits with E9223372036854775769 or E9223372036854775807 Number overflow not run Number overflow

DynamoDB starts each of these messages with 1 validation error detected: and ExtendDB does not. This PR does not change that difference.

Why

Closes #616. DynamoDB reads a number's exponent as a 64-bit value, and main reads it as a 32-bit value.

Testing done

  • Unit tests in number.rs: seven new tests, 28 tests in the module, all passing on 39a41fb. Run against main 817b240's number.rs, two of the first five new tests fail on cannot be converted: exponent_is_read_as_i64 at 1E2147483648 and zero_with_exponent_outside_i32_is_range_checked at 0E2147483648. exponent_near_i32_limits_is_an_overflow_or_underflow, which failed on abf8082 with the overflow panic, passes on main since feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394. On the merge commit e80d018, the two newest tests fail with the answers the review reported. zero_with_exponent_at_i64_max_is_an_underflow gets Number overflow there. magnitude_at_i64_max_is_an_overflow_before_the_digit_count gets the message for more than 38 digits.
  • Wire tests in tests/rust/src/number_validation.rs: five new tests. Against a dev-mode SQLite server built from 39a41fb, the number_validation filter passes 14 of 14 with --test-threads=1. The filter also matches main's update_item_number_validation.rs. In an earlier parallel run, 5 of the 14 failed with a transport error (DispatchGone) and no server error. The failing tests came from this branch and from main. Against main 817b240, put_item_exponent_near_i32_limits_is_overflow_or_underflow fails at 1E2147483648 and put_item_zero_with_exponent_outside_i32_is_rejected at 0E2147483648. put_item_exponent_outside_i64_is_malformed passes on main too.
  • cargo fmt --all -- --check and cargo +1.97.0 clippy --locked -p extenddb-core --all-targets -- -D warnings are clean on 39a41fb. cargo test --locked -p extenddb-core --lib passed 1089 tests, with 1 ignored, one revision earlier. That revision differs from 39a41fb only in tests and comments. The passing tests include feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394's exponents_at_the_ends_of_i32_classify_the_magnitude, doc_examples_normalize_as_stated and exponent_at_the_end_of_i32_is_refused_as_an_overflow.
  • At the first revision, a probe sent zeros such as 0E126, -0E126 and 0.0E-130 as keys and attributes. It covered GetItem, Query, BatchGetItem, BatchWriteItem, TransactWriteItems, UpdateItem and DeleteItem. The output on the branch was the same as on main abf8082. For those zeros this branch and main 817b240 both return 0. The only check this PR adds on that path applies to an exponent outside i32.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a

Breaking changes

None.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

validate_and_normalize_number parsed the exponent as an i32. #394 moved
its exponent sums to saturating i64 arithmetic, which fixed the overflow
panics of #616, but an exponent outside the i32 range was still refused
as malformed.

The exponent is now parsed as an i64 and the sums are done in i128
instead of saturating i64, so none of them can overflow. DynamoDB reads
the exponent the same way: 1E2147483648 is a Number overflow and
1E9223372036854775808 cannot be converted. A zero whose exponent does
not fit in an i32 is range checked too; main refused those as malformed.

A zero whose exponent fits in an i32 stays accepted as 0, although
DynamoDB refuses 0E126 and 0.0E-130. A refused zero keeps its raw
spelling in AttributeValue, and only the PutItem item and UpdateItem's
update values are validated afterwards, so UpdateItem's key and the
other operations would store 0E126 as a key separate from 0. #618
covers both.

Closes #616

@eliseong eliseong left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@LeeroyHannigan I checked this change against live DynamoDB. The panic from #616 is fixed, and
the wrong message in release builds is fixed. No value panicked a debug build,
and no value wrapped on the paths that check numbers.

Two values still get the wrong message. I measured each one on live DynamoDB in
us-east-1. I sent each value two times, and both times gave the same answer.

  1. A zero with the exponent at the largest 64-bit value.

    PutItem with a non-key attribute {"N": "0E9223372036854775807"}.
    Live DynamoDB answers Number underflow. This change answers Number overflow.

    The neighbors show the edge. 0E9223372036854775806 answers Number overflow,
    and 0.0E9223372036854775807 answers Number overflow. Only the exponent at
    exactly the largest 64-bit value, with no fraction, answers Number underflow.

  2. More than 38 significant digits together with an out-of-range magnitude.

    PutItem with
    {"N": "1234567890123456789012345678901234567890E9223372036854775768"}.
    Live DynamoDB answers Number overflow. This change answers
    Attempting to store more than 38 significant digits in a Number.

    Live DynamoDB checks the magnitude before the digit count. The same value with
    the exponent one lower (...767) answers the digit-count message on both. So
    when the magnitude is out of range, the overflow or underflow message comes first.

Please change the code to match live DynamoDB for both values, and add the
measured values as test cases. The 64-bit exponent range is new in this change,
so both values reach these paths for the first time here.

The branch is a long way behind main. Please rebase it on current main so the
checks run against the current tree.

I will approve once both values match live DynamoDB and the branch is rebased.

Main has not changed crates/core/src/validation/number.rs or
tests/rust/src/number_validation.rs since this branch was cut. The
merge has no conflicting file and no edit outside the merge itself.
The core crate and its tests compile on the merged tree.
DynamoDB computes the exponent of a number's normalized form, the
order of magnitude plus one, in 64 bits. Two answers at the edge of
that range differed from ExtendDB's. PutItem in us-east-1 gave each
value below the same answer on two sends, on 9 or 10 Nov 2026.

A zero whose exponent is i64::MAX, with no fraction digit, is an
underflow: the sum wraps to a negative exponent. A fraction digit
lowers the exponent before the sum. So 0E9223372036854775807 is an
underflow, while 0E9223372036854775806 and 0.0E9223372036854775807
are overflows.

For a nonzero value whose sum is above i64::MAX, DynamoDB reports the
overflow before it counts the digits. Forty digits with
E9223372036854775768, E9223372036854775769 and E9223372036854775807
(an order of magnitude of i64::MAX, i64::MAX + 1 and i64::MAX + 39)
are overflows. The same digits with E9223372036854775767 are too many
digits. The one-digit 1E9223372036854775807 and 10E9223372036854775807
are overflows as well.

validate_and_normalize_number now computes the order of magnitude
before the digit count and applies both rules. The measured values
are unit tests in number.rs and wire tests in tests/rust, which check
the full message text.
@LeeroyHannigan

Copy link
Copy Markdown
Collaborator Author

@eliseong Both values now answer as you measured. 39a41fb computes the exponent of the normalized form, the order of magnitude plus one, in 64 bits:

  • 0E9223372036854775807 answers Number underflow, because the sum wraps. 0E9223372036854775806 and 0.0E9223372036854775807 stay Number overflow.
  • The 40 digits with E9223372036854775768 answer Number overflow before the digit count. With E9223372036854775767 they answer the message for more than 38 digits.

On 2026-11-10 I measured two more values the same way. The 40 digits with E9223372036854775769 and with E9223372036854775807 also answer Number overflow on DynamoDB. Both are in the tests with your five values, in number.rs and in tests/rust/src/number_validation.rs, and the wire tests now check the full message text. On the merge commit, without the fix, the two new unit tests fail with the answers you reported. Other spellings of these values follow the same rule and are not measured, for example -0E9223372036854775807.

I merged main at 887b6e2 (e80d018) instead of rebasing, which avoids a force push. CI now runs against current main. Main had not changed either file, and the merge has no conflicts.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-blocker Must ship for its milestone's release security v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An N value with an exponent near the i32 limits panics the number validator in debug builds and gets the wrong error in release builds

3 participants