Repository navigation
fix(core): range check number exponents as DynamoDB does - #619
LeeroyHannigan wants to merge 3 commits into
Conversation
validate_and_normalize_number parsed the exponent as an i32. #394 moved its exponent sums to saturating i64 arithmetic, which fixed the overflow panics of #616, but an exponent outside the i32 range was still refused as malformed. The exponent is now parsed as an i64 and the sums are done in i128 instead of saturating i64, so none of them can overflow. DynamoDB reads the exponent the same way: 1E2147483648 is a Number overflow and 1E9223372036854775808 cannot be converted. A zero whose exponent does not fit in an i32 is range checked too; main refused those as malformed. A zero whose exponent fits in an i32 stays accepted as 0, although DynamoDB refuses 0E126 and 0.0E-130. A refused zero keeps its raw spelling in AttributeValue, and only the PutItem item and UpdateItem's update values are validated afterwards, so UpdateItem's key and the other operations would store 0E126 as a key separate from 0. #618 covers both. Closes #616
95997b6 to
bfde5fd
Compare
There was a problem hiding this comment.
@LeeroyHannigan I checked this change against live DynamoDB. The panic from #616 is fixed, and
the wrong message in release builds is fixed. No value panicked a debug build,
and no value wrapped on the paths that check numbers.
Two values still get the wrong message. I measured each one on live DynamoDB in
us-east-1. I sent each value two times, and both times gave the same answer.
-
A zero with the exponent at the largest 64-bit value.
PutItem with a non-key attribute
{"N": "0E9223372036854775807"}.
Live DynamoDB answersNumber underflow. This change answersNumber overflow.The neighbors show the edge.
0E9223372036854775806answersNumber overflow,
and0.0E9223372036854775807answersNumber overflow. Only the exponent at
exactly the largest 64-bit value, with no fraction, answersNumber underflow. -
More than 38 significant digits together with an out-of-range magnitude.
PutItem with
{"N": "1234567890123456789012345678901234567890E9223372036854775768"}.
Live DynamoDB answersNumber overflow. This change answers
Attempting to store more than 38 significant digits in a Number.Live DynamoDB checks the magnitude before the digit count. The same value with
the exponent one lower (...767) answers the digit-count message on both. So
when the magnitude is out of range, the overflow or underflow message comes first.
Please change the code to match live DynamoDB for both values, and add the
measured values as test cases. The 64-bit exponent range is new in this change,
so both values reach these paths for the first time here.
The branch is a long way behind main. Please rebase it on current main so the
checks run against the current tree.
I will approve once both values match live DynamoDB and the branch is rebased.
Main has not changed crates/core/src/validation/number.rs or tests/rust/src/number_validation.rs since this branch was cut. The merge has no conflicting file and no edit outside the merge itself. The core crate and its tests compile on the merged tree.
DynamoDB computes the exponent of a number's normalized form, the order of magnitude plus one, in 64 bits. Two answers at the edge of that range differed from ExtendDB's. PutItem in us-east-1 gave each value below the same answer on two sends, on 9 or 10 Nov 2026. A zero whose exponent is i64::MAX, with no fraction digit, is an underflow: the sum wraps to a negative exponent. A fraction digit lowers the exponent before the sum. So 0E9223372036854775807 is an underflow, while 0E9223372036854775806 and 0.0E9223372036854775807 are overflows. For a nonzero value whose sum is above i64::MAX, DynamoDB reports the overflow before it counts the digits. Forty digits with E9223372036854775768, E9223372036854775769 and E9223372036854775807 (an order of magnitude of i64::MAX, i64::MAX + 1 and i64::MAX + 39) are overflows. The same digits with E9223372036854775767 are too many digits. The one-digit 1E9223372036854775807 and 10E9223372036854775807 are overflows as well. validate_and_normalize_number now computes the order of magnitude before the digit count and applies both rules. The measured values are unit tests in number.rs and wire tests in tests/rust, which check the full message text.
|
@eliseong Both values now answer as you measured. 39a41fb computes the exponent of the normalized form, the order of magnitude plus one, in 64 bits:
On 2026-11-10 I measured two more values the same way. The 40 digits with I merged main at 887b6e2 (e80d018) instead of rebasing, which avoids a force push. CI now runs against current main. Main had not changed either file, and the merge has no conflicts. |
What
#394 (34aa2df, backup format v1) moved the three exponent sums in
validate_and_normalize_numberto saturatingi64arithmetic, which fixed the overflow #616 reports. Main 817b240 answers the first three rows of the table below as DynamoDB does. On abf8082 a debug build panicked on the first and third. Main still parses the exponent as ani32, so an exponent between thei32andi64limits is refused ascannot be converted to a numeric value, where DynamoDB answersNumber overfloworNumber underflow. This PR carries thei64exponent, the zero check that goes with it, and DynamoDB's answers at thei64edge.In
crates/core/src/validation/number.rs:i64, as DynamoDB reads it, and the sums are done ini128, where none of them can overflow. This replaces feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394's saturatingi64sums and itslen_i64helper withdigit_countandcheck_magnitude.1E2147483648is nowNumber overflowand1E-2147483649Number underflow. Main answerscannot be convertedfor both.1E9223372036854775808is stillcannot be converted, as on DynamoDB.i32is range checked (0E2147483648isNumber overflow). Main refuses those zeros ascannot be converted, so the set of refused inputs does not change.normalized_exponent_exceeds_i64, reports when that sum leaves thei64range. For a zero, the sum leaves the range only when the exponent is exactly9223372036854775807with no fraction digit. The sum then wraps to a negative exponent, and the answer isNumber underflow. A fraction digit lowers the exponent before the sum, so0.0E9223372036854775807staysNumber overflow. For a nonzero value, the order of magnitude is now computed before the digit count. A sum outside the range then answersNumber overflow, and the digit count does not run.i32is still accepted as0. DynamoDB refuses0E126and0.0E-130. But a zero refused here keeps its raw spelling inAttributeValue. Afterwards, only the PutItem item and UpdateItem's ExpressionAttributeValues and AttributeUpdates go through the number check. BatchWriteItem, TransactWriteItems, UpdateItem's key and the other operations would then store and look up0E126as a key separate from0. BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618 covers that validation gap and the zero check together. The unit testzero_with_i32_exponent_stays_accepted_until_618marks the place to change.The PR keeps the test #394 added in
number.rs,exponents_at_the_ends_of_i32_classify_the_magnitude.The branch merges main at 887b6e2 (e80d018). Main had not changed
number.rsortests/rust/src/number_validation.rssince the branch was cut, and the merge has no conflicting file.Measured
The DynamoDB column is PutItem with a non-key
Nattribute in us-east-1. Each value was sent twice, with the same answer both times. The four rows above the last one were measured on 2026-11-09, the last row on 2026-11-10, and the other rows on 2026-10-27. The ExtendDB columns are PutItem with the same attribute against dev-mode SQLite servers. Main 817b240 and the branch head of that day ran on 2026-10-30. For the last five rows, this branch ran at 39a41fb on 2026-11-10.10E2147483647,12E2147483647,100E2147483647,-10E2147483647Number overflowNumber overflowNumber overflow1E+2147483647,0.1E2147483647Number overflowNumber overflowNumber overflow1.5E-2147483648,-1.5E-2147483648,0.01E-2147483648Number underflowNumber underflowNumber underflow1E2147483648,1E4294967296,1E9223372036854775807Number overflowcannot be convertedNumber overflow1E-2147483649,1E-9223372036854775808Number underflowcannot be convertedNumber underflow1E9223372036854775808,1E-9223372036854775809cannot be convertedcannot be convertedcannot be converted0E2147483648,0E9223372036854775806Number overflowcannot be convertedNumber overflow0E-9223372036854775807,0E-9223372036854775808Number underflowcannot be convertedNumber underflow0E126,0.0E-130Number overflow,Number underflow00(#618)0E9223372036854775807Number underflowcannot be convertedNumber underflow0.0E9223372036854775807Number overflowNumber overflow1234567890123456789012345678901234567890) withE9223372036854775768Number overflowNumber overflowE9223372036854775767E9223372036854775769orE9223372036854775807Number overflowNumber overflowDynamoDB starts each of these messages with
1 validation error detected:and ExtendDB does not. This PR does not change that difference.Why
Closes #616. DynamoDB reads a number's exponent as a 64-bit value, and main reads it as a 32-bit value.
Testing done
number.rs: seven new tests, 28 tests in the module, all passing on 39a41fb. Run against main 817b240'snumber.rs, two of the first five new tests fail oncannot be converted:exponent_is_read_as_i64at1E2147483648andzero_with_exponent_outside_i32_is_range_checkedat0E2147483648.exponent_near_i32_limits_is_an_overflow_or_underflow, which failed on abf8082 with the overflow panic, passes on main since feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394. On the merge commit e80d018, the two newest tests fail with the answers the review reported.zero_with_exponent_at_i64_max_is_an_underflowgetsNumber overflowthere.magnitude_at_i64_max_is_an_overflow_before_the_digit_countgets the message for more than 38 digits.tests/rust/src/number_validation.rs: five new tests. Against a dev-mode SQLite server built from 39a41fb, thenumber_validationfilter passes 14 of 14 with--test-threads=1. The filter also matches main'supdate_item_number_validation.rs. In an earlier parallel run, 5 of the 14 failed with a transport error (DispatchGone) and no server error. The failing tests came from this branch and from main. Against main 817b240,put_item_exponent_near_i32_limits_is_overflow_or_underflowfails at1E2147483648andput_item_zero_with_exponent_outside_i32_is_rejectedat0E2147483648.put_item_exponent_outside_i64_is_malformedpasses on main too.cargo fmt --all -- --checkandcargo +1.97.0 clippy --locked -p extenddb-core --all-targets -- -D warningsare clean on 39a41fb.cargo test --locked -p extenddb-core --libpassed 1089 tests, with 1 ignored, one revision earlier. That revision differs from 39a41fb only in tests and comments. The passing tests include feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394'sexponents_at_the_ends_of_i32_classify_the_magnitude,doc_examples_normalize_as_statedandexponent_at_the_end_of_i32_is_refused_as_an_overflow.0E126,-0E126and0.0E-130as keys and attributes. It covered GetItem, Query, BatchGetItem, BatchWriteItem, TransactWriteItems, UpdateItem and DeleteItem. The output on the branch was the same as on main abf8082. For those zeros this branch and main 817b240 both return0. The only check this PR adds on that path applies to an exponent outsidei32.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a
Breaking changes
None.
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.