Skip to content

Refuse an update whose result nests beyond 32 levels - #590

Open
LeeroyHannigan wants to merge 12 commits into
mainfrom
fix/update-result-nesting-depth
Open

LeeroyHannigan wants to merge 12 commits into
mainfrom
fix/update-result-nesting-depth

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

What

ExtendDB now refuses an UpdateItem, a TransactWriteItems Update or a PartiQL UPDATE whose result nests an attribute more than 32 levels deep, as the service does. Before, every backend applied it. The one exception was a PartiQL UPDATE whose parameter is itself past the limit, which crates/engine/src/partiql/item_depth.rs refused already.

The depth check

apply_update_validated in crates/core/src/expression/update_evaluator.rs is the function the update paths call after their condition check when they evaluate the expression in Rust. The two exceptions, UpdateItem on SQLite and PostgreSQL, now call a wrapper of its body (below). The MongoDB native update path takes only one-element paths with a checked value, so it cannot produce an over-limit value.

After the actions apply, apply_update_validated now calls validate_written_nesting_depth (new file crates/core/src/expression/update_nesting.rs). It checks each top-level attribute a SET or an ADD wrote, once however many actions write into it. The walker is nests_beyond_item_limit in crates/core/src/validation/mod.rs, the walker of PutItem's validate_item_nesting_depth, so the limit (MAX_ITEM_NESTING_DEPTH) has one definition. The check covers ADD because ExtendDB creates the missing maps on an ADD path. REMOVE and DELETE cannot deepen a value. The check leaves attributes the update does not write alone. So an item stored deeper than the limit before this change stays updatable through its other attributes.

The order against the size checks

On SQLite and PostgreSQL, UpdateItem calls the new apply_update_item_validated in place of apply_update_validated followed by validate_update_item_size. It runs the same body. When the depth check refuses, it first sizes the statement with the new validate_update_statement_size (crates/core/src/validation/mod.rs, the statement half of validate_update_item_size). A statement over max_item_size_bytes answers the size message, and any other refused update answers the nesting message. When the depth check passes, the vector check runs, then validate_update_item_size sizes the statement and the updated item, as before.

The statement check passes a PartiQL UPDATE (finished_item_sizing), which ExtendDB sizes by its updated item alone. It also passes the legacy AttributeUpdates form, whose figure comes from the finished values of the attributes it names. MongoDB and Cassandra UpdateItem keep apply_update_validated. MongoDB applies no size limit on updates, and Cassandra sizes the updated item afterwards with validate_item_size and its own wording. The TransactWriteItems Update path on every backend keeps apply_update_validated. On SQLite and PostgreSQL it then sizes the updated item with validate_transact_update_item_size.

On the PartiQL paths of main (#411), an UPDATE whose parameter is within the parameter limit goes to UpdateItem with finished_item_sizing. A BatchExecuteStatement member takes the same path, and an ExecuteTransaction member takes the TransactWriteItems Update path. An UPDATE whose SET value is itself past the limit goes through unwritable_update_error in crates/engine/src/partiql/item_depth.rs, which calls apply_update_validated. The text of the depth check is the text of ITEM_NESTING_MESSAGE there, so the answer for such a value does not change.

Two orders are not measured: the depth check against the vector check (validate_vector_write_changed), and on MongoDB the depth check against validate_index_keys_after_update. In ExtendDB the depth check runs first in both. The doc comment of apply_update_item_validated says so for the vector check.

Docs

docs/differences-from-dynamodb.md: the sizing paragraph and the UpdateItem row no longer say that every transacted Update has the finished item as its only size figure. A new row records the TransactWriteItems Update the service refuses before the transaction runs (below), ExtendDB's answer, and #937. docs/dynamodb-limits.md: the nesting row names validate_attribute_values_nesting_depth for the values of UpdateItem, and validate_written_nesting_depth for the attributes that UpdateItem, the TransactWriteItems Update and the PartiQL UPDATE write.

Files

At e91da2e against the merge base 5ada959: 12 files, 893 insertions, 55 deletions. crates/core/src/expression/update_evaluator.rs is 642 lines on main and 716 at e91da2e. update_nesting.rs is 138 lines at e91da2e.

Not covered by this change

Merging

git merge-tree --write-tree of e91da2e against main 883f8c7 is clean. Squash-merge this PR. The message of its first commit, 1fbc671, says that REMOVE, ADD and DELETE cannot deepen a value, and 564e314 reverses that for ADD. The message of f29f8e1 says that the legacy AttributeUpdates form is not sized by its statement, and e91da2e fixes the code comments.

Why

Closes #574.

What happened on main

crates/engine/src/update_item.rs checks the nesting depth of ExpressionAttributeValues, and each value in such an update is within the limit. Nothing checked the item the update produced. So SET #m.#k...#k = :v with a value 2 wrappers deep at a path 30 wrappers deep stored an attribute nested past the limit.

What the service answers

Measured in us-east-1, each case twice with the same answer both times. Depth here counts M or L wrappers around a scalar: 31 wrappers is 32 levels, the PutItem limit. In every refused case below, the item did not change.

On 2026-10-25 at 19:53 UTC and 2026-10-26 at 00:55 UTC:

  • The service applies an update whose result has at most 31 wrappers in every attribute. That covers a SET at the top level, into a map, into a list, and through if_not_exists that keeps the existing value. It also covers a SET of a scalar at the deepest path of an item already at the limit.
  • One more wrapper answers ValidationException Nesting Levels have exceeded supported limits, with no 1 validation error detected: prefix and no suffix. That covers a SET into a map or list, into a new map member, through list_append, and through the default of if_not_exists.
  • The service cancels a TransactWriteItems Update past the limit with one ValidationError reason carrying the same text.
  • A condition that fails on the same request answers ConditionalCheckFailedException.

On 2026-11-10, on a table with the key pk, each item holding x and a map m of 30 wrappers:

  • 19:28 UTC. An UpdateItem that sets a 409,500-byte string beside a SET that takes m to 32 wrappers answers Item size to update has exceeded the maximum allowed size, with the SET actions in either order. The same request as a TransactWriteItems Update answers ValidationException with that text before the transaction runs. The control, the same strings with the SET at 28 keys, answers the size message. A PartiQL UPDATE with a value of 2 wrappers at a path 30 keys deep answers the nesting message. With 1 wrapper the service applies it, and m has 31 wrappers.
  • 19:30 to 19:38 UTC. The item also stores a 405,000-byte string, and the request sets 8,000 bytes beside the deep SET. The stored item is under the 409,600-byte limit, and the updated item is over it. UpdateItem answers the nesting message in either order. The service cancels the same TransactWriteItems Update with one ValidationError reason carrying the nesting message, in either order. The control through 28 keys answers the size message. An item of 300,000 bytes with 120,000 bytes in the request, grow SET first, answers the nesting message.
  • 20:39 UTC. A BatchExecuteStatement whose first member sets x on item B and whose second member is the deep UPDATE of item A answers 200. The first entry has no error, the second has ValidationError with the nesting message, B takes its update, and A does not change. The same two members in ExecuteTransaction cancel the transaction with the reasons None and ValidationError with the nesting message, and neither item changes. An ExecuteStatement UPDATE that sets a 409,500-byte parameter beside the deep SET answers the nesting message, with the SET clauses in either order. The control with the SET at 28 keys answers the size message.

So the service sizes an UpdateItem statement before the depth check and sizes the updated item after it. A PartiQL UPDATE meets the depth check ahead of the size of its own parameter.

Testing done

Tests

  • crates/core/src/expression/update_nesting.rs: 5 unit tests. They cover the limit for maps and lists, every branch of a list, ADD targets, and two actions into one attribute. The fifth checks that attributes outside the SET and ADD stay unchecked.

  • crates/core/src/expression/update_evaluator_tests.rs, module update_item_sizing: 4 unit tests of apply_update_item_validated:

    • a statement over the limit answers the size message ahead of the depth
    • an updated item over the limit meets the depth check first
    • the same update through a shallow path answers the size message
    • a PartiQL or legacy update keeps the depth refusal
  • tests/test_update_result_nesting_depth.py: 30 dual-target wire tests. 22 encode the 2026-10-25 and 2026-10-26 measurements (the ADD test compares only the error code and the stored item, as above). 8 encode the 2026-11-10 measurements:

    • the statement over the limit, in both orders, answers the size message
    • the updated item over the limit, in three shapes, answers the nesting message
    • the shallow-path control answers the size message
    • the TransactWriteItems Update, in both orders, answers a cancellation with one ValidationError reason carrying the nesting message

    The 2 tests that expect the size message skip MongoDB and Cassandra, as tests/test_item_size_at_limit.py does.

  • tests/test_partiql_item_depth.py, class TestPartiqlUpdateThroughADeepPath: 7 dual-target wire tests from the 2026-11-10 measurements:

    • the value of 2 maps answers the nesting message, and the service applies the value of 1 map
    • the batch member answers its own ValidationError entry, and the other item takes its update
    • the transaction member cancels the transaction with the reasons None and ValidationError
    • the 409,500-byte parameter beside the deep SET, in both orders, answers the nesting message
    • the control at 28 keys answers the size message, and skips MongoDB and Cassandra as tests/test_partiql_item_size.py does

Runs at e91da2e

The runs below are from 2026-11-10, against a dev-mode SQLite build of e91da2e on 127.0.0.1, with toolchain 1.97.0 for the CI checks.

Run Result
tests/test_update_result_nesting_depth.py 30 passed
tests/test_partiql_item_depth.py 43 passed, 3 xfailed
tests/test_partiql_transaction.py, tests/test_partiql_check_order.py 263 passed, 2 xfailed
cargo test -p extenddb-core --lib 1097 passed, 1 ignored
cargo test -p extenddb-engine --lib 702 passed
cargo fmt --all -- --check exit 0
cargo clippy --locked --all-targets --features extenddb-storage-cassandra/test-hooks,extenddb-storage-mongodb/test-hooks -- -D warnings exit 0, no warnings
cargo clippy --locked --all-targets --features extenddb-engine/backup-test-hooks -- -D warnings exit 0, no warnings

The 5 xfails over the three PartiQL files are the strict markers on main (UNSTORED_DEEP_VALUE, TRANSACTION_INDEX_UNITS), which apply when the target is ExtendDB.

These runs are from 2026-11-10 at f7e3da1, with the wire row on a build of f29f8e1's Rust code. That code is the code of e91da2e apart from comments and one unit test name:

Run Result
cargo test -p extenddb-storage-sqlite --lib 285 passed, 2 failed, 1 ignored. The 2 failures are backup_catalog::write_lock_tests, which depend on the host: the backup store refuses a root under a directory another uid owns.
cargo test -p extenddb-storage-postgres --lib 78 passed
wire: test_item_size_at_limit.py, test_list_map_item_size.py, test_partiql_item_size.py, test_item_operations.py, test_transaction_operations.py, test_batch_operations.py 273 passed

Negative controls

  • Against a SQLite build of 38ef819 (before apply_update_item_validated), tests/test_update_result_nesting_depth.py and tests/test_partiql_item_depth.py give 66 passed, 2 failed, 3 xfailed. The 2 failures are test_a_statement_over_the_size_limit_is_refused_ahead_of_the_depth[size] and [depth], which get the nesting message there.
  • With the statement check removed from the depth branch of apply_update_checked, a_statement_over_the_limit_is_refused_for_its_size_ahead_of_the_depth fails and the other 3 unit tests pass. With that branch sizing the updated item too (validate_update_item_size), an_updated_item_over_the_limit_meets_the_depth_check_first and the PartiQL or legacy test fail, and 2 pass. These two runs were at f29f8e1.
  • With validate_written_nesting_depth returning Ok(()), at e47a2ef, tests/test_partiql_item_depth.py gives 5 failed, 38 passed, 3 xfailed. ExtendDB then applies the deep value in ExecuteStatement, in the batch member and in the transaction member. The 409,500-byte parameter beside the deep SET answers the size message in both orders. The control at 28 keys passes.
  • At c05cbc6 on 2026-10-26, against main of that date, the new file's 22 tests give 10 failed, 12 passed. The 10 that fail are the refusals, the ADD one included. The 12 that pass are the updates within the limit, the condition test and the update of another attribute.

Earlier runs at c05cbc6 (2026-10-26)

The runs covered every file under tests/ whose test names match update, nest, item_size or transact:

Backend Result
SQLite 312 passed, 6 skipped. The new file: 22 passed
PostgreSQL 16 312 passed, 6 skipped
MongoDB 7 308 passed, 6 skipped, 4 failed

The 4 MongoDB failures are in tests/test_transact_absent_item_isolation.py, the write skew on main that the file marks XFAIL_UNTIL_MONGODB_FIX. The marker applies only when the run sets EXTENDDB_TEST_MONGODB_CONTAINER, and these runs did not set it. Also at c05cbc6: cargo test --workspace --exclude extenddb-storage-cassandra on stable and on 1.88.0 (1,772 passed each), and cargo test -p extenddb-storage-cassandra --lib (61 passed).

Not run locally at e91da2e: cargo test --workspace, the wire files on PostgreSQL, MongoDB and Cassandra, and the Cassandra integration tests. CI runs the new files in run-integration, run-integration-sqlite, mongodb-pytest and cassandra-pytest.

CI at e91da2e: 63 of 63 checks passed.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

UpdateItem and the TransactWriteItems Update checked the nesting depth of
ExpressionAttributeValues but not of the item the update produced, so a SET
of a 2-level map at a 30-level path stored an attribute nested past the
limit, on every backend. Amazon DynamoDB refuses an update whose result nests
an attribute more than 32 levels (31 M or L wrappers around a scalar) and
writes nothing: UpdateItem answers ValidationException `Nesting Levels have
exceeded supported limits`, and a TransactWriteItems Update is cancelled with
a ValidationError reason carrying that text (measured 2026-10-25 at 19:53 UTC
and 2026-10-26 at 00:55 UTC, each case twice). A failing condition is reported
first.

apply_update_validated, which every backend's update and transactional update
path calls after its condition check, now checks the top-level attributes the
SET actions wrote. REMOVE, ADD and DELETE cannot deepen a value, and other
attributes are not checked, so an item stored deeper than the limit before
this change stays updatable through them.
Dual-target wire tests from the 2026-10-25 and 2026-10-26 measurements: SET
into a deep map or list, at the top level, through list_append and through
if_not_exists, up to and past the limit; a failing condition next to a result
that is too deep; an update of another attribute on an item at the limit; and
the TransactWriteItems Update cancellation reason.
ExtendDB's ADD creates the missing maps on its path, so an ADD can nest an
attribute past the limit (`ADD #m.#k...#k.#a.#b.#c :one` on a value 30
wrappers deep stored 32). The check now covers the top-level attribute of
each ADD target as well as each SET target.

Each top-level attribute is walked once however many actions write into it,
and the walk matches maps and lists directly instead of boxing an iterator
per node. With 200 SET actions into one list of 4,000 maps, UpdateItem took
78 ms against 36 ms on main before this; the walk now runs once.
The service refuses `ADD path(29).#a.#b.#c :one` on a value 30 wrappers
deep for its 33-element path (measured 2026-10-26 at 01:26 UTC, twice);
ExtendDB refuses it for the depth of the maps the ADD would create. The test
compares the error code and the stored item only. The stored-depth helper
now takes the deepest branch, so the list_append case sees an appended
element.

@pdf-amzn pdf-amzn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of the 2nd finding, I haven't spent the time to read the actual code yet.

Findings (AI generated)

Should-fix: tests/test_update_result_nesting_depth.py has no test for the depth-before-size order. The PR body says this order was measured (2026-10-26 01:26 and 01:44 UTC) for UpdateItem, with the SET actions in either order, and for a TransactWriteItems Update. Add a dual-target case: one attribute that pushes the item near 400 KB plus one SET that goes past the depth limit, expecting the nesting message. Without it, that measured order is unprotected.

Should-fix: crates/core/src/expression/update_nesting.rs:16,65 reimplements the walker and constant that already exist in crates/core/src/validation/mod.rs:2539-2587 (MAX_ITEM_NESTING_DEPTH, check_attribute_value_depth). The only difference is the error text. Reuse that walker, for example as a pub(crate) predicate or with the error passed in, so the limit is defined in one place.

Should-fix: the PR's local runs used a base 246 commits behind main. Since then, #411 added the PartiQL UPDATE / ExecuteTransaction path (crates/engine/src/partiql/item_depth.rs:157), which also calls apply_update_validated. The message now coincides with ITEM_NESTING_MESSAGE, so the outcome should match. Still, rebase and rerun tests/test_partiql_item_depth.py, tests/test_partiql_transaction.py and tests/test_partiql_check_order.py before merging. The doc comment on unwritable_update_error ("None when the updated item passes the item check") now describes a check that apply_update_validated itself refuses first. Update the comment if it no longer holds.

Nit-or-follow-up: tests/test_update_result_nesting_depth.py:137-149. The ADD test only checks the code, because DynamoDB refuses for a different reason: the path is 33 elements (The document path has too many nesting levels). ExtendDB's ADD creates the missing intermediate maps (navigate_to_parent_map_or_create, update_evaluator.rs:376) and has no document-path length check. Those two gaps on main are why ADD needs a depth check at all. File an issue for each gap, or record them in docs/differences-from-dynamodb.md, so this test can later assert the service's message.

Measured 2026-11-10 18:38 UTC (us-east-1, each case twice, dual-target probe; results in discussions/probe-add-parent-{dynamodb,extenddb}.jsonl). DynamoDB refuses ADD through a missing parent map with ValidationException The document path provided in the update expression is invalid for update. That holds for a missing item, a missing a, a={} with ADD a.b.c, and a string-set ADD. ExtendDB on main creates the maps in all four cases. ADD whose parent exists (a={} with ADD a.b, a.b={} with ADD a.b.c) matches on both. A separate divergence turned up: DynamoDB accepts ADD a[0] on a=[] (result [1]), and ExtendDB refuses it with the document-path message. Once the missing-parent bug is fixed, ADD can add at most one level at an existing parent, so the ADD branch of validate_written_nesting_depth only has to cover that. Both are main-branch bugs, not #590's to fix; file them separately.

Nit-or-follow-up: update_nesting.rs is called before validate_vector_write_changed (update_evaluator.rs:116). On MongoDB it is also called before validate_index_keys_after_update. The PR does not cite a measurement for the order of depth against vector or index-key validation. Either note it as unmeasured or measure it.

Nit-or-follow-up: the first commit's message says "REMOVE, ADD and DELETE cannot deepen a value". Commit 564e314 reverses that for ADD. If the PR is merged without squashing, reword the first commit's message.

Nit-or-follow-up: update_nesting.rs:46. The Vec::contains dedup is O(actions²). That is fine at expression-size limits. A BTreeSet<Cow> would say what it means more directly. Optional.

Main is 247 commits ahead of the branch point, and the merge is clean.
crates/core/src/expression/mod.rs changed on both sides: main exports
more names from it, and this branch adds the update_nesting module.

Main adds the PartiQL ExecuteStatement, BatchExecuteStatement and
ExecuteTransaction operations (#411). When a PartiQL UPDATE sets a
value nested past the item limit,
crates/engine/src/partiql/item_depth.rs evaluates the update against
the stored item without writing it. It does this for a single
statement and for each ExecuteTransaction member. It calls
apply_update_validated and then checks the updated item. With this
branch, apply_update_validated refuses that update itself when the
update stores the value in an attribute that a SET writes. Its text is
"Nesting Levels have exceeded supported limits", the text item_depth.rs
gives for its item check, so the PartiQL answer stays the same. The
PartiQL code sends any other UPDATE to UpdateItem, where it meets the
same check.
crates/core/src/expression/update_nesting.rs had its own copy of the
walker and the limit in crates/core/src/validation/mod.rs. It counted
31 wrappers where validation/mod.rs counts MAX_ITEM_NESTING_DEPTH
levels. The walker in validation/mod.rs is now a predicate,
nests_beyond_item_limit, and the update check calls it. The limit now
has one definition.

validate_item_nesting_depth and validate_attribute_values_nesting_depth
give the same error as before. The update check keeps its own text,
"Nesting Levels have exceeded supported limits". Both walkers refused a
value at its 32nd M or L wrapper on any path, so no answer changes. The
unit tests of both files pass unchanged.
The service sizes an UpdateItem statement before it checks the nesting
depth of the updated item. It sizes the updated item after that check
(measured on 2026-11-10 from 19:28 to 19:37 UTC in us-east-1, each
case twice).

- A SET of a 409,500-byte string next to a SET that takes m to 32
  wrappers puts the statement over the 409,600-byte limit. UpdateItem
  answers "Item size to update has exceeded the maximum allowed size",
  with the SET actions in either order.
- An item holds a 405,000-byte string. A SET of 8,000 bytes next to
  the same SET of m keeps the statement under the limit and puts the
  updated item over it. UpdateItem answers the nesting message in
  either order. A SET of 120,000 bytes on an item of
  300,000 bytes gets the same answer. A TransactWriteItems Update is
  cancelled with one ValidationError reason that carries the nesting
  message, in either order. The same UpdateItem through a shallow path
  answers the size message.
- A PartiQL UPDATE that sets a value of 2 maps at a path 30 maps deep
  answers the nesting message and changes nothing. A value of 1 map is
  applied.

The tests that expect the size message skip MongoDB and Cassandra, as
tests/test_item_size_at_limit.py does. Against a server
built from the parent commit, the two statement tests fail, because
ExtendDB answers the nesting message there. The other new tests pass.
The service sizes an UpdateItem statement before it checks the nesting
depth of the updated item. It sizes the updated item after that check
(measured on 2026-11-10 from 19:28 to 19:37 UTC in us-east-1, each
case twice). Before this change, apply_update_validated refused a deep
result first. An update whose statement was over the item size limit
got the nesting message. The service answers "Item size to update has
exceeded the maximum allowed size".

The new apply_update_item_validated in
crates/core/src/expression/update_evaluator.rs applies a standalone
UpdateItem and checks the result as apply_update_validated does. Then
it sizes the update with validate_update_item_size. When the depth
check refuses, it first checks the statement figure with the new
validate_update_statement_size in crates/core/src/validation/mod.rs,
the statement half of validate_update_item_size. The SQLite and
PostgreSQL UpdateItem paths call it in place of their two calls.

The measurements give two orders: the statement size ahead of the
depth, and the depth ahead of the size of the updated item. The answer
changes only for an update whose statement is over the limit and whose
result nests too deep. The vector check keeps its place after the
depth check and ahead of the size check. A PartiQL UPDATE and the
legacy AttributeUpdates form are not sized by their statement, so
their checks keep their order. TransactWriteItems sizes the updated
item after the depth check already. The MongoDB and Cassandra
UpdateItem paths do not size the statement, and do not change.
The doc comment of unwritable_update_error said that the item check on
the updated item answers once the condition holds and the update
applies. With this branch, apply_update_validated refuses first when an
update stores a value past the nesting limit. It checks each attribute
that a SET or an ADD writes, and its text is the text of
ITEM_NESTING_MESSAGE.

The comment now says that. It also says that the item check, which runs
next, refuses only an attribute stored past the limit that the update
does not write. The code does not change.
The service was measured on 2026-11-10 at 20:39 UTC in us-east-1, two
sends each. Each item held x and 30 maps in m.

- BatchExecuteStatement: the first member sets x on one item and
  answers no error. The second member sets a value of 2 maps at 30
  keys below m on another item. Its entry is the ValidationError
  "Nesting Levels have exceeded supported limits". The first item
  takes the update, and the second item does not change.
- ExecuteTransaction with the same two members is cancelled with the
  reasons None and ValidationError, the second with the nesting
  message. Neither item changes.
- ExecuteStatement with SET big = ? (409,500 bytes) and the same deep
  SET answers ValidationException with the nesting message, with the
  SET clauses in either order. The item does not change.
- With the deep SET at 28 keys, the same statement answers "Item size
  to update has exceeded the maximum allowed size".

Five dual-target tests in tests/test_partiql_item_depth.py encode these
answers. The size test skips MongoDB and Cassandra, as
tests/test_partiql_item_size.py does. The items of the two earlier
tests in the class now hold x, as the measured items did. Against a
SQLite server built from the parent commit, the five tests pass, so the
code does not change.
docs/differences-from-dynamodb.md said that a transacted Update is
sized by the finished item alone. The service refused one transacted
Update for its size before the transaction ran (measured on 2026-11-10
at 19:28 UTC in us-east-1, two sends). The Update set a 409,500-byte
value, and a second SET action took a map past the nesting limit. The
answer was ValidationException "Item size to update has exceeded the
maximum allowed size", and the item did not change.

ExtendDB runs that transaction and cancels it with one ValidationError
reason that carries "Nesting Levels have exceeded supported limits".
Before this branch, PostgreSQL and SQLite cancelled it with a reason
that carried the size message. The rule behind the check of the
service is not measured. #937 tracks the measurement and the check.

The paragraph now gives this case, and a new row of the table records
the request, both answers and the issue. The row above it no longer
says that every transacted Update is sized on the finished item.
- The "# Errors" section of apply_update_validated now gives the
  nesting refusal. Its text is "Nesting Levels have exceeded supported
  limits", for an attribute that a SET or an ADD writes past the limit.
- Three comments said that the legacy AttributeUpdates form is not
  sized by its statement. They are the doc comments of
  apply_update_item_validated and validate_update_statement_size, and
  the comment of a unit test. validate_update_item_size sizes that form
  with a figure that it computes from the finished values of the
  attributes the form names. The comments now say that, and that the
  statement check does not apply to that form or to a PartiQL UPDATE.
  The unit test is renamed to
  a_partiql_or_legacy_update_keeps_the_depth_refusal. The comment of
  apply_update_item_validated also says that the order of the depth
  check against the vector check is not measured.
- docs/dynamodb-limits.md named validate_item_nesting_depth as the
  nesting check of UpdateItem. The row now names
  validate_attribute_values_nesting_depth for the values of UpdateItem.
  It names validate_written_nesting_depth for the attributes that
  UpdateItem, a TransactWriteItems Update or a PartiQL UPDATE writes.
- A comment in tests/test_update_result_nesting_depth.py cited
  docs/differences-from-dynamodb.md for the wording of the Cassandra
  size refusal, which that file does not give. The comment now gives
  the text, "Item size has exceeded the maximum allowed size", and
  names validate_item_size as its source.

The code does not change.
@LeeroyHannigan

Copy link
Copy Markdown
Collaborator Author

@pdf-amzn I pushed e91da2e. It merges main and answers each item. On 2026-11-10 I measured every service answer the new tests encode, two sends each.

  • The depth and size order: the order depends on where the size is. When the request's own values carry the size, UpdateItem answers the size message first, with the SET actions in either order. When the stored item carries the size and the request's values are small, it answers the nesting message. 81768cc adds tests of both shapes and the shallow-path control. On SQLite and PostgreSQL, f29f8e1 sizes the statement when the depth check refuses, and answers the size message when the statement is over the limit. The description now gives these rows in place of the old order sentence.
  • The walker: 38ef819 checks written attributes with nests_beyond_item_limit, the walker of PutItem's check. MAX_ITEM_NESTING_DEPTH has one definition, and the update path keeps its own text.
  • Main: I merged main (e1f2c92) and did not rebase, because the branch's pushed commits stay as they are. At e91da2e on SQLite, test_partiql_item_depth.py gives 43 passed and 3 xfailed. test_partiql_transaction.py and test_partiql_check_order.py give 263 passed and 2 xfailed. f7e3da1 updates the comment of unwritable_update_error. After the merge every PartiQL write meets the check. A BatchExecuteStatement member, an ExecuteTransaction member and a 409,500-byte parameter beside the deep SET all get the nesting answer on the service, and e47a2ef encodes them.
  • The ADD gaps: I measured both and filed UpdateItem: ADD through a missing parent map creates the map, and DynamoDB refuses the path #939 (ADD through a missing parent map) and UpdateItem: ADD on a list element is refused, and DynamoDB applies it #940 (ADD on a list element). The service also applies ADD a[0] on a = [5], and ExtendDB's code refuses any list element as the target.
  • The unmeasured orders: the comment of apply_update_item_validated says that the order against the vector check is not measured. The description lists that order and the MongoDB order against validate_index_keys_after_update.
  • The first commit's message: the description asks for a squash merge. The message of f29f8e1 has the same kind of error about the legacy AttributeUpdates form, and e91da2e fixes the code comments.
  • The Vec::contains dedup: I left it as it is. You marked it optional, and the list holds one entry for each attribute that the expression writes.

The measurements also found a gap in TransactWriteItems. The service refuses an Update that sets a 409,500-byte value beside the deep SET before the transaction runs, and ExtendDB cancels the transaction. #937 tracks it, and docs/differences-from-dynamodb.md records it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UpdateItem: a result nested more than 32 levels is applied; the service refuses it

3 participants