feat(partiql): implement ExecuteStatement, BatchExecuteStatement, ExecuteTransaction by LeeroyHannigan · Pull Request #411 · ExtendDB/extenddb

LeeroyHannigan · 2026-10-19T20:21:14Z

What

This PR adds the three PartiQL operations: ExecuteStatement, BatchExecuteStatement and ExecuteTransaction. They run SELECT, INSERT, UPDATE and DELETE statements, and EXISTS checks inside a transaction. The server authorizes each statement by itself, as Amazon DynamoDB does.

The engine translates each statement into operations that it already has. A statement runs as the point reads, Queries, Scans and item writes that the service makes for it, through the existing handlers. The PR changes no storage backend and no handler of a classic operation.

crates/core/src/partiql/ holds the lexer, the parser, and the planner that maps a WHERE to key reads.
crates/engine/src/partiql/ holds the executors and the code that authorizes each statement.
Both modules compile only with the default-on partiql Cargo feature. With the feature off, the three operations answer UnknownOperationException, as on main.
Outside the two modules and the tests, the PR changes 17 files, with 468 lines added and 6 lines deleted. The commit messages name them.

Reads

A SELECT reads by key when every disjunct of its WHERE names a key. Any other SELECT scans the table or the index.
A read by key splits into at most 50 disjuncts. A page visits at most 10 of them, in the order that the service uses. A page stops at Limit items or at 1 MB of item bytes read.
An index read takes one Query for each disjunct.
ORDER BY, projections, positional parameters and the NextToken follow the rules and messages of the service.

Writes, batch and transaction

INSERT creates an item and answers DuplicateItemException for a key that exists. UPDATE and DELETE need the full key. The other terms of the WHERE become the condition expression.
The service sizes an UPDATE by its finished item alone, and sizes a standalone UpdateItem by its statement. At a three-byte key an UPDATE writes an item of exactly 409,600 bytes, while UpdateItem stops 17 bytes short. The UPDATE executor asks the storage layer for the finished item rule.
BatchExecuteStatement takes 1 to 25 statements, all reads or all writes, with one response entry for each statement.
ExecuteTransaction takes 1 to 100 statements and runs all of them or none. It gives positional cancellation reasons and honours ClientRequestToken.

Authorization

The server authorizes each statement with dynamodb:PartiQLSelect, PartiQLInsert, PartiQLUpdate or PartiQLDelete. The resource is the table, or the index alone for a read of an index. The operation itself needs no action.
The engine asks an ItemAuthorizer in its OperationContext. The server's authorizer evaluates the caller's policies with check_authorization, and allows every access in developer mode. Without an authorizer, the engine refuses every statement.
The engine authorizes each disjunct of a read by key with its own dynamodb:LeadingKeys value. A refused statement in a batch gets its own AccessDenied entry. A refused transaction member cancels the transaction.
The condition keys LeadingKeys, FullTableScan, EnclosingOperation, Attributes, Select and ReturnValues hold the values that the service gives them.
Each check stays on the side of authorization where the service makes it. So a caller with no permission gets AccessDeniedException for a missing table, a missing index or a key of the wrong type.

Differences from DynamoDB

docs/differences-from-dynamodb.md records one difference: ExtendDB does not throttle a PartiQL request by table. The throttle reads the top-level TableName of a request, and a PartiQL request names its table in the statement.

Known gaps

The measurements do not cover these orderings for a caller that has no permission. In each case the code follows the measured refusals beside it, and the module documentation names the choice.

In a read transaction, ExtendDB refuses a member that reads an index, or whose WHERE lacks the full primary key, before authorization.
ExtendDB refuses two batch statements on one item before it authorizes them.
A page visits only a disjunct that the policy refuses, while another held-back disjunct fails a check. That page gets the refusal of the check.
A statement fails an ORDER BY check or has a key of the wrong type, and also has overlapping disjuncts or more than 50 disjuncts. ExtendDB authorizes that statement first.
ExtendDB refuses a NextToken that another statement gave before it authorizes the statement.

The measurements do not cover these orderings for a caller with permission either:

A WHERE that does not read by key, against a key value of the wrong type.
On an index, a path into a key attribute against the ORDER BY refusals.
Two maps in one statement that each repeat a name. ExtendDB reports the map whose repeated name comes first.

The refusal message for a role session differs from the service's message. ExtendDB names the session with an arn:aws:iam:: ARN and says that no identity-based policy allows the action. The service uses an arn:aws:sts:: ARN and names the session policy. Every operation uses this message, so one test stays a strict expected failure until the shared code changes.

The NextToken is not signed. The server authorizes each request that presents a NextToken again, from the statement in that request.

Commits

The parser and the planner, in core.
The executors and the docs, in the engine.
The dual-target wire tests.
The authorization of each statement.
The authorization tests.
The clippy pedantic fixes in the new code.
The item size of an UPDATE.

Each commit message lists the rules that its part implements.

Why

On main, the server answers the three operations with UnknownOperationException, so a client that uses PartiQL cannot run against ExtendDB. The tests expect the values and messages that Amazon DynamoDB gave in us-east-1, measured between 2026-10-19 and 2026-11-03.

Testing done

Wire tests: 35 files, tests/test_partiql_*.py, with 1,976 cases.
- 32 files with 1,919 cases are dual-target. On 2026-11-04 against Amazon DynamoDB, the new tests/test_partiql_item_size.py gave 13 passed. On 2026-11-03 the other 31 files gave 1,902 passed, 3 skipped and 1 failed. The failed case makes an eventually consistent read right after a PutItem, and the service returned no item. The case passed on a rerun, and so did the rest of its file.
- 3 files with 57 cases test authorization on ExtendDB only, because they make users and roles through the management API.
- On ExtendDB with SQLite, 1,967 passed, 3 skipped and 6 are expected failures:
  - 4 divergences in code that PartiQL shares with the classic operations, where the fix belongs. They are set_add on a list element (classic ADD), and the 32-level nesting check on 3 parameter values that a statement does not store.
  - The role session message above.
  - The order in which ExtendDB checks the members of a parameter map. The order depends on whether the build includes the MongoDB crate, so this expected failure is not strict.
- The pytest jobs of CI run the 35 files on PostgreSQL, MongoDB and Cassandra. On MongoDB and Cassandra the cases of tests/test_partiql_item_size.py that need a refusal skip, because those backends size updates differently (MongoDB applies no item size limit to updates, and Cassandra sizes updates on the finished item against a fixed 400 KiB with the PutItem wording #818).
The authorization tests on main (test_fgac_leading_keys.py, test_fgac_attributes.py, test_auth_permissions.py and test_authz_resource_arn.py): 143 passed and 1 skipped.
The item size and index key tests on main (test_item_size_at_limit.py and test_index_key_validation_on_update.py): 122 passed.
Unit tests of the core, engine, server and auth crates: 1,972 passed. The PartiQL unit tests also pass in the build that keeps JSON maps in request order, which includes the MongoDB crate.
The test jobs of CI run cargo test --workspace and cargo +1.88.0 check --workspace --all-targets.
cargo fmt --all -- --check, cargo +1.97.0 clippy --all-targets -- -D warnings and cargo check --no-default-features of the core and engine crates are clean.
Clippy pedantic reports no new warning other than doc_markdown. Its 123 doc_markdown warnings flag names such as PartiQL and NextToken in doc comments, which main also leaves unquoted, with more than 540 such warnings. The sixth commit clears the other 24.

Checklist

I have read CONTRIBUTING.md
All tests pass (cargo test --workspace)
Code is formatted (cargo fmt --check)
Clippy is clean (cargo clippy -- -W clippy::pedantic)
I have added or updated tests for new functionality
I have updated documentation if behavior changed
Breaking changes are noted below (if any)
If this changes the wire protocol, Storage trait, auth model, on-disk
format, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. The PR adds the three operations as the service runs them, and authorizes each statement as the service does. docs/differences-from-dynamodb.md records the one difference.

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

Adds crates/core/src/partiql, compiled only with the new default-on `partiql` Cargo feature. It turns a statement string into an AST and then into a plan made of the operations the engine already has. Nothing else in core depends on it. The lexer and parser take SELECT, INSERT, UPDATE (also in its FROM-first form), DELETE and, inside ExecuteTransaction, EXISTS, with positional parameters. They refuse what the service refuses, with its messages: a statement longer than 8192 characters, a WHERE of more than 300 operators, an IN list of more than 100 members, a value nested inside 32 maps or lists, a number with more than 38 significant digits or outside the service's magnitude range, and reserved words used as names. A statement that nests deeper than 32 levels is parsed on a thread with a 64 MiB stack, and nesting past the service's limits (1000 groups, 1000 literal levels) is refused. The planner reads a WHERE the way the service does. OR and IN on key attributes expand into disjuncts, and each disjunct is a point read, a Query of one partition, or part of a scan. Overlapping key conditions, key values of the wrong type and paths into key attributes are refused in the order the service checks them. DuplicateItemException is a new error variant. Only a PartiQL INSERT of a key that already exists returns it. Each rule and message was measured against Amazon DynamoDB in us-east-1 between 2026-10-19 and 2026-11-03, and the code cites the measurement where it implements the rule.

…eTransaction Adds crates/engine/src/partiql behind the same `partiql` feature. It translates each statement onto the handlers the engine already has (get, query, scan, put, update and delete, and their batch and transaction forms) and changes no storage backend and no classic handler. The dispatch gains three arms and is_known_operation three entries, each behind the feature. With the feature off the server answers the three operations with UnknownOperationException as before, and removing PartiQL is deleting the two partiql modules, those arms and entries, the DuplicateItemException variant and the feature. SELECT reads by key when every disjunct of its WHERE is keyed, and scans the table or index otherwise. A read by key that splits into more than 50 disjuncts is refused with "Too many decomposed read operations for a given query." A page visits at most 10 disjuncts, the first page in key order (or the ORDER BY order) and later pages in the order the service uses, and stops at Limit items or at 1 MB of item bytes read. A page of a read by key has a LastEvaluatedKey only when it holds Limit items, and every page reports the capacity of its own reads. An index read takes one Query for each disjunct. A disjunct that fails a check made when it is read (a key over the size limit, a key attribute in its filter) is held back: it takes a visit and reads nothing, and the request whose page would visit only held-back disjuncts gets its refusal. The NextToken carries where the next page goes on from and digests of the statement and its parameters, and a token sent with another statement or other parameter values is refused. INSERT creates an item and answers DuplicateItemException for a key that exists. UPDATE needs the full key, does not create a missing item, and supports SET (with arithmetic, list_append, set_add and set_delete), REMOVE and the RETURNING modes. DELETE needs the full key, succeeds on a missing item and allows RETURNING ALL OLD *. Every other term of a write's WHERE becomes its condition expression. BatchExecuteStatement takes 1 to 25 statements, all reads or all writes, and answers each statement in its own response entry. ExecuteTransaction takes 1 to 100 statements, all reads or all writes and EXISTS checks, cancels as a whole with positional cancellation reasons, and honours ClientRequestToken. The requests a statement is translated into run with the classic expression size and token limits raised to 131,072 (16 for each of the 8192 characters a statement may hold), so the translation of a statement the service accepts is not refused for its length. docs/differences-from-dynamodb.md lists the three operations as supported and adds a PartiQL section. The difference it records: a PartiQL request is not throttled by table, because the throttle reads the request's top-level TableName and a PartiQL request names its table in the statement. Behaviour was measured against Amazon DynamoDB in us-east-1 between 2026-10-19 and 2026-11-03.

Adds 30 files, tests/test_partiql_*.py, with 1,883 test cases. Each case runs against an ExtendDB server when EXTENDDB_TEST_ENDPOINT is set and against Amazon DynamoDB otherwise, with the same assertions. Every expected value was measured on the service in us-east-1, and on 2026-11-03 the whole set ran against the service: 1,880 passed and 3 were skipped. Where the service gives more than one answer to the same request, the tests accept each answer it gave: whether a LastEvaluatedKey or an empty page follows the last item of a partition, which depends on where the partition sits in storage, and the two forms in which a validation message echoes a statement list (its fields, or the name of the service's request object). A test that reads an index first waits until the index shows every item it wrote. Expected failures on ExtendDB: - The order in which parameter map members are checked: request order when the build includes the MongoDB crate, name order otherwise. Not strict. - Two divergences in code PartiQL shares with the classic operations, to be fixed there: set_add and set_delete on a list element (classic ADD and DELETE), and the 32-level nesting check on parameter values a statement does not store. Strict.

Until this commit the server authorized the three PartiQL operations once, before dispatch, with the operation's action against table/*. A Deny on one table did not apply, a dynamodb:LeadingKeys condition had no values to check, and a policy that allows dynamodb:PartiQLSelect and the other PartiQL actions allowed nothing. The service authorizes each statement by itself, with dynamodb:PartiQLSelect, PartiQLInsert, PartiQLUpdate or PartiQLDelete against the table it reads or writes, or against the index alone for a read of an index. It needs no action for the operation itself, and the classic actions grant nothing. The engine gains an ItemAuthorizer trait, carried in OperationContext. The server's implementation evaluates the caller's policies with check_authorization against this deployment's table or index ARN, with the request-level condition keys every other operation carries and the item-level keys of the statement. In developer mode it allows every access, as authorize_request does for the other operations. authorize_request no longer authorizes the three operations, and the engine refuses every statement when its context has no authorizer. The trait is not behind the partiql feature, and only PartiQL uses it. Each executor makes its checks in the order a permitted caller gets them and authorizes the statement before it reads or writes an item. The checks of the request and the refusals made at parse time come first, as before. Among the checks of a statement, these also come before authorization: a parameter shortfall, SELECT COUNT(*), ConsistentRead on a global secondary index, a path into a key attribute, overlapping disjuncts, more than 50 disjuncts, a NextToken that does not decode or that another statement gave, a DELETE RETURNING clause other than ALL OLD *, a BatchExecuteStatement read without the full primary key, and an EXISTS member of the wrong form or on an index. Every other check comes after it, so a caller with no permission gets AccessDeniedException for a missing table, a missing index, a key of the wrong type or a duplicate INSERT. A check made before authorization is answered first only when no check ahead of it fails. Otherwise the statement is authorized first. Three refusals are made before authorization by analogy with the measured refusals beside them, and none is measured for a caller with no permission: a read transaction member that reads an index, one that does not name the full primary key, and two batch statements on one item. The module documentation says so where each is made. A read by key authorizes each disjunct with its own partition key value, at most 50 values, the MAX_DECOMPOSED_READS limit. A refused disjunct is held back like one that fails a check made when it is read: it takes a visit and reads nothing, and the request whose page would visit only held-back disjuncts gets AccessDeniedException. Across the disjuncts of one read, an authorization refusal ranks below the refusals of the other checks. A refused statement in BatchExecuteStatement is its own response entry, {"Error": {"Code": "AccessDenied", ...}, "TableName": ...}. A refused member of ExecuteTransaction cancels the transaction before any condition is evaluated, with AccessDenied for each refused member and None for the others. The condition keys: - dynamodb:LeadingKeys holds the partition key value of each disjunct or written item. A number is compared in the form Java's new BigDecimal(s).toString() prints for the text the statement or the parameter gives it (NumberForm::Decimal): under ForAllValues ["10"], pk = 10 and pk = 010 run and pk = 10.0 is refused. - dynamodb:FullTableScan is "true" for a scan and "false" for a read by key in ExecuteStatement, and absent in BatchExecuteStatement, ExecuteTransaction and every write. RequestParams in the auth crate gains full_table_scan to carry "false". - dynamodb:EnclosingOperation is BatchExecuteStatement or ExecuteTransaction inside those operations, and absent in ExecuteStatement. - dynamodb:Attributes holds the table's key attributes and every top-level name the statement uses: projection, WHERE, SET values, REMOVE targets and conditions. - dynamodb:Select is ALL_ATTRIBUTES for * on a table, ALL_PROJECTED_ATTRIBUTES for * on an index, and SPECIFIC_ATTRIBUTES for a projection list. - dynamodb:ReturnValues is NONE for a write without RETURNING and follows RETURNING otherwise. An EXISTS member is authorized as a read. An index read of a missing table is authorized against the table ARN. Two refinements outside authorization, measured with a permitted caller: - A map literal that names a key twice is refused at parse time with "Duplicate keys in tuple. Key: <name> under <path> at <line>:<column>:<length>". At the top level of an INSERT VALUE the position is the second occurrence of the name, with the length of the quoted name. In any other map it is that map's opening brace, with length 1. Before this commit ExtendDB accepted the literal and kept the last value. common::insert_values is the one place that reads an INSERT's values, for authorization, the write and the duplicate check of a batch or transaction. - A SELECT makes its key checks in the service's order: a path into a key attribute, an ORDER BY term that is not a key attribute, the key values, the hash-key ORDER BY rule, overlapping disjuncts, and then the number of disjuncts. ExtendDB checked the key values and overlap first, so 8 measured statements got another refusal, for example (pk = 'a' OR pk = 'a') ORDER BY n. Outside the two partiql modules this commit changes 10 files, with 346 lines added and 3 removed: - crates/engine/src/item_authorizer.rs, new: the trait and ItemAccess, with the field and re-export in crates/engine/src/lib.rs. - crates/server/src/item_authz.rs, new: the server's authorizer, with its module line in crates/server/src/lib.rs and its construction in crates/server/src/handler.rs. - crates/server/src/request_helpers.rs: authorize_request returns early for the three operations. - crates/auth/src/policy/context.rs: RequestParams.full_table_scan, with a unit test. - crates/server/src/leading_keys.rs: extract_typed_value is pub(crate). - crates/engine/Cargo.toml and Cargo.lock: extenddb-storage-sqlite as a dev-dependency, for the unit tests that build an OperationContext. Removing PartiQL now also means deleting item_authz.rs and its lines in handler.rs and lib.rs, the early return in authorize_request, and the trait module, field and re-export in the engine. full_table_scan and the visibility of extract_typed_value change no other operation and can stay. Each rule was measured against Amazon DynamoDB in us-east-1 on 2026-11-03.

Adds ExtendDB-only wire tests for the authorization of each statement, and dual-target tests for the two refinements of the previous commit. The authorization tests, 57 cases in three files, are ExtendDB-only. They create users, roles and policies through the management API, so they need the builtin auth provider and the admin login in EXTENDDB_ADMIN_USER and EXTENDDB_ADMIN_PASSWORD. Each test gives an IAM user the permissions the measured role session had, in its identity policy, and expects the answer the service gave that session on 2026-11-03. - test_partiql_authorization.py: the action of each statement type, table and index resources, the answers of the three operations, dynamodb:LeadingKeys for each disjunct and each write, dynamodb:FullTableScan, dynamodb:EnclosingOperation and session policies. Two tests send every statement path, and a NextToken a permitted caller got, from a caller without a PartiQL action. - test_partiql_authorization_keys.py: EXISTS members, dynamodb:ReturnValues, dynamodb:Attributes, dynamodb:Select, number partition keys, writes without dynamodb:FullTableScan, and an index read of attributes the index does not hold. - test_partiql_authorization_order.py: the checks on each side of authorization, and the pages of a read by key with refused disjuncts. - partiql_authz_helpers.py holds the principals, policies and refusal checks the three files share. One case is a strict expected failure: the refusal of a role session. The shared authorization message names the session with an arn:aws:iam:: ARN and says that no identity-based policy allows the action. The service names it with an arn:aws:sts:: ARN and names the session policy. Every operation uses that message, so the fix belongs there. The dual-target cases, 23 in two files, ran against Amazon DynamoDB in us-east-1 on 2026-11-03, and all 23 passed. - test_partiql_repeated_names.py: a map literal that names a key twice, at the top level of a VALUE, in a nested map, a list, a SET value and a WHERE literal. Also names that differ only in case, a syntax error after the value, and the refusal in BatchExecuteStatement and ExecuteTransaction. - test_partiql_order_by.py, 12 new cases: the ORDER BY refusals among the key checks, with ORDER BY on a key and on another attribute.

`cargo clippy -- -W clippy::pedantic` reported 144 warnings in the code this PR adds that main does not have. This commit clears the 24 that are not doc_markdown, with no change in behaviour: - single_match_else, manual_let_else, if_not_else, match_same_arms, needless_for_each and many_single_char_names take the form clippy suggests. - next_prefix in plan/ranges.rs returns Option<Vec<u8>> instead of Option<Option<Bound>>, and the BeginsWith arm maps the result back to the type of its bound. - shape_response and write::reshape return DispatchResult instead of a Result that was always Ok. The internal helpers of base_join.rs and returning.rs take the serde_json error by reference. - detect_duplicate_keys keeps a HashSet instead of a HashMap with unit values. - The server boxes the check_authorization future, as large_futures suggests. - split in plan/split.rs moves the cross product of an AND into its own function. lex, batch::run and select::run keep their length, and Parser keeps its six flags, each with an allow and its reason. The 120 doc_markdown warnings stay. They flag names such as PartiQL and NextToken in doc comments, which main also leaves unquoted: main has 542 such warnings.

Since #416, the SQLite and PostgreSQL storage layers size a standalone UpdateItem by its statement: a base of 3, 18 per SET clause, and the written paths with their values, checked against the limit beside the finished item. A PartiQL UPDATE runs as an UpdateItem, so it inherited that rule. At a three-byte key it refused a finished item of 409,600 bytes, which the service writes. The service sizes a PartiQL UPDATE by its finished item alone, as it sizes a transacted Update (measured in us-east-1 on 2026-11-04): - UPDATE ... SET p = ? writes a finished item of exactly 409,600 bytes at a key where UpdateItem with SET p = :p stops 17 bytes short. - One byte more is refused with "Item size to update has exceeded the maximum allowed size". - A second SET clause or an if_not_exists adds nothing to the size. - BatchExecuteStatement gives the refusal as the statement's ValidationError entry. ExecuteTransaction cancels with a ValidationError reason with the same message. - A WHERE term that fails, and the implicit check that the item exists, are answered first: an UPDATE over the limit that would not run gets ConditionalCheckFailedException. ExpressionMaps gains finished_item_sizing. When it is set, validate_update_item_size checks only the finished item, as validate_transact_update_item_size does. handle_update_item now calls handle_update_item_with(body, ctx, false), and the PartiQL UPDATE executor calls handle_update_item_with with true, in ExecuteStatement and BatchExecuteStatement. ExecuteTransaction already ran through the transacted Update and its sizing. Outside the two partiql modules this commit changes: - crates/core/src/expression/resolver.rs: the finished_item_sizing field. - crates/core/src/validation/mod.rs: the branch in validate_update_item_size, with a unit test. - crates/engine/src/update_item.rs: handle_update_item_with. - docs/differences-from-dynamodb.md: the Item Size section names the PartiQL UPDATE beside the transacted Update. Removing PartiQL now also means deleting the field, the branch and handle_update_item_with. tests/test_partiql_item_size.py adds 13 dual-target cases. All 13 passed against the service on 2026-11-04. The cases that need a refusal skip on MongoDB and Cassandra, whose update sizing differs from the service's, as docs/differences-from-dynamodb.md records.

test_a_statement_nested_to_the_length_limit_is_refused_and_the_server_keeps_serving accepted any ClientError, so an InternalServerError from the local server passed it. DynamoDB answers some of the shapes with InternalFailure, so the dual-target test cannot require HTTP 400 on both targets. On the local target it now requires HTTP 400 and the code ValidationException for each of the 42 shapes. The live target keeps the broad check.

The SET value walk in values.rs reads the values in written order. It does not check the operand type of an arithmetic literal ('x' + 1). The translation of the statement does, after the walk. DynamoDB puts that refusal in the same place. It comes after every number rule, empty bag and call refusal in the SET values, in whatever order the statement gives them. Measured 2026-11-06 in ExecuteStatement, a batch and a transaction. SET a = 'x' + 1, b = size(m) and SET b = size(m), a = 'x' + 1 get the size message. SET a = 'x' + 1, b = <<>> gets the bag message. SET a = 'x' + 1e126 and SET b = 1e126, a = 'x' + 1 get the number message. SET a = 'x' + 1, b = 2 - true names the + and the S. The module comment in values.rs records this. The dual-target tests test_the_operand_type_of_an_arithmetic_literal_comes_after_the_other_set_value_checks and test_the_operand_type_of_an_arithmetic_literal_comes_after_a_call_in_a_batch_and_a_transaction in tests/test_partiql_grammar.py hold the measured answers.

hitesh96db

Re-review of 0e4b4a8d (the 11 partiql commits from f9255f1a on, plus the merge of main, whose only non-main edits add item_authorizer: None to two OperationContext literals).

Setup. Measured 2026-11-06. ExtendDB: this PR at 0e4b4a8d, cargo build --release of the workspace, PostgreSQL backend, on 127.0.0.1. Amazon DynamoDB: us-east-1. Last round's experiments use the table and seed item from that review (partiql_probe_sf, hash pk S / range sk S, item {pk: 'h', sk: 's', m: {k: 1}, a: 1}), created and deleted by the same script, which I re-ran unchanged against ExtendDB.

The three blockers from last round are fixed. All 12 experiments now give DynamoDB's answers, read-backs included:

Blocker	`c745a3c0`	`0e4b4a8d` (same as DynamoDB)
A write the batch can't build (`SET a = size(m)`, `attribute_exists`, `begins_with`, `upper`)	request-level `Syntax error; token: "("`, neither INSERT written	that statement's own `ValidationError`, both INSERTs written
`<<>>` in a batch INSERT or UPDATE	HTTP 500 `InternalServerError`, nothing written	`Empty bags are not supported` for that statement, the other INSERT written; ExecuteStatement and ExecuteTransaction give the same message
`m[k[k[...]]]` at 10, 1,000 and 2,700 levels	refused at the innermost `k`	`Unexpected path component at 1:53:1`; the server stays healthy

Claims I checked rather than took on trust:

A path in brackets is read in a loop. rest_of_path_in_brackets loops with a counter, and anything else in an inner bracket goes through expression_in_brackets, inside self.nested(Nest::Operand, ..), so it is within the operand cap. every_nesting_form_at_the_length_limit_parses_on_a_small_stack passes here.
A parameter of the wrong JSON type fails the request first. check_types recurses over M/L, but the handler's serde_json::from_slice keeps its default recursion limit (no crate enables unbounded_depth): raw signed requests with a parameter nested 100 to 100,000 levels deep (up to 800 KB) all get SerializationException ... recursion limit exceeded, and the server stays healthy.
Translation errors under 500 stay with their statement. translate_write sends every error through untranslated(); check_translation no longer runs for UPDATE/INSERT, but for those it only called build_*_parts, which still run.
The dual-target tests match the service. tests/test_partiql_{batch,grammar,limits,transaction,write}.py: against DynamoDB 1,783 passed, 3 skipped; against this PR 1,781 passed, 3 skipped, 2 xfailed (the transaction index units, #866). The whole tests/test_partiql_*.py against this PR: 3,061 passed, apart from the authorization files, which I could not run (they need the instance's admin password); this round does not touch authorize/, and CI's run-tests passes.

One finding, inline: the_deepest_statements_that_parse_are_walked_on_a_small_stack overflows its stack on macOS arm64 and aborts the whole extenddb-core unit-test binary. Linux CI passes, so it is a portability problem of the test, not a server crash that I could show.

Correction to my previous review. In the first blocker's Experiment 1 I quoted DynamoDB's answer for the size statement as Unrecognized function: upper at 1:33:5. That is Experiment 4's message. DynamoDB answered The function is not allowed in an update expression; function: size, as your reply says.

the_deepest_statements_that_parse_are_walked_on_a_small_stack ran the server's walks of a parsed statement on a thread with a 1 MiB stack. The deepest walk, the literal checks of a condition with a value 1000 levels deep, needs 870 KB on a Linux x86_64 debug build. Debug frames are larger on macOS arm64, where the same walk needs about 1.1 MB. The thread overflowed there and aborted the whole extenddb-core test binary. The server runs these walks on a tokio worker thread, whose stack is tokio's default of 2 MiB. The test now gives the walks that stack, so it checks the bound that the server depends on.

hitesh96db

Approving at ba6f6426. The three blockers from the second round are fixed (re-measured against DynamoDB and this PR on 2026-11-06), the changed PartiQL wire tests pass against both, and the macOS stack-test overflow is fixed: core and engine unit tests pass on an M-series Mac. Thanks for the thorough fixes and the tracking issues for what remains (#884 to #888).

run-integration and run-integration-sqlite build a release server and run the Python suite of devtools/run-tests against it. This branch adds the PartiQL wire tests to that suite. On main the two jobs take 30 to 38 minutes. On this branch they took 37:55 and 38:24 at 0e4b4a8. At ba6f642 both hit the 40-minute limit twice, and GitHub cancelled them with no test failed. Their timeout-minutes goes from 40 to 60. The other jobs keep 40.

.github/workflows/integration.yml: main (#740) set timeout-minutes to 55 for run-integration and run-integration-sqlite, with comments that give its measured durations. This branch set both to 60 for the PartiQL wire tests. The result keeps 60, and each comment gives the durations from both changes. Main added crates/storage-sqlite/tests/restore_job_resume.rs, which builds an OperationContext. The test now sets item_authorizer, the field this branch adds, to None.

The merge of #740 adds its restore suites to the jobs that also run this branch's PartiQL tests. Each estimate below starts from the slowest run measured on this branch. It adds #740's increase outside the build step, measured on its merge queue run. It also adds the extra build time of a cold cache. - run-integration: 48:04 + 2:19 = about 50 min, against 60. The bound goes to 70. - run-rust-integration: 29:39 + 4:29 + 1:44 = about 36 min, against 40. The bound goes to 50. - cassandra-pytest: 42:10 + 1:59 = about 44 min, against 45. #740 added no time to this job outside the build. The bound goes to 60. - mongodb-rust: the slowest run on main took 38:42, against 40. Neither change adds time to it. The bound goes to 55. Each new bound is about 1.4 times its estimate. #740 used the same ratio when it set 55 for a 40-minute run. run-integration-sqlite (about 45 min, against 60) and run-tests (about 12 min, against 15) keep their bounds.

Main adds PartiQL SELECT (#411), which builds a Scan body and calls handle_scan with two arguments, and this branch adds the raw_body argument to handle_scan. The call now passes an empty raw body. Scan reads the raw body only for the request order of the members of ExclusiveStartKey, ExpressionAttributeValues and ScanFilter. With an empty raw body, the parsed body's member order stands in for the request's order. The order sets which fault the Scan reports first. PartiQL builds this body itself, so no request text exists for it. The body holds TableName and, when the statement or the request gives them, FilterExpression, IndexName, ExpressionAttributeNames, ExpressionAttributeValues, Limit, ExclusiveStartKey, ConsistentRead and ReturnConsumedCapacity. It never holds ScanFilter.

crates/engine/src/lib.rs: main (#411) added the partiql module behind its feature where this branch added per_table_capacity. The result keeps both modules, in name order. This branch replaced parse_condition_expr, parse_update_expr and parse_optional_filter in expression_helpers.rs with parse_condition_checked and parse_update_expr_checked. These run the grammar pass and then the semantic pass against the request's expression maps. parse_optional_filter stays for the parser tests only. The PartiQL code from main called the replaced functions in three files. base_join.rs parses the filter of an index read that reaches back to the base table with parse_condition_checked. It keeps its two operand checks after the parse. item_depth.rs works out the answer of a statement or transaction that sets a value past the item nesting limit. It parses the conditions and the update with the checked functions, the condition first, as UpdateItem checks it. The condition and filter calls pass include_length as true, so an over-size expression keeps the " expression size: N" suffix that main's functions gave it. Three PartiQL unit tests check rendered text against the grammar. They now call parse_condition_expr_with and parse_optional_filter with the same argument. write/update.rs reads the actions of an UPDATE for its key and index checks before the statement runs as UpdateItem. For it, the merge keeps parse_update_expr as main has it, with the grammar checks and the repeated-section rule only. The semantic refusals then come from UpdateItem, in its order, and pass through bare_expression_error. A checked parse there would make BatchExecuteStatement answer a list_append operand of the wrong type with the "Invalid UpdateExpression: " prefix. The batch would also report that defect ahead of a condition defect, which ExecuteStatement reports first. This branch's grammar accepts any call, so the translation no longer refuses size(m) in a SET value. The statement checks refuse it before the translation. The batch unit test a_refusal_of_the_translated_request_is_the_statements_own drops that row, and checks that a list_append operand of the wrong type translates. On this branch PutItem, UpdateItem and DeleteItem put "1 validation error detected: " in front of their expression-stage messages. PartiQL INSERT, UPDATE and DELETE run through these handlers. partiql::bare_expression_error stripped only the "Invalid ...Expression: " prefix, so a statement's message would have kept both the wrapper and the prefix. strip_single_item_write_prefix in expression_helpers.rs undoes single_item_write_prefix, and both functions now read the wrapped message starts from one constant. bare_expression_error calls it first. A unit test covers a wrapped message, and an enum refusal that keeps its "1 validation error detected: ". This branch added creation_date_time to TableKeyInfo. The Cassandra restore engine from main (#740) builds a TableKeyInfo for its restore target. It sets the field to 0.0, as the legacy restore copy in backup_engine.rs does, because only the transaction capacity order reads it.

LeeroyHannigan requested review from amrith, c33howard, jcshepherd, pdf-amzn, robinnsc and yesyayen as code owners October 19, 2026 20:21

LeeroyHannigan force-pushed the feat/partiql branch 2 times, most recently from 2c60924 to afe7993 Compare October 20, 2026 11:06

This was referenced Oct 20, 2026

storage-cassandra: a PutItem that loses the create race can leave its LSI row behind #414

Closed

fix(query): check legacy conditions against the key schema #507

Open

LeeroyHannigan force-pushed the feat/partiql branch from afe7993 to 2036786 Compare November 3, 2026 20:48

LeeroyHannigan requested review from eliseong, hitesh96db and navareamol as code owners November 3, 2026 20:48

LeeroyHannigan added 7 commits November 4, 2026 01:09

LeeroyHannigan force-pushed the feat/partiql branch from 2036786 to af0fac7 Compare November 4, 2026 01:38

navareamol added the v1 Needed for the 1.0 release label Nov 4, 2026

LeeroyHannigan added 2 commits November 6, 2026 13:49

LeeroyHannigan requested a review from hitesh96db November 6, 2026 14:20

hitesh96db reviewed Nov 6, 2026

View reviewed changes

Comment thread crates/core/src/partiql/parser/limits/tests.rs Resolved

LeeroyHannigan requested a review from hitesh96db November 6, 2026 20:05

hitesh96db previously approved these changes Nov 6, 2026

View reviewed changes

LeeroyHannigan added 2 commits November 6, 2026 22:12

Merge branch 'main' into feat/partiql

5a677f2

LeeroyHannigan enabled auto-merge November 6, 2026 22:41

LeeroyHannigan dismissed hitesh96db’s stale review via 54c4256 November 6, 2026 23:07

LeeroyHannigan requested a review from hitesh96db November 7, 2026 00:16

LeeroyHannigan added 2 commits November 7, 2026 01:18

LeeroyHannigan mentioned this pull request Nov 7, 2026

Test a_retry_waits_on_its_registration_and_another_backup_is_refused fails when the first restore finishes before the third request #908

Open

hitesh96db approved these changes Nov 8, 2026

View reviewed changes

LeeroyHannigan added this pull request to the merge queue Nov 8, 2026

Merged via the queue into main with commit 0cfb0d9 Nov 8, 2026
47 checks passed

LeeroyHannigan deleted the feat/partiql branch November 8, 2026 21:20

LeeroyHannigan mentioned this pull request Nov 9, 2026

fix(scan): check AttributesToGet, ScanFilter, ExclusiveStartKey and ExpressionAttributeValues in the service's order #854

Open

8 tasks done

LeeroyHannigan mentioned this pull request Nov 9, 2026

docs: the README list of supported operations omits PartiQL and backups #913

Open

Navigation Menu

Repository navigation

feat(partiql): implement ExecuteStatement, BatchExecuteStatement, ExecuteTransaction - #411

feat(partiql): implement ExecuteStatement, BatchExecuteStatement, ExecuteTransaction#411
LeeroyHannigan merged 39 commits into
mainfrom
feat/partiql

LeeroyHannigan commented Oct 19, 2026 •

edited

Loading

Uh oh!

hitesh96db left a comment

Uh oh!

Uh oh!

hitesh96db left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

LeeroyHannigan commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Reads

Writes, batch and transaction

Authorization

Differences from DynamoDB

Known gaps

Commits

Why

Testing done

Checklist

Uh oh!

hitesh96db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hitesh96db left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LeeroyHannigan commented Oct 19, 2026 •

edited

Loading