Repository navigation
PartiQL (#411): ExtendDB refuses SET a = a + 1 + 1, and other parser forms and syntax texts differ #887
Description
Describe the bug
Follow-up to #411. This issue lists the parser forms and syntax texts at 0e4b4a8 that differ from the service. The first case is a statement the service runs and ExtendDB refuses. The rest are refusals on both sides with different texts, or with a different order among refusals.
Measured against Amazon DynamoDB in us-east-1 on 2026-11-06. Table: pk S hash key, sk S range key. The item pk = 'h', sk = 's' holds a 1, x 0, m {'k': 1} and l [1, 2]. Operation: ExecuteStatement unless a row says otherwise. In the tables, U stands for UPDATE "<t>" SET, W for WHERE pk = 'h' AND sk = 's', and a condition form is the tail of SELECT * FROM "<t>" WHERE pk = 'h' AND . syntax: stands for Statement wasn't well formed, can't be processed: and refused at L:C:N for Unexpected path component at L:C:N. The positions are for the table names the measurement used: 15 characters in the rows written with U and W, and 18 in the other rows. With the 18-character name, the first token inside m[ in a condition is at column 57. The ExtendDB answers are from 0e4b4a8 with the SQLite backend, debug build.
A statement the service runs:
| Statement | DynamoDB | ExtendDB at 0e4b4a8 |
|---|---|---|
U a = a + 1 + 1 W |
runs, a goes from 1 to 3 |
syntax: Unexpected token after expression |
operand_tail (crates/core/src/partiql/parser/operands.rs) reads one operator and one right operand, and trailing_error (crates/core/src/partiql/parser.rs) refuses the second + as a token left over after the statement. (a + 1) + 1 and ((a + 1) + 1) + 1 run on both sides.
Calls in a SET value and in a WHERE condition:
| Statement | DynamoDB | ExtendDB at 0e4b4a8 |
|---|---|---|
U a = count() W |
syntax: Aggregate functions are always unary |
Unsupported token in expression: count at 1:34:5 |
U a = nullif(m) W |
syntax: Expected COMMA |
Unsupported token in expression: UNKNOWN at 1:34:6 |
U a = cast(m AS INT) W |
Unsupported token in expression: cast at 1:34:4 |
syntax: Expected RIGHT_PAREN |
U a = "upper"(m) W |
Unrecognized function: upper at 1:34:7 |
syntax: Unexpected token after expression |
U x = 1 W AND count(m) = 1 |
Unsupported token in expression: count at 1:87:5 |
Unrecognized function: count at 1:87:5 |
U x = 1 W AND coalesce(m, 1) = 1 |
Unsupported token in expression: UNKNOWN at 1:98:1 |
Unrecognized function: coalesce at 1:90:8 |
U x = 1 W AND nullif(m, 1) = 1 |
Unsupported token in expression: UNKNOWN at 1:88:6 |
Unrecognized function: nullif at 1:88:6 |
U x = 1 W AND if_not_exists(m, 1) = 1 |
The function is not allowed in a condition expression; function: if_not_exists |
Unrecognized function: if_not_exists at 1:95:13 |
U x = 1 W AND list_append(l, l) = 1 |
The function is not allowed in a condition expression; function: list_append |
Unrecognized function: list_append at 1:93:11 |
INSERT INTO "<t>" VALUE {'pk': 'i', 'sk': 's', 'v': <<>> (no closing brace) |
syntax: Expected RIGHT_CURLY |
syntax: Expected } to close item value |
UPDATE "<t>" SET x = NOT x 10 a WHERE pk = 'h' AND sk = 'w', and the same with 99 |
Unsupported operator in Update Expression. Operator: not |
syntax: Unexpected operator |
(size( x 10 a )) x 10 = 1, and the same with 99 and 100 |
Incorrect operand type for operator or function; operator or functi (the measurement cut the text there) |
runs, 0 items |
INSERT INTO "<t>" VALUE {'pk': 'h', 'sk': 'i', 'x': m[k[0]]} |
Unsupported data type: PATH under key root at 1:68:1 |
syntax: Expected a literal value |
The SET value texts for count() and nullif(m) come from SetWalk::call in crates/core/src/partiql/values.rs. The WHERE texts come from where_function_refusal (crates/core/src/partiql/parser/forms.rs). The two INSERT texts come from item_map and literal_value in crates/core/src/partiql/parser/literals.rs.
An INSERT item key that is not a string:
| Statement | DynamoDB | ExtendDB at 0e4b4a8 |
|---|---|---|
INSERT INTO "<t>" VALUE {1: 'a', 'sk': 's'} |
Tuple keys must be of type String. Actual type: INT under root at 1:41:1 |
Tuple keys must be of type String. Actual type: Id under root at 1:41:1 |
Measured in ExecuteStatement on 2026-11-06. The service names the type of the key. item_map and map_members (literals.rs) write Id for every key that is not a string. {pk: 'a', 'sk': 's'} gets Actual type: Id on both sides. Not measured: {1.5: 'a'} and {true: 'a'}, for the type names.
An expression in the outer bracket of a path (21 cases, the rows below among them):
| Statement | DynamoDB | ExtendDB at 0e4b4a8 |
|---|---|---|
m[a = 1] = 1, m[k - 1] = 1, m[1 + 1] = 1, m[k + 1] = 1 |
refused at 1:59:1, the operator |
syntax: Expected RIGHT_BRACKET |
m[f(x)] = 1 |
refused at 1:57:1 |
syntax: Expected RIGHT_BRACKET |
m[NOT a] = 1 |
refused at 1:57:3 |
syntax: Unexpected operator |
m[(k)] = 1 and m[('k')] = 1 |
runs, 1 item (reads m.k) |
refused at 1:57:1 |
m[(1)] = 1 and m[+1] = 1 |
runs, 0 items (reads m[1]) |
refused at 1:57:1 |
m[*] = 1 |
Wildcard path component like a[*].b are not supported at 1:57:1 |
refused at 1:57:1 |
m[{] = 1 |
syntax: Unexpected term |
refused at 1:57:1 |
UPDATE "<t>" SET m[(a NOT = 1)] = 1 WHERE pk = 'h' AND sk = 'w' |
syntax: Expected RIGHT_PAREN |
refused at 1:35:1 |
The service also runs an Ion literal, a name or a number in parentheses, and +1 in the brackets of a SET or REMOVE target. A match needs the position of the root operator of the expression in the bracket, which the parser does not track (inner_bracket and expression_in_brackets, crates/core/src/partiql/parser/operands.rs).
Where a bracket refusal in the select list comes (11 cases):
| Statement | DynamoDB | ExtendDB at 0e4b4a8 |
|---|---|---|
SELECT m[k[0]] FROM "t" WHERE pk = 'h' |
Table name should be within the length [3, 255] ... (the name rule) |
refused at 1:10:1 |
SELECT m[k[0]] FROM t z WHERE pk = 'h' |
Aliasing is not supported |
refused at 1:10:1 |
SELECT m[k[0]] FROM t WHERE pk = 'h' LIMIT 1 |
Unsupported clause: LIMIT at 1:63:1 |
refused at 1:10:1 |
SELECT a AS z, m[k[0]] FROM t WHERE pk = 'h' |
Aliasing is not supported |
refused at 1:18:1 |
The service reports four refusals ahead of the bracket refusal. They are the table name rule, an alias on the table, a LIMIT clause, and an alias or a call written earlier in the select list. ExtendDB reports the bracket refusal first. The order of the other refusals is in the module comment of crates/core/src/partiql/parser/walk.rs.
Reserved words that the service reads to their end (10 cases):
| Statement | DynamoDB | ExtendDB at 0e4b4a8 |
|---|---|---|
a = CASE x 10 b, and the same with 99 |
syntax: Expected a WHEN clause in CASE |
syntax: Unexpected keyword |
m[k[SELECT a]] = 1 |
syntax: Expected FROM after SELECT list |
syntax: Unexpected keyword |
m[k[a = SELECT b]] = 1 |
syntax: Expected FROM after SELECT list |
syntax: Unexpected keyword |
m[k[PIVOT a]] = 1 |
syntax: Expected AT keyword |
syntax: Unexpected keyword |
m[k[DATE '2020-01-01']] = 1 |
refused at 1:57:1 (the literal parses) |
syntax: Unexpected keyword |
EXISTS(SELECT * FROM t) in a bracket parses on the service too. ExtendDB gives these forms the term error of the word (reserved_word_term, crates/core/src/partiql/parser/terms.rs), as terms.rs documents for the statement-level forms, which are not measured.
A map literal without its colon (3 cases): a = {'a'} gets syntax: Expected COLON from the service and syntax: Expected : in map literal from ExtendDB (map_members, crates/core/src/partiql/parser/literals.rs).
A key condition compared with true in a batch or transaction SELECT (4 cases): SELECT * FROM "<t>" WHERE pk = 'h' AND sk = 's' = true, with one = true, with two and with 297. In BatchExecuteStatement and in ExecuteTransaction the service refuses the statement for not naming the key, as it refuses a batch or transaction SELECT without its key. The transaction prefixes TransactStatements[0]. ExtendDB reads the item by its key. In ExecuteStatement the statement runs on both sides.
Not measured: BatchExecuteStatement and ExecuteTransaction for the rows that do not name them. The measurement of the NOT, size, bracket, select-list and reserved-word rows used an earlier tree of the #411 branch. That tree came before the change that fixed calls in a SET value. Those rows are not re-measured at 0e4b4a8. The measurement of the SET value calls (count(), nullif(m), cast, "upper") came after that change.
To Reproduce
- Start ExtendDB at 0e4b4a8 with the SQLite backend.
- Create the table above and put the item.
- Send the statements in the tables.
- Compare the answers with the DynamoDB column.
Expected behavior
The DynamoDB column of each table.
Actual behavior
The ExtendDB column of each table.
Environment
- ExtendDB version: the feat(partiql): implement ExecuteStatement, BatchExecuteStatement, ExecuteTransaction #411 branch at 0e4b4a8, SQLite backend, debug build
- Operating system: Linux
- Rust version (if building from source): 1.97.0
- Client SDK/driver and version: boto3
- Deployment method (binary, container, source): source
Proposed change
Start with a + 1 + 1: let operand_tail read a chain of + and - operators, and let ExprBuilder::operand (crates/engine/src/partiql/write/expr_builder.rs) render the chain for the classic update expression. Measure the chain in a SET value and in a condition before the change. Only the SET value form in the first table has a measurement.
The texts in the second table are local edits in the files named under it. #886 covers the order of the first refusal in a statement with two faults. The change proposed there, moving the value checks into the parser's walk, covers the WHERE texts of this issue too.
The bracket expressions need the parser to keep the position of the root operator of an expression in a bracket. The select-list order needs the bracket refusal to take its place in the walk order. The reserved words need the parser to read each form to its end before it refuses the word. Each of these is a change in crates/core/src/partiql/parser/. Each needs a measurement of the forms this pass did not cover: the statement-level reserved-word forms, and the batch and transaction operations.
Additional context
#885 covers the depth caps of the parser. They refuse the size, NOT and CASE forms from 100 levels, where the service reads them to their end.
Activity