SQLRDD: ERP write path - EOF like ADS, Append fixes, less work per statement, optional result cache by ecosSystem · Pull Request #2128 · X-Sharp/XSharpPublic

ecosSystem · 2026-10-27T11:32:35Z

The branch is rebased onto the current dev (after the squash merge of #2123) and contains no commits of #2123 any more. It has the 5 commits described below, Robert's backslash fix, and one commit per point of the Copilot review (see Review fixes).

These came out of profiling an order copy in our ERP. Cpy_Auftrag() copies an order with its lines, and every line runs the usual DBF-era code: save the position of a dozen work areas, seek into tax, item group, weight and parameter tables, restore the positions. Under ADS one copy took 0.33 s. Under SQLRDD it took 1.45-1.56 s, and repeated copies 1.02 s. It sent about 4,200 statements per copy, and only about 80 of them wrote anything.

What changed

Append() without MAX(recno), report a failed insert (1af27e9385).
- Append() asked the server for MAX(recno) + 1 as the key of the new row. With a recno column the insert itself returns the new record number, which then replaced that key, so the query was one round trip per Append() for nothing.
- A failed insert (e.g. a CHECK constraint) was rolled back and reported as success. The work area then stood on a row that did not exist on the server, and the following field changes went nowhere. The error is now registered as an EG_WRITE error and Append() returns FALSE, as a failed append does under DBF.
EOF positioning like ADS, without extra queries (f42092477c). Saving and restoring a position at EOF means SetOrder() plus GoTo(LastRec()+1). In the order copy this ran thousands of times, mostly on empty tables. Each time SQLRDD sent four queries (single record, keyset window, ROW_NUMBER(), single record again) and counted the order to compute BOF.
- GoTo() to a missing record: It goes to EOF as soon as the lookup by recno finds nothing. Before, it stayed on the previous record. Like ADS it sets BOF as well, also for GoTo(0).
- Backward skip at EOF+BOF: A backward Skip() at EOF and BOF lands on the first record with BOF, as with ADS.
- SetOrder() at EOF: SetOrder() to the active order stays at EOF instead of calling GoTo(). An order change at EOF keeps BOF as it was (ADS), so the order is no longer counted.
- Pending EOF: At a pending EOF (phantom record, buffer not ending at the last record), Skip(1) now also stays at EOF the second time; before, it read on behind the buffer.
- Found: It is reset by GoTo(), GoTop(), GoBottom() and Skip(), as in VO.
GoTo() returns TRUE at the phantom record (83ba209d47). The DBF RDD does the same. The VO DbServer:GoTo() treats FALSE as an error: __DBSGoTo() builds an error object and BREAKs, which took 7 % of the order copy.
Less client work per statement (8e971446ae).
- Every query copied the buffer structure (_tableTemplate:Clone()), which took 10 % of the time for tables with 80-180 columns. The replaced buffer, or a page whose rows were copied into the buffer, is now reused (GetDataTable(cTable, oTemplate, oTarget)).
- ColumnList() built the quoted column list for every statement (4 %). It is now built once per table definition.
Short-lived result cache for repeated reads (b5eff1aa43). New connection option ResultCacheMs (default 0 = off): the RDD may answer a read statement it sent within that time from its result.
- Scope: only the RDD's own reads of table work areas (SqlDbCommand.CacheTable): a single SELECT without parameters, results of up to 20 rows.
- Dropped per table: as soon as this connection writes to the table (any work area) or locks one of its records, so a locked record is always read fresh.
- Dropped entirely: on a rollback, and when a statement that does not come from the RDD writes.
- Other users: their changes are seen at the latest after ResultCacheMs. That is the trade-off, and why the option is off by default.

Measurements

Order copy in the ERP (copy of an order with 10 lines, SQL Server 2022 Express on the same machine; ADS 0.33 s for one copy):

	before	after 1-4	with `ResultCacheMs=1000`
first copy (opens ~30 tables)	1.45-1.56 s	1.13 s	0.95 s
further copies	1.02 s	0.81-0.83 s	0.56-0.61 s
statements per further copy	~4,200	1,625	(1,115 of the 1,625 were exact repeats without a write in between)

The remaining time is mostly the application's own work, which ADS has as well, plus one round trip per statement that is not repeated.

Test program on the same database:

	before	after
Seek, 178 column table	1.24 ms	0.89 ms
SetOrder(0) / GoTo() / SetOrder(n), 178 column table	1.29 ms	0.93 ms
statements for the EOF sequence below	44-61	21
statements of the GoTo/Skip/Seek pattern from #2123 with `ResultCacheMs=1000`	171-802	25-55 % fewer

Testing

EOF/BOF against AXDBFCDX: 1, 338, 1,025 and 1,245 row tables. The sequence covers GoTo past the end, GoTo(0), backward skips from there, SetOrder() at EOF (same and other order), a failed seek followed by order changes, and restoring a saved EOF position. EOF, BOF and RecNo are identical, except LastRec where the data differ. Empty tables differ in one ADS quirk: ADS clears EOF after Skip(-1), the DBF RDD and SQLRDD keep it.
Navigation, sequences and seeks: The GoTo/Skip/Seek pattern from SQLRDD: cheaper interactive navigation (lazy GoTo, order switches, keyset windows and backward reads) #2123 (8 table/tag combinations), full-order sequences, and sequences and seek checks for the 115 DTOS tags of our ERP. All return the same results as before, with and without the cache (494 result files compared, again after the review fixes). The only intended difference is GoTo() to a missing record, which now goes to EOF.
Cache: tested with two work areas and a second connection as another user. Own writes and appends are seen at once; another user's change is seen after ResultCacheMs or after RLock().
Append: checked against a table with a CHECK constraint. Before, Append() returned TRUE and nothing was written; now it returns FALSE.

Review fixes

One commit per point of the Copilot review. Points 5, 9, 2 and 6 were reproduced with the code before the fix and checked again after it (see the test below).

Copilot point	Commit	Change
Skip discards dirty edits after GoTo	`3899c46a95`	The pending changes are written before Skip() loads the window around the record. Before, `GoTo(n)`, `FieldPut()`, `Skip()` lost the change without an error. This was already in `dev` (lazy GoTo from #2123). A `DbServer` with the default ccOptimistic hid it, because it commits before every move; with `CCNONE` the change was lost.
Insert and identity failures are reported as success	`ac58e73479`	Each step of the insert (insert, identity, commit) is checked, and the original error is reported.
Failed append corrupts pending EOF state	`459b5cc9d6`	The EOF flags change only after a successful insert.
Query errors are misreported as successful EOF	`cb8c99b1c1`	When the lookup in GoTo() registered a new error, GoTo() reports it and returns FALSE instead of going to EOF.
Disabling cache reads preserves stale cached results	`58fa629108`	Setting `ResultCacheMs` to 0 drops the cached results.
Cache keys ignore callback-rewritten SQL	`0e5ea36a07`	The CommandText callback runs first, exactly once, and the key is the statement that is sent. A cached result passes the callback as well.
SELECT prefix check misses write batches	`0e5ea36a07`	A statement with a word that writes (INSERT, UPDATE, DELETE, MERGE, INTO, EXEC, DDL) counts as a write.
RLock reads stale cached current row	`7be52398fe`	Like ADS, a new lock of the record the work area stands on reads it again, unless the work area has pending changes. With `UseLocks` off nothing is locked and nothing is read again. With ccOptimistic, the `DbServer` now sees the other user's change and refuses to overwrite it, as under ADS; before, it overwrote it.
Backslashes break MySQL keyset predicates	`0f5012b8f6`	Robert's commit.

Test for the reproduced points (DbServer with CCNONE, ResultCacheMs=1000, page size 50, 3,000 rows):

	before	after
`GoTo(1500)`, `FieldPut()`, `Skip(1)`: value on the server	old value	new value
`GoTo(0)` at a pending EOF, rejected `Append()`, `Skip(1)`	record 113, not EOF	EOF, record 3062
`SELECT 1` + line break + `UPDATE`, then the same `GoTo()`	old value from the cache	new value
other user changes the record, `RLock()`, `FieldGet()`	old value	new value

🤖 Generated with Claude Code

Append() asked the server for MAX(recno) + 1 as the key of the new row, but with a recno column the insert itself returns the new record number, which then replaced it. That was one round trip per Append() for nothing; the returned record number now also updates LastRec(). A failed insert (e.g. a CHECK constraint) was rolled back and reported as success: the work area stood on a row that did not exist on the server and the following field changes went nowhere. The error is now registered as an EG_WRITE error and Append() returns FALSE, as a failed append does under DBF. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Code that saves and restores the position of a work area (order and recno, e.g. around a lookup) does SetOrder() and GoTo(saved recno). At EOF that recno is LastRec()+1, and in an ERP order copy this ran thousands of times on empty tables (tax and dimension tables). Each time SQLRDD sent four queries (single record, keyset window, ROW_NUMBER(), single record again) and counted the order to compute BOF. - GoTo() to a record that does not exist goes to EOF as soon as the lookup by its recno finds nothing. Before, it stayed on the previous record. Like ADS it sets BOF as well, also for GoTo(0). - A backward Skip() at EOF and BOF lands on the first record with BOF, as with ADS. - SetOrder() to the active order at EOF stays there instead of GoTo(). - An order change at EOF keeps BOF as it was (ADS), so the order is no longer counted for it. - At a pending EOF (phantom record with a buffer that does not end at the last record) Skip(1) stays at EOF also the second time; before, it read on behind the buffer. - Found is reset by GoTo(), GoTop(), GoBottom() and Skip(), as in VO. - GoToId() no longer recomputes EOF/BOF after GoTo(), which cleared BOF at the phantom record. Checked against AXDBFCDX on tables with 1, 338, 1,025 and 1,245 rows (GoTo past the end, GoTo(0), order changes and seeks at EOF, backward skips): identical EOF/BOF/RecNo except LastRec where the data differ. Empty tables differ in one ADS quirk (ADS clears EOF after Skip(-1)). Statements for that sequence: 44-61 -> 21. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

GoTo() to a record that does not exist positions on the phantom record (EOF, and BOF as with ADS) and returned FALSE. The DBF RDD returns TRUE there, the positioning succeeded. The VO DbServer:GoTo() treats FALSE as an error: __DBSGoTo() builds an error object and BREAKs out, which in an ERP order copy (thousands of GoTo() back to a saved EOF position) took 7 % of the time. The eager path (a second GoTo() in a row) now also lands on the phantom record instead of staying on the previous one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Profiling an ERP order copy (1,600 mostly single-row statements) showed the client side costs per statement next to the server round trips: - Every query copied the structure of the buffer (_tableTemplate:Clone()) to load its rows into: 10 % of the time for tables with 80-180 columns. The replaced buffer, or a page whose rows were copied into the buffer, is now kept and the next result is loaded into it (SqlDbCommand.GetDataTable(cTable, oTemplate, oTarget)). - SqlDbTableCommandBuilder.ColumnList() built the quoted column list for every statement (4 %). It is now built once per table definition. Seek on a 178 column table: 1.24 -> 0.89 ms, SetOrder/GoTo/SetOrder 1.29 -> 0.93 ms. Results unchanged (navigation, EOF/BOF, sequences and seeks of 115 DTOS tags). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Code written for DBF/ADS repeats the same lookups again and again: in an ERP order copy 1,115 of 1,625 statements were exact repeats of an earlier read, with no write to the table in between (the same seek into a tax, item group or parameter table for every line). With a local index these are free, against a server each is a round trip plus loading the result. The new connection option ResultCacheMs (default 0 = off) lets the RDD answer a read statement it sent within that time from its result: - Only the RDD's own reads (SqlDbCommand.CacheTable, set for table work areas): a single SELECT without parameters, results of up to 20 rows. Larger results (pages of a sequential read) are not kept. - The entries of a table are dropped as soon as this connection writes to it (insert, update, delete of any work area) or locks one of its records, so a locked record is always read fresh. - All entries are dropped when a transaction is rolled back or when a statement that does not come from the RDD writes. - Changes of other users are seen at the latest after ResultCacheMs. Checked with two work areas and a second connection as another user: own writes and appends are seen at once, another user's change after the time or after RLock(). Results of the navigation, EOF/BOF, sequence and DTOS seek tests are identical with and without the cache; the navigation test sends 25-55 % fewer statements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

RobertvanderHulst · 2026-10-27T11:55:22Z

@copilot resolve the merge conflicts in this pull request

Copilot

🟡 Changes recommended

Unresolved edit preservation, database-error propagation, and cache-consistency issues can lose updates or return stale or incorrectly scoped data.

9 open issues

Cache keys ignore callback-rewritten SQL · New
SELECT prefix check misses write batches · New
Disabling cache reads preserves stale cached results · New
Query errors are misreported as successful EOF · New
Skip discards dirty edits after GoTo · New
RLock reads stale cached current row · New
Insert and identity failures are reported as success · New
Backslashes break MySQL keyset predicates · New
Failed append corrupts pending EOF state · New

What changed in this PR

This PR builds on #2123’s SQLRDD navigation optimizations to reduce database round trips and client-side work in ERP order-copy operations, improve append and EOF behavior, and add an optional short-lived read cache.

Changes:

Adjusts append failure handling and record navigation to better match DBF/ADS behavior.
Reuses table structures and buffers, caches column lists, and expands keyset-based navigation.
Adds connection-level ResultCacheMs support with expiration and invalidation hooks.

File	Description
`src/Runtime/XSharp.SQLRdd/XSharp.SQLRDD.xsproj`	Includes the new result-cache implementation.
`src/Runtime/XSharp.SQLRdd/Support/SqlDbTableCommandBuilder.prg`	Adds reverse queries, configurable last-page reads, and column-list caching.
`src/Runtime/XSharp.SQLRdd/RDD/SQLRDD-Private.prg`	Implements buffer reuse, lazy positioning, keyset navigation, and insert error reporting.
`src/Runtime/XSharp.SQLRdd/RDD/SQLRDD-Orders.prg`	Preserves buffered records and EOF/BOF state across order changes.
`src/Runtime/XSharp.SQLRdd/RDD/SQLRDD-Main.prg`	Updates append, navigation, lock handling, and cache integration.
`src/Runtime/XSharp.SQLRdd/Classes/ResultCache.prg`	Adds bounded result storage with expiration and table-level invalidation.
`src/Runtime/XSharp.SQLRdd/Classes/Connection.prg`	Exposes cache configuration and clears cached results on rollback.
`src/Runtime/XSharp.SQLRdd/Classes/Command.prg`	Adds template-based result loading, cache lookups, and write invalidation.

_{Review effort: Balanced}

+        var cText := SELF:CommandText
+        if cText == null .or. cText:IndexOf(';') >= 0 .or. !cText:TrimStart():StartsWith("select", StringComparison.OrdinalIgnoreCase)
+            return null
+        endif
+        return cText


+        if cText == null .or. cText:TrimStart():StartsWith("select", StringComparison.OrdinalIgnoreCase) .and. cText:IndexOf(';') < 0
+            return
+        endif


+            if SELF:ResultCacheMs <= 0
+                return null
+            endif


+        if !found .and. SELF:_tableMode == TableMode.Table
+            // does not exist, as above
+            SELF:_GoToPhantom()
+            SELF:_SetBOF(TRUE)
+            RETURN TRUE


+        IF nToSkip != 0
+            SELF:_Found := FALSE
+        ENDIF


+        // Read fresh after a lock: the record may be written next.
+        if self:_oTd != null
+            self:Connection:ResultCache?:Invalidate(self:_oTd:RealName)


+        catch e as Exception
+            // Reporting success here made a failed insert look like a new record whose changes
+            // then went nowhere.
            _command:Connection:RollBackTrans()
+            self:_dbfError(ERDD.WRITE, XSharp.Gencode.EG_WRITE, "SqlRDD:Append", e:Message, FALSE)


+            if !self:_ExecuteInsertStatement(row)
+                // the error is reported, stay where we were
+                self:DataTable:Rows:Remove(row)
+                return false


RobertvanderHulst · 2026-10-27T16:04:13Z

Karl,
I have a problem merging this PR. Can you send me the full code again?

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

After GoTo() loaded only the record, Skip() looked up its position in the order first and wrote the pending changes afterwards. Loading the window replaced the buffer, so GoTo(n), FieldPut(), Skip() lost the change without an error. The changes are now written before the window is loaded, and Skip() returns FALSE when that write fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The command methods report a failure through their result and LastException instead of throwing. A failed insert was only noticed when assigning the missing identity happened to throw, and for providers without an identity query, the failed insert and the commit were not checked at all. Each step is now checked, and the original error is reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Append() cleared the pending EOF and marked the buffer as not contiguous before the insert. When the insert failed, the work area stayed where it was, but with these flags changed: at a pending EOF, the next Skip(1) read on behind the buffer. They are now changed only after a successful insert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

GoTo() went to the phantom record whenever the record was not found. A failed query (timeout, missing permission) is no proof that the record does not exist, but it ended at EOF and GoTo() returned TRUE. When the lookup registered a new error, GoTo() now reports it and returns FALSE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

While the cache is off, nothing is invalidated. Results kept from before could come back when it was switched on again within their lifetime, even after this connection had changed the rows. Switching it off now drops them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…atches - The cache key was taken before the CommandText callback rewrote the statement, and a cached result skipped the callback. Two work areas whose statements only differ after the callback could share results. The callback now runs first, exactly once, also for a cached result. - A batch needs no semicolon between its statements, so a SELECT followed by an UPDATE looked like a read: it was not invalidating the cache. A statement with a word that writes (INSERT, UPDATE, DELETE, MERGE, INTO, EXEC, DDL) now counts as a write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Code written for DBF/ADS locks a record, reads it and writes it back. The work area kept the values it had read before the lock (from its buffer or from the result cache), so a change another user committed in the meantime could be overwritten. Like ADS, a new lock of the record the work area stands on now reads it again, unless the work area has pending changes. With UseLocks off nothing is locked and nothing is read again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ecosSystem · 2026-10-28T08:37:55Z

Robert,

the conflicts came from the 7 commits of #2123 that this branch still carried: after the squash merge of #2123 they collided with the same changes in dev. I have replaced the branch with a version rebased onto the current dev, so it should merge cleanly now. It contains:

the 5 commits of this PR (unchanged in content),
your backslash fix (0f5012b8f6),
one commit for each of the other points of the Copilot review.

The most important one is "Skip discards dirty edits after GoTo": GoTo(n), FieldPut(), Skip() lost the change without an error. That was already in dev (lazy GoTo from #2123). A DbServer with ccOptimistic hides it, because it commits before every move; with CCNONE the change was lost. The description has a table of all points, the commits and the tests.

🤖 Generated with Claude Code

ecosSystem mentioned this pull request Oct 27, 2026

SQLRDD: cheaper interactive navigation (lazy GoTo, order switches, keyset windows and backward reads) #2123

Merged

ecosSystem and others added 5 commits October 27, 2026 12:55

RobertvanderHulst requested a balanced review from Copilot October 27, 2026 14:42

Copilot started reviewing on behalf of RobertvanderHulst October 27, 2026 14:43

View session

Copilot AI reviewed Oct 27, 2026

View reviewed changes

RobertvanderHulst and others added 8 commits October 28, 2026 08:02

Update condition to include backslash check

0f5012b

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

ecosSystem force-pushed the homebase/sqlrdd-write-path branch from 9ac26f6 to 7be5239 Compare October 28, 2026 08:37

RobertvanderHulst merged commit 41dad77 into X-Sharp:dev Oct 28, 2026
2 checks passed

ecosSystem mentioned this pull request Oct 28, 2026

SQLRDD: large tables - date and non-ASCII keyset literals, direct SeekLast and GoBottom, no deep OFFSET, OPTION (RECOMPILE) for keysets #2133

Merged

Navigation Menu

Repository navigation

SQLRDD: ERP write path - EOF like ADS, Append fixes, less work per statement, optional result cache - #2128

SQLRDD: ERP write path - EOF like ADS, Append fixes, less work per statement, optional result cache#2128
RobertvanderHulst merged 13 commits into
X-Sharp:devfrom
ecosSystem:homebase/sqlrdd-write-path

ecosSystem commented Oct 27, 2026 •

edited

Loading

Uh oh!

RobertvanderHulst commented Oct 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

RobertvanderHulst commented Oct 27, 2026

Uh oh!

ecosSystem commented Oct 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ecosSystem commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed

Measurements

Testing

Review fixes

Uh oh!

RobertvanderHulst commented Oct 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

RobertvanderHulst commented Oct 27, 2026

Uh oh!

ecosSystem commented Oct 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ecosSystem commented Oct 27, 2026 •

edited

Loading