[ecosSystem]

Rebased onto the current dev after #2121 was squash-merged: the 7 commits are unchanged, only their parents and hashes differ. #2128 builds on this branch.

Interactive work in an ERP (an order entry form that pages from one order to the next) is a stream of small operations: seeks into related tables, GoTo() back to a saved record, order switches around a lookup, one Skip(). Under DBF/ADS these are local and nearly free. Under SQLRDD each one was a round trip, and many of them read far more than needed. In our order entry form one paging step sent 183-300 statements; it now sends about 100.

What changed

1. Load query results into the buffer's structure (e147489333). Every query built a new DataTable from the reader's schema. For one or a few rows of a wide table that was most of the client side cost (1.5 ms per query for 178 columns, 0.5 ms when loaded into an existing structure). The work area keeps an empty clone of its first buffer, and SqlDbCommand.GetDataTable(cTable, oTemplate) loads the rows into a copy of it with GetValues()/LoadDataRow(). If the columns do not match, the rows are loaded as before.
2. GoTo() loads only the record (758712232a, a91d9ae087). GoTo() to a record that is not buffered used to look up its position right away, with a ROW_NUMBER() over the order plus a page around the record. It now loads just the record by its recno. The position is looked up when Skip(), OrderKeyNo etc. need it. A second GoTo() in a row still positions fully, so a browse that calls GoTo() for every row it paints gets one page instead of one query per row; a record loaded by an order change does not count for that. DBI_GETLOCKARRAY/DBI_LOCKCOUNT no longer query the lock table when UseLocks is off.
3. Order changes (88083310d0, fea0681d12).
   • At EOF an order change no longer loads and counts the first page of the new order. An empty buffer is built from the structure of the previous one, and EOF is marked as pending as with GoTo(0).
   • Otherwise the current record, which was in the buffer a moment before, is kept instead of being read again.
   • LastRec() stays valid across the switch.
   • A seek that finds nothing no longer counts the order to compute BOF.
4. Keyset conditions with literals (b9d4aabfae). The keyset condition from SQLRDD: fast sequential reads (keyset paging, read-ahead, no constraint indexes on loaded pages) #2121 read every key value of the last row with a subquery. Optimizing those took the server 30-45 ms per statement, against 3 ms to execute it, and the plan was never reused because the text changes with every page. Plain key columns with ASCII string or integer/decimal values are now written as literals. Other values (non-ASCII strings, dates, expressions) still use the subquery.
5. Small first reads (b9d4aabfae). GoBottom() and looking up the position of a record read PageSize / 10 rows instead of a full page, as a seek already does in SQLRDD: fast sequential reads (keyset paging, read-ahead, no constraint indexes on loaded pages) #2121. Code that goes to a record mostly reads only it and its neighbour. Reading on makes the pages grow.
6. Window after GoTo and backward reads by key (eb225987cb).
   • Window after GoTo: The page loaded when a lazily positioned record is moved from now starts at the record and is found by its key, instead of OFFSET n around it. With OFFSET the server reads through all rows before the record: 13 ms for 10 rows at offset 502 of a 1,025 row table, 2 ms by key. When the position is not known yet, the buffer is anchored at the record (like a seek) and the position is only looked up when needed.
   • Backward reads: Reading backwards finds the preceding rows by the key of the first buffered row, sorted in reverse (BuildSqlStatement(..., lReverse)), also on a seek-anchored buffer, which is then anchored at its new first row.
   • Limits: Both are only used when all key values can be written as literals. An order on a key expression keeps using offsets, because there the keyset subquery costs more to compile than the OFFSET.

Measurements

SQL Server 2022 Express on the same machine, with the database's default PARAMETERIZATION SIMPLE. AUFHDRDB has 1,025 rows and 178 columns, ADRESSDB has 52,000 rows and 154 columns.

test	before	after
SetOrder(0) / GoTo() / SetOrder(n), AUFHDRDB	11.1 ms	1.4 ms
DbEval() over ADRESSDB, tag REF_NR FOR TYP == 'K' (#2121)	3.9 s	2.3 s
GoTo/Skip pattern, ADRESSDB tag 1	26 s	3.3 s
GoTo/Skip pattern, AUFHDRDB tag 1 (descending)	5.4 s	4.5 s
GoTo/Skip pattern, PARADB (338 rows)	0.6 s	0.7 s

The GoTo/Skip pattern covers 40 records per table. For each one it does GoTo(), Skip(+1, +1, -3, -10, +25), an order switch with GoTo() and Skip(-1), Skip(+150) and Skip(-1), and Seek() of the record's key followed by Skip(-1, -15, +20) and a soft seek with Skip(-2). It ends with GoBottom(), Skip(-1) three times, Skip(-30) and Skip(+1)/Skip(-1) at EOF.

On small tables the keyset statements are slightly slower than OFFSET, which costs nearly nothing there: a few tenths of a millisecond per statement.

Deployment note: with ALTER DATABASE ... SET PARAMETERIZATION FORCED, the server reuses plans for seeks with new values instead of compiling every statement. In our measurements this saved another third of the SQL time per paging step: replaying the statements of one step took 110-145 ms with SIMPLE and 73-95 ms with FORCED.

Testing

• Full reads: The recno sequence of a full read is identical to the OFFSET version, for natural order, tags with FOR conditions, function keys and descending tags.
• DTOS keys: For 115 tags with DTOS keys of our ERP, the sequence and a seek check per sampled key prefix are unchanged. The seek check covers hard seek, the number of matching records, soft seek and SeekLast.
• Navigation: The GoTo/Skip pattern above returns the same positions (recno, EOF, BOF) as before on 8 table/tag combinations: ascending, descending, FOR condition, key expression and natural order.

🤖 Generated with Claude Code

[RobertvanderHulst]

@copilot resolve the merge conflicts in this pull request