[ecosSystem]

These came out of moving a real customer dataset of our ERP to SQL Server: 3.8 million orders, 5.4 million order lines, 1.1 million addresses, 72 GB. With the small test database everything was fine. With this one, the program start took almost a minute, the order entry did not open within two minutes, and selection lists (F8) and the article master hung, a search in the suppliers did not finish within two minutes, and the purchase orders took 15 seconds to open.

The branch is based on the current dev (after #2128). Each commit is one fix. Measurements are on SQL Server 2025 Developer with the database set to PARAMETERIZATION FORCED (which saves about a third of the SQL time for the many small statements of the ERP).

What changed

1. Dates as literals in keyset conditions (9fc4cee490). The keyset condition of a page wrote a subquery for every value that is not a string or a number. SQL Server could then not seek on the index: paging through BEL_ART + DTOS(DAT_HINZU) + BEL_NR of 3.8 million orders read the whole table for every page (665,762 logical reads, against 1,941 with a literal). New provider method DateLiteral() (default NULL = subquery as before); SQL Server returns 'yyyyMMdd' for a date without time, which is read the same way for all date types, whatever the language and DATEFORMAT.
2. field = 'literal' as an equality when the literal covers the field (492e94ca46). A single equals is a prefix compare in xBase, so TYP = 'L' became TYP LIKE 'L%'. For a C(1) field that is an equality, but for the server a LIKE is a range: with TYP = 'L' as the FOR condition of ART_NR+REF_NR it sorted all 477,757 rows of type L for the first page (8.1 s); with = 0.19 s. The column length comes from the table definition, because the fields of the work area do not exist yet when the production index is parsed.
3. SeekLast reads the last record of the key directly (8270cb620b). Seek() with Last found the first record of the key and skipped to the last one. Our order entry opens with a SeekLast for the newest order of a type; it read every order of that type page by page and had not arrived after 80 seconds. The last record is now read in reverse (one row) and positioned on like after GoTo(). With a filter, without a recno column, or when the key has no record, Seek() continues as before.
4. OPTION (RECOMPILE) for OR conditions on large SQL Server tables (37c15bb6cd). A keyset condition k1 >= v1 AND (k1 > v1 OR (k1 = v1 AND k2 > v2) OR ...) only becomes seeks when the server knows the values. With forced parameterization (or a cached plan) it seeks on k1 alone and reads all rows of v1 before the position: paging the lines of an order near the end of 5.4 million rows took 675 ms per page, with the hint 4 ms. On small tables compiling every page costs more than it saves (navigation tests took 1.5 to 2 times as long), so the hint is only added for tables with at least 100,000 rows, read once per connection from sys.partitions. New provider members (renamed in commit 10) RangeQueryHint, RangeQueryHintMinRows, TableRowsStatement, empty by default.
5. GoBottom() without counting the order first (0264e53a1e). GoBottom() read the last page in reverse, but counted the order before, only to know the position of the page. The count is invalidated by every Seek(), and a list of order lines calls GoBottom() on every move to another order: 72 counts of 5.4 million rows (46 ms each) while paging through orders. Without a valid count the last page is now anchored at its first row, as after GoTo(), and the position is looked up when something needs it.
6. No deep OFFSET for GoBottom() on descending orders and for far skips back (a0b95477b2). On a descending order GoBottom() still counted and read the page with OFFSET 3768900, which did not finish within 30 seconds (the order selection list). The reverse of a descending order is now built item by item, like the backward keyset reads do. A skip back by more than a page also used the position (OFFSET 3768805, two minutes for 150 rows back from the end); up to what the buffer holds, the rows before it are now read by key.
7. Soft seek on part of a column without SUBSTRING() (a1d9307e49). The column where a soft-seek value ends was compared as SUBSTRING(c, 1, n) >= v, which keeps the server from using the index. Now c >= v, and for the upper bound c < v OR c LIKE 'v%'. The lower bound is fast now (610 instead of about 17,000 reads); the upper bound still is not, because SQL Server does not seek on a LIKE inside an OR, and under a collation like Latin1_General_CI_AS there is no exact "next value" to use instead. Both now compare like the ORDER BY: with an accent-sensitive collation SUBSTRING(NAME,1,3) >= 'Zäh' was FALSE for 'Zahlungseingang', which sorts after 'Zählprotokoll', so that record was left out of the seek result and skipping on ended at EOF.
8. Non-ASCII keys as literals when the database code page has them (ae71f77367). Keyset literals were ASCII only, because a literal is converted to the code page of the database. A soft seek on 'GÜN' in the customers by short name therefore fell back to the position of the record: OFFSET 323815 of 1.1 million addresses, more than a minute. The connection now reads the code page of the database once (new provider member CodePageStatement, SQL Server: the code page of the database collation), and a string is written as a literal when all its characters survive a round trip through that code page.
9. Keys with STR(column, n, d) sorted and sought by their columns (cf36f8b65e). A key like ART_NR+LISTE+STR(MENGE,5,0) has a function in it, so it was sorted, sought and paged as the whole concatenated text, which the server computes for every row: a price lookup in 7 million rows read 118,000 pages, the page before the found record (keyset condition with subqueries on the text) 212,000 pages in 16.7 s. STR() is now a key part of width n, like DTOS() since SQLRDD: sort keys like REF_NR+DTOS(DATUM) by their columns #2122, and seeks are conditions per column. In this commit the STR() part was still sorted by its text; commit 11 sorts it by the number. The same skip back takes 1.1 s instead of 39.9 s.
10. OPTION (RECOMPILE) also for range and LIKE conditions on large tables (55e632a88a). Not only a keyset condition with OR depends on the values: with forced parameterization a range shares one plan for all values as well. A plan from the cache for TYP = @0 AND KURZNAME >= @1 ORDER BY KURZNAME, made for the 1.1 million customers (TYP 'K'), read the short name index in order and looked up TYP for every row. For a soft seek among the 488 suppliers (TYP 'L') near the end of the names it read 720,398 pages in 107 s to find nothing; recompiled, 418 pages in 6 ms. The hint now applies when the whole condition of the query (with the condition of the order and the scope) has OR, LIKE or range terms: keyset conditions, soft seeks, scopes, prefix seeks. A condition of equalities only, such as the many relation seeks, keeps the plan from the cache; it is a seek for any value. The provider members of commit 4 are renamed to RangeQueryHint and RangeQueryHintMinRows.
11. STR() key parts sorted and sought by the number (9daffc4311). Sorted by its text, the STR() part kept an index on (ART_NR, LISTE, MENGE) from giving the order: for the rows behind a position (the next page after a seek) the server sorted all rows behind it first. Opening the purchase orders looks up prices this way for every line: 118,278 reads and 6.5 s CPU per lookup, against 43 reads sorted by the number. The STR() part is now sorted by the column, and a seek value that covers the part compares the number ([MENGE] = 1 instead of STR([MENGE],5,0) = ' 1'). Behaviour change: the text sorts like the number for values from 0 up to the width; unlike DBF, negative values now come before 0, and values wider than the part ('*****') at the end. Neither occurs in the price table of the dataset (7 million rows).

Result

Program start plus order entry, measured on the SQL Server side:

	server time	logical reads
before	159 s (aborted)	284 million
after	about 4 s	0.3-1.2 million

The remaining time is mostly client side (round trips: a list page of orders runs about 100 relation seeks) and, on this 16 GB test machine, physical reads.

Testing

• Our regression suite on the test database, comparing the result files of each version with the one before, with and without ResultCacheMs: navigation patterns on 9 table/order combinations, EOF/BOF sequences, sequential reads, and full-order sequences plus Seek/SoftSeek/SeekLast checks for the 115 DTOS tags and 23 tags with field = 'x' conditions. New in this round: a GoBottom test (far skips, OrderKeyNo, Seek/GoBottom/SeekLast, on 10 orders including descending ones) and Skip(-1)/Skip(2) after SeekLast plus a soft SeekLast behind the key. 602 result files (with ARTPRSDB orders that have a STR() part added for commit 9; the test database has no table above 100,000 rows, so commit 10 is tested on the large one), no differences, except the two of the 'Zäh' case in commit 7, where the new result is the correct one.
• On the large database: the real statements of the slow steps replayed with and without each change (numbers above), and the navigation tests that hung before (GoBottom on the descending order: 3.4 s for the whole test including program start, before aborted after 2 minutes; soft seek on 'GÜN' and paging: 0-7 ms per step, before not finished after 150 s).
• Each commit was built and tested on top of the previous one.

Not in this PR: in our ERP the SQL indexes for tags with a FOR condition now include the columns of the condition (equality columns first, the others as INCLUDE). Without them the server has to read every row of the table to find those of the tag (571,427 instead of 343 reads for a tag with 81 of 1.7 million rows). We create those indexes in the application; SqlDbTableCommandBuilder.CreateIndex() might want the same.

🤖 Generated with Claude Code