Repository navigation
Parallel native writes, lighter parallel paths, faster first open, 11 bug fixes - #148
Merged
Merged
Conversation
…tions The struct grows by two appended fields. The 7.0 size (32) and a NULL options pointer are still accepted and decode as serial, so existing callers keep today's behavior. Negative values are rejected.
When xl_write_options asks for more than one degree of parallelism, xl_write_typed and xl_write_typed_to_memory hand the data rows to the core's parallel record writer. Styles and the header row are written as before. XLS and CSV keep the serial loop.
xl::WriteOptions gains degree_of_parallelism (default 1, serial) and batch_rows (default 0, library default).
WriteOptions gains degree_of_parallelism and batch_rows builders. Unset means a serial write and the library's default batch size.
WriteOptions gains degree_of_parallelism and batch_rows. None means a serial write and the library's default batch size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a degree of one, batching only added the cutting and sync flushes: par1 ran 3-5% slower than the serial write. The parallel entry points now hand a degree of one (or zero on a single processor) to the serial record write, and still end the sheet when it throws. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each 64 KB chunk built its own string cache when InternStrings was on, and every read of a file chunk went through RandomAccess.ReadAsync, allocating a state machine box and an overlapped source. A worker now keeps one cache across its chunks, and chunk reads are blocking positional reads on a handle opened without FileOptions.Asynchronous. 4.3M rows, dop 16: 385 to 332 ms by default, 282 to 220 ms with InternStrings, and no GCs left for struct rows with InternStrings.
9455837 reopened the chunk handle for synchronous I/O to drop the per-read allocations. Windows serializes all I/O on a synchronous file object, so with a cold cache the workers would have read one at a time. The handle is overlapped again, and RangedFileStream.ReadAsync uses a pooling builder instead of allocating a state machine box per read. 4.3M rows, dop 16, struct rows with InternStrings: 40 MB allocated before 9455837, 14 MB now.
Opening the first workbook of a process is dominated by JIT compilation. Cut what the open path makes the JIT do: - Huffman table building and the CRC finish no longer mix a loop with a stackalloc, which made the JIT skip tier 0 and compile them fully optimized on first call. - Byte fills in inflate write 8 bytes at a time instead of calling Span<byte>.Fill, whose Vector<T> memset is never precompiled. - The decoder rents its tables from ArrayPool<byte>, which ships precompiled, instead of instantiating ArrayPool<uint>. - Duplicate ZIP entry names are found by comparing pairs for up to 64 entries; the sort, with its comparer, moves to its own method and only larger directories reach it. Cold open of a 200-row XLSX: 24.3 to 21.1 ms. Repeated small opens are ~1% faster and allocate 3520 B instead of 3712 B; large reads unchanged.
…the file The FileStream was opened before the try block, and the export has no handler of its own, so a missing directory or a locked file threw out of an UnmanagedCallersOnly method and terminated the host process. The open now sits inside the try and reports XL_ERROR with the IO message.
The chunk plan's cursor only moves forward and was shared by every enumeration. A second await foreach found no chunks left, its workers exited, and the merge waited for a chunk nobody would produce, holding the reader's file open. Each enumeration now restarts the plan.
… sectors WorkbookStream placed sector n at 512 + n * sectorSize, which only matches the compound file layout when sectors are 512 bytes. A version 4 file has 4096-byte sectors and a header that fills a whole sector, so sector n starts at (n + 1) * sectorSize, as CfbContainer already computes; the old offset read from 3584 bytes too early.
An ISO date cell was stored as its OADate, which is the 1900 serial only from 1900-03-01 on and never the 1904 one. A 1904 workbook read 2020-01-01 back as 2024-01-02, and 1900-01-01 came back a day late. The serial now comes from ExcelEpoch for the workbook's date system; a date before the epoch, which has no serial, keeps its text so TryGetDateTime parses it directly. A value with a UTC offset was converted to the machine's local time; it now keeps the clock time as written, as a "Z" value already did.
A sheet that spills past 65,536 rows continues in "<name> (n)", but that name was never claimed in the workbook's set of sheet names. A later AddSheet could take it, and a sheet already named that way was not avoided, so the file ended up with two sheets of the same name, which Excel reports as damaged. The continuation now claims its name and moves to the next number when it is taken.
A number format reached styles.xml through SecurityElement.Escape, which leaves control characters and U+FFFE/U+FFFF in place, so a format such as "0\u0001" produced a styles part that does not parse. XML 1.0 cannot carry those characters even as references, so AddStyle now refuses them, as AddSheet already does for sheet names.
The merge numbered a failing row from the models it had yielded, so every blank record before the failure, which yields nothing, moved the reported row up by one. Chunks now report how many records they read, blank ones included, and the merge numbers from that.
With PrefetchDecompression the producer compared the CRC only when the running total landed exactly on the declared size. Reads come in 64 KB chunks, so an entry whose deflate data produced more than declared skipped the check, and a consumer that stopped at the declared size took damaged data. The CRC now covers exactly the declared bytes and is compared when they are all in.
ReadMiniStream stopped at the end of the chain and returned the rest of the stream as zeros, while the regular FAT path throws for the same damage. It now throws InvalidDataException too.
Loading Workbook sectors from a stream always asked for whole sectors, so a file that ends inside the stream's last sector threw EndOfStreamException, while the same bytes opened from memory read fine. The read now stops at the end of the stream.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #148 +/- ##
==========================================
+ Coverage 91.06% 91.30% +0.23%
==========================================
Files 252 276 +24
Lines 17501 21232 +3731
Branches 3162 3233 +71
==========================================
+ Hits 15938 19385 +3447
- Misses 1072 1352 +280
- Partials 491 495 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…add tests for lossless parsing feat(xlsx): implement canonical cell parsing and add tests for cell reading consistency
.NET 10.0.13 fixes the NativeAOT crash that killed partitioned aggregation on macOS arm64, so the native library is built with the SDK that ships it and the Rust tests run on macOS again.
… comparing number formatting Apple's libc++ may not ship floating-point from_chars, which every translation unit including the header compiles through CellView. Guard on __cpp_lib_to_chars and parse/format through the classic locale otherwise; a second test target builds that path. The for_each_row parity tests compared text() on both sides, so a number-only cell checked the binding's formatting against the library's. Compare the stored text instead; a generated XLSB with 1e20-scale numbers pins it.
…hars Apple's libc++ leaves __cpp_lib_to_chars undefined, so macOS always took the fallback, which printed the first %g precision that round-tripped: 1000 came out as "1e+03" and the rows test failed against to_chars. Build the fewest round-tripping digits in scientific form, then pick fixed unless scientific is shorter, printing integers exactly, as to_chars does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
Exposes parallel writes through the C ABI and the bindings, trims memory and JIT work on the parallel and cold-start paths, fixes 11 bugs found in a whole-repo review, and adds struct +
InternStringsvariants to the CSV benchmarks. The managed public API is unchanged; the C ABI only appends fields, so this fits a minor release.Parallel writes from the native bindings
xl_write_optionsgainsdegree_of_parallelismandbatch_rows, appended to the struct. The 7.0 struct size (32) and a NULL options pointer are still accepted and decode as a serial write.xl_write_typed/xl_write_typed_to_memoryhand XLSX and XLSB data rows to the core's parallel record writer when the degree is above 1. XLS and CSV stay serial.xl::WriteOptions), Rust (WriteOptionsbuilders) and Python (WriteOptions) expose both options, with docs in each README.Performance
Parallel write memory. Each batch was copied into a rented 2 MB deflate window. Batches are now encoded straight from their own buffer, with byte-identical output. Peak working set at 1M rows x 4 columns: +12 MB to +6 MB at 4 workers, +22 MB to +13 MB at 8. It is also slightly faster (XLSX at 8 workers: 40 to 35 ms, about 4.8x the serial write).
One worker writes serially. With
DegreeOfParallelism = 1(or 0 on a single processor) the parallel entry points use the serial record write, which was 3-5% faster than batching with one worker. A throw still ends the sheet, as documented.Parallel typed CSV parse. A worker keeps one
Utf8StringCacheacross its chunks instead of allocating one per 64 KB chunk, andRangedFileStream.ReadAsyncusesPoolingAsyncValueTaskMethodBuilder. The chunk handle stays overlapped on Windows. 4.3M rows at dop 16: 438 to 352 ms by default, 237 to 206 ms withInternStrings; struct rows withInternStringsallocate 14 MB instead of 40 MB.First open of a process. Cuts what the JIT has to compile on the open path, without changing the managed inflate or CRC:
stackallocin Huffman table building or the CRC finish (that combination is compiled fully optimized on first call);Span<byte>.Fill, whoseVector<T>memset is never precompiled;ArrayPool<byte>instead ofArrayPool<uint>for the decoder tables;Cold open of a 200-row XLSX: 24.3 to 21.1 ms. Repeated small opens are ~1% faster and allocate 3520 B instead of 3712 B; large reads are unchanged.
Bug fixes
xl_open_write_handlethrew out of anUnmanagedCallersOnlyexport, terminating the host process, when the file could not be created. It now returnsXL_ERROR.EndOfStreamExceptionwhen read from a file, though it read fine from memory.t="d"dates ignored the 1904 date system (2020-01-01 read as 2024-01-02), came back a day late before 1900-03-01, and were shifted into the machine's time zone when they had an offset.styles.xml;AddStylenow rejects it.PrefetchDecompression, an entry inflating past its declared size skipped the CRC check.Behavior changes worth knowing:
AddStylethrowsArgumentExceptionfor such number formats.t="d"date before the workbook epoch has no serial. It is now kept as text withCellType.Date, soTryGetDateTimeworks butTryGetDoubledoes not.t="d"value with a UTC offset keeps its clock time and drops the offset, as aZvalue already did.Benchmarks
New rows next to the existing class baselines, which stay as the default case:
CsvParallelParseBenchmarkandLargeSheetParallelBenchmarkparse CSV into astructwithInternStrings.CsvParallelVsSepBenchmarkadds a struct pair for both libraries, with string pooling on each side (InternStrings; Sep'sSepToString.PoolPerCol()/PoolPerColThreadSafe()), so tuned is compared with tuned.ParseBenchmarkgains anInternStringsvariant.docs/performance/benchmarks.mdis updated from a BenchmarkDotNet run (Short job; Sep's parallel struct rows rerun with Medium because the Short run spread too much). Highlights:InternStrings, with no garbage collections. The struct rows do not slow down past dop 8 the way the class rows do.The write, cold-start and allocation figures in the Performance section come from console harnesses (interleaved runs on the Ryzen 7 5700X), not BenchmarkDotNet.
Checklist
dotnet build ExcelReader.slnx --configuration Releasebuilds clean (warnings are errors)dotnet test tests/ExcelReader.Tests/ExcelReader.Tests.csproj --configuration ReleasepassesPublicAPI.Unshipped.txtupdated (no managed API change; the C ABI struct is append-only)Test plan
ReadsAVersion4CompoundFileWith4096ByteSectors,EnumeratingTheSameResultTwiceYieldsEveryRowBothTimes,IsoDateReadsBackTheSameDayInEitherDateSystem,ContinuationSheetSkipsANameAlreadyTakenandPrefetchChecksTheDeclaredBytesOfAnEntryThatInflatesPastItsSize.FastestSegmentMatchesTheStreamingEncoder: the windowless segment encoder is byte-identical to the streaming one.CsvParallelParseBenchmark,LargeSheetParallelBenchmarkandCsvParallelVsSepBenchmark, andcheck-benchmark-groups.shpasses for the new methods.