Skip to content

Read files in bulk and report output stream failures - #753

Open
Altren wants to merge 2 commits into
munich-data-engineering:mainfrom
Altren:perf/bulk-file-io
Open

Altren wants to merge 2 commits into
munich-data-engineering:mainfrom
Altren:perf/bulk-file-io

Conversation

@Altren

@Altren Altren commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

This change uses the reported file size as an allocation hint and reads directly into the buffer. When the size is unavailable or more data follows, it continues in 64 KiB blocks without seeking. Binary and text modes remain distinct, including native text conversion; the hint is not treated as a guaranteed byte count.

The shared save helpers now check stream state after closing, returning an error for failed writes or buffered-output flushes instead of reporting success. Reader stream errors and directories also return errors. Public APIs remain unchanged.

Measurements

Measurements compare upstream main (bb0c1dcb) with the new shared file loaders. Both sides use the same parser and serializer code. Apple Clang 21, macOS arm64, C++20 Release -O3 -DNDEBUG, no LTO. Six process runs with rotated/reversed case order; at least 0.1 s warmup and 0.25 s measured CPU time per case. Tables report median wall-clock milliseconds, including file open/read/close, allocations, and result destruction. Files are in the OS cache; these are not cold-disk throughput measurements. Fixture generation and equality checks are outside timing.

File loading only

Case File bytes Main, ms New, ms Speedup
Binary/1024 1,024 0.0122 0.0123 —
Text/1024 1,024 0.0127 0.0124 1.03×
Binary/1048576 1,048,576 2.0545 0.1486 13.83×
Text/1048576 1,048,576 2.4888 0.1512 16.46×
Binary/33554432 33,554,432 66.7553 5.0514 13.22×
Text/33554432 33,554,432 79.5736 4.9843 15.96×

Complete file load and deserialization

Case File bytes Main, ms New, ms Speedup
Cereal/Dataset 36,008,009 88.8233 23.1746 3.83×
YAS/Dataset 36,008,015 88.0490 22.5152 3.91×
JSON/Dataset 70,124,332 214.4066 57.6431 3.72×

Dataset contains 500 groups, each with 3,000 points (double x, y) and 3,000 uint64 IDs. Both loaders reproduce every original field.

Validation

  • Added BM_load_file cases to the all-format benchmark target for binary/text files at 1 KiB, 1 MiB, and 32 MiB; all six compiled and ran.
  • A Linux-only /dev/full test checks buffered output failure at close. Linux and Windows were not run locally; the text-mode equivalence test is portable and intended to exercise Windows newline conversion in CI.

@Altren

Altren commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

The only test failure is an apt update issue

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant