Skip to content

Add reader core scaling measurements with hot and cold caches - #46

Merged
tom-clickhouse merged 4 commits into
mainfrom
codex/reader-cpu-scaling
Oct 13, 2026
Merged

tom-clickhouse merged 4 commits into
mainfrom
codex/reader-cpu-scaling

Conversation

@tom-clickhouse

@tom-clickhouse tom-clickhouse commented Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator

This adds reproducible reader CPU scaling measurements for Blog 6 11c, using the original D1/D2 drill-down queries against the fixed 113.2B-quote dataset. The current cache-controlled sweep measures 2, 4, 8, 16, 30 and 59 cores on one fixed 59-vCPU / 236-GiB node, with max_threads controlling query parallelism. It does not compare six physically resized services.

The selected visual compares Hot caches with Cold data caches and query-condition caching off. Pooled uncompressed throughput rises from 0.803 to 15.560 GB/s for Hot and 0.731 to 9.848 GB/s for Cold. Gains taper at larger settings; the package does not claim perfect linear scaling. A separate condition-cache-on cold treatment and earlier 16/59-vCPU exploratory ledgers remain intact, without being spliced into the selected curves.

The cache pilot demonstrated that filesystem-cache-off alone still read through the userspace page cache. Cold treatments therefore bypass filesystem, page, and distributed data-cache reads per query. Profile counters verify positive S3 reads and zero page/distributed reads for Cold, and page hits with zero S3 reads/misses for Hot. Query-result caching remains disabled. Metadata/mark caches are not globally purged.

Included: exact source/treatment SQL, shuffled trial plans, append-only HTTP records, full query-log CSVs/ProfileEvents/Settings, capacity/version/data/cache checks, derived statistics, reports, offline analysis, live repeat instructions, and SHA-256 manifests. derive_visual.py reproduces the exact 11c payload and resource CSV: CPU used/headroom relative to the selected core budget and average measured per-query memory peaks. Throughput per core uses consumed CPU seconds, matching 11a.

Validation:

  • 432 cache-controlled measured queries plus 24 pilot/warmup executions; all 456 server records collected.
  • Zero failures, missing logs, result-cache reads or removed outliers; unchanged active data parts and service identity/capacity/version.
  • Offline cache analysis and resource derivation pass; exact 11c JSON matches the production visual input.
  • All 526 cache-package file hashes verify; private-secret scan, README links, and staged patch checks pass.

Start at full-path-realtime/quotes/clickhouse-cloud/experiments/reader-thread-scaling/README.md, then 2026-10-11-cache-modes/README.md for selected cases and reproduction steps. Original Run 2 continuous-ingestion evidence remains separate in PR #44; this experiment does not recreate its historical cache state or 3.25 GB/s display endpoint.

@tom-clickhouse tom-clickhouse changed the title Add reproducible reader CPU scaling measurements at 2-59 cores Add reader core scaling measurements with hot and cold caches Oct 11, 2026
@tom-clickhouse
tom-clickhouse marked this pull request as ready for review October 13, 2026 11:38
@tom-clickhouse
tom-clickhouse merged commit 71617f1 into main Oct 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants