Store logs in batches as they are collected instead of dropping them

[define-null]

Each round inserts the whole buffer only after every worker is done, so no worker is requested while it is inserted, and a full buffer drops pages that were already fetched and verified, only to request them again next round. With a large buffer, the insert at the end takes a good part of each round.

Changes

• A writer stores the buffer a batch at a time (MAX_INSERT_BATCH_BYTES) as soon as one has accumulated, while collection goes on; the rest is stored at the end of the round.
• A full buffer makes the worker wait for room instead of dropping its page. MAX_BUFFER_BYTES now bounds memory under a slow ClickHouse; nothing is re-fetched.
• Workers furthest behind (lowest watermark) are requested first, instead of in HashSet order.
• A failed INSERT ends the round as before: collection in progress is cancelled and the rest of the buffer discarded, so the next round collects it again from the watermark. The batch is not retried, since an INSERT that reached ClickHouse before failing would be stored twice.
• Metrics: logs_collector_logs_deferred_total is removed; buffer_max_bytes is now the size at which collection waits for the writer.
• futures and tokio-util are no longer dependencies. Version 2.5.0 → 2.6.0.

Notes

• Rounds, the per-round watermark query, MAX_PAGES, timeouts and CONCURRENT_WORKERS are unchanged.
• Memory: MAX_BUFFER_BYTES plus up to CONCURRENT_WORKERS pages held by workers waiting for room. With batches stored as they fill, the buffer stays near one batch unless ClickHouse falls behind, so a raised MAX_BUFFER_BYTES can go back towards the 256 MiB default.
• Round logs: Dumped logs to storage (with logs) now reports the whole round's rows; per-batch lines are debug.

Rollout: deploy, check that round_duration_seconds drops and requests continue through the end of each round, then lower MAX_BUFFER_BYTES if it was raised.

Tested with 14 simulated-time tests, including: batches stored within a round, workers waiting for room without a page being requested twice, a failed batch ending the round without loss or duplicates, an INSERT that fails after writing not being stored twice, and furthest-behind-first order. fmt, clippy (-D warnings) and workspace tests pass on 1.98. Not yet run against real workers.

🤖 Generated with Claude Code