[Darkheir]

Description

All buckets share the same pool of GetObject connection permits (QW_S3_MAX_GET_OBJECT_CONNECTIONS). During large bursts the pool saturates (up to 1023/1024 permits in use and 824 reads waiting on one searcher on fra1), but we cannot tell which bucket fills it, nor whether reads on other buckets suffer from it. Since #117, S3 timeouts also no longer log anything. This PR adds the bucket to both, to decide whether a per-bucket cap on connection permits is worth it.

	Before	After
object_storage_get_object_connections_in_use / _waiting	One value per searcher	One value per searcher and bucket (bucket label)
get_slice deadline exceeded	Metric only	Metric + rate-limited warning with bucket, key and size
get_slice giving up after a timed-out attempt	Metric only	Metric + rate-limited warning with bucket, key, size and number of timed-out attempts

Each warning is rate-limited to 60 per minute per searcher, so their counts are lower bounds.

Dashboards: per-searcher totals of the two gauges now need sum by (kubernetes_pod_name) (...).

How was this PR tested?

Added a test checking that a connection permit raises the gauge of its bucket only, and releases it when dropped. Existing timeout and deadline tests cover the warning paths.

[Copilot]

🟡 Changes recommended

The timeout warning misrepresents mixed retry failures, and independent rate limiters allow twice the stated combined warning budget.

2 open issues

• Share the warning rate limit across both failure modes · New
• Describe only the final attempt as timed out · New

What changed in this PR

This PR adds per-bucket visibility into S3 connection contention and restores timeout warnings, helping assess whether the shared connection pool needs per-bucket limits.

Changes:

• Adds bucket labels to connection-in-use and waiting gauges while retaining the shared permit pool.
• Adds rate-limited warnings with bucket, key, and size for deadline and terminal timeout failures.
• Adds a test for bucket-isolated in-use gauges and cleanup when permits are dropped.

File	Description
quickwit/​quickwit-storage/​src/​object_storage/​s3_compatible_storage.rs	Tracks permits by bucket, logs timeout outcomes, and tests gauge isolation.
quickwit/​quickwit-storage/​src/​metrics.rs	Converts connection gauges to bucket-labelled metric vectors and updates their descriptions.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues are reported for the change adding bucket-level S3 connection metrics and timeout diagnostics.

No open issues 🎉

2 resolved since last review

• Describe only the final attempt as timed out
• Share the warning rate limit across both failure modes

_{Review effort: Balanced}