Improve HTTP server autoscaling signals, overload handling, warmup, and startup recovery

I have opened a pull request implementing all four improvements described below.

Summary

I would like to contribute four focused improvements to the Rust HTTP query server:

1. A Prometheus endpoint exposing time-weighted workspace demand for dynamic scaling.
2. Opt-in, atomic rejectWhenBusy admission control, scoped to each repository.
3. Startup warmup that actually exercises Bazel instead of returning server cache hits.
4. Bounded recovery when the initial Git fetch explicitly recommends pruning origin.

Together, these changes address the operational lifecycle of a replicated service: deciding when to add capacity, avoiding unnecessary queueing, preparing new replicas for real work, and recovering from a specific startup failure without manual intervention.

The implementation builds on the existing multi-repository architecture, cancellation lifecycle, and cache tiers. It does not introduce a new cache backend, replace the existing JSON metrics, or change the affected-target calculation.

Motivation

The existing server already provides useful JSON snapshots, including activeQueries and workspaceBusy, and supports queued requests, timeouts, and cancellation. However, several deployment concerns remain:

• An instantaneous busy flag cannot distinguish a brief burst from sustained contention or show how much work is waiting behind a workspace lock.
• Requests can wait behind expensive work even when a caller would prefer to try another replica or retry later.
• A successful warmup can be satisfied by an existing server cache entry, leaving the Bazel execution path cold for the first real cache miss.
• A startup fetch can leave a repository unready even when Git identifies a concrete recovery command.

These improvements make the server easier to operate behind a load balancer and in an autoscaled deployment, while retaining existing request behavior unless callers explicitly opt into busy rejection.

1. Prometheus workspace-demand metric

Proposed interface

• Endpoint: GET /metrics/prometheus.
• Format: Prometheus text exposition (text/plain; version=0.0.4; charset=utf-8).
• Metric: bazel_diff_compute_slot_demand_avg, a gauge with a repo label.
• Configuration: --computeSlotDemandAvgWindow, defaulting to 5s.
• Other supported window examples: 10s, 30s, and 1m; zero and invalid durations are rejected.

The existing JSON /metrics endpoint remains unchanged. The new endpoint is also available before readiness, allowing monitoring to observe startup activity.

What it measures

Demand is the number of operations holding or waiting to acquire a workspace lock. The metric is a rolling, time-weighted average of that demand:

average demand = sum(demand level × time spent at that level within the window) / window duration

During startup, if a repository's metric history is shorter than the configured window, the denominator is the elapsed time since that history was initialized instead.

For example, with a five-second window:

• One operation holding the workspace throughout the window produces an average of 1.
• One holder and two waiters throughout the window produce an average of 3.
• Demand of 3 for one second followed by four idle seconds produces an average of 0.6.

Each repository has its own demand counter and rolling history, protected by its workspace's metrics mutex. RAII guards remove demand on release, cancellation, or error without affecting other repositories.

The endpoint always emits one series per configured repository, including idle repositories. Ordinary single-repository serve uses repo="default"; serve-multi uses the configured repository names. For example:

bazel_diff_compute_slot_demand_avg{repo="backend"} 2.4
bazel_diff_compute_slot_demand_avg{repo="frontend"} 0

This is workspace demand, not CPU utilization or a percentage. It includes the full workspace-protected operation, including cache-hit paths, Git work, hash generation, and warmup. Separate series distinguish demand at one repository from parallel activity across independent workspaces. An average above 1 for one repository indicates that waiting operations contributed during the window, not necessarily that a queue exists at scrape time.

No additional unlabeled total is emitted under the same metric name, preventing double-counting in aggregations. Consumers can calculate totals themselves, for example sum(bazel_diff_compute_slot_demand_avg) across all matching series, or sum by (repo) (bazel_diff_compute_slot_demand_avg) across replicas for each repository.

Why this is useful for KEDA and dynamic scaling

This metric is particularly well suited to dynamic scaling through a Prometheus-based scaler such as KEDA:

• It measures the constrained resource directly. Requests serialize on a repository's workspace, so CPU utilization alone may not reveal the queue forming behind it.
• It captures both service and waiting demand. A busy boolean remains true whether one operation is executing or several others are waiting.
• It preserves duration, not just snapshots. Integrating state transitions retains the contribution of bursts between scrapes; short spikes contribute proportionally rather than looking identical to sustained saturation.
• It offers a configurable stability/responsiveness trade-off. Operators can choose a shorter window for faster reaction or a longer one to smooth transient bursts.
• It is directly scrapeable. Prometheus can collect the gauge without a custom JSON exporter, and KEDA can query it to drive replica decisions.

The window should be tuned together with the scrape interval, scaler polling interval, cooldown, and actual per-instance capacity. The repo label lets operators identify hot repositories and choose repository-specific scaling signals or aggregate demand, accounting for traffic distribution and which repositories new replicas actually serve.

For a single repository, adding the label preserves the numeric signal. For consumers migrating from the earlier unlabeled implementation, this is a time-series schema change: verify downstream exporter/metric-descriptor compatibility and avoid double-counting old and new series that coexist within a monitoring lookback window during rollout.

Requests rejected before workspace acquisition do not contribute to this metric. Deployments using busy rejection should therefore also consider ingress request/rejection metrics when evaluating total offered load. This proposal provides the scaling signal, not a bundled KEDA configuration or a mechanism for waking a deployment from zero replicas.

2. Opt-in rejectWhenBusy admission control

Proposed behavior

Clients can request immediate rejection instead of queueing:

• GET: add rejectWhenBusy=true to the query string.
• POST: set "rejectWhenBusy": true in the JSON body.

This applies to the impacted-target, impacted-target-with-distances, and dependency-edge query endpoints.

If the selected repository already has an accepted, unfinished query, the server returns HTTP 503 with:

{"error":"server busy: another query is already processing"}

Admission uses an atomic compare-and-exchange on the existing per-repository active-query counter, before spawning the worker. Two simultaneous opt-in requests cannot both observe the same repository as idle and reserve admission.

Benefits and compatibility

• Lets a caller or routing layer choose another replica, or apply its own retry/backoff policy, rather than consuming its timeout in a local queue.
• Keeps independent repositories independent: activity in one repository does not reject a request targeting another.
• Reuses the existing active-query guard, including its completion, panic, and cancellation bookkeeping.
• Requires no new server startup flag.
• Preserves existing queueing for requests without the parameter or with false.

GET enables the feature only for the exact value true; POST accepts a JSON boolean. Busy rejection is not an exclusive reservation against later ordinary requests: unflagged requests can still be accepted and wait normally.

This complements existing timeout cancellation rather than replacing it. Rejection avoids admitting new work; cancellation stops previously admitted work after its timeout.

Integration with a service mesh or retry-aware routing layer

In a replicated deployment, rejectWhenBusy allows application-level admission control to cooperate with infrastructure-level traffic management. Rather than accepting a query into a busy replica's workspace queue, the server can immediately return HTTP 503, allowing a suitably configured routing layer—for example, Istio with Envoy—to retry the query against another eligible replica. This can reduce queue-induced tail latency and make better use of idle capacity elsewhere in the deployment, provided those replicas serve the requested repository.

This is retry-based load redistribution, not an HTTP redirect or an automatic server-side forwarding mechanism. Returning 503 alone does not guarantee a retry or selection of a different replica. The mesh must explicitly enable retries for the relevant response status and configure host selection to prefer a different eligible endpoint where supported. Requests must also opt into rejectWhenBusy; the mesh does not enable that application behavior merely by installing a retry policy.

Retry policies should use bounded attempts, backoff, retry budgets, and an overall request deadline to avoid amplifying overload when all replicas are busy. Query replay must be safe, including for POST requests, and retries must preserve the request body and repository selection. Since 503 is also used for readiness failures, a status-based retry policy may handle those responses as well; the busy response is not a distinct HTTP status. These deployment policies remain outside the server implementation.

3. Warm Bazel, not just the server cache

Problem

When --warmupRevision uses the ordinary cache-retrieval path, an in-memory, local-disk, or remote cache hit can make warmup finish without exercising Bazel. A newly started replica can therefore appear warmed while its first real cache miss still pays the cold Bazel/JVM/query cost.

Proposed behavior

For each configured warmup revision:

• Bypass reads from all three existing server cache tiers: in-process RAM, local disk, and configured remote storage.
• Run fresh hash generation, without deriving the result from a cached partial-graph base.
• Preserve the normal cache-write path: refresh local storage, attempt the configured remote write, and update the in-memory cache when the entry fits its budget.
• Leave ordinary query cache retrieval unchanged.

This also forces a full generation when partial-graph mode is enabled. It does not disable Bazel's own caches or imply that Bazel must rebuild every action.

Related RAM-cache correctness fix

A same-key refresh first removes the old in-memory entry, even if the new result is too large for the memory-cache budget. Otherwise, a stale RAM value could continue shadowing the refreshed on-disk result. If the refreshed entry cannot fit, subsequent queries fall back to the fresh lower-tier entry instead.

The RAM cache is pre-existing infrastructure; this is a refresh-consistency fix, not an additional cache type.

Benefits and trade-off

Explicit warmup prepares the execution path a replica will need for uncached queries, making it more useful during scale-out and rolling updates. It also refreshes server cache contents rather than merely confirming their presence.

The trade-off is intentional additional startup work when warmup revisions are configured. Existing warmup failure handling remains unchanged; this proposal does not make every warmup failure fatal or a readiness blocker.

4. Recover from Git's explicit prune recommendation

Proposed behavior

For the initial git fetch --all --tags:

1. If fetch succeeds, continue normally.
2. If fetch fails, inspect both stdout and stderr.
3. Only if either stream contains git remote prune origin, run that fixed command.
4. If pruning succeeds, retry the fetch once.
5. If pruning or the retry fails, retain the existing unready/lame-duck behavior.

The recovery command is fixed; the server does not execute arbitrary instructions from Git output. It does not prune on unrelated failures or merely because a successful fetch mentions the command. Disabling the initial fetch continues to bypass this path.

Benefits and safety boundaries

This can recover unattended from stale remote-tracking references for which Git explicitly recommends pruning, avoiding a manual repair or repeated restart cycle.

Recovery is deliberately narrow and bounded. There is no generic retry loop or attempt to repair authentication, network, or unrelated repository failures. An unrecovered repository remains unready: health and query requests retain their existing 503 behavior, metrics remain available, and warmup is not attempted for that failed startup path.

Scope

The proposed implementation is limited to the HTTP server and its CLI configuration in src/server.rs and src/main.rs, including their embedded tests. It adds no dependencies and preserves the existing cancellation model, multi-repository support, and normal cache behavior outside explicit warmup.

The goal is a small operational improvement set with clear boundaries: a better scaling signal, optional fail-fast admission, meaningful warmup, and a targeted startup recovery path.