[haozzz (haozzzdev)]

Summary

System design for #278: capacity-aware Kueue admission with Cluster Autoscaler, plus a runnable laptop demo of the success path.

• docs/design/capacity-provisioning-admission.md: the design. It gates admission on capacity with Kueue's ProvisioningRequest AdmissionCheck (best-effort-atomic-scale-up). It bounds every wait in three layers: submit-time preflight, CA ValidUntilSeconds plus Kueue retryStrategy, and a TauGrid admission deadline that sets Workload.spec.active=false. It sets up the Kueue objects once per cluster through TauCluster External / Adopt / Manage modes, leaves the researcher workflow unchanged, and handles topology with a workload label that CA resolves from its own in-memory domain map (§10).
• docs/design/images/capacity-provisioning-admission.{png,svg}: the component diagram, with every addition in red.
• docs/design/examples/capacity-provisioning-demo/: real Kueue v0.19.2 and the real ProvisioningRequest CRD on kind, with a small fake autoscaler and KWOK nodes. A 2 × 8 GPU Job stays suspended until both nodes exist, then both pods start together, and the pool scales back to 0.

Documentation and demo only. No change to TauGrid controllers, CLI, CRDs or charts.

Related issue

Closes #278

Validation

Ran the demo on kind (Docker, kind v0.29.0, Go 1.22.2):

./setup.sh
./bin/tau-demo fake-ca          # terminal 1
AUTO=1 ./scenario.sh            # terminal 2

preflight: profile gpu-a100-8 ProvisioningReady=True
preflight: gang needs 2 node(s) of gpupool (max 2): fits
  ✓ Submitted              Job applied (suspended)
  ✓ Quota reserved         16 nvidia.com/gpu on ClusterQueue gpu-a100 (flavor a100-scale)
  ✓ Capacity provisioning  capacity booked on 2 node(s) of gpupool
  ✓ Pod scheduling         2/2 on aks-gpupool-vmss438424, aks-gpupool-vmss913705
  ✓ Running                Succeeded (2/2 completions)
Node pool gpupool: 0/2 nodes

The full captured run is in docs/design/examples/capacity-provisioning-demo/sample-output.txt.

Compatibility and operational impact

None. Docs and a self-contained example with its own Go module.

Checklist

• The change is focused and does not include unrelated edits.
• Tests cover the changed behavior, or I explained why tests are not needed. (Design doc; the demo was run end to end as shown above.)
• Documentation and examples are updated when public behavior changes.
• Generated files are updated from their source and included in this pull request. (None.)
• New dependencies are publicly available and their license and provenance were reviewed. (The demo uses upstream Kueue, KWOK and client-go only.)
• The diff contains no secrets, private endpoints, customer data, or Microsoft-internal information.
• Fork-based CI can validate the change without repository secrets, or maintainer-only validation is clearly identified.

[haozzz (haozzzdev)]

@microsoft-github-policy-service agree

[Pengfei Ni (feiskyer)]

Kueue can admit the Workload after markTerminal succeeds but before deactivate. The Workload patch then conflicts, but the owner keeps its terminal annotations; the next reconcile takes admitted(wl) and calls maybeTTLCleanup. Under the proposed status and TTL rules, a running job is reported failed and can later be deleted. Please commit the admission-versus-timeout decision on one object, then mirror only a committed timeout to the owner and arm cleanup, with recovery for an interrupted mirror.