Skip to content

fix: prepare backup workers before quiesce and bound CSI captures - #836

Open
Baylem wants to merge 36 commits into
stack/pr-769from
stack/pr-772
Open

Baylem wants to merge 36 commits into
stack/pr-769from
stack/pr-772

Conversation

@Baylem

@Baylem Baylem commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Supersedes #772 using the same reviewed commits on an upstream branch, enabling GitHub's native stacking. Prior review discussion remains on that PR. Current-head CI passed.

Native stack, bottom to top: #833 → #834 → #835 → #836.

Backup preparation could disable saving while a worker was still waiting for storage or repository access, and a stalled CSI capture could hold quiesce indefinitely. This prepares and verifies the real worker before save-off, then applies persisted deadlines to preparation, live copying and CSI capture.

  • Restic workers mount the source, initialize/check the repository and become Ready before quiesce. An owned projected ConfigMap authorizes only the exact Backup, worker Pod, source identity and expiry. Workers use resource defaults and prefer the source node, with an optional hard placement requirement. First binding waits for an initial source Pod to be created and scheduled within the preparation deadline; a replacement after binding still fails closed. They receive no Kubernetes API token.
  • Preparation, quiesce and CSI deadlines survive restarts and configuration changes. Copy failure/deletion revokes authorization and waits for actual worker drainage before save-on. Raw and already-stopped copies retain the full Job budget.
  • Aborted CSI snapshots remain invalid even if the driver later reports readiness. Cleanup verifies snapshot/content UIDs and respects Retain policy. Restore and new snapshot-backed PVCs reject incomplete, aborted or replaced captures.
  • Prepared workers record volume coverage before copy authorization and use the effective template under lifecycle ownership, including when a newer catalog revision is pending.

Includes controller, worker-script and snapshot regressions plus a dedicated two-node RWO preparation/restore CI drill. The Kind fixture does not claim qualification of attachable CSI drivers. Worker expiry assumes synchronized clocks; restoring saving after an operator outage still requires operator/agent recovery.

Dependencies included: #760 lifecycle exclusion, #833 stopped-server preparation, #767 capture metadata, #835 staged templates and its security/resource dependencies, #756 snapshot authorization, and #759 shared web build prerequisite. Review the commits for this issue group after those prerequisites. Shared restore cleanup retains ownership until failed startup drains, including force-deletion recovery. The preparation, placement and Restricted workload buckets remain registered in upstream change-based test selection.

Validation: production and regression-source compilation passed; required actionlint/offline zizmor and whitespace checks passed. Runtime unit, envtest and E2E execution remains in CI under the repository's compile-only local policy.

Closes #717.
Closes #719.

Issue coordination: #717 and #719 build on #718's exclusive ownership (#760) and #716's stopped-source path (#833).

The dependency stack now includes verified effective-template API redaction, clone password protection and diagnostics namespace isolation. Backup detail UI follows Pencil nodes zhLZN (achieved consistency and legacy volume coverage) and bHJy6 (recorded per-volume coverage), with saved JSON/PNG exports and screenshot coverage inherited from #835.

Final integrated API/operator builds and handler/controller regression-source compilation passed.

Transient source API errors retry within the persisted copy deadlines while unverified worker authorization is revoked. Confirmed identity changes still abort. Cleanup retains lifecycle ownership and quiesce obligations until workers drain. The Garry’s Mod CI job releases unused hosted-runner SDKs and its already-loaded image archive before Kind starts, preserving the real game image and test assertions.

Inherited password protection now covers creation, cloning and unpinned edits plus unsafe template retyping. The finalized Pencil references include corrected drawer spacing, typography and centered action labels. CI uses the shared checksum-verified Trivy release installer; current runtime results are recorded below.

Visual verification on 0a976382 passed all 115 design comparisons and expanded-text clipping checks without relaxing thresholds (visual run).

November 6 follow-up: Source-change errors now preserve both the sentinel and JSON cause. The postcheck retry regression waits for persisted capture metadata before injecting a transient read failure, and asserts that injection occurred. Includes the API and initial-pin fixture repairs from #835. All six affected standalone Go modules now select github.com/klauspost/compress v1.20.2 to fix GO-2026-6865. Module and workspace checksums are included. The patch was independently reviewed; standalone Linux builds, affected feature builds and controller regression-source compilation passed. Runtime tests and security scans run in CI under repository policy.

Current validation: all 106 main CI jobs passed on c4288100 (CI run). Visual comparison, expanded-text clipping and portable-path checks also passed. Owner-delete E2E retries only HTTP 409 through fresh authorization, preserves the created object's UID, and requires HTTP 204 plus deletion of that object. The webhook fixture verifies a separate French readiness send through the API-to-Service CGI path and retains bounded failure diagnostics. Both corrected scenarios passed on AMD64 and ARM64; actual language/status/payload assertions and the production timeout remain unchanged.

Baylem added 28 commits October 21, 2026 11:58
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Revoke unverified worker gates while retaining lifecycle and cleanup obligations. Reserve disk for the Garrys Mod CI image before starting kind.

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 42 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 4128a3b and 975c6ec.

Files selected for processing: 44

  • .github/workflows/ci.yaml
  • charts/gameplane/templates/operator.yaml
  • charts/gameplane/values.yaml
  • docs/operator-operations.md
  • hack/ci_scope.py
  • hack/test_ci_scope.py
  • i18n/locales/en/app.json
  • operator/cmd/main.go
  • operator/config/rbac/role.yaml
  • operator/hack/condition-ids/main.go
  • operator/internal/controller/backup_budgets.go
  • operator/internal/controller/backup_budgets_test.go
  • operator/internal/controller/backup_controller.go
  • operator/internal/controller/backup_csi_deadline_order_test.go
  • operator/internal/controller/backup_envtest_test.go
  • operator/internal/controller/backup_forget.go
  • operator/internal/controller/backup_guard.go
  • operator/internal/controller/backup_guard_test.go
  • operator/internal/controller/backup_manifest.go
  • operator/internal/controller/backup_manifest_test.go
  • operator/internal/controller/backup_preparation.go
  • operator/internal/controller/backup_preparation_test.go
  • operator/internal/controller/backup_snapshot_cleanup.go
  • operator/internal/controller/backup_snapshot_source.go
  • operator/internal/controller/backup_snapshot_source_test.go
  • operator/internal/controller/backup_stopped_test.go
  • operator/internal/controller/backup_volumesnapshot.go
  • operator/internal/controller/backup_volumesnapshot_envtest_test.go
  • operator/internal/controller/backup_worker.go
  • operator/internal/controller/backup_worker_replacement_test.go
  • operator/internal/controller/backup_worker_retry_test.go
  • operator/internal/controller/gameserver_controller.go
  • operator/internal/controller/helpers_envtest_test.go
  • operator/internal/controller/restore_controller.go
  • operator/specs.md
  • test/e2e/api_owner_only_ops_e2e_test.go
  • test/e2e/backup_preparation_e2e_test.go
  • test/e2e/buckets.sh
  • test/e2e/ci_selection_e2e_test.go
  • test/e2e/failure_paths_e2e_test.go
  • test/e2e/fixtures/notify-sink.yaml
  • test/e2e/garrysmod_bot_e2e_test.go
  • test/e2e/i18n_notify_language_e2e_test.go
  • test/e2e/specs.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Baylem Baylem added the area: agent in-pod sidecar label Nov 5, 2026 — with ChatGPT Codex Connector
@Baylem Baylem added area: api REST/WebSocket gateway area: chart Helm chart and deployment area: e2e kind-based end-to-end suite area: modules game template OCI bundles area: operator controller-runtime operator and CRDs area: optional-components sentinel, capture-sidecar, mcp-server, tunnel, audit-syslog-bridge, telemetry-receiver area: shared netguard, gameaction, gameproto, svcutil and other shared packages area: web React dashboard type: fix Bug fix labels Nov 5, 2026 — with ChatGPT Codex Connector
@Baylem
Baylem added this pull request to stack #841 November 5, 2026 03:57
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

CI Report - 975c6ec

run 42274538040 - attempt 1

0 failed - 0 passed - 0 skipped - 33 cancelled - 1s elapsed

Test selection: unknown - per-suite table

Coverage

no coverage/* commit statuses found on 975c6ec

Changed

Area Files +/-
operator/ 28 +2442/-154
test/ 9 +312/-27
docs/ 1 +76/-14
i18n/ 1 +32/-0
charts/ 2 +24/-0
.github/ 1 +18/-4
hack/ 2 +10/-2

Slowest jobs

Job Duration
move parity (spec 028) 0s
telemetry default gate 0s
detect changes 0s
codegen drift 0s
standalone Compose smoke 0s

e2e buckets run

bot-fast, multicluster, upgrade

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Probe the API-to-Service CGI path with a separate French sink before checking language payloads. Keep actual send assertions and production timeouts unchanged, and retain bounded receiver diagnostics before cleanup when a test fails.

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem marked this pull request as ready for review November 6, 2026 16:30
@Baylem
Baylem requested a review from ValgulNecron as a code owner November 6, 2026 16:30
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: agent in-pod sidecar area: api REST/WebSocket gateway area: chart Helm chart and deployment area: e2e kind-based end-to-end suite area: modules game template OCI bundles area: operator controller-runtime operator and CRDs area: optional-components sentinel, capture-sidecar, mcp-server, tunnel, audit-syslog-bridge, telemetry-receiver area: shared netguard, gameaction, gameproto, svcutil and other shared packages area: web React dashboard type: fix Bug fix

Projects

None yet

1 participant