Skip to content

test: exercise panel database and encryption-key recovery - #839

Open
Baylem wants to merge 24 commits into
masterfrom
stack/pr-770
Open

Baylem wants to merge 24 commits into
masterfrom
stack/pr-770

Conversation

@Baylem

@Baylem Baylem commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Supersedes #770 using the same reviewed commits on an upstream branch, enabling GitHub's native stacking. Prior review discussion remains on that PR. Current-head CI passed.

This PR independently targets master.

Panel database recovery needs to preserve live games, restored authorization and the encryption key paired with the backup. This adds a recovery runbook for combined and standalone deployments and an isolated SQLite recovery drill in CI.

The drill backs up a private standalone panel, destroys its working database, restores a copy, invalidates old bearer credentials and checks users, scoped permissions, encrypted cluster registration, audit integrity and continued game progress. A wrong-key startup must exit without serving HTTP; the probe has a positive control. A sanitized CI artifact records actual recovery-point age and recovery time, including the wrong-key trial. The original archive remains read-only.

The runbook covers database outage/loss, lost keys, Kubernetes/SQL identity reconciliation and operator-selected targets. PostgreSQL PITR and complete storage-cluster recovery remain deployment-specific procedures.

Validation: E2E Go source and Python compilation passed; actionlint, offline zizmor and whitespace checks passed. Runtime tests and the recovery drill passed in CI; local checks remained compile-only under repository policy.

Includes the shared web build prerequisite from #759 so full CI can build the existing UI.

Includes the shared NuclearOption agent correction also present in #838: one absolute deadline covers dialing, request writes and response reads, and a stalled partial request closes the connection. Regression sources cover blocked length and body writes; the original closed-response assertion is preserved.

Closes #728.

The recovery drill waits for Healthy cluster discovery, grants the isolated administrator access to the fixture cluster and namespace, and logs in again after that grant invalidates its session. The Kubernetes-created game is assigned to the administrator through the scoped transfer API; share creation must fail before that assignment and succeed afterward. Both administrator and reader cookies are proved valid before backup and rejected through the authenticated self-read endpoint after recovery. Its template update changes only the intended command, avoiding conflicts with controller status updates.

CI installs Trivy v0.74.0 directly from its official GitHub release with bounded retries and a pinned SHA-256 checksum. The existing vulnerability scans and failure gates remain enabled.

November 6 follow-up: All six affected standalone Go modules now select github.com/klauspost/compress v1.20.2 to fix GO-2026-6865. Module and workspace checksums are included. The patch was independently reviewed; standalone Linux builds, affected feature builds and controller regression-source compilation passed. Runtime tests and security scans run in CI under repository policy.

Current validation: all 104 main CI jobs passed on fb428ebb (CI run). All other applicable workflows passed.

Baylem added 20 commits October 21, 2026 12:31
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 37 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 973aa12 and fb428eb.

Files ignored due to path filters: 7

  • agent/go.sum is excluded by !**/*.sum
  • api/go.sum is excluded by !**/*.sum
  • go.work.sum is excluded by !**/*.sum
  • mcp-server/go.sum is excluded by !**/*.sum
  • operator/go.sum is excluded by !**/*.sum
  • telemetry-receiver/go.sum is excluded by !**/*.sum
  • tunnel/go.sum is excluded by !**/*.sum
Files selected for processing: 26

  • .github/actions/install-trivy/action.yaml
  • .github/workflows/ci.yaml
  • .github/workflows/publish-edge.yaml
  • .github/workflows/release.yaml
  • agent/go.mod
  • agent/internal/rcon/nuclearoption.go
  • agent/internal/rcon/nuclearoption_deadline_test.go
  • agent/internal/rcon/nuclearoption_test.go
  • agent/specs.md
  • api/go.mod
  • docs/panel-recovery.md
  • docs/standalone-panel.md
  • hack/ci_scope.py
  • hack/test_ci_scope.py
  • mcp-server/go.mod
  • operator/go.mod
  • telemetry-receiver/go.mod
  • test/e2e/api_client_response_e2e_test.go
  • test/e2e/buckets.sh
  • test/e2e/ci_selection_e2e_test.go
  • test/e2e/env.go
  • test/e2e/panel_recovery_e2e_test.go
  • test/e2e/specs.md
  • test/e2e/suite-components.json
  • test/e2e/testdata/recovery/maintenance.py
  • tunnel/go.mod
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Baylem Baylem added area: e2e kind-based end-to-end suite area: shared netguard, gameaction, gameproto, svcutil and other shared packages area: web React dashboard type: test Test additions or changes labels Nov 5, 2026 — with ChatGPT Codex Connector
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

CI Report - fb428eb

run 42180081179 - attempt 1

0 failed - 33 passed - 0 skipped - 149m27s elapsed

Test selection: full (selection-rule change) - per-suite table

Coverage

Module Measured Gate Margin
web L95.93 S94.44 B89.33 F91.9 L92 S92 B82 F76 +2.44

Changed

Area Files +/-
test/ 8 +876/-3
docs/ 2 +180/-0
agent/ 6 +67/-16
.github/ 4 +50/-13
hack/ 2 +12/-2
api/ 2 +3/-3
mcp-server/ 2 +3/-3
operator/ 2 +3/-3
tunnel/ 2 +3/-3
telemetry-receiver/ 2 +3/-2
(root) 1 +1/-0

Slowest jobs

Job Duration
e2e telemetry / arm64 (kind) 23m4s
e2e web live / amd64 (kind) 12m27s
e2e capture overhead (kind) 12m27s
build e2e images (arm64) 12m14s
build e2e images 12m13s

e2e buckets run

api-agent, api-auth, api-mods, api-rbac, api-roles, bot-fast, multicluster, operator, optional-components, postgres-availability, ratelimit, telemetry, upgrade

@Baylem
Baylem marked this pull request as ready for review November 6, 2026 06:56
@Baylem
Baylem requested a review from ValgulNecron as a code owner November 6, 2026 06:56
Select github.com/klauspost/compress v1.20.2 in all six affected standalone module graphs, including transitive consumers, and refresh their checksums.

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
Keep the historical compress go.mod checksum required when loading transitive workspace module metadata. The selected compress source remains v1.20.2.

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e kind-based end-to-end suite area: shared netguard, gameaction, gameproto, svcutil and other shared packages area: web React dashboard type: test Test additions or changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs/ops: define and exercise panel database, Kubernetes state, and key recovery

1 participant