Skip to content

feat: add bounded game performance and relay monitoring - #838

Open
Baylem wants to merge 29 commits into
stack/pr-763from
stack/pr-771
Open

feat: add bounded game performance and relay monitoring#838
Baylem wants to merge 29 commits into
stack/pr-763from
stack/pr-771

Conversation

@Baylem

@Baylem Baylem commented Nov 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Supersedes #771 using the same reviewed commits on an upstream branch, enabling GitHub's native stacking. Prior review discussion remains on that PR. Current-head CI passed.

Native stack, bottom to top: #837 → #838.

Operators need to distinguish missing measurements, game degradation, resource pressure and relay failures. This adds cached numeric game-performance measurements and relay metrics, with a complete Prometheus configuration and a real policy-enforced scrape scenario in CI.

Modules can explicitly declare tick/frame rates or tick duration, units and an optional degradation threshold. One bounded source feeds both dashboard routes and Prometheus; scrapes never execute RCON. Unknown, invalid and stale readings have no value series, while genuine zero remains visible. Source RCON receives an aggregate response limit and an absolute exchange deadline so a streaming response cannot defeat the collector's resource bounds.

Tunnel metrics consume the same sanitized health evidence as readiness. The chart adds scrape-policy validation, tunnel discovery, pressure/game/relay alerts and a workload dashboard. The documented Prometheus example aligns monitor labels, discovery namespaces and ingress permissions; explicitly managed external policies remain supported. The CI fixture uses Prometheus Operator and Cilium, verifies actual agent/tunnel scraping, revokes/restores ingress and checks denial from an unapproved namespace. Promtool fixtures cover alert firing/recovery and duplicate operator readings.

No game module is automatically opted in. Playit's pinned client still reports unknown live connectivity, and infrastructure rules require kubelet/cAdvisor and kube-state-metrics collectors as documented.

Includes #768 relay health and its #765/#762 prerequisites, #837 network isolation, and #759 shared web build correction. The embedded Tailscale SDK follows upstream v1.104.2 with a Go 1.27.3 minimum. Management-placement, network-isolation and monitoring buckets are registered in change-based test selection. Monitoring bootstrap retries transient bundle-download failures within a fixed time budget and retains checksum verification.

Expected controller retries use explicit polling delays while genuine errors retain backoff. Mod deletion and its HTTP DELETE manifest update use the same opened directory handle, preventing symlink aliases or directory replacement from redirecting removal. Regression sources cover retry pacing, retained lifecycle fences, link-target preservation and atomic metadata cleanup.

Validation: Go production/regression-source and web type compilation passed locally. Runtime tests run in CI under the repository's compile-only local policy; verification of the current head passed.

The NuclearOption RCON client also applies its absolute command deadline before writing the request, preventing a peer that stops reading from holding the shared client indefinitely. Both blocked-write stages have regression coverage. Monitoring selector validation follows the operator NetworkPolicy gate, so standalone panel monitoring remains usable without a game scrape selector.

Closes #725.

The backup-forget envtest fixture seeds terminal status before starting reconciliation, preventing a normal-copy Job race in environments without garbage collection.

CI installs Trivy v0.74.0 directly from its official GitHub release with bounded retries and a pinned SHA-256 checksum. The existing vulnerability scans and failure gates remain enabled.

November 6 follow-up: All six affected standalone Go modules now select github.com/klauspost/compress v1.20.2 to fix GO-2026-6865. Module and workspace checksums are included. The patch was independently reviewed; standalone Linux builds, affected feature builds and controller regression-source compilation passed. Runtime tests and security scans run in CI under repository policy.

Current validation: all 105 main CI jobs passed on e5647400 (CI run). All other applicable workflows passed.

The operator E2E job initially hit a notification delivery timeout. One targeted rerun passed on the unchanged commit (attempt 2); no timeout, assertion, or production behavior was changed.

Baylem added 27 commits October 21, 2026 12:10
@Baylem
merge: combine relay and enforced-network monitoring prerequisites
2e109f9 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
feat: expose bounded game performance and relay monitoring
94d56a4 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
fix(agent): bound NuclearOption request writes
0c5fdf5 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
fix: align monitoring validation and relay regression fixtures
6cd3f48 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
fix: include safe operator upgrades in monitoring installs
6b9681e 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
fix(e2e): read metrics probes from completed pod logs
55ddd2d 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
fix(e2e): distinguish policy verdicts from probe diagnostics
7b794f2 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
test(e2e): drive event streams and retry monitoring downloads
6923113 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
fix(ci): integrate upstream selection and current Tailscale SDK
a072caf 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
chore: synchronize CRDs with workspace generator
d31f279 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
ci: use Go 1.27 compatible golangci-lint
3248d53 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
test: synchronize closed RCON response fixtures
dcc7161 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
test: clarify closed response deadline synchronization
59a0f44 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
ci: align lint tooling with Go 1.27 export data
9522fff 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
docs: classify lint tool dependency versions
9197f6c 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
fix(operator): use explicit waits for expected reconcile retries
71cd384 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
fix(agent): confine mod deletion to an open directory root
5042ab1 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
fix: align tailscale condition documentation and coverage
d1ded7f 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
fix: preserve known relay outages and reviewed prerequisites
cfe8c19 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
chore: merge upstream namespace controls into monitoring
18b4999 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
chore: reconcile monitoring branch with upstream
4137e81 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
test: exercise production relay observation and scoped wipe inventory
cb5e73b 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
chore: reconcile relay monitoring with upstream forwarding
8a59abb 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Merge upstream FRP dependency security fix
a80d3c2 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
chore: stack monitoring on current network isolation
42e047f 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
test(operator): seed finished backups before forget manager startup
4701667 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
merge: update network isolation base for observability
843d3b0 
Signed-off-by: Baylem <zelathis@gmail.com>
@coderabbitai

coderabbitai Bot commented Nov 5, 2026 •
edited
Loading

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 36 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 1bfcce0 and e564740.

Files selected for processing: 70

  • .github/workflows/ci.yaml
  • agent/cmd/main.go
  • agent/internal/caps/caps.go
  • agent/internal/rcon/nuclearoption.go
  • agent/internal/rcon/nuclearoption_deadline_test.go
  • agent/internal/rcon/nuclearoption_test.go
  • agent/internal/rcon/rcon.go
  • agent/internal/rcon/rcon_bounds_test.go
  • agent/internal/status/performance.go
  • agent/internal/status/performance_test.go
  • agent/internal/status/status.go
  • agent/specs.md
  • charts/gameplane/charts/gameplane-crds/crds/gameplane.local_backups.yaml
  • charts/gameplane/charts/gameplane-crds/crds/gameplane.local_backupschedules.yaml
  • charts/gameplane/charts/gameplane-crds/crds/gameplane.local_clusters.yaml
  • charts/gameplane/charts/gameplane-crds/crds/gameplane.local_gameservers.yaml
  • charts/gameplane/charts/gameplane-crds/crds/gameplane.local_gametemplates.yaml
  • charts/gameplane/charts/gameplane-crds/crds/gameplane.local_modules.yaml
  • charts/gameplane/charts/gameplane-crds/crds/gameplane.local_modulesources.yaml
  • charts/gameplane/charts/gameplane-crds/crds/gameplane.local_networkcaptures.yaml
  • charts/gameplane/charts/gameplane-crds/crds/gameplane.local_restores.yaml
  • charts/gameplane/crd-manifests/gameplane.local_backups.yaml
  • charts/gameplane/crd-manifests/gameplane.local_backupschedules.yaml
  • charts/gameplane/crd-manifests/gameplane.local_clusters.yaml
  • charts/gameplane/crd-manifests/gameplane.local_gameservers.yaml
  • charts/gameplane/crd-manifests/gameplane.local_gametemplates.yaml
  • charts/gameplane/crd-manifests/gameplane.local_modules.yaml
  • charts/gameplane/crd-manifests/gameplane.local_modulesources.yaml
  • charts/gameplane/crd-manifests/gameplane.local_networkcaptures.yaml
  • charts/gameplane/crd-manifests/gameplane.local_restores.yaml
  • charts/gameplane/dashboards/gameplane-workloads.json
  • charts/gameplane/templates/NOTES.txt
  • charts/gameplane/templates/grafana-dashboard.yaml
  • charts/gameplane/templates/monitoring-rules.yaml
  • charts/gameplane/templates/monitoring-validation.yaml
  • charts/gameplane/templates/networkpolicies.yaml
  • charts/gameplane/templates/servicemonitors.yaml
  • charts/gameplane/tests/monitoring-rules.sh
  • charts/gameplane/tests/monitoring-rules.test.yaml
  • charts/gameplane/values.yaml
  • charts/tests/check_gateway.py
  • charts/tests/check_standalone.py
  • deploy/kind/monitoring.sh
  • docs/examples/monitoring/gameplane-values.yaml
  • docs/examples/monitoring/prometheus.yaml
  • docs/monitoring.md
  • docs/tunnels.md
  • hack/ci_scope.py
  • hack/test_ci_scope.py
  • operator/api/v1alpha1/gametemplate_types.go
  • operator/api/v1alpha1/zz_generated.deepcopy.go
  • operator/config/crd/gameplane.local_gametemplates.yaml
  • operator/internal/controller/backup_forget_envtest_test.go
  • operator/internal/controller/gameserver_relay_health_test.go
  • operator/internal/controller/gameserver_tunnel.go
  • operator/internal/controller/gametemplate_performance_envtest_test.go
  • operator/specs.md
  • test/e2e/buckets.sh
  • test/e2e/ci_selection_e2e_test.go
  • test/e2e/monitoring_e2e_test.go
  • test/e2e/specs.md
  • test/e2e/suite-components.json
  • tunnel/main.go
  • tunnel/relay_health.go
  • tunnel/relay_metrics.go
  • tunnel/relay_metrics_test.go
  • tunnel/specs.md
  • tunnel/tailscale.go
  • tunnel/tailscale_test.go
  • web/src/types.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Baylem Baylem added area: agent in-pod sidecar area: chart Helm chart and deployment labels Nov 5, 2026 — with ChatGPT Codex Connector
@Baylem Baylem added area: e2e kind-based end-to-end suite area: modules game template OCI bundles area: operator controller-runtime operator and CRDs area: optional-components sentinel, capture-sidecar, mcp-server, tunnel, audit-syslog-bridge, telemetry-receiver area: shared netguard, gameaction, gameproto, svcutil and other shared packages area: specs spec-kit feature specs and design docs area: web React dashboard type: feature New capability or user-facing addition labels Nov 5, 2026 — with ChatGPT Codex Connector
@Baylem
Baylem added this pull request to stack #842 November 5, 2026 03:57
This was referenced Nov 5, 2026
Closed
Open
Open
@github-actions

github-actions Bot commented Nov 5, 2026 •
edited
Loading

Copy link
Copy Markdown

CI Report - e564740

run 42180083464 - attempt 2

0 failed - 33 passed - 0 skipped - unknown elapsed

Test selection: full (selection-rule change) - per-suite table

Coverage

Module Measured Gate Margin
web L95.93 S94.44 B89.33 F91.9 L92 S92 B82 F76 +2.44

Changed

Area Files +/-
agent/ 11 +691/-99
charts/ 30 +752/-22
tunnel/ 7 +266/-30
docs/ 4 +243/-0
test/ 5 +233/-3
operator/ 8 +164/-21
.github/ 1 +26/-10
deploy/ 1 +25/-0
hack/ 2 +12/-2
web/ 1 +5/-0

Slowest jobs

no job duration data available

e2e buckets run

bot-fast

@Baylem
Baylem marked this pull request as ready for review November 6, 2026 06:56
@Baylem
Baylem requested a review from ValgulNecron as a code owner November 6, 2026 06:56
Baylem added 2 commits November 6, 2026 02:14
@Baylem
Merge updated network isolation parent into observability
ea8de2b 
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Merge network isolation workspace checksum into observability
e564740 
Signed-off-by: Baylem <zelathis@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ValgulNecron ValgulNecron Awaiting requested review from ValgulNecron ValgulNecron is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

area: agent in-pod sidecar area: chart Helm chart and deployment area: e2e kind-based end-to-end suite area: modules game template OCI bundles area: operator controller-runtime operator and CRDs area: optional-components sentinel, capture-sidecar, mcp-server, tunnel, audit-syslog-bridge, telemetry-receiver area: shared netguard, gameaction, gameproto, svcutil and other shared packages area: specs spec-kit feature specs and design docs area: web React dashboard type: feature New capability or user-facing addition

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(observability): add game-performance and relay signals with a complete scrape setup

1 participant

@Baylem