feat(terraform): add shared Terraform plan/apply workflows [NOJIRA]

[tobias0106]

Why

Three Terraform repos run three divergent pipelines — 1,820 lines of CI between them:

repo	CI lines
terraform	468
terraform-core	554 + 139 (composite action)
terraform-db-access	659

The problem isn't the duplication, it's that correctness fixes have to be applied N times. terraform apply -auto-approve — re-plan on merge, the pattern behind the earlier Cloudflare outage — is live in 5 places across the three repos. monta-app/terraform#156 fixes 2 of them and defers the ClickHouse jobs by its own admission. Same story for the missing state lock, and for the "Argument list too long" plan-comment bug, which is fixed in exactly one of the three copies.

What this adds

Two reusable workflows (461 lines) that the three repos call with ~40 lines each.

terraform-discover.yml — emits the JSON matrix of stacks affected by the event. A stack is any directory containing backend.tf, which makes that directory simultaneously the unit of state, the CI job, the lock scope and the blast radius. Discovery walks the diff and maps each changed file to its nearest ancestor stack, so adding a stack needs no workflow edit — this replaces terraform-core's hand-written plan-X/apply-X job pairs and the hand-maintained paths-filter lists.

The practical goal: a ClickHouse change can init, plan and apply while Route53 state is drifted.

terraform-stack.yml — plans or applies exactly one stack.

• Apply downloads the artifact uploaded by the plan and runs terraform apply tfplan. It never re-plans, so what merges is what was reviewed (generalises monta-app/terraform#156 to every stack instead of two).
• Per-stack locking lives here, not in the caller: plans cancel per ref, applies queue and are never cancelled mid-apply.
• Terraform version comes from .terraform-version in the stack dir, falling back to the repo root — so stacks upgrade one at a time instead of a four-repo big bang.
• On a failed apply: re-plans in place, opens one issue per stack labelled terraform-failed assigned to whoever merged, comments the link on the originating PR, and banners later plans of that stack until the next successful apply closes the issue. Recovery is always a forward apply; there is deliberately no rollback path.
• OIDC role per account, with a static-key fallback so repos can adopt this before their IAM roles exist.

Caller example, required permissions and the merge_group guidance are in docs/workflow-guide.md.

Verification

• actionlint clean on both workflows and on the caller example extracted from the docs.
• Discovery logic tested against the real terraform-core tree — 8 cases: single stack, two stacks across two accounts, modules/** fan-out (11 stacks), markdown-only → [], deep nested file → owning stack, duplicate paths → deduped, force-all → 11, stack with no account mapping → hard fail.

Known issue to fix before terraform-db-access adopts this

Fan-out is too wide. A change to a path owned by no stack currently selects every stack in the repo. terraform-db-access keeps _shared-users-*.tf in the environment root, outside any stack — today its own workflow fans out to the DBs in that environment only. Under this PR it would fan out to all 56, which is harmless for plan and wrong for apply.

Fix is to scope fan-out to the changed file's nearest ancestor directory (~10 lines in the discover script). It also stops a modules/cloudflare/** change planning Grafana in the terraform repo. Not blocking terraform-core, blocking for db-access.

Adoption notes

• Each repo needs a one-time terraform-failed label.
• Lockfiles must be generated for the runner arch: terraform providers lock -platform=linux_arm64 -platform=darwin_arm64. terraform-db-access currently has 3 of 56.
• tf-vars-json is a migration bridge for terraform's seven TF_VAR_* secrets. The destination is stacks reading their own secrets via data "aws_secretsmanager_secret_version" so CI holds nothing but the AWS role.
• Callers should pin a tag rather than @main before terraform and terraform-db-access adopt it — a bad commit here would otherwise break apply in three repos at once, and the failure mode is a half-applied production stack.

Suggested first adopter is terraform-core: 11 stacks already have backend.tf, it deletes ~693 lines, and it's the lowest-stakes way to find out if this design is wrong.

🤖 Generated with Claude Code