From 18c7e5b96e0b2a45fe054aac7c5259e1c9970228 Mon Sep 17 00:00:00 2001 From: Harley Trung Date: Thu, 24 Sep 2026 15:16:18 +0700 Subject: [PATCH 1/2] docs: record verified upstream production upgrade and recovery --- AGENTS.md | 1 + docs/operations/coderpush-production.md | 41 +++++++++++---- docs/operations/upstream-upgrade-20260924.md | 52 ++++++++++++++++++++ 3 files changed, 85 insertions(+), 9 deletions(-) create mode 100644 docs/operations/upstream-upgrade-20260924.md diff --git a/AGENTS.md b/AGENTS.md index c87d399ee1e..50dcd6e57c0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,6 +4,7 @@ Multica is a task management platform where people and agents collaborate on iss ## Scope and Reading Order +- For CoderPush production settings, deployment, or fork delivery, read [the production runbook](docs/operations/coderpush-production.md) first. This fork is the working repository; production secrets stay on the server. Verify live state before acting on dated inventory. - Before changing `apps/mobile/`, also read [apps/mobile/AGENTS.md](apps/mobile/AGENTS.md), even if your tool does not load nested instructions automatically. Platform-specific sections below apply only to the named platform. - For naming, translations, or Chinese UI/docs copy, read [conventions.mdx](apps/docs/content/docs/developers/conventions.mdx) and [conventions.zh.mdx](apps/docs/content/docs/developers/conventions.zh.mdx). - Maintain shared rules here and mobile-specific rules in the mobile file. `CLAUDE.md` files only import them. Update instructions in the same change that alters the referenced workflow or boundary; do not add incident timelines, dependency version lists, or duplicate rules. diff --git a/docs/operations/coderpush-production.md b/docs/operations/coderpush-production.md index 59682206f7d..f6c61938b9f 100644 --- a/docs/operations/coderpush-production.md +++ b/docs/operations/coderpush-production.md @@ -8,7 +8,30 @@ non-secret intent and verification here. Never commit credentials or database du For the new team workspace, squad, runtime setup and remaining onboarding steps, see [CoderFactory team setup](coderfactory.md). -## Verified inventory — 18 September 2026 +## Current release — 24 September 2026 + +Production runs CoderPush main commit `355006fc62cc379eabeb61b2a9ee50e2707e2dd7`, +which merges upstream `e909e9c89d524cd54c1d5f4fa5963882093efdc9` and preserves +our handoff fixes. Backend and frontend are pinned to verified local image IDs; +schema is 547. Cutover completed at 08:13:28 UTC. Health, readiness, signed-in UI, +all 79 previously online runtime registrations and Lark websocket reconnection +were verified. Existing signup configuration and secrets were preserved. + +See [the upgrade receipt](upstream-upgrade-20260924.md) for image identities, +CI, migration/restore rehearsal, final backups, and rollback requirements. +The original local pilot containers and volumes were removed. The development +database is retained but stopped; it is only needed for fork development/tests. + +The base `/opt/multica/compose.yml` now pins the new images and disables pulling; +the matching `/opt/multica/coderpush-images.yml` overlay is retained. Base-only +Compose operations have the same effective configuration. GHCR packages remain +private. Use the manual **Export CoderPush release images** workflow when the +host lacks registry credentials; validate archive checksum, original registry +digests and imported OCI image identity as the receipt describes. Never copy +personal or runtime GitHub tokens to the host. `DO_NOT_TRACK=1` disables the new +upstream telemetry sender. + +## Historical inventory — 18 September 2026 Production is **AWS Lightsail Singapore**, not Hetzner. Hetzner was evaluated before the Lightsail deployment on 15 September. DNS and SSH verified the current @@ -93,18 +116,19 @@ after policy changes and verify its effective environment, not just the file. ## Deployment from this fork's main -**Prepared locally; not activated in production.** Production still uses upstream -v0.4.43. There is no automatic main-to-server rollout. The fork checkout inspected -was `7e4758ac1a94e9ff843696333364610bb8d4bbf7`: 78 commits after v0.4.43, -with 32 new migrations, 468–499. Migration 468 deletes obsolete link rows and drops -columns; an image-only rollback is not sufficient after a schema upgrade. +**Activated on 24 September 2026.** The first fork release and restore rehearsal +are complete; see the current release and receipt above. There is no automatic +main-to-server rollout. Each future release still needs matched image builds, +CI and migration review. The first upgrade applied 77 migrations from schema +467 through 547. Migration 468 deletes obsolete link rows and drops columns; +an image-only rollback to v0.4.43 is not sufficient. The fork workflow `.github/workflows/coderpush-images.yml` is manually dispatched on `main` and publishes Linux AMD64 backend/frontend images tagged with the full commit SHA. It uses the workflow's package token and needs no production SSH key. It does not publish a moving `latest` tag or deploy anything. Both build jobs must succeed for the same SHA; a partial publication is not a release. Existing CI must -also pass for that SHA. The workflow must first be committed and merged to `main`. +also pass for that SHA. The build and export workflows are committed and available on `main`. 1. Review the intended `main` commit, changes since the running version, migration compatibility, and existing CI results. Build both images using **CoderPush main @@ -204,8 +228,7 @@ a claim that each integration was retested on 18 September. - `01a0adbe-0b3c-7cf0-bdbe-371a8ae36194` — **Investigate multica setup failure**: Singapore model authentication diagnosis. -Open items: first fork image build and deployment rehearsal; backup restore testing -and retention/off-host verification; reconcile newer CoderInternals/NanoHome runtime +Open items: backup retention/off-host verification; reconcile newer CoderInternals/NanoHome runtime configuration before changing shared services. Keep live runtime edits and repository delivery status separate: a host change does not mean a PR was committed or merged. diff --git a/docs/operations/upstream-upgrade-20260924.md b/docs/operations/upstream-upgrade-20260924.md new file mode 100644 index 00000000000..04653fa3253 --- /dev/null +++ b/docs/operations/upstream-upgrade-20260924.md @@ -0,0 +1,52 @@ +# Upstream upgrade — 24 September 2026 + +## Release identity + +- Upstream: `e909e9c89d524cd54c1d5f4fa5963882093efdc9` (latest main fetched at preparation). +- Fork release: `355006fc62cc379eabeb61b2a9ee50e2707e2dd7`, [PR #9](https://github.com/CoderPush/multica/pull/9). +- Preserves the duplicate-assignee handoff guard and acceptance-based parent completion. Merge resolution also preserves upstream cancellation/dependency warnings. +- Previous backend: `013385718f731be728fd7b2c0d09828edb868009`, v0.4.43 handoff backport; previous frontend: v0.4.43. +- Schema transition: 467 to 547, 77 new migration files. Image-only rollback is unsafe. + +## Verification + +- [PR CI](https://github.com/CoderPush/multica/actions/runs/35971843482) and [release-commit CI](https://github.com/CoderPush/multica/actions/runs/35972509064) passed. +- [Backend and frontend builds](https://github.com/CoderPush/multica/actions/runs/35972515607) passed for the same release SHA. +- Focused database-backed handoff and child-completion race tests passed, with verbose output proving execution. Full local handler suite passed. The full local Go invocation hit Dsh-probe timeouts under load; its isolated race rerun passed. This is not a claim that the complete local invocation passed. +- Focused integration source review and Amazon Q review found no concrete blockers; this is not a new exhaustive audit of all upstream changes. +- A fresh production backup was restored on Singapore into an internal Docker network. All migrations passed; the copy retained 6 workspaces, 106 issues and 46 agents, with no invalid indexes. +- Rehearsal schedules were disabled, no production integration keys or workers were supplied, and network egress was blocked. Both final images then passed backend health/readiness and frontend HTTP smoke checks. +- macOS source archives must use `COPYFILE_DISABLE=1 tar --no-xattrs` for migrations. Metadata sidecars named `._*.up.sql` are otherwise mistaken for migrations. The first rehearsal caught this before applying the upgrade; the clean archive passed. + +## Private image transfer + +GHCR created private packages. Singapore has no registry credentials. [PR #10](https://github.com/CoderPush/multica/pull/10) adds a manual export workflow using only the Actions token's package-read permission. [Export run](https://github.com/CoderPush/multica/actions/runs/35973305156) passed. The artifact expires after three days; retain the verified images and recovery archives on the host. + +Archive SHA-256 was checked before and after transfer. Registry digests matched the build logs. The archive's config hashes matched the runner's recorded image IDs; the imported OCI manifest hashes and root filesystem layer hashes were then verified on Singapore, along with AMD64/Linux and the full revision label. + +Docker 29's containerd image store reports the imported OCI manifest hash as its image ID, whereas the export runner recorded config hashes. These differ without a payload change. Do not compare these two representations directly or assume `docker save/load` preserves the registry index digest. + +| Image | Registry index digest | Imported manifest / local image ID | +| --- | --- | --- | +| Backend | `sha256:64b1b5376e18f3f175d8ab85d707f2564c414082741a6a16d1e4cf4bcdda2fd7` | `sha256:42f0f2f6cf0b716b5cc99c90de6800f0b9a37e012904e757eb5193d1efe5165f` | +| Frontend | `sha256:f856eee7a2f0f1cfb5a11ec5ea09b85f66417ae6545a8b338c0d73a543cb3563` | `sha256:90e09df4734b63943d9ec3e24c7a20125420af400f808f016267f326ee705236` | + +Deployment pins the imported image IDs with `pull_policy: never`. No personal or agent token was copied to the host and package visibility was not changed. `DO_NOT_TRACK=1` disables the new upstream telemetry sender. + +## Cutover and recovery + +Cutover completed at **08:13:28 UTC / 15:13:28 Vietnam** after the global unfinished-task count reached zero. The guard refused two earlier attempts while a task was still running; those attempts left services unchanged. + +- Final database backup: `/opt/multica/backups/database-20260924T081256Z.dump`; archive listing validated. Restore rehearsal was completed against the preceding fresh backup before cutover. +- Upload archive: `/opt/multica/releases/upstream-20260924/uploads.before.tar.gz` (about 247 MiB). +- Public health reports the full release SHA; readiness reports database and migrations OK; schema is 547 and invalid-index count is zero. +- Exact pre/post-cutover counts match: 6 workspaces, 106 issues, 46 agents. The protected `.env` remained byte-identical, preserving JWT/integration keys and signup policy. +- Worker, Caddy and backup timer are active. All 79 recently online runtime registrations reconnected with the same provider counts. Signed-in runtime UI shows Singapore and Hogan Web Worker online; the pre-existing offline machine remains offline. +- Lark websocket connected. No backend error lines were observed in the initial post-cutover window; neither app container restarted. +- Existing browser session loaded the issue board and runtimes successfully. Before/after screenshots remain in the local, gitignored `.screenshots/` directory. No fresh email login, new model inference or outbound Lark message was initiated for acceptance. + +`/opt/multica/compose.yml` is now pinned to the new local image IDs, with pull disabled. Its effective configuration was compared byte-for-byte as parsed JSON with the tested two-file configuration before atomic replacement. Thus an ordinary base-only Compose operation cannot accidentally revive the incompatible v0.4.43 images. The matching `coderpush-images.yml` overlay is also retained. Future registry releases may replace the IDs with verified registry digest references and restore the appropriate pull policy. + +The host release directory is `/opt/multica/releases/upstream-20260924/`. Keep its protected environment, Compose, Caddy, worker-service, uploads and database backup references private. Never commit those files. + +For a rollback, first stop admission and drain work, stop the worker/backend/frontend, preserve any post-upgrade writes and current configuration, then restore the pre-upgrade database together with its matching old images and uploads/configuration as needed. Retain the new database before a restore; schema downgrades cannot recover rows deleted by migration 468. Reconcile writes made since cutover before replacing production data. From bc2a5b9f74c092f8fb39e6bc5997d347ceedb5a9 Mon Sep 17 00:00:00 2001 From: Harley Trung Date: Thu, 24 Sep 2026 15:17:44 +0700 Subject: [PATCH 2/2] docs: remove reference to unshipped setup note --- docs/operations/coderpush-production.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/operations/coderpush-production.md b/docs/operations/coderpush-production.md index f6c61938b9f..7aac287665c 100644 --- a/docs/operations/coderpush-production.md +++ b/docs/operations/coderpush-production.md @@ -5,9 +5,6 @@ and production operations. Open enhancement PRs against this fork's `main`, not upstream. Production environment changes are performed on the host; record their non-secret intent and verification here. Never commit credentials or database dumps. -For the new team workspace, squad, runtime setup and remaining onboarding steps, -see [CoderFactory team setup](coderfactory.md). - ## Current release — 24 September 2026 Production runs CoderPush main commit `355006fc62cc379eabeb61b2a9ee50e2707e2dd7`,