Releases: sync new R2 releases from the API every 30 minutes by adamshiervani · Pull Request #81 · jetkvm/cloud-api

adamshiervani · 2026-09-19T00:45:48Z

A firmware upload sat in R2 until an operator ran npm run sync-releases:production by hand. The API process now runs the same sync on start and every 30 minutes, registering each new stable version at the default 10% rollout. No new process, no cron container.

What moved

 scripts/
-└── sync-releases.ts     # listing + artifact walk + DB insert + GPG + prompts
+└── sync-releases.ts     # GPG check + confirmation prompt (ReleaseDecider)
 src/
+├── release-sync.ts      # listing + artifact walk + DB insert + scheduler
+├── s3.ts                # one R2 client, s3ObjectExists, versionHasSkuSupport
 ├── releases.ts          # uses src/s3.ts instead of its own client + helpers
 └── index.ts             # scheduleReleaseSync({ prisma, s3Client }, ...)

The build compiles src only, so the shared core has to live there. The script imports it and passes the interactive parts in as a callback.

How a tick runs

scheduleReleaseSync                      (src/index.ts, on start + every 30 min)
  run
    if previous run still going → warn, skip tick
    syncReleases(decide = createAtDefaultRollout)
      for each OTA prefix (app, system, mini)
        listStableVersions (R2)  ‖  listSyncedVersions (DB)   ← one call each per prefix
        for each version not in DB
          collectReleaseArtifacts (R2, SKUs probed concurrently)
          newestUploadTime (R2)   → changed < 10 min ago? "uploading", retry next tick
          decide(...)             → { create, rolloutPercentage: 10 }
          release.create
            P2002 → "created concurrently elsewhere", counted as already-synced
    catch → log, schedule continues

Script vs scheduler

sequenceDiagram
    participant R2
    participant API as API (scheduler)
    participant Op as Operator (script)
    participant DB
    R2-->>API: new stable version
    API->>DB: findMany versions for prefix
    API->>DB: create release @10%
    Op->>DB: findMany versions for prefix
    Note over Op,DB: already synced → no prompt

The script still works and still prompts, but only for versions the scheduler has not registered yet. After this lands, answering N or a in the script delays a release by at most one tick; a custom rollout percentage only sticks if the script runs before the next tick. Signature verification stays script-only and never gated anything in the DB before either. Adjusting rollout on an existing row is unchanged: it is not something sync does.

Behaviour notes

Existing rows are never touched. Sync only inserts brand-new (version, type) pairs, as before.
Scheduled runs get a settle window. After scanning artifacts, sync lists the version folder; if the newest object changed in the last 10 minutes the snapshot is discarded and the version retried next tick. Listing after the scan means any upload active during the scan is seen, so a tick cannot freeze a half-uploaded release with a partial SKU set or a stale hash. The operator script does not set uploadSettleMs: it shows the artifact list before the write and has no next run.
R2 config lives in src/s3.ts. Client, bucket name and CDN origin are exported from one module; the request handlers, the scheduler and the script all import them.
Multiple API instances each run the schedule. The unique constraint on (version, type) settles the race; the loser logs and moves on.
Known versions load once per prefix instead of one findUnique per version, so a tick over an already-synced bucket costs one R2 list and one DB query per prefix.
Dev with --watch re-runs the startup sync on every save against whatever bucket .env.development points at.

Known limits

The settle window is a heuristic. A version that was quiet for the full 10 minutes and then resumes uploading, or is re-uploaded after registration, is registered from the pre-resume snapshot and never updated. Closing that needs either a completion marker written by the firmware release tooling, or letting sync append artifacts to an existing row. Both are follow-ups, not part of this PR.

Tests

A version with an object modified inside the settle window is deferred; one outside it is created in the same run.
syncReleases honours a decider's custom rollout, skip and abort answers.
A row created by another instance between the version query and the insert counts as already synced (real P2002 from the constraint).
Scheduler: runs at start, survives a failed run and creates on the next tick.
Scheduler: skips ticks while a run is in progress and resumes after (fails if the guard is removed).

Note

^{Cursor Bugbot is generating a summary for commit 092b16d. Configure here.}

The release sync only ran when an operator invoked scripts/sync-releases.ts by hand, so a firmware upload sat in R2 until someone remembered to run it. The API process now runs the same sync on start and every 30 minutes, registering each new stable version at the default 10% rollout. The non-interactive core (bucket listing, artifact collection, DB insert) moves from the script into src/release-sync.ts so the API can import it; the script keeps the GPG check and the confirmation prompt and passes them in as the release decider. Both entry points share one R2 client via src/s3.ts. Scheduled runs skip a tick while the previous run is still going, log and survive a failed run, and treat a unique-constraint hit on insert as "already synced" so several API instances can race without an error. The existence check now precedes the R2 artifact walk, so a tick over an already-synced bucket costs one list call per prefix plus one DB lookup per version.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit efe7825. Configure here.}

A scheduled tick can land while a version is still being uploaded and register it with a partial SKU set or a hash that changes afterwards. Sync never rewrites a row, so that snapshot would be permanent. Before collecting artifacts, list every object under the version folder and skip the version when the newest object changed in the last 10 minutes. The window is shorter than the schedule interval, so a real upload costs at most one extra tick.

adamshiervani · 2026-09-19T00:52:48Z

@codex review

chatgpt-codex-connector · 2026-09-19T00:53:02Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-19T11:15:39.295703Z	`79a3c74`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-09-19T00:55:18Z

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 4e885fa636

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… concurrently * src/s3.ts now exports bucketName and baseUrl next to the client, so the request handlers, the scheduler and the script read R2 config from one place instead of three. * The upload settle window moves into SyncConfig.uploadSettleMs. The scheduler sets it; the one-shot operator script leaves it unset because an operator sees the artifact list before confirming and has no next run. * newestUploadTime uses the SDK's paginateListObjectsV2 instead of a hand-rolled continuation loop. * collectReleaseArtifacts probes SKUs concurrently and folds the results in SKU order, so artifact order and the primary artifact are unchanged. * Tests drop empty per-prefix listing stubs that the file-level default already covers.

adamshiervani · 2026-09-19T08:54:19Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f7fc64a504

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-19T08:57:53Z

+  if (config.uploadSettleMs) {
+    const newest = await newestUploadTime(clients.s3Client, config.bucketName, type, version);
+    if (newest && Date.now() - newest.getTime() < config.uploadSettleMs) {


Gate creation on upload completion, not a preflight timestamp

When an upload begins or resumes after newestUploadTime returns, this check can pass and the subsequent artifact scan and database insert can capture only the objects available at that moment. Because later runs skip existing (version, type) rows, any subsequently uploaded SKUs or corrected hashes are never registered. This occurs whenever a scheduled tick overlaps an active upload after the timestamp preflight, so creation should instead depend on an explicit completion marker or otherwise verify completion atomically with artifact collection.

Useful? React with 👍 / 👎.

Fixed in 79a3c74. The settle listing now runs after the artifact scan instead of before it. An upload active at any point during the scan leaves an object newer than the window, so that snapshot is discarded and the version is retried on the next tick. An upload that starts only after a version has been quiet for the full window is a re-upload to an already-registered version, which is outside the contract for the manual script as well. A completion marker written by the uploader would need changes in the firmware release tooling, so it is out of scope here. The settle test now also asserts the listing happens after the scan.

Listing the version folder before the scan left a gap: an upload that started between the listing and the scan passed the check and was captured partially. Listing after the scan closes it, because any upload active during the scan leaves an object newer than the window and the snapshot is discarded instead of registered.

adamshiervani · 2026-09-19T08:59:56Z

@codex review

chatgpt-codex-connector · 2026-09-19T09:00:48Z

Codex Review: Something went wrong. Try again later by commenting “@codex review”.

Provided git ref 79a3c7442d73f88fa73476684d492c280132e309 does not exist

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

adamshiervani · 2026-09-19T09:01:37Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 79a3c7442d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-19T09:05:17Z

+    const newest = await newestUploadTime(clients.s3Client, config.bucketName, type, version);
+    if (newest && Date.now() - newest.getTime() < config.uploadSettleMs) {


Make upload completion atomic with release creation

If an older, partially uploaded version resumes after its objects have been returned by this listing but before release.create, newest can remain outside the settle window while artifacts contains the pre-resume snapshot; the row is then inserted, and subsequent runs permanently skip that version. Fresh evidence after the earlier fix is that the database insert still occurs after this non-atomic newestUploadTime check. Gate creation on an explicit upload-completion marker or otherwise verify completion atomically with the captured artifact set.

Useful? React with 👍 / 👎.

Not changing this one. The remaining window is the gap between the post-scan listing and the insert, and it only matters for a version that was quiet for the full settle window and then resumes uploading in that instant. A resume one second after the insert has the same effect, so no ordering of timestamp checks closes it. The two real fixes are outside this PR: a completion marker written by the firmware release tooling (rv1106-system and the kvm/kvm-mini release scripts), or a policy change that lets sync append artifacts to an existing row, which the never-rewrite rule currently forbids. Recorded under "Known limits" in the PR description so the human reviewer can decide whether to follow up with either.

adamshiervani · 2026-09-19T09:25:32Z

@codex review using model astra-6 with high reasoning effort

chatgpt-codex-connector · 2026-09-19T09:28:07Z

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 79a3c7442d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot

Copilot review overview

🟡 Changes recommended

Version discovery does not paginate R2 results, permanently omitting releases beyond the first page.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1

Open (1)

Handle paginated S3 listings to process all release versions · New

What changed in this PR

Adds automatic R2 release synchronization at API startup and every 30 minutes.

Changes:

Extracts reusable release synchronization and shared R2 utilities.
Adds upload settling, concurrent-run protection, and race handling.
Retains interactive operator workflows and expands test coverage.

File	Description
`src/release-sync.ts`	Implements synchronization and scheduling.
`src/s3.ts`	Centralizes R2 configuration and helpers.
`src/releases.ts`	Uses shared R2 utilities.
`src/index.ts`	Starts the scheduler.
`scripts/sync-releases.ts`	Retains interactive release decisions.
`test/sync-releases.test.ts`	Tests synchronization and scheduling behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

adamshiervani · 2026-09-19T11:19:52Z

+  const response = await s3Client.send(
+    new ListObjectsV2Command({
+      Bucket: bucketName,
+      Prefix: `${type}/`,
+      Delimiter: "/",
+    }),
+  );
+
+  return (response.CommonPrefixes ?? [])
+    .map(cp => cp.Prefix?.split("/")[1])
+    .filter((version): version is string => Boolean(version))
+    .filter(
+      version => Boolean(semver.valid(version)) && semver.prerelease(version) === null,
+    )
+    .sort(semver.compare);


Fixed in 092b16d: listStableVersions now walks every page with paginateListObjectsV2, and the sync tests cover a truncated listing (registers versions from every page of a truncated listing).

ListObjectsV2 returns at most 1,000 common prefixes per page. Once a release prefix grows past that, versions on later pages were never seen on any tick. listStableVersions now walks every page with the SDK paginator, as newestUploadTime already does, and the mock in the sync tests can serve a truncated listing.

* fix(releases): serve staged releases before any release reaches 100% (#78) A prefix with no release at 100% made the default lookup throw a 500 before eligibility was checked, so no device got the staged build. That is every device for a new prefix until its first release is fully rolled out, and every JetKVM device the day no app or system row sits at 100%. The default lookup now returns null when nothing is at 100%. A device inside the rollout bucket gets the staged release as before; a device outside it gets a 404 saying no release is rolled out yet for its SKU, instead of a 500. Found by Bugbot on the release PR (#77). * Releases: sync new R2 releases from the API every 30 minutes (#81) * feat(releases): sync new R2 releases from the API every 30 minutes The release sync only ran when an operator invoked scripts/sync-releases.ts by hand, so a firmware upload sat in R2 until someone remembered to run it. The API process now runs the same sync on start and every 30 minutes, registering each new stable version at the default 10% rollout. The non-interactive core (bucket listing, artifact collection, DB insert) moves from the script into src/release-sync.ts so the API can import it; the script keeps the GPG check and the confirmation prompt and passes them in as the release decider. Both entry points share one R2 client via src/s3.ts. Scheduled runs skip a tick while the previous run is still going, log and survive a failed run, and treat a unique-constraint hit on insert as "already synced" so several API instances can race without an error. The existence check now precedes the R2 artifact walk, so a tick over an already-synced bucket costs one list call per prefix plus one DB lookup per version. * fix(release-sync): defer versions whose upload is still settling A scheduled tick can land while a version is still being uploaded and register it with a partial SKU set or a hash that changes afterwards. Sync never rewrites a row, so that snapshot would be permanent. Before collecting artifacts, list every object under the version folder and skip the version when the newest object changed in the last 10 minutes. The window is shorter than the schedule interval, so a real upload costs at most one extra tick. * refactor(release-sync): share R2 config, paginate via SDK, probe SKUs concurrently * src/s3.ts now exports bucketName and baseUrl next to the client, so the request handlers, the scheduler and the script read R2 config from one place instead of three. * The upload settle window moves into SyncConfig.uploadSettleMs. The scheduler sets it; the one-shot operator script leaves it unset because an operator sees the artifact list before confirming and has no next run. * newestUploadTime uses the SDK's paginateListObjectsV2 instead of a hand-rolled continuation loop. * collectReleaseArtifacts probes SKUs concurrently and folds the results in SKU order, so artifact order and the primary artifact are unchanged. * Tests drop empty per-prefix listing stubs that the file-level default already covers. * fix(release-sync): take the settle listing after the artifact scan Listing the version folder before the scan left a gap: an upload that started between the listing and the scan passed the check and was captured partially. Listing after the scan closes it, because any upload active during the scan leaves an object newer than the window and the snapshot is discarded instead of registered. * fix(release-sync): paginate the version listing ListObjectsV2 returns at most 1,000 common prefixes per page. Once a release prefix grows past that, versions on later pages were never seen on any tick. listStableVersions now walks every page with the SDK paginator, as newestUploadTime already does, and the mock in the sync tests can serve a truncated listing. * Release sync: a unique violation is only a race when the row exists (#82) * fix(release-sync): only treat a unique violation as a race when the row exists Sync caught every P2002 from the release insert as "created concurrently elsewhere". On staging the id sequences were behind the rows after a data import, so each insert failed on the primary key, was logged as a race, and left nothing in the table. After a unique violation, createRelease now looks the (version, type) row up. Present means another instance registered it first; absent means the insert really failed, and the error is rethrown with the type and version in its message so the scheduled run log names the release. * fix(release-sync): name the release in every per-version failure Wrapping only the insert error left the row lookup, and the S3 scan before it, free to escape without the type and version. syncReleases now wraps whatever createRelease throws for a version. * feat(releases): POST /releases/sync for the upload script (#83) The scheduled tick found a new version at most 30 minutes plus the settle window after upload. The upload script knows when its last object is written, so it can trigger the sync itself. One runner now owns the in-progress flag and serves both the timer and the endpoint. The endpoint is guarded by RELEASE_SYNC_TOKEN as a bearer token, compared in constant time, and is not registered without it. It runs with the settle window off and answers with the per-outcome counts, or 409 while a run is in progress. * fix(auth): accept the bearer scheme in any case (#85) Scheme names are case-insensitive (RFC 9110). The token is still compared exactly. * fix(releases): scope the sync endpoint's settle bypass to one version (#84) The endpoint turned the settle window off for every version in the bucket. A call made while a different upload was still running registered that upload half done, and sync never revisits a row. The body now names the version the caller finished, and only that version skips the window. Without a body the call is a normal tick.

cursor Bot reviewed Sep 19, 2026

View reviewed changes

Comment thread src/release-sync.ts

chatgpt-codex-connector Bot reviewed Sep 19, 2026

View reviewed changes

adamshiervani marked this pull request as ready for review September 19, 2026 11:13

adamshiervani requested a balanced review from Copilot September 19, 2026 11:13

Copilot started reviewing on behalf of adamshiervani September 19, 2026 11:13 View session

Copilot AI reviewed Sep 19, 2026

View reviewed changes

adamshiervani merged commit a01f7a4 into dev Sep 19, 2026
3 checks passed

adamshiervani deleted the feat/auto-release-sync branch September 19, 2026 11:24

		const newest = await newestUploadTime(clients.s3Client, config.bucketName, type, version);
		if (newest && Date.now() - newest.getTime() < config.uploadSettleMs) {

Conversation

adamshiervani commented Sep 19, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What moved

How a tick runs

Script vs scheduler

Behaviour notes

Known limits

Tests

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adamshiervani commented Sep 19, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Sep 19, 2026

Uh oh!

adamshiervani commented Sep 19, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 19, 2026

Choose a reason for hiding this comment

Uh oh!

adamshiervani Sep 19, 2026

Choose a reason for hiding this comment

Uh oh!

adamshiervani commented Sep 19, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 19, 2026

Uh oh!

adamshiervani commented Sep 19, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 19, 2026

Choose a reason for hiding this comment

Uh oh!

adamshiervani Sep 19, 2026

Choose a reason for hiding this comment

Uh oh!

adamshiervani commented Sep 19, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

adamshiervani Sep 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adamshiervani commented Sep 19, 2026 •

edited by cursor Bot

Loading

chatgpt-codex-connector Bot commented Sep 19, 2026 •

edited

Loading