diff --git a/docs/runbooks/skill-harness-proof.md b/docs/runbooks/skill-harness-proof.md index 34461295..5d3763e1 100644 --- a/docs/runbooks/skill-harness-proof.md +++ b/docs/runbooks/skill-harness-proof.md @@ -51,18 +51,19 @@ They do not prove every example works. Inspect source attempts and review findin The container image includes the pinned Agent bootstrap. Each session receives its own copy. Every job gets a fresh container. The Worker destroys it after saving the result. -Durable Object alarms check progress and enforce the fifteen minute deadline. +Durable Object alarms check progress and enforce the forty-five minute deadline. +A GLM 5.3 step takes about 26 seconds. The runner tells the Agent to write `SKILL.md` by turn 20, so a deadline still leaves a candidate. ## Limits and failures The proof allows one active job and 96 model calls. Generation targets 35 turns. Independent review targets 20 turns. These turn targets guide the model. The gateway enforces the total call limit. -OpenCode Go calls allow 8,192 output tokens. The gateway caps `max_tokens` and `max_completion_tokens`. +OpenCode Go calls allow 32,768 output tokens, because GLM 5.3 reasoning counts as output. The gateway caps `max_tokens` and `max_completion_tokens`. Anthropic calls allow 8,192 output tokens, including a 2,048-token thinking budget. Google calls allow 4,096 output tokens. It limits each request to 512 KiB and each result to 1 MiB. -The runner uses `skilld-harness@3.3.0` and returns its usage reports. +The runner uses `skilld-harness@3.6.4` and returns its usage reports. That version returns failed output checks to the Agent for up to two repair turns. Reports sum generation, repair, and review attempts separately. If a token count is missing, the total stays unknown. Failed runs retain bounded tool traces and any earlier valid candidate. diff --git a/docs/runbooks/skillgen.md b/docs/runbooks/skillgen.md index 2c103cf2..e4788a08 100644 --- a/docs/runbooks/skillgen.md +++ b/docs/runbooks/skillgen.md @@ -62,7 +62,7 @@ So a maintenance release, such as `1.3.13` while `main` is at `2.0.0-beta.14`, n If several packages match one tag, the job splits into one job per package. Each split job opens its own pull request on a branch named after the package directory. If the package publication follows the tag, the job waits for it. -The whole job has a thirty minute deadline. +The whole job has a sixty minute deadline, counted from when it leaves the queue. The job matches npm's `gitHead`, or npm's HTTPS provenance record, to that commit. The provenance match checks the package digest, repository, tag, and source commit. @@ -102,6 +102,6 @@ Different tags resolving to the same commit reuse the recorded target. Generation failures persist as job outcomes. They produce no pull request. The App has one global queue and one active container. -Jobs run one at a time, and the thirty minute deadline includes time in the queue. +Jobs run one at a time. Time a job waits in the queue does not count toward its deadline. If 64 jobs are queued, the webhook answers 503. GitHub does not redeliver it. Organization keys and optional shared capacity require separate onboarding and budgets. diff --git a/layers/marketing/app/pages/skillgen.vue b/layers/marketing/app/pages/skillgen.vue index de7530bf..9d27cb58 100644 --- a/layers/marketing/app/pages/skillgen.vue +++ b/layers/marketing/app/pages/skillgen.vue @@ -129,7 +129,7 @@ const codeClass = 'font-mono text-xs text-default' Review the pull request

- Skillgen starts with your latest tag when you install the App. If you turned the repository on after the install, it starts at your next tag. Each new tag starts another run. If npm has not published the version yet, the run waits. Each run must finish within 30 minutes. + Skillgen starts with your latest tag when you install the App. If you turned the repository on after the install, it starts at your next tag. Each new tag starts another run. If npm has not published the version yet, the run waits. Each run must finish within an hour of starting.

A model drafts the update from the tagged source with the skilld Harness. A separate review must accept it before anything reaches your repository. If the Skill needs no change, Skillgen opens nothing. diff --git a/scripts/skill-harness-proof.mjs b/scripts/skill-harness-proof.mjs index f97a63da..2b070c74 100644 --- a/scripts/skill-harness-proof.mjs +++ b/scripts/skill-harness-proof.mjs @@ -21,7 +21,7 @@ async function main() { throw new Error(`Proof start returned ${started.status}: ${await started.text()}`) const { id } = await started.json() console.log(`Proof ${id} started.`) - const deadline = Date.now() + 17 * 60 * 1000 + const deadline = Date.now() + 47 * 60 * 1000 while (Date.now() < deadline) { const response = await fetch(new URL(`/proofs/${id}`, endpoint), { headers }) if (!response.ok) diff --git a/workers/skill-harness/runner/package-lock.json b/workers/skill-harness/runner/package-lock.json index df7070ab..7fa2e5fd 100644 --- a/workers/skill-harness/runner/package-lock.json +++ b/workers/skill-harness/runner/package-lock.json @@ -8,7 +8,7 @@ "dependencies": { "@ai-sdk/harness": "1.0.138", "@ai-sdk/harness-opencode": "1.0.140", - "skilld-harness": "3.3.0", + "skilld-harness": "3.6.4", "ws": "8.21.0", "zod": "4.5.4" } @@ -156,9 +156,9 @@ "license": "(AFL-2.1 OR BSD-3-Clause)" }, "node_modules/skilld-harness": { - "version": "3.3.0", - "resolved": "https://registry.npmjs.org/skilld-harness/-/skilld-harness-3.3.0.tgz", - "integrity": "sha512-pwN8EJxrP9JZKVUaLwe2I84yTxjXiEKeWXyQtaNOG8dWOJcnSZPWFxCJsT4b6MDUWsEhso+hVE8qXU9M6/Kfhg==", + "version": "3.6.4", + "resolved": "https://registry.npmjs.org/skilld-harness/-/skilld-harness-3.6.4.tgz", + "integrity": "sha512-HrtQ8zp998epmTAXz/RXUHtlzMKk4hUSgY3fRKSVtdyNeZy0yCOglkIJcTxuJL99ly3yQY1BiscK9bnaWqtmtA==", "license": "MIT", "dependencies": { "@ai-sdk/harness": "^1.0.128", diff --git a/workers/skill-harness/runner/package.json b/workers/skill-harness/runner/package.json index 878ce970..eec3f0fa 100644 --- a/workers/skill-harness/runner/package.json +++ b/workers/skill-harness/runner/package.json @@ -5,7 +5,7 @@ "dependencies": { "@ai-sdk/harness": "1.0.138", "@ai-sdk/harness-opencode": "1.0.140", - "skilld-harness": "3.3.0", + "skilld-harness": "3.6.4", "ws": "8.21.0", "zod": "4.5.4" } diff --git a/workers/skill-harness/runner/run.mjs b/workers/skill-harness/runner/run.mjs index 685994bf..593301bb 100644 --- a/workers/skill-harness/runner/run.mjs +++ b/workers/skill-harness/runner/run.mjs @@ -9,7 +9,8 @@ import { runGeneration } from './orchestrate.ts' import { createRunTrace } from './trace.ts' const startedAt = Date.now() -const signal = AbortSignal.timeout(14 * 60 * 1000) +// One minute under JOB_TIMEOUT_MS, so the runner writes its result before the Worker gives up. +const signal = AbortSignal.timeout(44 * 60 * 1000) const trace = createRunTrace() async function execute() { @@ -54,7 +55,7 @@ async function execute() { 'Use one final command to check frontmatter, paths, and examples. Do not repeat counts, greps, or confirmation reads.', 'After writing and checking the output, finish immediately. A separate independent review follows generation.', stage === 'generation' - ? 'Aim to finish within 35 model turns. Reserve the remaining run budget for independent review and any repair.' + ? 'Write SKILL.md to the output directory by model turn 20, then verify and edit it there. A deadline then still leaves a candidate. Aim to finish within 35 model turns. Reserve the remaining run budget for independent review and any repair.' : 'Review source claims in batches. Aim to finish within 20 model turns. Return concrete findings when checks finish.', ...(findings.length ? ['Correct these review findings against the exact package source. Preserve other supported guidance.', JSON.stringify(findings)] : []), ].join('\n\n')) diff --git a/workers/skill-harness/src/contracts.ts b/workers/skill-harness/src/contracts.ts index ed1329be..583dc645 100644 --- a/workers/skill-harness/src/contracts.ts +++ b/workers/skill-harness/src/contracts.ts @@ -3,7 +3,8 @@ import { z } from 'zod' export const MAX_REQUEST_BYTES = 512 * 1024 export const MAX_RESULT_BYTES = 1024 * 1024 export const MAX_MODEL_CALLS = 96 -export const JOB_TIMEOUT_MS = 15 * 60 * 1000 +/** GLM 5.3 takes about 26 seconds per step, so generation, review, and one repair need most of this. */ +export const JOB_TIMEOUT_MS = 45 * 60 * 1000 const skillPath = z.string().max(200).refine(path => path === 'SKILL.md' diff --git a/workers/skill-harness/src/gateway.ts b/workers/skill-harness/src/gateway.ts index aca4e623..e29860af 100644 --- a/workers/skill-harness/src/gateway.ts +++ b/workers/skill-harness/src/gateway.ts @@ -20,7 +20,8 @@ function openCodeRoutingHeaders(request: Request): Record { } return headers } -const OPENCODE_GO_MAX_OUTPUT_TOKENS = 8192 +// GLM 5.3 reasoning counts as output. A proof step that hit 8,192 ended its turn with finish reason length. +const OPENCODE_GO_MAX_OUTPUT_TOKENS = 32768 // Web search, hosted plugins, live search, and remote MCP run on the provider at a separate cost. const OPENAI_COMPATIBLE_HOSTED_FIELDS = ['web_search_options', 'plugins', 'search_parameters', 'mcp_servers'] diff --git a/workers/skill-harness/src/github-jobs.ts b/workers/skill-harness/src/github-jobs.ts index 9bfe158a..6b59bea7 100644 --- a/workers/skill-harness/src/github-jobs.ts +++ b/workers/skill-harness/src/github-jobs.ts @@ -2,10 +2,12 @@ import type { PreparedTag } from './github-client' import type { TagRequest } from './github-events' import { DurableObject } from 'cloudflare:workers' import { githubInstallationClient, prepareTag, publishSkill } from './github-client' +import { jobExpired } from './job-deadline' import { followsRetryChain } from './job-retry' type Outcome = Awaited> | { _tag: 'Skipped', reason: string } | { _tag: 'Failed', code: string, detail: string } -interface Job { id: string, request: TagRequest, receivedAt: number } +/** `startedAt` is when the job reached the head of the queue. */ +interface Job { id: string, request: TagRequest, receivedAt: number, startedAt?: number } type State = Job & ( { _tag: 'Queued' } | { _tag: 'Ready' | 'Generating' | 'Publishing', context: PreparedTag } @@ -54,7 +56,7 @@ export class GithubJobs extends DurableObject { if (!job) return null // The operator gets outcomes and source identity, without baseline file contents. - return { id: job.id, request: job.request, receivedAt: job.receivedAt, _tag: job._tag, ...(job._tag === 'Finished' ? { outcome: job.outcome } : {}) } + return { id: job.id, request: job.request, receivedAt: job.receivedAt, startedAt: job.startedAt, _tag: job._tag, ...(job._tag === 'Finished' ? { outcome: job.outcome } : {}) } } async alarm(): Promise { @@ -63,9 +65,12 @@ export class GithubJobs extends DurableObject { if (!id) return await this.ctx.storage.setAlarm(Date.now() + 10_000) - const state = await this.ctx.storage.get(`job-${id}`) - if (!state) + const stored = await this.ctx.storage.get(`job-${id}`) + if (!stored) throw new Error('APP_JOB_STATE_MISSING') + const state = stored.startedAt === undefined && stored._tag !== 'Finished' ? { ...stored, startedAt: Date.now() } : stored + if (state !== stored) + await this.ctx.storage.put(`job-${id}`, state) if (state._tag === 'Finished') { await this.ctx.storage.transaction(async (storage) => { const current = await storage.get('queue') ?? [] @@ -73,8 +78,8 @@ export class GithubJobs extends DurableObject { }) return } - if (Date.now() - state.receivedAt > 30 * 60 * 1000) { - await this.finish(state, { _tag: 'Failed', code: 'APP_JOB_DEADLINE', detail: 'The job exceeded thirty minutes.' }) + if (jobExpired(state, Date.now())) { + await this.finish(state, { _tag: 'Failed', code: 'APP_JOB_DEADLINE', detail: 'The job exceeded its deadline after it started.' }) return } await this.advance(state).catch(async (cause: unknown) => { diff --git a/workers/skill-harness/src/job-deadline.ts b/workers/skill-harness/src/job-deadline.ts new file mode 100644 index 00000000..9118b898 --- /dev/null +++ b/workers/skill-harness/src/job-deadline.ts @@ -0,0 +1,13 @@ +import { JOB_TIMEOUT_MS } from './contracts' + +/** + * A GitHub job's deadline: one full Harness run plus tag preparation, npm + * publication waits, and the pull request. It counts from when the job reached + * the head of the queue, because jobs run one at a time and a queued job + * would otherwise spend its deadline waiting behind others. + */ +export const APP_JOB_DEADLINE_MS = JOB_TIMEOUT_MS + 15 * 60 * 1000 + +export function jobExpired(job: { receivedAt: number, startedAt?: number }, now: number): boolean { + return now - (job.startedAt ?? job.receivedAt) > APP_JOB_DEADLINE_MS +} diff --git a/workers/skill-harness/test/gateway.test.ts b/workers/skill-harness/test/gateway.test.ts index a5491f54..f898316d 100644 --- a/workers/skill-harness/test/gateway.test.ts +++ b/workers/skill-harness/test/gateway.test.ts @@ -39,7 +39,7 @@ describe('sandbox credential gateway', () => { const options = openCodeGo() const response = await forwardSandboxRequest(openCodeGoRequest({ model: 'expensive-model', - max_tokens: 32000, + max_tokens: 100000, max_completion_tokens: 131072, n: 4, service_tier: 'priority', @@ -57,7 +57,7 @@ describe('sandbox credential gateway', () => { ...Object.entries(openCodeRouting), ]) const body = JSON.parse(String(forwarded?.body)) - expect(body).toMatchObject({ model: 'glm-5.3', max_tokens: 8192, max_completion_tokens: 8192, stream: true, tools: [{ type: 'function' }] }) + expect(body).toMatchObject({ model: 'glm-5.3', max_tokens: 32768, max_completion_tokens: 32768, stream: true, tools: [{ type: 'function' }] }) expect(body).not.toHaveProperty('n') expect(body).not.toHaveProperty('service_tier') expect(options.consumeModelCall).toHaveBeenCalledOnce() @@ -75,9 +75,10 @@ describe('sandbox credential gateway', () => { it.each([ [{ max_tokens: 1000 }, { max_tokens: 1000 }], - [{}, { max_tokens: 8192 }], - [{ max_tokens: 'unbounded' }, { max_tokens: 8192 }], - [{ max_completion_tokens: 500 }, { max_tokens: 8192, max_completion_tokens: 500 }], + // GLM 5.3 reasons before it answers. Its thinking counts as output, so 8,192 cut off whole steps. + [{}, { max_tokens: 32768 }], + [{ max_tokens: 'unbounded' }, { max_tokens: 32768 }], + [{ max_completion_tokens: 500 }, { max_tokens: 32768, max_completion_tokens: 500 }], ])('bounds OpenCode Go output for %j', async (limits, expected) => { const options = openCodeGo() await forwardSandboxRequest(openCodeGoRequest({ messages: [], ...limits }), options) diff --git a/workers/skill-harness/test/job-deadline.test.ts b/workers/skill-harness/test/job-deadline.test.ts new file mode 100644 index 00000000..caabbf41 --- /dev/null +++ b/workers/skill-harness/test/job-deadline.test.ts @@ -0,0 +1,21 @@ +import { describe, expect, it } from 'vitest' +import { APP_JOB_DEADLINE_MS, jobExpired } from '../src/job-deadline' + +const minute = 60 * 1000 + +describe('app job deadline', () => { + it('does not count time a job spent waiting behind other jobs', () => { + const now = Date.now() + expect(jobExpired({ receivedAt: now - 3 * 60 * minute, startedAt: now - 5 * minute }, now)).toBe(false) + }) + + it('expires a job that has run past the deadline since it started', () => { + const now = Date.now() + expect(jobExpired({ receivedAt: now - APP_JOB_DEADLINE_MS - 2 * minute, startedAt: now - APP_JOB_DEADLINE_MS - minute }, now)).toBe(true) + }) + + it('leaves room for a full Harness run and its publication', () => { + const now = Date.now() + expect(jobExpired({ receivedAt: now - 50 * minute, startedAt: now - 50 * minute }, now)).toBe(false) + }) +})