Background subagents

README.md

@@ -87,6 +87,30 @@ Existing log identity is authoritative. A conflicting `sessionId`, a nonempty lo
 history, or multiple session starts fails rather than silently initializing a new session. `cwd` and
 `meta` apply only to initialization; resumed model/prompt changes are recorded as configuration epochs.
 
+## Background subagents
+
+A `subagent` tool call with `run_in_background: true` returns as soon as the agent has started, so the
+calling agent can finish its turn while the subagent works. When the subagent finishes, fold writes a
+synthetic `<task-notification>` user message for the agent that launched it and, if the root agent is
+idle, starts a new root turn on it (a "wake"). Completions that arrive while the root is busy join that
+run as one coalesced follow-up.
+
+```ts
+import { agentControlTools, defineAgent, Session, subagentTool } from '@humanlayer/fold-core'
+
+const agent = defineAgent({ model, tools: [subagentTool([researcher]), ...agentControlTools()] })
+const session = yield * Session.open({ agent })
+yield * session.send('Investigate the three slow endpoints in parallel.')
+yield * session.awaitBackgroundIdle // background agents done, notification turns delivered
+const tasks = yield * session.backgroundTasks // status, usage, summary per task
+```
+
+`agentControlTools()` gives the model `list_agents`, `stop_agent`, and `send_message`. Hosts get
+`backgroundTasks`, `backgroundTaskChanges`, `stopTask(id)`, and `awaitBackgroundIdle`. `interrupt()` stops
+only the foreground turn; `stop()` and closing the session stop background agents too. The durable rows
+are `task_started` and `task_finished`; see [docs/background-subagents.md](docs/background-subagents.md)
+for the full behavior, events, and API.
+
 ### API migration
 
 This is a breaking API change: replace `startSession` / `resumeSession` with `Session.open` and remove

docs/background-subagents.md

@@ -0,0 +1,158 @@
+# Background subagents
+
+Fold can run a subagent in the background: the tool call that launches it returns at once, the
+launching agent keeps working (or ends its turn), and fold tells the launcher when the subagent
+finishes. The design follows Claude Code's background agents, and the data maps one to one onto a
+host's task table (HumanLayer's `session_background_tasks`).
+
+## Launching
+
+The `subagent` tool takes `run_in_background: true` together with any of its selectors:
+
+| Call                                                       | Effect                                           |
+| ---------------------------------------------------------- | ------------------------------------------------ |
+| `{ agent, prompt, description?, run_in_background: true }` | dispatch a fresh subagent as a background task   |
+| `{ fork: true, prompt, run_in_background: true }`          | fork the caller's context into a background task |
+| `{ agent_id, prompt, run_in_background: true }`            | resume an earlier subagent as a background task  |
+
+The call returns a receipt such as:
+
+```text
+Background agent started. agent_id: agent_ab12cd34
+It is working in the background. You will be notified with a <task-notification> message when it
+finishes, so do not poll for it. ...
+```
+
+The task id is the subagent's agent id. Without `run_in_background` the subagent runs in the
+foreground exactly as before: the call blocks and returns the subagent's result.
+
+## What the launcher sees when it finishes
+
+When a background run ends, fold writes a user message for the launcher:
+
+```xml
+<task-notification>
+<task-id>agent_ab12cd34</task-id>
+<status>completed</status>
+<summary>Background agent "find the config loader" completed</summary>
+<result>
+...the subagent's final message...
+</result>
+<usage>turns: 3, tool uses: 7</usage>
+<note>The agent keeps its context. Send it a message with agent_id agent_ab12cd34 to continue it.</note>
+</task-notification>
+```
+
+`status` is `completed`, `failed` (with an `<error>`), or `stopped`. The message carries
+`origin: { _tag: 'task-notification', taskIds }`, so UIs can render it as a system row; the model sees
+the text like any user message.
+
+Delivery:
+
+1. If the launcher is a subagent that is still running, the notification is queued as a follow-up for
+   it and drains when its run would otherwise complete.
+2. Otherwise the notification goes to the root agent:
+    - root running: a follow-up of the current run (drained at its natural completion);
+    - root idle: a **wake** - a new root run whose input is the notification. Wakes go through the same
+      gate and running-agent claim as `send`, so they never overlap a user turn or a model switch.
+3. Coalescing: everything queued when a wake takes the gate goes into that one wake, and adjacent queued
+   notifications drain as a single user message. Two agents finishing together produce one turn.
+4. If a run ends before consuming its follow-up (it was stopped or interrupted), the notification goes
+   back to the queue and is delivered again.
+
+## Controls
+
+### Model tools
+
+`agentControlTools()` returns three tools to install next to a `subagentTool` value (the fold-agent
+coding and RLM modes do this):
+
+| Tool           | Parameters              | Behavior                                                                                                                                                      |
+| -------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `list_agents`  | none                    | running and finished tasks with status, description, turns, tool uses, last tool, and the start of the result or error                                        |
+| `stop_agent`   | `{ agent_id }`          | interrupts a running task; it ends `stopped`. No notification follows: the caller already knows                                                               |
+| `send_message` | `{ agent_id, message }` | running agent: steered (read after its current step). Finished or stopped agent: resumed in the background with its full context; it notifies again when done |
+
+### Session API
+
+```ts
+type FoldSession = {
+	// ...existing members
+	readonly backgroundTasks: Effect<ReadonlyArray<BackgroundTask>>
+	readonly backgroundTaskChanges: Stream<ReadonlyArray<BackgroundTask>>
+	readonly stopTask: (taskId: AgentId | string) => Effect<boolean, SubagentNotFoundError>
+	readonly awaitBackgroundIdle: Effect<void>
+}
+```
+
+- `stopTask(id)` accepts a full id or a unique short reference. It interrupts the run, records it
+  `stopped`, and notifies the launcher like any completion. It resolves `false` when the task is not
+  running.
+- `awaitBackgroundIdle` resolves when no task is running, no wake is scheduled or running, and no
+  notification is waiting for delivery. Headless hosts await it after `send`.
+- `interrupt()` with no target interrupts the foreground only: the root's run and any running agent that
+  is not a background task or beneath one. Background tasks keep running and still wake the root when
+  they finish. `interrupt({ agentId })` on a background task stops that task.
+- `stop()` stops running background tasks (recorded `stopped`) and requests the usual graceful stop for
+  the foreground. Their notifications do not wake the root; they are steered into the root's next run,
+  before its first model call.
+- Closing the session scope interrupts every background run; each one is recorded `stopped` before the
+  session's services are released.
+- Opening an existing log rebuilds the task list from its rows. A run that has `task_started` but no
+  `task_finished` (the process died) is recorded `stopped`, and its notification waits for the root's
+  next run.
+
+The `send(text, { agentId })` and `steer(text, { agentId })` targets also work on background tasks.
+
+`BackgroundTask`:
+
+| Field                                | Meaning                                                                                                |
+| ------------------------------------ | ------------------------------------------------------------------------------------------------------ |
+| `taskId`                             | the background agent's id                                                                              |
+| `toolCallId`                         | the tool call that launched the latest run (a `subagent` or `send_message` call)                       |
+| `parentAgentId`                      | the launcher; it is notified                                                                           |
+| `taskType`                           | `'subagent'`                                                                                           |
+| `agentType`                          | registry type name; null for forks                                                                     |
+| `description`, `prompt`              | the label and the message the latest run started with                                                  |
+| `status`                             | `running`, `completed`, `failed`, or `stopped`                                                         |
+| `isBackgrounded`                     | always true today (foreground runs are not recorded as tasks)                                          |
+| `startedAt`, `endedAt`, `notifiedAt` | epoch millis for the latest run                                                                        |
+| `usage`                              | `{ turns, toolUses, totalTokens }` for the latest run (`totalTokens`: context size at the latest turn) |
+| `lastToolName`, `progressSummary`    | live activity while running                                                                            |
+| `summary`, `error`                   | the latest run's final text or error                                                                   |
+| `runCount`                           | 1, plus one per resume                                                                                 |
+
+## Events
+
+Durable log rows (additive to the v1 log format; older logs decode unchanged):
+
+- `task_started` on the task agent's envelope: `taskType`, `agentType`, `description`, `prompt`,
+  `isBackgrounded`, `runCount`. Written after the agent's `agent_started` (or resume) rows and before its
+  first message.
+- `task_finished`: `status`, `summary`, `error`, `usage`, `durationMs`, `lastToolName`. Written by an
+  uninterruptible finalizer, so every run that started gets one.
+- `user-message.origin` (optional): `{ _tag: 'task-notification', taskIds }` on runtime-written
+  notification messages.
+
+Ephemeral delta on `session.events()`: `{ type: 'task-progress', taskId, turns, toolUses, totalTokens,
+lastToolName, progressSummary }`, published after each of the task agent's turns.
+
+## CLI and TUI
+
+- `foldcode --prompt` waits after its own turn until background agents and their notification turns are
+  done, then exits with the root's latest outcome. `--background-timeout <seconds>` caps the wait
+  (default 600); agents still running then are stopped. Ctrl+C while waiting stops them.
+- The line renderer prints `[background] agent_x started: ...` and
+  `[background] agent_x completed (3 turns, 7 tool uses)` lines, and a wake as
+  `[notification] background agent agent_x finished`.
+- In the TUI, the SUBAGENTS rail marks background agents with `BG`, the rail header and the activity
+  label count them (`1 RUN · 1 BG`), `x` stops the selected running background agent, and a wake shows
+  as an `AGENTS` row with each agent's summary instead of a `USER` row.
+
+## Limits
+
+- Only subagents run in the background; there are no background shells or monitors.
+- A foreground subagent cannot be moved to the background mid-run (no Ctrl+B equivalent).
+- There is no concurrency cap on background tasks.
+- Notifications are in memory until delivered. A process that dies with an undelivered notification
+  loses it; the durable `task_finished` row remains.

packages/fold-agent/src/Mode/Mode.ts

@@ -6,12 +6,12 @@
  *
  * `defaultCodingMode` is the batteries-included local coding agent: the full filesystem toolset (read,
  * write, edit, apply_patch, bash - the family policy advertises the right editing subset per model),
- * the disk skill tool, and the default subagent roster (general-purpose, bash, researcher - see
- * Mode/Subagents). Other modes are just other `FoldMode` values (Mode/Rlm); Mode/ModeName maps the
+ * the disk skill tool, the default subagent roster (general-purpose, bash, researcher - see
+ * Mode/Subagents), and the background-agent controls (list_agents, stop_agent, send_message). Other modes are just other `FoldMode` values (Mode/Rlm); Mode/ModeName maps the
  * selectable names to them. RPI is not a mode: the `rpi` context flag appends the RPI specialist
  * roster to ANY mode's subagents (Mode/Rpi).
  */
-import { skillTool, subagentTool, type FoldTool } from '@humanlayer/fold-core'
+import { agentControlTools, skillTool, subagentTool, type FoldTool } from '@humanlayer/fold-core'
 import type { FileSystem, Path } from 'effect'
 import type { HttpClient } from 'effect/http'
 import type { ChildProcessSpawner } from 'effect/process'
@@ -74,6 +74,8 @@ export const DEFAULT_CODING_PROMPT: string =
 	'Delegate to a subagent when a step would burn context you need for the main task: researcher for ' +
 	'locating and explaining code, web-search-researcher for current web research, bash for running a ' +
 	'command and summarizing its output, and general-purpose for a self-contained multi-step task. Do simple work yourself. ' +
+	'Launch independent subagents with run_in_background when you can keep working meanwhile; each result ' +
+	'arrives later as a <task-notification> message, and list_agents, send_message, and stop_agent manage them. ' +
 	'Keep responses concise; let the tools do the work.'
 
 /** The batteries-included local coding agent: full filesystem toolset, disk skills, default subagents. */
@@ -85,5 +87,6 @@ export const defaultCodingMode: FoldMode = {
 		...codingTools({ cwd }),
 		skillTool(skillsFromDisk({ cwd })),
 		subagentTool(modeSubagents({ cwd, rpi })),
+		...agentControlTools(),
 	],
 }

packages/fold-agent/src/Mode/Rlm.ts

@@ -8,7 +8,7 @@
  * The mode runs on the `orchestrator` config role, which falls back to `smart` when the config
  * declares no orchestrator binding (D25).
  */
-import { skillTool, subagentTool } from '@humanlayer/fold-core'
+import { agentControlTools, skillTool, subagentTool } from '@humanlayer/fold-core'
 
 import { skillsFromDisk } from '../Skills/DiskSkills'
 import { applyPatchTool } from '../Tools/ApplyPatchTool'
@@ -64,5 +64,6 @@ export const rlmMode: FoldMode = {
 		applyPatchTool({ cwd }),
 		skillTool(skillsFromDisk({ cwd })),
 		subagentTool(modeSubagents({ cwd, rpi })),
+		...agentControlTools(),
 	],
 }

packages/fold-agent/test/Mode/Launch.vi.test.ts

@@ -127,6 +127,7 @@ it.effect('launchSession composes the model, agentfiles, and mode tools over Ses
 				expect(tools).toContain('bash')
 				expect(tools).toContain('skill')
 				expect(tools).toContain('subagent')
+				expect(tools).toEqual(expect.arrayContaining(['list_agents', 'stop_agent', 'send_message']))
 			}),
 		)
 	}).pipe(Effect.provide(launchTestServices)),

packages/fold-agent/test/Mode/Rlm.vi.test.ts

@@ -68,6 +68,7 @@ it('holds the file tools, skills, and subagents - and deliberately no bash', ()
 
 	expect(names).not.toContain('bash')
 	expect(names.filter((name) => name === 'subagent')).toHaveLength(1)
+	expect(names).toEqual(expect.arrayContaining(['list_agents', 'stop_agent', 'send_message']))
 	expect(names.filter((name) => name === 'skill')).toHaveLength(1)
 	expect(names).toContain('read')
 	expect(names).toContain('write')

packages/fold-agent/test/Mode/ToolSchemas.vi.test.ts

@@ -0,0 +1,71 @@
+/**
+ * Every tool the coding and RLM modes advertise - root tools and every subagent's and fork agent's
+ * tools, with and without the RPI roster - must convert to a JSON Schema that OpenAI's strict tool
+ * schemas accept. The conversion is the one the OpenAI provider runs in `prepareTools`
+ * (`Tool.getJsonSchema` with `toCodecOpenAI`), which throws for a non-object root or a root anyOf
+ * (for example `Schema.Struct({})`, which also accepts arrays). Regression for list_agents.
+ */
+import { tmpdir } from 'node:os'
+
+import { expect, it } from '@effect/vitest'
+import { collectAgentDefinitions, customModel, type FoldModel } from '@humanlayer/fold-core'
+import { Effect, Stream } from 'effect'
+import { LanguageModel, Tool } from 'effect/ai'
+import { toCodecOpenAI } from 'effect/ai/OpenAiStructuredOutput'
+
+import { defaultCodingMode, layerCodingToolServices, rlmMode, type FoldMode, type ModeModels } from '../../src/index'
+
+const model: FoldModel = customModel({
+	activeModel: {
+		providerId: 'test',
+		providerKind: 'openai-compatible',
+		modelId: 'schema-check',
+		role: null,
+		requestedReasoningLevel: 'off',
+		reasoning: { _tag: 'disabled' },
+	},
+	make: LanguageModel.make({ generateText: () => Effect.die(new Error('unused')), streamText: () => Stream.empty }),
+})
+const models: ModeModels = { primary: model, smart: model, fast: model, orchestrator: model }
+
+/** Root tools plus every reachable subagent and fork-agent tool, deduplicated by value. */
+const advertisedTools = (mode: FoldMode, rpi: boolean) =>
+	Effect.gen(function* () {
+		const rootTools = mode.buildTools({ cwd: tmpdir(), models, rpi })
+		const definitions = yield* collectAgentDefinitions(rootTools)
+		const all = [
+			...rootTools,
+			...definitions.subagents.flatMap((definition) => definition.tools ?? []),
+			...definitions.forkAgents.flatMap((definition) => definition.tools),
+		]
+		return [...new Set(all)]
+	})
+
+for (const mode of [defaultCodingMode, rlmMode]) {
+	for (const rpi of [false, true]) {
+		it.effect(`${mode.name} mode (rpi ${rpi}) advertises only OpenAI-compatible tool schemas`, () =>
+			Effect.gen(function* () {
+				const tools = yield* advertisedTools(mode, rpi)
+				const names = new Set(tools.map((tool) => tool.name))
+				expect(names).toContain('subagent')
+				expect(names).toContain('list_agents')
+				expect(names).toContain('stop_agent')
+				expect(names).toContain('send_message')
+
+				for (const foldTool of tools) {
+					const contribution = yield* foldTool.init
+					const schema = (() => {
+						try {
+							return Tool.getJsonSchema(contribution.tool, { transformer: toCodecOpenAI })
+						} catch (error) {
+							throw new Error(`tool ${foldTool.name}: ${String(error)}`, { cause: error })
+						}
+					})()
+					expect(schema, foldTool.name).toMatchObject({ type: 'object' })
+					expect(schema, foldTool.name).not.toHaveProperty('anyOf')
+					expect(schema, foldTool.name).not.toHaveProperty('oneOf')
+				}
+			}).pipe(Effect.scoped, Effect.provide(layerCodingToolServices({ outputDirectory: tmpdir() }))),
+		)
+	}
+}