Skip to content

The scripted test model answers codeaf's helpers note instead of the person on the fourth message #2563

Open

Description

@AbirAbbas

What happened

On the person's fourth message in a conversation, the scripted test model (test/remote/stub) answers codeaf's own helpers note instead of the person. A scripted PROBE-PWD on that turn never calls bash. The same thing happens on the first message after a long chat is opened in a new process, because a new process counts the person's earlier messages and lands the note on that turn.

Seen on app/v3 a687c92 and on the idle hand-off branch (#2287), on Linux and macOS, with codeaf chat --no-host --model stub/scripted --one-model:

› PROBE-ECHO d
STUB-REPLY: Instructions from codeaf, not from the person: when to suggest helpers. If …
› PROBE-PWD
<the working folder, from a real bash call>

The note lands once per process, at the step boundary, as a message after the person's latest words (landHelpersNoteLocked, at helpersNoteAfter = 4 person messages). The stub's script takes the last user-role message as the person's ask, so on that one turn it reads the note.

Replication

Deterministic (no model).

  1. make build, then go build -o modelstub ./test/remote/stub, then start it: ./modelstub -addr <stub address>.
  2. Use a throwaway home: set HOME, CODEAF_HOME and CODEAF_PROFILE_DIR to an empty folder, plus CODEAF_SYNC_URL=off, CODEAF_TELEMETRY=off, CODEAF_BASE_URL=<stub address>/api/v1 and OPENROUTER_API_KEY=stub. Put {"model.talk":"stub/scripted","tools.approvalMode":"allow"} in $CODEAF_HOME/config.json.
  3. Run bin/codeaf chat --no-host --model stub/scripted --one-model, and press esc at setup.
  4. Send PROBE-ECHO a, PROBE-ECHO b, PROBE-ECHO c, then PROBE-PWD.
    • Today: the fourth reply is STUB-REPLY: Instructions from codeaf, not from the person: when to suggest helpers. … and no bash call is made.
    • Expected: a bash call with pwd, quoted back.
  5. A fifth PROBE-PWD works. To see the new-process variant, /quit, reopen with --session <that chat's .cell/transcript.jsonl>, and send PROBE-PWD first.

Field (real models). Not shown to misbehave. A real model reads the note as codeaf's, because of its opening words. This defect is about scripted runs, where it makes any e2e or hand test that scripts a tool call on the fourth message (or right after a reopen) flaky.

Where

  • script in test/remote/stub/main.go (search SCANNING BACKWARDS FOR THE LAST USER MESSAGE).
  • landHelpersNoteLocked and helpersNoteOpening in internal/session/helpersnote.go.
  • The engine's own list of note openings, in internal/session/agent.go (search strings.HasPrefix(text, helpersNoteOpening)).

The fix

The stub looks for the person's ask the way the engine tells a note from a person: it skips user-role messages that open with Instructions from codeaf, not from the person. Keep the product's placement of the note, because tail notes are where volatile instructions land without disturbing the cached prefix. The opening string should come from one shared place, so the stub cannot drift from the engine.

Acceptance

  • e2e: through the real door, with bin/codeaf chat driven in tmux against the stub (keyless). Four messages, the fourth PROBE-PWD. Assert that the fourth reply contains the working folder from a bash call and does not contain Instructions from codeaf.
  • e2e: the control. The same run, reopened with --session and PROBE-PWD sent first, gets a bash call.
  • Unit: script returns the person's last message when a later user-role message opens with Instructions from codeaf, not from the person.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions