Skip to content

Shape a finished job's report after the task's brief: a few sentences, no list, one for what to check - #1726

Merged
davidmckayv merged 2 commits into
mainfrom
fixpass/claude-sdk-brevity
Nov 16, 2026
Merged

davidmckayv merged 2 commits into
mainfrom
fixpass/claude-sdk-brevity

Conversation

@davidmckayv

@davidmckayv davidmckayv commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Read first: one did-the-task signal the checks miss, found and fixed

On the Claude Agent SDK, the multi-step task asks for customer emails "in the last day". In 2 of 3 live runs at the first head (a7e5253), the sheet included an email 30 hours old. On main this happened in 1 of 4 runs. The live checks passed those runs, and so did the claims grader (see "A gap in the checks" below). The other three setups never included that email, before or after.

Cause. The first head also told a delegating Bot to end its brief at what to check, with no line on how to report. Claude then wrote shorter briefs that dropped the work's own checks. I replayed the delegating step of four captured Claude SDK runs 6 times each (24 briefs per wording) and counted briefs that keep a rolling 24-hour window instead of a calendar day ("since 2026-11-15"):

Wording Briefs with a rolling 24 hours
Main 17 of 24
First head (with the line) 4 of 24
This head (without the line) 15 of 24

Later steps were replayed 6 times per capture on main's wording and the PR's. When the brief is held the same, the wording makes no difference:

  • the Gmail search reached past the last day in 28 of 48 replays on main and 30 of 48 on the PR
  • the step that writes the rows included the 30-hour-old email in 12 of 36 on main and 12 of 35 on the PR

So the brief is what decides it.

Fix (now on main ea7f43a as 1914f40). The delegate line is gone. A test now fails if it comes back: red on the first head, green here. The report is already shaped after the brief, whatever the brief asks it to include. The offline report lengths below were measured on briefs written without that line.

Live on this head (95a49e2), multi-step case, 3 runs each:

Setup Words Did the task Claims 30-hour-old email in the sheet
Built-in Claude 112, 69, 68 3/3 3/3 0/3
Claude Agent SDK 76, 99, 81 1/3 2/3 1/3 (main: 1/4)

The two Claude Agent SDK runs that missed:

  • Run 2. The task finished: sheet, post and report all done. The conversation's turn then ended with "The Bot stopped before it finished answering", which no other run showed.
  • Run 3.
    • It searched Gmail by calendar day and took in the 30-hour-old email. That is the fault on main described above.
    • It then reported "4 customer emails from the last day", which failed claims.
    • Its first Slack send failed on invalid arguments, so the corrected send asked for approval again.

Counted side by side: main b825ff5 vs this head, multi-step case, live.

Setup, runs Did the task Claims 30-hour-old email First Slack send failed Ended badly Words
Claude Agent SDK, main, 8 6 6 4 2 0 192-249
Claude Agent SDK, PR, 8 7 6 2 1 0 79-110
Built-in Claude, main, 8 8 8 0 0 0 156-229
Built-in Claude, PR, 8 valid of 12 8 8 0 0 0 75-118
  • First Slack send failures are argument mistakes, not the post's text. One sent {"text": …} with a field the schema refuses; another sent {"channel": "sales"} with no text. In both, the corrected send asked for approval again.
  • "The Bot stopped before it finished answering" (head run 2) is a carry-on fault after an approval. The Claude Agent SDK harness's stream for the carried-on run ended 0.3 s after its model request, with no finish event. The server retried, and the retry posted the report. It happened 0 times in the 16 counted runs.
  • Invalid runs. In 4 of 12 PR runs, built-in Claude read the sheet back with GOOGLESHEETS_QUERY_TABLE, a real action the test's fake Sheets does not play yet, so those runs are invalid. Main had 0 of 8. PR runs also read the sheet after posting more often: 4 of 17 on PR heads, 0 of 14 on main.

After the last-day fix (#1730). This PR was rebased onto main 01e5460, which includes #1730, then onto ea7f43a (a test-only change). The Claude Agent SDK multi-step case then ran live 6 times each:

Claude Agent SDK, 6 runs Did the task Claims 30-hour-old email First Slack send failed Ended badly Words
Main 01e5460 6 5 0 0 0 177-270
This PR on it 6 5 0 0 0 78-140

Each side had the same one claims miss: the report said the excluded Wingtip email was "about 25 hours" old when it was 30. One PR run had 140 words.

Why

Claude Sonnet 5.5 reported a three-app job (go through the inbox, build a sheet, post a summary in #sales) in 166 to 252 words. That happened on OpenBot's own runner and on the Claude Agent SDK alike. GPT-5.6 Terra and LangGraph took 38 to 112 words under the same rules.

Claude repeated the sheet's rows, said it had read the sheet back, listed what it left out and why, and gave each inference its own bullet. It follows the task's brief, which comes last and often asks for "anything you could not confirm", over a rule that comes before it.

What changed

  • After the brief, a delegated task is told how to report work done. One paragraph of at most four sentences, with no list or headings, whatever the brief asks it to include. It says what was done, with the link, then at most one sentence on what the person should check. A worked example on apps no test uses shows the shape. A question's answer is shaped as before, by the frame ahead of the brief. readBrief takes the new text off, so the task thread still shows the brief alone. A thread opened before this change still reads back.
  • The reply rules say a finished job the same way, with the same example.
  • The claims rule adds one line: an inference, however likely, is still unconfirmed. GPT had written that company names taken from email domains left "none unconfirmed", which the claims grader fails.
  • The Claude Agent SDK fixture of what the server sends is rewritten, and the harness test checks that Claude Code starts with the new rules.

Tests

New and changed tests in shared/bot-prompt.test.ts and shared/tests/orchestration.test.ts:

  • On main: 5 fail.
  • On this branch: 56 pass.
  • The new guard that a brief is not told to stop short fails on the first head and passes here.
  • All of shared: 392 pass.
  • agent-claude-sdk/tests: 137 pass.
  • app/tests/task-worked-for.test.tsx: 4 pass (the task thread shows the brief alone).

Measured

Offline. I replayed each case's captured final task step on the real model with the new wording, 6 answers per case. Each answer was graded by the same claims grader the live runs use (Claude Opus 5.5).

Wording Claude Agent SDK Built-in Claude Built-in GPT
Main 164-195 words 173-244 words 55-84 words, claims failed 1 of 6
This PR 81-114 words 93-134 words 45-62 words, claims failed 0 of 6

Wordings that did not get Claude under 120:

  • reworded rules ahead of the brief: 145-190 words
  • "under 60 words": 104-168
  • a stronger brief rule alone: 172-218

Live, at the first head (a7e5253, with the delegate line). A real stack with local CopilotKit Intelligence 0.16.0-rc.13 and real models, graded by the cases' code checks and the Claude grader. Main first, then the PR.

Multi-step case ("go through my inbox, build a sheet, post in #sales"):

Setup Main: words PR: words Did the task (main → PR) Claims (main → PR)
Claude Agent SDK, claude-sonnet-5-5 166, 252, 167 108, 99, 94 3/3 → 3/3 3/3 → 3/3
Built-in, claude-sonnet-5-5 175, 186, 220 105, 104, 102, 86, 72, 123 3/3 → 5/6 3/3 → 6/6
Built-in, gpt-5.6-terra 67, 112, 97 43, 48, 35 3/3 → 3/3 3/3 → 3/3
LangGraph 45, 38, 57 34, 34, 48 3/3 → 3/3 3/3 → 3/3

Six cases, one run each:

Setup Did the task (main → PR) Claims (main → PR) Multi-step words (main → PR)
GPT 6/6 → 6/6 5/6 → 6/6 36 → 42
Claude Agent SDK 5 of 5 valid runs → 5/6 5/6 → 5/6 247 → 62

Notes on the live misses:

  • Built-in Claude, the 5/6. One run asked twice to approve the Slack post. Its first send had no text and failed, so the corrected send asked again. The post was made once, and the run did every step. It did not happen in the other 5 runs.
  • Built-in Claude, the 123-word run. One of its 6 runs was 3 words over 120.
  • Claude Agent SDK, sheets-template. It asked twice to approve one conditional-format change. That is the same double approval as on main, where that run was invalid (the fake Sheets lacks CREATE_CHART).
  • Claude Agent SDK, Slack search. The claims miss ("none of them has replies" after reading one thread) is on main too.
  • Claude Agent SDK, last-day window. See "Read first" at the top.

A gap in the checks

The multi-step case's "did the task" check (one "Customer follow-ups" sheet with the three customers in it) tests that Contoso, Fabrikam and Tailspin are present and the internal HR email is not. It does not test that an email outside the last day stays out. The claims grader judges what the Bot says, not the window. So a sheet holding the 30-hour-old Wingtip email passes both. The check should fail a sheet that holds Wingtip.

@davidmckayv
davidmckayv force-pushed the fixpass/claude-sdk-brevity branch 2 times, most recently from 95a49e2 to dae1412 Compare November 16, 2026 20:52
…, no list, one for what to check

Claude Sonnet 5.5 reported a three-app job (read the inbox, build a sheet, post in #sales) in 166 to 252 words, on OpenBot's own runner and on the Claude Agent SDK alike, where GPT-5.6 Terra took 45 to 84. It repeated the sheet's rows, said it had read the sheet back, listed what it left out, and gave each inference its own bullet. It follows the task's brief, which comes last and often asks for "anything you could not confirm", over a rule that comes before it.

- After the brief, a task is told how a report of work done is shaped: one paragraph of at most four sentences, no list or headings, what was done with its link, then at most one sentence on what to check. A worked example shows it, on apps no test uses. A question's answer is shaped as before. A thread opened without this text still reads back as the Bot's brief.
- A delegating Bot's brief ends at what to do and check, with no line on what to report.
- The reply rules say a finished job the same way, with the same example.
- The claims rule adds that an inference, however likely, is still unconfirmed. GPT had written that company names taken from email domains left "none unconfirmed".
- The Claude Agent SDK fixture of what the server sends is rewritten, and the harness test checks Claude Code starts with the new rules.
…d at what to check

Told to end its brief at what to check, with no line on how to report, Claude wrote shorter briefs that also dropped the work's own checks. Replaying the delegating step of four captured multi-step runs six times each, briefs for "emails from the last day" kept a rolling 24 hours in 4 of 24 with that line and 15 of 24 without it (17 of 24 on main). Live, the task then counted a 30-hour-old email as the last day's in 2 of 3 runs on the Claude Agent SDK. The report is already shaped after the brief, whatever the brief asks it to include, so the line is not needed for length: the offline report lengths were measured on briefs written without it.
@davidmckayv
davidmckayv force-pushed the fixpass/claude-sdk-brevity branch from dae1412 to 1914f40 Compare November 16, 2026 21:08
@davidmckayv
davidmckayv merged commit 0c9d387 into main Nov 16, 2026
16 checks passed
@davidmckayv
davidmckayv deleted the fixpass/claude-sdk-brevity branch November 16, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant