Repository navigation
Shape a finished job's report after the task's brief: a few sentences, no list, one for what to check - #1726
Merged
Merged
Conversation
davidmckayv
requested review from
MikeRyanDev,
dustywusty,
guidovizoso,
mxmzb,
samjulien and
tylerslaton
as code owners
November 16, 2026 19:43
davidmckayv
force-pushed
the
fixpass/claude-sdk-brevity
branch
2 times, most recently
from
November 16, 2026 20:52
95a49e2 to
dae1412
Compare
…, no list, one for what to check Claude Sonnet 5.5 reported a three-app job (read the inbox, build a sheet, post in #sales) in 166 to 252 words, on OpenBot's own runner and on the Claude Agent SDK alike, where GPT-5.6 Terra took 45 to 84. It repeated the sheet's rows, said it had read the sheet back, listed what it left out, and gave each inference its own bullet. It follows the task's brief, which comes last and often asks for "anything you could not confirm", over a rule that comes before it. - After the brief, a task is told how a report of work done is shaped: one paragraph of at most four sentences, no list or headings, what was done with its link, then at most one sentence on what to check. A worked example shows it, on apps no test uses. A question's answer is shaped as before. A thread opened without this text still reads back as the Bot's brief. - A delegating Bot's brief ends at what to do and check, with no line on what to report. - The reply rules say a finished job the same way, with the same example. - The claims rule adds that an inference, however likely, is still unconfirmed. GPT had written that company names taken from email domains left "none unconfirmed". - The Claude Agent SDK fixture of what the server sends is rewritten, and the harness test checks Claude Code starts with the new rules.
…d at what to check Told to end its brief at what to check, with no line on how to report, Claude wrote shorter briefs that also dropped the work's own checks. Replaying the delegating step of four captured multi-step runs six times each, briefs for "emails from the last day" kept a rolling 24 hours in 4 of 24 with that line and 15 of 24 without it (17 of 24 on main). Live, the task then counted a 30-hour-old email as the last day's in 2 of 3 runs on the Claude Agent SDK. The report is already shaped after the brief, whatever the brief asks it to include, so the line is not needed for length: the offline report lengths were measured on briefs written without it.
davidmckayv
force-pushed
the
fixpass/claude-sdk-brevity
branch
from
November 16, 2026 21:08
dae1412 to
1914f40
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Read first: one did-the-task signal the checks miss, found and fixed
On the Claude Agent SDK, the multi-step task asks for customer emails "in the last day". In 2 of 3 live runs at the first head (a7e5253), the sheet included an email 30 hours old. On main this happened in 1 of 4 runs. The live checks passed those runs, and so did the claims grader (see "A gap in the checks" below). The other three setups never included that email, before or after.
Cause. The first head also told a delegating Bot to end its brief at what to check, with no line on how to report. Claude then wrote shorter briefs that dropped the work's own checks. I replayed the delegating step of four captured Claude SDK runs 6 times each (24 briefs per wording) and counted briefs that keep a rolling 24-hour window instead of a calendar day ("since 2026-11-15"):
Later steps were replayed 6 times per capture on main's wording and the PR's. When the brief is held the same, the wording makes no difference:
So the brief is what decides it.
Fix (now on main ea7f43a as 1914f40). The delegate line is gone. A test now fails if it comes back: red on the first head, green here. The report is already shaped after the brief, whatever the brief asks it to include. The offline report lengths below were measured on briefs written without that line.
Live on this head (95a49e2), multi-step case, 3 runs each:
The two Claude Agent SDK runs that missed:
Counted side by side: main b825ff5 vs this head, multi-step case, live.
{"text": …}with a field the schema refuses; another sent{"channel": "sales"}with no text. In both, the corrected send asked for approval again.After the last-day fix (#1730). This PR was rebased onto main 01e5460, which includes #1730, then onto ea7f43a (a test-only change). The Claude Agent SDK multi-step case then ran live 6 times each:
Each side had the same one claims miss: the report said the excluded Wingtip email was "about 25 hours" old when it was 30. One PR run had 140 words.
Why
Claude Sonnet 5.5 reported a three-app job (go through the inbox, build a sheet, post a summary in #sales) in 166 to 252 words. That happened on OpenBot's own runner and on the Claude Agent SDK alike. GPT-5.6 Terra and LangGraph took 38 to 112 words under the same rules.
Claude repeated the sheet's rows, said it had read the sheet back, listed what it left out and why, and gave each inference its own bullet. It follows the task's brief, which comes last and often asks for "anything you could not confirm", over a rule that comes before it.
What changed
readBrieftakes the new text off, so the task thread still shows the brief alone. A thread opened before this change still reads back.Tests
New and changed tests in
shared/bot-prompt.test.tsandshared/tests/orchestration.test.ts:shared: 392 pass.agent-claude-sdk/tests: 137 pass.app/tests/task-worked-for.test.tsx: 4 pass (the task thread shows the brief alone).Measured
Offline. I replayed each case's captured final task step on the real model with the new wording, 6 answers per case. Each answer was graded by the same claims grader the live runs use (Claude Opus 5.5).
Wordings that did not get Claude under 120:
Live, at the first head (a7e5253, with the delegate line). A real stack with local CopilotKit Intelligence 0.16.0-rc.13 and real models, graded by the cases' code checks and the Claude grader. Main first, then the PR.
Multi-step case ("go through my inbox, build a sheet, post in #sales"):
Six cases, one run each:
Notes on the live misses:
A gap in the checks
The multi-step case's "did the task" check (
one "Customer follow-ups" sheet with the three customers in it) tests that Contoso, Fabrikam and Tailspin are present and the internal HR email is not. It does not test that an email outside the last day stays out. The claims grader judges what the Bot says, not the window. So a sheet holding the 30-hour-old Wingtip email passes both. The check should fail a sheet that holds Wingtip.