Skip to content

evaluation-adapters: classifyEntrypointResult drops stdout diagnostics on blocked (issue #147) - #148

Open
ruvnet wants to merge 9 commits into
mainfrom
dream/2026-10-02-evaluation-adapters
Open

ruvnet wants to merge 9 commits into
mainfrom
dream/2026-10-02-evaluation-adapters

Conversation

@ruvnet

@ruvnet ruvnet commented Oct 2, 2026

Copy link
Copy Markdown
Owner

Nightly Dream Cycle, 2026-10-02. DEEP=evaluation-adapters, SCAN=flywheel,darwin. Full report: docs/dream-cycle/2026-10-02-evaluation-adapters-report.md. Issue: #147.

Hypothesis (frozen before implementation)

Given classifyEntrypointResult() receives an ExecResult from a real evaluator-entrypoint failure where the underlying tool (Node's node --test, the governance half of this repo's own bench entrypoint) writes its failure diagnostics to stdout and leaves stderr empty, then falling back to r.stdout (clearly labeled) when r.stderr is empty on a nonzero exit should surface that diagnostic instead of the current "exited N with no stderr", subject to: a failure with real stderr content must produce a byte-identical reason to today (stdout is never consulted when stderr is non-empty); a failure with both streams empty still produces a clear, distinct generic message; the existing stale-state carve-out (which only inspects stderr) is unchanged and runs first; npm test stays green with 0 regressions. Not modified after evaluation began.

Candidate

packages/cli/src/entrypoint.ts (+19/−1) and packages/cli/src/entrypoint.test.ts (+34 lines, 4 new tests). One conceptual change: a stdout fallback in classifyEntrypointResult's blocked branch, gated strictly behind stderr being empty.

// after:
if (!stderr) {
  const stdout = r.stdout.trim();
  if (stdout) {
    return { verdict: 'blocked', code: r.code, reason: `(stdout) ${stdout}` };
  }
}
return {
  verdict: 'blocked',
  code: r.code,
  reason: stderr || `exited ${r.code} with no output on either stream`,
};

Distinct from open PR #137 (bin.ts's execFile catch-block for a spawn failure, e.g. ENOENT — its fallback only fires when err.code is non-numeric, inert for a normal nonzero exit). Read #137's full diff first to confirm no overlap.

Evaluation Receipt

Real evaluator: npm test (this repo's own bench entrypoint).

Baseline (main@23a5770) Candidate
vitest 769 773 (+4 new, 0 regressions)
governance 150 150 (unchanged)
Total 919 923
npm run typecheck clean clean
npm run lint clean clean

Live pre/post-fix proof through the real built CLI:

$ node packages/cli/dist/bin.js verify-entrypoint stream-probe --cmd "node --test <failing-file>"
# BASELINE (pre-fix):
stream-probe: blocked (exit 1) — exited 1 with no stderr
# CANDIDATE (post-fix):
stream-probe: blocked (exit 1) — (stdout) TAP version 13
not ok 1 - intentionally failing to observe which stream node:test writes to
  ...
  error: 'STREAM_PROBE_MARKER\n\n1 !== 2\n'

Stream-separation proof backing the hypothesis: a throwaway failing node --test file produced 1040 stdout bytes / 0 stderr bytes; npm run test:governance on this repo's own real governance suite produced 27057 stdout bytes / 0 stderr bytes.

Darwin entrypoint: not exercised live tonight — this session's own execution-permission policy denied npx @metaharness/darwin ... outright before completion. Recorded as DARWIN_LIVE_CHECK=blocked (sandbox policy), not worked around. LLM_EVAL was not blocked (OPENROUTER_API_KEY present, unused by this candidate).

Baseline

Parent commit 23a577082aad39750051dc7ef0ec9d0d72900adf (main). Initial npm test on this session's shallow checkout showed 3 unrelated failures in scripts/benchmark-ticket-codec.test.mjs (pinned-commit git show on a path absent from that commit); confirmed environmental (shallow clone — both ci.yml and dream-nightly.yml already set fetch-depth: 0 for this exact "pinned historical codec differential oracle"), fixed with git fetch --unshallow, then 919/919 green.

Darwin Lineage

DARWIN=not-applicable — a single-branch fallback in one classifier function has no evolvable population to fitness-search, same precedent as every prior single-conceptual-change night on this surface (#65, #91, #116, #132, #137).

Evidence

OBSERVATION (classifyEntrypointResult's blocked branch reads only r.stderr) → OBSERVATION (live repro: failing node --test → 1040/0 stdout/stderr bytes) → OBSERVATION (same pattern on real npm run test:governance: 27057/0) → MEASUREMENT (pre/post real-CLI repro) → MEASUREMENT (919→923 tests, 0 regressions) → INFERENCE (independent critic, separate context) → DECISION (critic CLEAR, one disclosed note) → DECISION (verdict ACCEPT).

Reward-Hack Check

Independent adversarial-critic subagent (separate context): CLEAR. Verified via git diff HEAD that the change is purely additive — one new conditional block in entrypoint.ts, four new tests in entrypoint.test.ts, zero existing tests modified/weakened/removed. The fix checks r.stderr.trim() truthiness generically, not special-cased to any fixture string; one new test actually strengthens the contract (stdout must NOT leak into reason when stderr is present). Confirmed the only two call sites (index.ts) just log check.reason, no hidden string-matching coupling elsewhere in the repo. No benchmark/gold data touched; no hidden cost; no undocumented cache; no threshold changed.

Security Review

classifyEntrypointResult never executes anything — it only classifies an already-completed ExecResult; no new network/filesystem/credential surface. Disclosed, non-blocking: this modestly widens what can appear in a blocked reason — previously only stderr content, now stdout too when stderr is empty, with no truncation. Mirrors the trust boundary already applied to stderr (same consumer, same lack of a size cap) — a widening of an existing boundary, not a new exposure class. No evidence either configured entrypoint (bench, darwin) emits secrets to stdout on failure.

Regression Analysis

0 pre-existing tests modified or removed. All 919 baseline tests pass unchanged; 4 new vitest tests added. npm run lint/npm run typecheck clean. npm run build clean, no wasm/NAPI degradation.

ADR

None — a diagnostic-completeness fix to an existing classifier's blocked branch, not a new architectural decision or repo-wide invariant.

Gist

GIST=LOCAL — no gh CLI gist capability (gh auth status reports an invalid GH_TOKEN) or MCP gist-creation tool available this session. Full report committed at docs/dream-cycle/2026-10-02-evaluation-adapters-report.md.

Issue

#147

Witness

report_sha256 : 643ad95981d507ff636d8a14a0525dc59dcff80158f8d4a722414783cdd07272
session_commit: 23a577082aad39750051dc7ef0ec9d0d72900adf
witness       : 29c998e2cef84d47a01469e871f3322d3dc5d7c0622128548b3f276b13c0a57b

Reproduce: node packages/cli/dist/bin.js witness verify-report docs/dream-cycle/2026-10-02-evaluation-adapters-report.md --commit 23a577082aad39750051dc7ef0ec9d0d72900adf

Merge Policy

Human review required. This session never self-merges and never autonomously promotes candidate state. dream.config.json's autoMerge: false; no automerge-safe label applied.


🤖 Generated with Claude Code

https://claude.ai/code/session_01TAHRTU6Nrq1aPvuS2eBoK1


Generated by Claude Code

claude added 2 commits October 2, 2026 09:27
…s on a nonzero exit with empty stderr

bench's own governance half (`node --test`) writes its entire TAP report,
failures included, to stdout and leaves stderr empty — live-confirmed with a
throwaway failing test file (1040 stdout bytes, 0 stderr bytes) and against
the real `npm run test:governance` (27057/0). classifyEntrypointResult's
`blocked` branch only ever read stderr, so the one failure mode it exists to
surface clearly reported the least informative message available ("exited N
with no stderr") while a full diagnostic sat one field away. Falls back to
stdout, clearly labeled, only when stderr is empty; stderr-bearing failures
are byte-identical to before. Distinct from open PR #137 (bin.ts's execFile
spawn-failure catch-block, inert for a normal nonzero exit).

npm test 919->923 (+4 new, 0 regressions). Independent critic: CLEAR.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAHRTU6Nrq1aPvuS2eBoK1
Issue #147, PR #148, verdict ACCEPT, witness 29c998e2...

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAHRTU6Nrq1aPvuS2eBoK1
…6-10-02-evaluation-adapters

# Conflicts:
#	docs/dream-cycle/LEDGER.md
MODE=CONSOLIDATE (20 open dream/* candidate PRs, well over
implementationTrack.reviewBacklogLimit=6). Merged main@3fc3222 into this
branch (merge commit, resolving one LEDGER.md conflict by keeping both
sides' rows), re-ran the real evaluator on the merged head (vitest
886->890, governance 195 unchanged, 0 regressions), re-ran an independent
critic and reward-hack check on the unchanged candidate diff (CLEAR), and
refreshed the report's Evaluation Receipt and Witness section accordingly.
Verdict ACCEPT still holds.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012E88SNbhsvrWBNezM4ErZK

ruvnet commented Oct 23, 2026

Copy link
Copy Markdown
Owner Author

Consolidation refresh — 2026-10-23 (MODE=CONSOLIDATE)

20 open dream/* candidate PRs tonight, well over implementationTrack.reviewBacklogLimit=6. This was the oldest eligible draft (verdict ACCEPT in its own Evaluation Receipt, no prior dream-consolidate marker for today's SHA).

Done: merged main@3fc3222d2e5566a203331d3686c4d2eaf06713a8 into this branch (merge commit 5d29d638, no rebase, no force-push; one conflict in LEDGER.md, resolved by keeping both sides' rows). Re-ran the real evaluator on the new head: npm test 886→890 vitest (+4, this candidate's own tests, 0 regressions), governance 195 unchanged; typecheck/lint/check:edge-contracts all clean. Re-ran an independent critic + reward-hack pass on the diff (byte-identical to the one originally reviewed — no competing changes landed on main for entrypoint.ts/entrypoint.test.ts since baseline): CLEAR. Refreshed the Evaluation Receipt and Witness in docs/dream-cycle/2026-10-02-evaluation-adapters-report.md (new witness f0fe18ae…, verified self-contained via dream-machine witness verify-report). Verdict ACCEPT still holds.

Not done — staying draft. The dependency-audit check (npm audit --audit-level=high) fails on this head. This is not this PR's failure: package-lock.json on this branch is byte-identical to main's (git diff origin/main...HEAD -- package-lock.json is empty), and running npm audit --audit-level=high locally against that exact unmodified lockfile reproduces the identical two findings (@vitest/mocker moderate, vite high — both transitive npm-registry advisories, not a lockfile change on this branch). It's the same class of pre-existing, already-tracked dev-toolchain audit drift as issue #37 (open, unresolved, no fix PR exists yet to port — remediation there is a semver-breaking vitest/vite major bump, out of scope for this single-classifier-function diagnostic fix). Not re-running the check: this isn't a flake (reproduced deterministically, twice, against an unmodified lockfile), so a re-run would just fail the same way. Per policy this PR never self-merges and never gets automerge-safe; leaving it draft rather than ready-for-review since not every required check on the head is green, even though the gate the Dream Cycle itself controls (evaluation, critic, reward-hack, witness) all clear.


Generated by Claude Code

STEP 25. MODE=CONSOLIDATE night; row records the refresh of PR #148
(merge main forward, re-run evaluator + critic, verdict ACCEPT still
holds) and why it stayed draft (dependency-audit fails on a pre-existing,
registry-side devDependency advisory unrelated to this candidate, tracked
in issue #37).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012E88SNbhsvrWBNezM4ErZK
Intent/result receipts for tonight's external side effects (2 pushes,
1 PR comment, 1 issue comment) per the schedule contract (ruvnet/core-memory
AGENTS.md rule 5, issue #187).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012E88SNbhsvrWBNezM4ErZK
…6-11-08)

# Conflicts:
#	docs/dream-cycle/LEDGER.md
Merge main@71d607b forward (10 commits, 1 LEDGER.md conflict resolved
additively); re-ran real evaluator (vitest 908->912 +4 own, governance
197 unchanged, 0 regressions); fresh independent critic + reward-hack
pass CLEAR; dependency-audit gate now passes (resolved by #205, no
longer blocking); darwin ran live (bounded mock sandbox, generic
control-plane check); refreshed Evaluation Receipt and Witness.
@ruvnet
ruvnet marked this pull request as ready for review November 8, 2026 09:28

ruvnet commented Nov 8, 2026

Copy link
Copy Markdown
Owner Author

Consolidation refresh — 2026-11-08 (MODE=CONSOLIDATE)

18 open dream/* candidate PRs tonight (#98,#101,#137,#141,#145,#148,#152,#154,#156,#161,#180,#182,#184,#189,#192,#199,#202,#204), well over implementationTrack.reviewBacklogLimit=6. #145/#156 are already ready-for-review (non-draft) and #199/#202/#204 are routine-self-review PRs — both excluded from eligibility, same as #98 (compound multi-verdict portfolio receipt, not a single PR-level ACCEPT) and #101 (incompatible legacy ledger-schema conflict, per last night's #145 writeup). This PR (#148) was again the oldest eligible draft: verdict ACCEPT in its own Evaluation Receipt, not self-review, no dream-consolidate marker for today's default-branch SHA (the prior marker, from 2026-10-23, named 3fc3222…, which main has since moved past).

Done: merged main@71d607bea6608180b8c41980dc5386526d3f1f16 into this branch (merge commit 3bd4c4fd9cf3817974e97f544aafea60c7aaccaa, no rebase, no force-push; one conflict in LEDGER.md, resolved by taking the union of both sides' rows, de-duplicated by date, chronological order — dream-machine ledger verify --since-row 38 reports ✓ ledger OK — 60 rows). packages/cli/src/entrypoint.ts/entrypoint.test.ts (this candidate's only files) received zero competing changes on main since the last refresh — the diff is still byte-identical to the one originally reviewed on 2026-10-02.

Re-ran the real evaluator on the new head: npm test 908→912 vitest (+4, this candidate's own tests, 0 regressions), governance 197 unchanged; typecheck/lint/check:edge-contracts/check:development-policy all clean. Ran a fresh independent critic (separate subagent, no prior context, instructed to try to kill the candidate) + reward-hack pass on the diff: CLEAR (full checklist and two non-blocking disclosures in the refreshed report, docs/dream-cycle/2026-10-02-evaluation-adapters-report.md). Darwin ran live tonight, bounded mock sandbox: Winner: g2_v5 · Delta over baseline: +0.110, exit 0 — generic control-plane check, DARWIN=not-applicable to this single-conceptual-change candidate (unchanged precedent). Refreshed the Evaluation Receipt and Witness (new witness 8c431495…, self-verified via dream-machine witness verify-report). Verdict ACCEPT still holds.

The dependency-audit blocker from the two prior consolidation rounds (2026-10-23/10-24/10-28) is resolved. npm audit --audit-level=high — the exact command ci.yml's dependency-audit job runs — now exits 0 on this head (10 moderate-only dev-toolchain findings remain, 0 high/critical; cleared by #205's vite 8.3.6/vitest 5.1.0 bump, already on main, confirmed via issue #197). npm audit --omit=dev and the production-scoped audit-gate both report 0 findings at any severity.

CI on the pushed head (3bd4c4f). All 9 checks green: guard, test (22), test (24), no-optional-deps, dependency-security, dependency-audit, software-evidence, CodeQL/analyze. analyze (CodeQL) failed on the first run with a transient HTTP 504 from GitHub's own CodeQL-bundle CDN during codeql-action/init (before any code analysis ran — confirmed via job logs, three retries, all 504, nothing candidate-related); re-ran just that job (rerun_failed_jobs) and it passed clean on retry. This is the first night this exact PR has had every required check green.

Decision. evaluation_complete ∧ effect_positive ∧ significance_sufficient ∧ no_material_regression ∧ tests_green ∧ reward_hack_clear ∧ critic_clear ∧ witness_valid ∧ receipt_reproducible all hold, and every required check on the exact pushed head is green → marking ready for review. This is not approval — human review required, this session never self-merges and never applies automerge-safe. Ledger row appended: docs/dream-cycle/LEDGER.md (2026-11-08).

No merge, no close, no force-push, no self-promotion.


Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants