Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
549 changes: 549 additions & 0 deletions docs/dream-cycle/2026-10-02-evaluation-adapters-report.md

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions docs/dream-cycle/LEDGER.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,13 @@
| 2026-09-22 | evaluation-adapters | classifyEntrypointResult's stale-state regex (/already exists/i) matched any entrypoint's unrelated failure message, not just darwin's collision; narrowed to darwin's exact child-id-already-exists shape | #131 | #132 | yes | ACCEPT | npm test 785->787 (+2 new, 0 regressions); live repro proves false-positive stale-state misclassification fixed; darwin entrypoint live-reconfirmed end-to-end (one invocation, sandbox then blocked repeats) | 648c4e953a82143d9bd662c5c10cc7da708f6b52ccf2953aba505e0369035b9a | 18 open PRs, 0 merged since #84 (2026-09-06); PR #116 (2026-09-17, execFile verify-entrypoints automation) still open/draft/CI-green, awaiting post-fix re-review, untouched by this PR; zeroMergeStreak=true; ledger 15d stale on main before this row |
| 2026-09-28 | security-adversarial | findUnpinnedNpxInvocations only recognized npx/npm exec, leaving pnpm dlx/yarn dlx/bunx unpinned invocations entirely undetected; added unified matchTrigger() covering all 5, reusing existing hardened extraction/pinning pipeline | #138 | #139 | yes | REJECT | npm test 905->915 (765 passed+3 skipped+150 governance), 0 regressions; live pre/post-fix repro confirms bunx/pnpm dlx/yarn dlx now detected, no false positives; independent critic found unresolved global-flag-before-subcommand bypass (pre-existing pattern for npm exec, extended not introduced), disclosed via describe.skip + comment, not fixed -- critic_clear fails, so REJECT not ACCEPT | bb2bc7bd371d0b9901455cf2acd3ee6e145d3fca7a981604a822f42aad25345c | #100 (2026-09-08 automerge-guard) MERGED 2026-09-26 but issue #99 itself still open (human housekeeping gap, not fixed here); #119/#120 (duplicate CI audit-gate job) MERGED, confirmed live in git log at 36954d0; #68 (2026-09-03 unpinned-npx cross-check) still OPEN as an issue, but its recommended isUnpinned exact-semver fix is already live on main; #137 (2026-09-27 evaluation-adapters) remains OPEN/draft; zeroMergeStreak=true is the CLI's unverified worst-case default, not re-litigated a 5th time (ground truth via pull_request_read shows real recent merges); no session merge or self-promotion |
| 2026-09-30 | compiler-parity | buildStep.cmd was completely unvalidated and interpolated directly into an executed bash block in the compiled nightly prompt (live repro: missing cmd compiled the literal command 'undefined'); also degradeOnWasmFailure silently inverted policy on a truthy non-boolean like 'no'; explicitly named as this repo's own highest-severity open compiler-parity gap in PR #127's report (2026-09-20), unclaimed until tonight; added object/type validation mirroring adrConvention/ruosEvaluation | #142 | #143 | yes | ACCEPT | npm test 905->919 (+14 vitest, 150 governance unchanged), 0 regressions; lint/typecheck clean; self-hosted dream.config.json compile output byte-identical; live repro confirms both the 'undefined' bash-block corruption and the degradeOnWasmFailure policy inversion fixed | 6ba8cc25e4e1a5e43f485e1f42d0624b28eee803d26275d173b37e61c3cd9e33 | #105/#111/#127 (same compiler-parity defect-class lineage) confirmed MERGED via git log ground truth despite LEDGER.md file listing them open/draft; LEDGER.md 8 days stale on main (daysSinceLastRow=8) despite 51 PRs merged since the last row (ledger-signals surface, flagged not fixed); zeroMergeStreak=false; 5 open unmerged dream/* PRs (#98,#101,#137,#139,#141), none overlapping this candidate; no session merge or self-promotion |
| 2026-10-02 | evaluation-adapters | classifyEntrypointResult's blocked branch read only stderr, reporting exited N with no stderr even when node --test (bench's governance half) wrote the full failure diagnostic to stdout and left stderr empty; added stdout fallback, strictly scoped to empty-stderr nonzero exits | #147 | #148 | yes | ACCEPT | npm test 919->923 (+4 new, 0 regressions); live pre/post-fix real-CLI repro confirms full TAP diagnostic now surfaced on blocked (was: exited 1 with no stderr); darwin live invocation denied by this session's own sandbox policy before completion | 29c998e2cef84d47a01469e871f3322d3dc5d7c0622128548b3f276b13c0a57b | zeroMergeStreak=false via ground-truth search_pull_requests (61 merged incl. #143,#135,#133,#120,#132,#116), tool default (no --merged) falsely reports true; 6 open dream/* candidate PRs ahead of this one (#145,#141,#139,#137,#101,#98), none overlapping this candidate (#137 touches a different function, bin.ts's execFile spawn-failure path); ledgerStale=true (2d since last row #52/PR#143); no session merge or self-promotion |
| 2026-10-03 | security-adversarial | withReference()'s BASELINE_COMMIT git-show oracle failed opaquely (not self-healing) on a shallow clone, live-reproduced on this session's own checkout; added hasCommit()/tryFetchCommit() self-heal (git fetch --depth 1 origin <fixed-sha> only) + clear error, never making BASELINE_COMMIT configurable | #149 | #150 | yes | ACCEPT | npm test 769+150->769+152 (+2 new), 3 previously-failing governance tests (0 pre-existing) -> 0 fail; typecheck/lint clean; independent critic CLEAR | 3c3dc0728d1b327253b66d0021e80ffd0d5084056644f7f424baf31206ba416f | zeroMergeStreak tool default (true) is a known false negative (merged_at timestamps on #103-#143 show ongoing real merges, latest #143 2026-09-30, matching this ledger's prior row); PR #139 (security-adversarial, unpinned-execution detector widening) remains open, different finding, no overlap; investigated and ruled out ci.yml's dependency-audit job as a #119/#120 regression -- confirmed intentional via git show 36954d0; no session merge or self-promotion |
| 2026-10-08 | security-adversarial | npm <global-flag> exec <pkg> bypassed findUnpinnedNpxInvocations's exact-adjacency check (npm --loglevel=silent/-s/--yes exec pkg undetected); pre-existing gap PR #139 disclosed but left unfixed; added findNpmExecIndex tolerating self-contained flag tokens, disclosed 2 residual edge cases (separate-value flag bypass unfixed, separate-value-exec-literal mislabel found by independent critic) | #157 | #158 | yes | ACCEPT | npm test vitest 769->776 passed +1 disclosed skip, governance 150/150 unchanged, 0 regressions; live pre/post repro confirms 3 bypass cases fixed, 0 false positives on npm run exec-something / npm test / npm ci | eeac6e3d1f9ee2887694582eb4a1b3e3ee2a94c26359e95f3858806e6af1d6a9 | issue #149/PR #150 (shallow-clone BASELINE_COMMIT oracle, same surface) and issue #138/PR #139 (dlx/bunx trigger widening, same surface) both remain open/unreviewed, distinct from tonight's finding; 9 open unmerged dream/* PRs (#137,#139,#141,#145,#148,#150,#152,#154,#156) as of tonight, zeroMergeStreak=true (0 merged since #143 on 2026-09-30, 8 nights); ledgerStale=true carried forward; no session merge or self-promotion |
| 2026-10-10 | compiler-parity | slots[].deep/scan[]/bonusModuli/cron/evaluatorEntrypoints/labels/extraDisciplines/competitors/branchPrefix/ledgerPath/adrConvention.dir values could break out of the compiled prompt's fences/inline-spans, injecting fabricated instructions into this self-hosting system's own authoritative control prompt; named-but-unclaimed since PR #143; closed across the full class via 3 independent-critic rounds (BLOCKED, BLOCKED, CLEAR) plus 1 self-found packageSpec-escaping gap | #164 | #165 | yes | ACCEPT | npm test 769->795 vitest (+26), 159/159 governance unchanged, 0 regressions; lint/typecheck clean; self-hosted dream.config.json compile output byte-identical throughout | faed331494a93b8d038533828f8402a4a8d5eac0b788cf60c916f3cb49191a8c | 15 open dream-cycle/maintenance PRs (#98,#101,#137,#139,#141,#145,#148,#150,#152,#154,#156,#158,#159,#161,#163), 0 overlapping this candidate; LEDGER.md 10d stale on main before this row (ledger-signals-surface finding, out of scope for compiler-parity slot); no session merge or self-promotion |
| 2026-10-11 | ledger-signals | duplicateDirections counted raw row occurrences instead of distinct nights; a single re-appended night (2026-08-28) false-positived as a 3-night repeat; dedupe by (directionKey, date) | #166 | #167 | yes | ACCEPT | npm test 928->929 (+1), 0 regressions; live: real ledger's 2026-08-28 duplicateDirections false positive gone post-fix | 54824cd4528bcc15932c98bd5d97dd2222271f571a1f7fd0ce32a29ddf790401 | zeroMergeStreak=false (36 dream/* PRs confirmed merged via search_pull_requests ground truth, last #143 on 2026-09-30); 14 open unmerged dream/* PRs since then (#98,#101,#137,#139,#141,#145,#148,#150,#152,#154,#156,#158,#161,#165), none overlapping this candidate's file; ledgerStale=true (11d) on main before this row; issue #149/PR #150 (shallow-clone git-show oracle) found already filed, withheld not re-fixed; no session merge or self-promotion |
| 2026-10-12 | evaluation-adapters | evaluateHarnessCandidate's hardReject keyword list omitted 'no declared harness surface'; a zero-surface patch lacking holdout evidence was misclassified INCONCLUSIVE instead of REJECT (structural invalidity masked as insufficient evidence); added the missing keyword | #176 | #177 | yes | ACCEPT | npm test 1001->1004 (+3 vitest, 0 regressions); live repro: zero-surface+no-holdout candidate flips INCONCLUSIVE->REJECT, same score/reasons; lint/typecheck clean | 6fb594bd01e09a8ed25b67833f50601a8b27c772bc202ccce4c84f48b74af9e4 | zeroMergeStreak=false via ground-truth search_pull_requests (38 dream/* PRs merged, most recent #167 on 2026-10-11); #137/#148/#156 (same evaluation-adapters surface) remain open/draft/unreviewed, read in full, no overlap (different files/functions); harness-optimizer.ts (PR #30) has zero call sites anywhere in the live pipeline, flagged as a wiring gap, not fixed tonight; no session merge or self-promotion |
| 2026-10-23 | [consolidate #148] | 20 open dream/* candidate PRs (reviewBacklogLimit=6); consolidated oldest eligible PR #148 (evaluation-adapters, classifyEntrypointResult stdout fallback): merged main@3fc3222 forward (21d/~70 commits, 1 LEDGER.md conflict resolved additively), re-ran real evaluator and independent critic on the unchanged diff | 147 | 148 | yes | ACCEPT | npm test 886->890 vitest (+4 candidate's own, 0 regressions), governance 195 unchanged; typecheck/lint/check:edge-contracts clean; critic+reward-hack CLEAR; verdict ACCEPT still holds; stayed draft (not ready-for-review) because dependency-audit (npm audit --audit-level=high) fails on pre-existing registry-side vitest/vite devDependency advisories reproduced identically against an unmodified lockfile (tracked separately, issue #37), unrelated to this candidate's diff | f0fe18aef3864a2c428f80f2b1a9ee2de370a37ec41f06533f5d5e52be4701e7 | 20 open dream/* candidate PRs (#98,#101,#137,#139,#141,#145,#148,#150,#152,#154,#156,#158,#161,#180,#182,#184,#189,#192,#194,#199), 0 merged since last ledger row (2026-10-12, #177); #145 already carried a same-SHA dream-consolidate marker from 2026-10-21, skipped; no session merge or self-promotion |
| 2026-10-24 | security-adversarial (CONSOLIDATE refresh of 2026-10-03) | CONSOLIDATE: merged main@3fc3222 into draft PR #150 (BASELINE_COMMIT git-show oracle shallow-clone self-heal); re-ran real evaluator and a fresh independent critic/reward-hack pass on the unchanged diff; stays draft (dependency-audit red, pre-existing vite/launch-editor advisory per issue #197, not this PR's) | #149 | #150 | yes | ACCEPT | npm test vitest 886/886 unchanged, governance 195->197 (+2 this candidate's own tests), 0 regressions; typecheck/lint clean; dependency-audit red (pre-existing, unrelated, issue #197); 8/9 required checks green | e4b8f464dac872ad98b32bf1bf9db870099cdfdf5ec657efd2954c1b7d7f8b18 | 20 open dream/* PRs tonight; #145 and #148 (nearer-front eligible drafts) already carry a dream-consolidate marker for today's SHA from the two preceding nights; #150 was the oldest eligible draft with no prior marker; no merge, no close, no force-push, no self-promotion |
| 2026-10-28 | security-adversarial | [consolidate #158] merged main (3fc3222, static since 2026-10-19) into PR #158 branch; re-ran real evaluator on new head (0 regressions); fresh critic CLEAR (independent subagent, own adversarial inputs vs the live compiled function, no counter-example beyond the 2 already-disclosed gaps); darwin ran (bounded mock sandbox, generic control-plane check, not candidate-specific); dependency-audit stays red (pre-existing vite/launch-editor advisory, issue #197, lockfile byte-identical to main); PR stays draft | #157 | #158 | yes | ACCEPT | npm test vitest 886->893+1skip (this candidates own +7/+1-skip), governance 195 unchanged, 0 regressions; typecheck/lint/build clean (8 packages); ledger verify --since-row 38 OK 54 rows; live repro reconfirmed against the new two-arg signature (3 bypass cases fixed, 0 false positives) | de820209193875b8d8fe1777f151891c12a83e817508425446f8348f047bf699 | #145/#148/#150/#152/#154 already carry a dream-consolidate marker for todays unchanged SHA (3fc3222) from 2026-10-21/23/24/26/25; 20 open dream/* PRs tonight, 0 merged since mains last move (#196, 2026-10-19); #139 (older, same surface) is REJECT and excluded; zeroMergeStreak=true; no session merge or self-promotion |
| 2026-11-08 | [consolidate #148] | 18 open dream/* candidate PRs (reviewBacklogLimit=6); consolidated oldest eligible PR #148 (evaluation-adapters, classifyEntrypointResult stdout fallback): merged main@71d607b forward (10 commits, 1 LEDGER.md conflict resolved additively), re-ran real evaluator and a fresh independent critic on the unchanged diff; dependency-audit blocker from the two prior rounds now resolved (#205); CodeQL transient 504 fixed by one job rerun; all 9 CI checks green, PR marked ready for review | 147 | 148 | yes | ACCEPT | npm test 908->912 vitest (+4 candidate's own, 0 regressions), governance 197 unchanged; typecheck/lint/check:edge-contracts/check:development-policy clean; critic+reward-hack CLEAR; darwin live (bounded mock, winner g2_v5, +0.110, not candidate-specific); verdict ACCEPT still holds; dependency-audit now green (resolved by #205); CodeQL recovered after one rerun (transient 504, not candidate-related); 9/9 required checks green -> marked ready for review | 8c431495b857ddc886ca4afcff5ccf714c166304094b8fd34675cb191ce47d29 | 18 open dream/* candidate PRs (#98,#101,#137,#141,#145,#148,#152,#154,#156,#161,#180,#182,#184,#189,#192,#199,#202,#204), 10 merged since last ledger row 2026-10-28 (#139,#150,#158 dream-cycle candidates plus 7 dependency/maintenance PRs); #145 marked ready-for-review last night (2026-11-07 consolidate, same main SHA); #98/#101 remain excluded (compound-verdict portfolio receipt; incompatible legacy ledger schema); no session merge or self-promotion |
8 changes: 8 additions & 0 deletions docs/dream-cycle/receipts/2026-10-23.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{"ts":"2026-10-23T09:19:30Z","kind":"intent","action":"push","target":"ruvnet/dream-machine:dream/2026-10-02-evaluation-adapters","inputs_sha256":"d52ae9db2581cf750fa2bfc56f9fdfb63995c922d8865bc0bff538375ca9693c"}
{"ts":"2026-10-23T09:19:35Z","kind":"result","action":"push","target":"ruvnet/dream-machine:dream/2026-10-02-evaluation-adapters","outcome":"ok","ref":"d21c7f0"}
{"ts":"2026-10-23T09:23:00Z","kind":"intent","action":"pr_comment","target":"ruvnet/dream-machine#148","inputs_sha256":"8a166a06d550a36a69a6b8f5fafe846bc92acf4f222a6571b5866555ecaaaf95"}
{"ts":"2026-10-23T09:23:30Z","kind":"result","action":"pr_comment","target":"ruvnet/dream-machine#148","outcome":"ok","ref":"https://github.com/ruvnet/dream-machine/pull/148#issuecomment-6361743977"}
{"ts":"2026-10-23T09:24:30Z","kind":"intent","action":"push","target":"ruvnet/dream-machine:dream/2026-10-02-evaluation-adapters","inputs_sha256":"9f3bbd8a7ebd116b366862d417be51f03e39023d6a324b7d03e1ae750efa0ae3"}
{"ts":"2026-10-23T09:24:35Z","kind":"result","action":"push","target":"ruvnet/dream-machine:dream/2026-10-02-evaluation-adapters","outcome":"ok","ref":"3d3285c"}
{"ts":"2026-10-23T09:25:00Z","kind":"intent","action":"issue_comment","target":"ruvnet/dream-machine#147","inputs_sha256":"a7c8ebc2a53767f2ad8d465220fae093bc5e749c320f02e7d090f03b796977b9"}
{"ts":"2026-10-23T09:25:05Z","kind":"result","action":"issue_comment","target":"ruvnet/dream-machine#147","outcome":"ok","ref":"https://github.com/ruvnet/dream-machine/issues/147#issuecomment-6361763075"}
9 changes: 9 additions & 0 deletions docs/dream-cycle/receipts/2026-11-08.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{"ts":"2026-11-08T09:24:24Z","kind":"intent","action":"push","target":"dream/2026-10-02-evaluation-adapters","inputs_sha256":"8f3c5c594735a92625383a5fee3d9e9f92d285a2c66b69214724162a697b7e9f"}
{"ts":"2026-11-08T09:24:34Z","kind":"result","action":"push","target":"dream/2026-10-02-evaluation-adapters","outcome":"ok","ref":"3bd4c4f"}
{"ts":"2026-11-08T09:28:05Z","kind":"intent","action":"issue-comment","target":"dream-machine#197","inputs_sha256":"e0f3016b8e971acc09c55abdd71649cc907f299c1539de80e37cefe9067ef6f9"}
{"ts":"2026-11-08T09:28:26Z","kind":"result","action":"issue-comment","target":"dream-machine#197","outcome":"ok","ref":"6719813028"}
{"ts":"2026-11-08T09:28:49Z","kind":"intent","action":"pr-ready-for-review","target":"dream-machine#148","inputs_sha256":"bc4968be636ffbba1452ec073e183ab999b32c78f3612e1708b5611ad9af3c4f"}
{"ts":"2026-11-08T09:29:03Z","kind":"result","action":"pr-ready-for-review","target":"dream-machine#148","outcome":"ok","ref":"3bd4c4f"}
{"ts":"2026-11-08T09:29:13Z","kind":"intent","action":"pr-comment","target":"dream-machine#148","inputs_sha256":"d3427629981549ff15cc092b196a99f4851a48e95310152ddab9b1199117e51b"}
{"ts":"2026-11-08T09:29:42Z","kind":"result","action":"pr-comment","target":"dream-machine#148","outcome":"ok","ref":"6719825183"}
{"ts":"2026-11-08T09:30:21Z","kind":"intent","action":"push","target":"dream/2026-10-02-evaluation-adapters","inputs_sha256":"dd94e60b7025e295c5dc6c109f532182cf4aff316f10e15e461b397dca38c088"}
38 changes: 38 additions & 0 deletions packages/cli/src/entrypoint.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,44 @@ describe('classifyEntrypointResult', () => {
expect(r.verdict).toBe('blocked');
});

it('falls back to stdout when a nonzero exit has empty stderr (reproduces `bench`: node --test writes its entire TAP report, failures included, to stdout, never stderr)', () => {
const r = classifyEntrypointResult({
code: 1,
stdout: "not ok 1 - widget renders\n ---\n error: 'expected 1 to equal 2'\n ...\n1..1\n# fail 1",
stderr: '',
});
expect(r.verdict).toBe('blocked');
expect(r.reason).toContain('not ok 1 - widget renders');
expect(r.reason).toContain('(stdout)');
});

it('prefers stderr over stdout when both are present on a nonzero exit (unchanged precedent)', () => {
const r = classifyEntrypointResult({
code: 1,
stdout: 'some informational progress output',
stderr: 'the real error',
});
expect(r.verdict).toBe('blocked');
expect(r.reason).toBe('the real error');
expect(r.reason).not.toContain('informational progress output');
});

it('reports a clear generic message when a nonzero exit has both streams empty', () => {
const r = classifyEntrypointResult({ code: 1, stdout: '', stderr: '' });
expect(r.verdict).toBe('blocked');
expect(r.reason).toBe('exited 1 with no output on either stream');
});

it('does not let a stale-state-shaped message in stdout alone override the stdout fallback into stale-state (scope: only stderr is checked for stale-state, matching all prior reproductions)', () => {
const r = classifyEntrypointResult({
code: 1,
stdout: 'Error: darwin: autonomous or generated child id already exists: g1_v0',
stderr: '',
});
expect(r.verdict).toBe('blocked');
expect(r.reason).toContain('(stdout)');
});

it('does not flag an unrelated "already exists" failure from a non-darwin entrypoint as stale-state', () => {
// classifyEntrypointResult is shared across every configured evaluatorEntrypoints
// value (bench, flywheel, redblue, darwin, ...) — a real bench-test failure whose
Expand Down
Loading
Loading