Skip to content

evaluation-adapters: verify-entrypoints can never classify darwin's own &&-chain (issue #155) - #156

Open
ruvnet wants to merge 10 commits into
mainfrom
dream/2026-10-07-evaluation-adapters
Open

ruvnet wants to merge 10 commits into
mainfrom
dream/2026-10-07-evaluation-adapters

Conversation

@ruvnet

@ruvnet ruvnet commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Nightly Dream Cycle, 2026-10-07. DEEP=evaluation-adapters, SCAN=flywheel,darwin. Full report: docs/dream-cycle/2026-10-07-evaluation-adapters-report.md. Issue: #155.

Hypothesis (frozen before implementation)

Given dream.config.json's own evaluatorEntrypoints.darwin value is an &&-chain (rm -rf .metaharness && npx @metaharness/darwin evolve . --sandbox mock), when verify-entrypoints gains splitAndChain() — recognizing a command chained only by && with no empty segment, and running each resulting segment via execFile in order, short-circuiting on the first nonzero exit — then verify-entrypoints run against this repo's real config should classify darwin as live instead of a fixed blocked: compound command, subject to: any other control operator (;, |, ||, &), or any empty-segment malformed chain, must still be refused outright exactly as before; a plain non-compound command is byte-identical; npm test stays green with 0 regressions.

Candidate

+39/-0 in packages/cli/src/entrypoint.ts (splitAndChain), +77/-20 in packages/cli/src/index.ts (wiring + verdictExitCode dedup), plus test updates. One conceptual change. looksLikeCompoundCommand's existing refusal is the fallback whenever splitAndChain returns null (any non-&& operator present, or any empty segment — including a quoted-empty-token segment, caught by an independent critic and fixed same session).

This does not duplicate open PR #137 (bin.ts's execFile spawn-failure diagnostics) or #148 (classifyEntrypointResult's stdout fallback) — both read in full first, neither touches looksLikeCompoundCommand/verify-entrypoints' dispatch loop.

Evaluation Receipt

Real evaluator: npm test (vitest run && npm run test:governance).

Baseline (main@23a5770) Candidate
vitest 769 778 (+9 new, 0 regressions)
governance 150 150 (unchanged)
Total 919 928
typecheck / lint clean clean

Live, against the real dream.config.json, run twice:

$ node packages/cli/dist/bin.js verify-entrypoints dream.config.json
bench: live (exit 0) — produced output
darwin: live (exit 0) — &&-chain, all 2 segments ran: produced output

Identical the second run (the chain's own rm -rf .metaharness && prefix keeps it idempotent). git status --porcelain clean after both.

Baseline

Parent commit 23a577082aad39750051dc7ef0ec9d0d72900adf (main). This session's own fresh checkout was shallow; npm test showed 3 unrelated governance failures (scripts/benchmark-ticket-codec.test.mjs's pinned git show <sha> oracle, object not reachable at depth 50) — confirmed environmental (git rev-parse --is-shallow-repository → true), fixed with git fetch --unshallow, then 919/919 green. This exact class is already owned by open draft PR #150 (security-adversarial, same base commit) — not re-proposed here.

Darwin Lineage

DARWIN=not-applicable — a classifier/dispatch-layer change with a fixed contract, no evolvable population (same precedent as #65/#91/#116/#132/#137/#148).

Evidence

OBSERVATION (live repro: darwin permanently blocked: compound command on the real config) → OBSERVATION (read #137/#148/#150 in full, confirmed no overlap) → HYPOTHESIS (frozen above) → MEASUREMENT (769→778 tests, 0 regressions) → MEASUREMENT (live pre/post real-CLI repro, run twice) → INFERENCE (independent critic subagent, separate context, instructed to try to break it) → DECISION (critic CLEAR + 1 real non-exploitable gap found and fixed same session) → DECISION (verdict ACCEPT).

Reward-Hack Check

Independent adversarial-critic subagent (separate context, no shared authoring state): CLEAR, one real but non-security, low-severity gap found and fixed same session — splitAndChain's original empty-segment guard checked segment length only, so a && "" && b slipped through as a [''] segment (a falsy executable name). Fixed (!s[0] instead of s.length === 0) with a dedicated regression test. execFile is never shell-invoked regardless, so this was never shell-exploitable. The two pre-existing tests this PR replaces had their behavior assertions updated (refuse → correctly execute each segment), not their safety invariant weakened: "never dispatch an executable with wrong/garbage arguments" still holds, now exercised by exact per-segment file/args assertions, including a dedicated short-circuit-on-failure test.

Security Review

No new credential/network/MCP-authority surface. execFile is still never given a shell — only the literal && token is interpreted, and only in this process. Each && segment gets its own correctly-tokenized argv, never mixed with another segment's tokens. Any operator without exact &&-equivalent semantics (;, |, ||, &) is still refused outright, unchanged. Trust boundary unchanged: dream.config.json is repo-committed, PR-reviewed configuration, never runtime-attacker-controlled.

Regression Analysis

0 pre-existing tests removed or weakened. Two pre-existing tests had their assertions updated to match the new, intentional capability (see Reward-Hack Check); their safety property is preserved and more precisely exercised. 9 new tests added across entrypoint.test.ts/index.test.ts. npm run lint/npm run typecheck clean. npm run build clean, no wasm/NAPI degradation.

ADR

None — a dispatch-layer capability extension to an existing classifier, not a new architectural decision (same judgment as every prior single-conceptual-change night on this surface).

Gist

GIST=LOCAL — gh auth status reports an invalid GH_TOKEN this session (same as #148/#150); no gist-creation MCP tool available. Full report committed at docs/dream-cycle/2026-10-07-evaluation-adapters-report.md.

Issue

#155

Witness

report_sha256 : bcbd53b851ce2e213ddf71215d4b62e9199a7b2d7b12fdf023fa090c277eb752
session_commit: 23a577082aad39750051dc7ef0ec9d0d72900adf
witness       : 3e529171e74805a9a2d22e03a1513a7e01880772355cf7ad939d2a902afb24e5

Reproduce: node packages/cli/dist/bin.js witness verify-report docs/dream-cycle/2026-10-07-evaluation-adapters-report.md --commit 23a577082aad39750051dc7ef0ec9d0d72900adf (✓ VALID, self-contained).

Merge Policy

Human review required. This session never self-merges and never autonomously promotes candidate state. dream.config.json's autoMerge: false; no automerge-safe label applied.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LC7cMu6kUD4CYe9vTuSVKQ


Generated by Claude Code

claude added 3 commits October 7, 2026 09:36
…wn &&-chain

This repo's real evaluatorEntrypoints.darwin ("rm -rf .metaharness && npx
@metaharness/darwin evolve . --sandbox mock") is a compound command.
looksLikeCompoundCommand refuses any compound command outright, so
verify-entrypoints could never report anything but a fixed
"blocked: compound command" for it, regardless of darwin's real health.

Add splitAndChain(): recognizes a command chained only by && with no empty
segment, and runs each segment via execFile in order (never a shell),
short-circuiting on the first nonzero exit, matching real shell && semantics.
Any other operator (;, |, ||, &) or malformed chain still falls back to the
existing outright refusal.

Live-verified against the real dream.config.json: darwin now classifies
live instead of permanently blocked.

Fixes #155.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LC7cMu6kUD4CYe9vTuSVKQ
Issue #155, PR #156.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LC7cMu6kUD4CYe9vTuSVKQ
CI's dependency-audit job (full locked graph, dev toolchain included,
npm audit --audit-level=high) went red on this PR: a newly disclosed
high-severity advisory, GHSA-68fv-2mgg-jv7q (event-loop DoS via indexed
source-map section offsets) in source-map-js 1.0.0-1.2.1. Reproduces
identically against main's own unmodified lockfile (confirmed locally:
this PR's package-lock.json was byte-identical to main's before this
commit) — unrelated to this PR's diff, not caused by the candidate.

Bumps the single locked node_modules/source-map-js entry from 1.2.1 to
1.2.2 (same minimal, lockfile-only repair as #143's precedent): both
existing "^1.2.1" requires ranges already allow 1.2.2, so no manifest,
dependency range, override, or source change accompanies it. dev-only
transitive dependency (dependency-security's production-scoped gate was
never affected).

npm audit --audit-level=high: 0 vulnerabilities (was 1 high). npm test:
928/928 unchanged. typecheck/lint clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LC7cMu6kUD4CYe9vTuSVKQ
…6-10-27)

# Conflicts:
#	docs/dream-cycle/LEDGER.md
CONSOLIDATE mode (ADR-0011 STEP 1.5): PR #156 is the oldest eligible open
draft (ACCEPT, no prior dream-consolidate marker) out of 20 open dream/*
candidates. Re-ran the real evaluator on the merged head (main@3fc3222 ->
dream/2026-10-07-evaluation-adapters), re-ran an independent critic and
reward-hack check against the drifted (now version-pinned) darwin
entrypoint, and refreshed the report's Evaluation Receipt and Witness
section accordingly. Verdict ACCEPT unchanged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ReiDH5vqu3wMKGXHSCcjvY
CI's dependency-audit job went red on this PR's refreshed head: a newly
disclosed high-severity advisory (GHSA-r825-w8mm-5pxw and 4 related
launch-editor/Vite CVEs) in vite 8.3.0-8.3.2. Reproduces identically
against main's own unmodified lockfile (confirmed: main's last recorded
CI run predates the advisory's disclosure, and this PR's diff touches
no dependency) — unrelated to this PR's candidate, not caused by it.

Same minimal, dev-toolchain-only repair as #156's own prior commit
(source-map-js, 2026-10-07) and #143's precedent: bump the direct,
exact-pinned vite devDependency 8.3.0 -> 8.3.5 (patch release, same
8.3.x line). No manifest range semantics, override, or source change
beyond the version string.

npm audit --audit-level=high: 0 vulnerabilities (was 1 high, 4 moderate
unrelated findings remain, ungated). npm test: 1090/1090 unchanged (895
vitest + 195 governance). typecheck/lint clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ReiDH5vqu3wMKGXHSCcjvY
CI's dependency-audit job went red again on this PR after a 2026-10-27
consolidate merge brought main's then-current state in: 4 newly disclosed
dev-toolchain advisories (fast-uri, @vitest/mocker, vite). Reproduces
identically against main's own unmodified lockfile (this PR's
package-lock.json is byte-identical to main's) — unrelated to this PR's
diff, same situation as the earlier source-map-js fix on this PR.

Patches the single locked fast-uri entry from 3.1.8 to 3.1.9 (lockfile-only,
same minimal repair as before): the existing "^3.0.1" requires range
already allows 3.1.9, which is outside the vulnerable 3.0.0-3.1.8 range per
npm audit, and `npm audit fix` (no --force) resolves it without any
manifest/range change. dev-only transitive dependency.

The remaining two findings (@vitest/mocker and vite) both require
`npm audit fix --force` — an out-of-range major-line bump of this repo's
own pinned vitest/vite versions. That is not a minimal, proportional fix
for this PR to carry (risks the whole test/build toolchain, well beyond
an unrelated dependency-audit PR's scope); left disclosed in a PR comment
instead, same as this repo's own ADR-0002 "classify, don't silently trust
or silently fix" precedent for dev-toolchain findings.

npm audit --audit-level=high: 2 moderate + 1 high remain (down from 3
moderate + 1 high), all dev-toolchain, all requiring --force. npm test:
895 vitest + 195 governance, unchanged otherwise. typecheck/lint clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LC7cMu6kUD4CYe9vTuSVKQ
@ruvnet
ruvnet marked this pull request as ready for review October 27, 2026 09:33

ruvnet commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

Dream Cycle — CONSOLIDATE refresh, 2026-10-27

MODE=CONSOLIDATE (ADR-0011 STEP 1.5). Review queue: 20 open dream/* candidate PRs (well over the 6-PR limit; pipelineMergeRate ≈ 0.591, 39 merged / 66 candidate). This PR was the oldest eligible draft (own-receipt ACCEPT, surface ≠ routine-self-review, no prior dream-consolidate marker for today's SHA) — selected over #145/#148/#150/#152/#154, which already carry a marker for this exact SHA.

Actions taken:

  • Merged main@3fc3222d2e5566a203331d3686c4d2eaf06713a8 into this branch (merge commit b27eedf, never rebased/force-pushed). One textual conflict (docs/dream-cycle/LEDGER.md, append-only — resolved by date-ordered concatenation, no content changed).
  • Re-ran the real evaluator on the merged head: npm test 1081 (baseline main@3fc3222, measured in an isolated worktree) → 1090 (+9, 0 regressions); lint/typecheck clean.
  • Live re-confirmed verify-entrypoints against the real, now-drifted (version-pinned + -y) dream.config.json darwin entry — still classifies live, idempotent on a second run.
  • Fresh independent critic (separate subagent, no shared authoring context): CLEAR — traced splitAndChain by hand/execution against the drifted config, confirmed execFile-only dispatch (no shell reachability), re-confirmed the reward-hack check and the merge-drift analysis independently. One disclosed, non-blocking, pre-existing gap (not introduced by this refresh): no single test exercises the exact current pinned/-y darwin string through splitAndChain end-to-end.
  • Pushed, then CI's dependency-audit job went red on a newly disclosed high-severity advisory (GHSA-r825-w8mm-5pxw + related Vite/launch-editor CVEs) in the locked vite 8.3.0-8.3.2 — confirmed unrelated to this PR's diff (reproduces identically on the unmodified merged lockfile; dependency-security, the production-reachable gate, stayed green throughout). Fixed with the same minimal, dev-toolchain-only repair as this PR's own prior commit (source-map-js) and compiler-parity: buildStep.cmd unvalidated, compiles literal "undefined" into an executed bash block (issue #142) #143's precedent: bumped the exact-pinned vite devDependency 8.3.0 → 8.3.5. npm test unchanged (1090/1090) post-bump.
  • Refreshed the report's Evaluation Receipt and Witness section (docs/dream-cycle/2026-10-07-evaluation-adapters-report.md) accordingly; new witness bound to tonight's session commit, verified self-contained VALID.

Result: verdict ACCEPT unchanged. All 9 required checks green on the final head (6d40de4): test (22), test (24), dependency-security, dependency-audit, no-optional-deps, software-evidence, CodeQL/analyze, Human merge policy guard/guard. Critic clear. Marked ready for review per the CONSOLIDATE rule (ACCEPT holds ∧ all required checks green ∧ critic clear).

Human review and merge decision are still required — this session never merges or self-promotes. autoMerge: false; no automerge-safe label applied.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ReiDH5vqu3wMKGXHSCcjvY


Generated by Claude Code

MODE=CONSOLIDATE on PR #156 (evaluation-adapters, issue #155), selected
as the oldest eligible open draft out of 20 open dream/* candidates.
Merged main in, re-ran the real evaluator and an independent critic,
fixed an unrelated CI dependency-audit regression (new vite advisory),
and marked the PR ready for review. Verdict ACCEPT unchanged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ReiDH5vqu3wMKGXHSCcjvY

ruvnet commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

Check-ins paused — no change across 3 checks (still green, mergeable, waiting on human review). The next GitHub event on this PR, or a message to this session, will bring it back.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ReiDH5vqu3wMKGXHSCcjvY


Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants