Skip to content

RFC-64 M1 6/7: expose bounded runtime coverage evidence - #2016

Open
branarakic wants to merge 14 commits into
codex/rfc64-m1-adaptive-capacityfrom
codex/rfc64-m1-runtime-evidence
Open

RFC-64 M1 6/7: expose bounded runtime coverage evidence#2016
branarakic wants to merge 14 commits into
codex/rfc64-m1-adaptive-capacityfrom
codex/rfc64-m1-runtime-evidence

Conversation

@branarakic

@branarakic branarakic commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

M1 stack

This is PR 6 of 7 in the RFC-64 M1 stack. It is stacked on exact adaptive-capacity head 6992dd36ec23bf298e039ad5fd7a9b517b7a855d and supplies the runtime evidence contract consumed by PR 7. The validated review head is f123856165e168f7d15664d7e07d26badcf9ad81.

Summary

This PR adds bounded, node-admin-only evidence for automatic RFC-64 synchronization work.

It records immutable running and terminal complete or failed entries for:

  • Core automatic public-CG rounds: the explicit-selection snapshot used for evidence, the frozen automatic tail, configured/effective batch sizes, planning lane, one real round job ID, and per-CG completions;
  • Edge periodic reconciler work for startup-rehydrated always-on selections;
  • observed metadata, durable, and shared-memory completion signals.

Operators can poll:

GET /api/diagnostics/sync-coverage-evidence?afterSequence=N

The endpoint requires the node-admin token. Agent-scoped tokens cannot inspect node-wide synchronization work. The exact authorization predicate is shared with the existing node-wide backpressure diagnostics route, while each route retains its own 403 response.

This is observability only. It does not change CG selection, scheduler priority, admission, retry, persistence, single-flight, or sync completion decisions.

User and operator impact

Before this PR, aggregate status and final store state could not prove which CGs a Core scheduler actually admitted, whether work was automatic or explicit, or whether an Edge refresh came from its post-restart always-on reconciler.

After this PR, operators and the release harness can bind exact observations to actual automatic runtime work. The journal is not itself a convergence verdict: PR 7 independently verifies exact VM/SWM heads, inventories, assets, and triples.

Edge behavior is unchanged:

  • on-demand CGs move only after explicit requests;
  • always-on CGs reconcile automatically;
  • unselected CGs are not activated.

Bounds and fail-closed behavior

  • 256 retained entries;
  • at most 32 CG identifiers per entry;
  • at most 256 characters per identifier;
  • actual Node process start from performance.timeOrigin plus a distinct wave ID;
  • nextSequence and droppedBeforeSequence expose overwritten evidence.

Evidence fails closed when IDs are truncated, metadata/durable/SWM is incomplete, SWM reports failure/timeout/denial/backpressure/dropped triples, or the overall operation does not cleanly terminate.

The aggregate compatibility result carries no terminal field, while the production detailed SWM boundary requires the canonical per-CG terminal list. A legacy JavaScript override returning only the aggregate result, malformed terminal entries, or throwing terminal getters leaves automatic evidence unverifiable and failed closed without changing successful sync accounting or retry behavior.

The negative runtime coverage explicitly proves:

  • a non-throwing durable result with complete: false records progress but cannot produce a successful evidence terminal;
  • an empty metadata confirmation set fails only the metadata plane;
  • a partial metadata confirmation set completes only the confirmed CG and fails the unconfirmed CG;
  • a legacy metadata observer returning void, a non-iterable, or a hostile iterable cannot abort synchronization and leaves the metadata plane unverified;
  • stop-policy and continuation-stopped SWM terminals leave the affected CG evidence unverified;
  • an already-syncing automatic admission no-op creates no synthetic journal row;
  • public stop-policy and private swm_recovery detailed terminals retain their exact lane and disposition, and their frozen results cannot be mutated by callers.

Every Core per-CG completion carries the scheduler round's actual job ID, preventing detached or synthetic completion claims.

Before

sequenceDiagram
    participant T as Internal sync trigger
    participant A as DKGAgent
    participant P as Existing planner
    participant S as Existing sync runtime
    participant D as Triple store
    participant O as Node operator

    T->>A: Connection or reconciler event
    A->>P: Plan synchronization work
    P-->>A: Selected and automatic CGs
    A->>S: Execute existing work
    S->>D: Persist verified results
    S-->>A: Aggregate result
    O->>A: Read aggregate status
    Note over O,A: No immutable per-round provenance
Loading

After

sequenceDiagram
    participant T as Internal sync trigger
    participant A as DKGAgent
    participant P as Existing planner
    participant J as Bounded evidence journal
    participant S as Existing sync runtime
    participant D as Triple store
    participant O as Node operator

    T->>A: Explicit module-private automatic command
    A->>P: Plan automatic round
    P-->>A: Evidence snapshot and frozen automatic tail
    A->>J: Start journal-owned running entry
    A->>S: Execute or join existing single-flight
    Note over A,S: Explicit selections remain live during later phases
    S->>D: Persist verified results once
    S-->>A: Aggregate plus mandatory per-CG SWM terminals
    A->>J: Finish through opaque journal handle
    O->>A: GET evidence after sequence N
    A->>J: Read bounded snapshot
    J-->>A: Entries and overwrite boundary
    A-->>O: Node-admin evidence
    Note over A,S: Every joiner receives the same canonical terminals
Loading

Compatibility and risk

  • The existing syncSharedMemoryFromPeerDetailed(peer, ids, options) call shape is unchanged.
  • The broad aggregate SharedMemorySyncResult remains aggregate-only for structural compatibility. The production detailed method returns SharedMemorySyncDetailedResult with mandatory terminals, and one detailed-boundary translator maps canonical scheduler executions to those terminals.
  • Exact settled, deferred, skipped, disabled, and ineligible outcomes are frozen before single-flight fanout, so a reconciler joining a manual call receives the same evidence without duplicate fetching.
  • Runtime terminal and metadata-observer normalization is isolated inside the evidence recorder; malformed host values cannot escape into sync control flow.
  • Untriggered compatibility paths retain the exact established three-argument call shape. Connection-open, peer-update, and periodic automatic paths alone pass the module-private branded fourth token.
  • Planning and evidence transitions are explicit: plan the round, start the recorder once, run sync, then finish. Wrong-finisher calls fail before consuming the live journal entry.
  • Evidence bookkeeping lives in a focused recorder; UUID creation, canonical running state, transitions, and wire rows live in the journal behind opaque frozen handles.
  • Only stable evidence schema, entry, and snapshot types remain in the package root; journal construction, capacity constants, and bounding helpers stay internal.
  • Metadata evidence reuses the existing confirmed set without duplicate store reads.
  • Edge startup provenance is owned by canonical rehydration state and intersects with current selected, subscribed, explicit, always-on state.
  • /api/context-graph/subscriptions gains additive effective syncMode.
  • The journal is intentionally process-local; consumers bind evidence to process start and wave IDs.
  • The shared isNodeAdminCaller helper preserves auth-disabled behavior and fails closed for missing, unknown, and agent-scoped tokens when authentication is enabled.

Validation

  • Exact base: 6992dd36ec23bf298e039ad5fd7a9b517b7a855d.
  • Exact validated head: f123856165e168f7d15664d7e07d26badcf9ad81.
  • Full review-focused Agent lane: 7 files / 149 tests passed.
  • Evidence runtime plus journal subset: 2 files / 26 tests passed (19 + 7).
  • Focused observer and scheduler terminal lane: 3 files / 38 tests passed.
  • CLI evidence plus backpressure authorization: 2 files / 7 tests passed (4 + 3).
  • Agent build passed: TypeScript, public type-contract tests, and package-root validation.
  • CLI build passed: dependency prebuild, TypeScript, public CLI type-contract tests, and runtime-asset assembly.
  • Full CLI dependency closure: 17 workspace packages built successfully, including the EVM no-op compile check, Agent build, and CLI build.
  • Full diff: 21 files, +2,581/-104; no generated or untracked artifacts.
  • Rebase range-diff: 14/14 commits patch-equivalent. The 13 previously published commits moved unchanged onto the corrected parent, and the held hidden-review fix remains the distinct 14th commit.
  • git diff --check: pass.
  • Live GitHub checks and review state remain the merge-readiness source of truth.

Review focus

  1. Evidence is bounded, authorized, truthful, and overwrite-detectable.
  2. Every single-flight joiner receives exact immutable per-CG outcomes.
  3. Automatic provenance remains explicit and module-private inside the trusted process; the node-admin endpoint remains the external authorization boundary.
  4. Journal shapes match the downstream fail-closed verifier contract.

Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
Comment thread packages/agent/src/sync/coverage-evidence-journal.ts Outdated
Comment thread packages/agent/test/sync-coverage-evidence-runtime.test.ts Outdated
Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
Comment thread packages/agent/src/dkg-agent-types.ts Outdated
Comment thread packages/agent/src/index.ts Outdated
Comment thread packages/agent/src/sync/requester/ordered-sync.ts Outdated
Comment thread packages/agent/src/dkg-agent-lifecycle.ts
Comment thread packages/cli/src/daemon/routes/sync-coverage-evidence.ts Outdated
@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-capacity branch 2 times, most recently from 69c1dec to 072249e Compare August 2, 2026 12:29
Comment thread packages/cli/src/daemon/context-graph-catchup-coordinator.ts Outdated
Comment thread packages/agent/src/dkg-agent-types.ts
Comment thread packages/agent/src/sync/backpressure.ts Outdated
Comment thread packages/cli/src/daemon/context-graph-catchup-coordinator.ts Outdated
@branarakic
branarakic force-pushed the codex/rfc64-m1-runtime-evidence branch from fe9d92c to 175f486 Compare August 2, 2026 12:39
Comment thread packages/cli/src/daemon/handle-request.ts
@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-capacity branch from 072249e to b166ef8 Compare August 2, 2026 12:59
Comment thread packages/agent/vitest.unit.config.ts
Comment thread packages/agent/src/dkg-agent.ts
@branarakic
branarakic force-pushed the codex/rfc64-m1-runtime-evidence branch from 175f486 to 0b053ba Compare August 2, 2026 13:26
Comment thread packages/agent/src/sync/coverage-evidence-journal.ts Outdated
Comment thread packages/agent/src/sync/requester/ordered-sync.ts
Comment thread packages/agent/src/dkg-agent-lifecycle.ts Outdated
Comment thread packages/agent/src/sync/requester/ordered-sync.ts
Comment thread packages/agent/src/dkg-agent-base.ts Outdated
Comment thread packages/agent/src/sync/coverage-evidence-journal.ts
Comment thread packages/cli/src/daemon/routes/sync-coverage-evidence.ts Outdated

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: retry_exhausted

@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-capacity branch from 8be9d57 to 13759eb Compare August 2, 2026 14:22
@branarakic
branarakic force-pushed the codex/rfc64-m1-runtime-evidence branch from e7cb778 to 785b434 Compare August 2, 2026 14:28
Comment thread packages/agent/src/sync/coverage-evidence-recorder.ts Outdated
@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-capacity branch from 13759eb to 5793aed Compare August 2, 2026 15:20
@branarakic
branarakic force-pushed the codex/rfc64-m1-runtime-evidence branch from d9a70d4 to e176c85 Compare August 2, 2026 15:26

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Verification reviewer failed: retry_exhausted

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: retry_exhausted

@branarakic
branarakic force-pushed the codex/rfc64-m1-adaptive-capacity branch from 5793aed to 6992dd3 Compare August 2, 2026 15:49

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: retry_exhausted

@branarakic
branarakic force-pushed the codex/rfc64-m1-runtime-evidence branch from e176c85 to f123856 Compare August 2, 2026 15:51

@otReviewAgent otReviewAgent left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational Notice: Review Agent could not complete this review.

Business logic reviewer failed: retry_exhausted

}

async attemptSyncFromPeerWithReconcilerAccounting(
protected async attemptSyncFromPeerWithReconcilerAccounting(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Bug: Changing the reconciler sync method to protected breaks typed callers

What's wrong
The PR changes an existing method on the exported agent class from public to protected. Any TypeScript consumer or subclass-adjacent integration that used this method as part of the agent lifecycle surface will be blocked at compile time by this release.

Example
A typed integration that previously did await agent.attemptSyncFromPeerWithReconcilerAccounting(peer, probe) will now fail to compile because the method is only accessible inside the class hierarchy, even though the runtime method still exists.

Suggested direction
Keep the method public for compatibility and hide only the new automatic trigger token behind an internal overload/helper, or introduce a public wrapper with the old call shape before making the internal implementation protected.

Confidence note
This is a TypeScript API compatibility issue: JavaScript callers can still reach the method at runtime, but typed consumers of the exported DKGAgent surface can no longer call it.

For Agents
Look at LifecycleSyncMethods in packages/agent/src/dkg-agent-lifecycle.ts. Preserve the previous public callable surface, or add a public compatibility wrapper that keeps the old signature while routing internal evidence-triggered calls through the new path. A type-level compatibility test should prove existing public calls still compile.

* enabled, the caller must present a recognized token that is not bound to an
* agent identity; missing, unknown, and agent-scoped tokens all fail closed.
*/
export function isNodeAdminCaller(ctx: NodeAdminAuthContext): boolean {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Node-admin auth was only partially centralized

What's wrong
The new helper is a good direction, but landing it without migrating the existing duplicate gate leaves the daemon with two implementations of the same security boundary. That is architectural drift: future maintainers will not know which one is authoritative.

Example
Changing the node-admin rule later would require finding both isNodeAdminCaller(ctx) users and the local closure in context-graph.ts; missing one route would leave different node-wide diagnostics with different authorization semantics.

Suggested direction
Since this PR creates a canonical helper, route the existing node-wide context-graph subscription gates through it as well. Otherwise this extraction is a partial abstraction that reduces duplication in two routes while leaving a second source of truth in place.

Confidence note
This is a maintainability concern rather than a behavior defect; the duplicated helper may currently be equivalent, but the new helper’s name/comment make it look canonical.

For Agents
Use the new helper consistently across daemon routes that enforce the same node-admin boundary. Start with packages/cli/src/daemon/routes/context-graph.ts and replace the local closure with the shared helper while preserving response bodies and existing agent-scoped denial behavior.

result: Object.freeze({ ...outcome.result } satisfies SharedMemoryContextGraphResult),
};
}
if (outcome.disposition === 'deferred') {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Issue: Deferred shared-memory terminals are not verified at the lifecycle boundary

What's wrong
The new public detailed result is supposed to expose exact per-CG terminal state. The tests validate the generic scheduler can emit a deferred outcome, but they do not prove the lifecycle conversion returns the expected shared-memory terminal entries for a real admission deferral.

Example
For a shared-memory request over ['first', 'second', 'third'] where second throws SyncBackpressureBusyError, the detailed result should include first as settled, second as deferred, and third as skipped with reason: 'prior-deferral'. Today that mapping could be dropped or mislabeled without failing the lifecycle test.

Suggested direction
Assert the detailed terminal contract on a real syncSharedMemoryFromPeerDetailed backpressure-deferral scenario, not only the generic ordered scheduler outcome.

Confidence note
Based on diff and nearby tests; the generic scheduler outcome is tested, but the shared-memory detailed-result mapping is not exercised for this branch.

For Agents
Extend the existing preserves completed shared-memory progress when a later admission is deferred test in packages/agent/test/sync-requester-priority.test.ts, or add a focused lifecycle test, to assert the exact contextGraphTerminals list and immutability for the deferred and prior-deferral tail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants