Record every UK build attempt in Logbook (#666)

[juaristi22]

Closes #666. Workstream A of the UK-migration master epic #665 — this lands before P1 calibration starts generating intermediate builds, since Logbook is the migration's build-attempt memory.

What changed

#628 delivered the append-only Logbook, but adoption is per-tool and only tools/build_us_multispine_pool.py called the record_build_attempt seam. The three UK drivers now append exactly one validated, hash-chained row per non-dry invocation, so discarded intermediate builds stay visible. Rows spool beside the build outputs; receipts live under build-scoped logbook-receipts/<build_id>/.

Dispositions

Exit path	Disposition
Success (all three drivers)	iterating
Gate-battery block; calibrated-candidate refusal; unexpected exception	failed
Named dev-rung SPI singleton abort	discarded
Dry runs (both routes, success and failure)	no row
Argparse / config refusals, including a malformed or conflicting Logbook chain head	no row — recording starts post-parse and after chain-config validation, matching the US driver

The rung abort keeps its existing uk_rung_abort_receipt payload, including its own disposition: "aborted_with_receipt" field; only the Logbook row says discarded. Exit codes, exception types, and stdout payloads are unchanged — the spool notice goes to stderr.

Verdicts and pins. gate_verdicts anchor into durable local receipts, referenced checkout- or home-relative where possible (rows are exported publicly; paths under neither root fall back to the absolute path without its leading slash — see the local_artifact_reference docstring). The national rows carry the schema-4 battery statuses uncollapsed (all five of passed/failed/not_applicable/evidence_absent/unreached), read from error.report_path on a block; rung aborts point at the named-edge receipt; rowwise and candidate rows point into their manifests. The national pin surface hoists the source digests out of the checkpoint-only path so failure rows carry real input pins, and names the future ledger_facts slot for #622/#623 without inventing a placeholder.

New module. microcosm.build.logbook_adoption is a driver-agnostic extraction of the generic US helpers: attempt state, predecessor resolution, git code pins, exportable artifact references, normalized role-pin digests, atomic receipts, and a record-once terminal write. The US driver deliberately keeps its local copy in this PR — its contract suite pins those seams by monkeypatch — so migrating it is load-bearing convergence work tracked as #689. Also repoints the Logbook docstring from the merged #616 at #665/#666.

Design notes for review

• Record-once is structural. A second record_build_attempt call in one run produces a different row (wall_seconds differs), and two rows chaining the same predecessor make a fork the archive refuses. The national driver's handlers are therefore sibling except arms — each records once and then returns or re-raises. GateBatteryBlockedError subclasses RuntimeError, so the except ValueError arm cannot shadow it; the unrecognized-ValueError case records inline before re-raising, since a re-raise never falls through to a later arm on the same try. Every driver test asserts len(rows) == 1.
• Non-nullable digests. input_pins_digest and identity_digest are required 64-hex, so a placeholder preflight digest is seeded before anything can fail; code_pin keeps the unresolved-local-git-code-pin sentinel until the real pin resolves. That is what lets a preflight failure still record a valid row.
• Two calls were judgment, not derivation, and are easy to flip: dry-runs recording nothing, and the rung abort as discarded rather than failed.

Verification

• test_logbook_adoption.py + the four UK driver suites: 169 passed
• Logbook contract, backfill, archive, CLI, and the untouched US driver suite: 204 passed
• ruff check . clean; all 10 changed files ruff format clean. Full-repo ruff format --check shows 29 pre-existing unrelated diffs, confirmed identical at the base commit and left alone.
• Acceptance: a UK row chained onto the committed archive's tail (in a temp copy) exports, validates as one 29-row chain, and renders in the same table beside the US rows — uk-national-staging | f001 | iterating.

🤖 Generated with Claude Code

[juaristi22]

An adversarial review (Codex, run locally) surfaced two failure-path findings; both are fixed in 206ff6e.

[high] Invalid Logbook predecessor could erase national sidecars before aborting. The national driver resolved --logbook-prev-row-digest / POPULACE_LOGBOOK_PREV_ROW_DIGEST after unlinking the prior attempt's evidence, replay, build-record, and rung-abort sidecars — so an operator typo in an optional logging feature destroyed local evidence and then crashed. Chain config now resolves before any side effect in all three drivers (the rowwise dataset driver had the same ordering against its stale coverage/crosswalk unlinks). A malformed or conflicting head refuses the run with no row and untouched sidecars, matching the argparse-refusal posture, and every recorded row now carries the resolved head rather than the raw CLI value.

[medium] Rowwise candidate setup failures escaped unrecorded. Input hashing, frame/ladder loading, cloning, and target binding ran before the candidate's recording envelope opened, so those failures spooled no row — contradicting the one-row-per-non-dry-invocation contract. The envelope now opens right after CLI validation; pins promote as soon as both inputs hash, the identity digest waits only on the frame-derived source year, and dry-runs are routed past recording explicitly.

Both regression tests were verified to fail against the unfixed drivers before the fix landed:

• test_invalid_logbook_predecessor_refuses_before_sidecar_cleanup (malformed-env + cli-env-conflict): pre-existing sidecars keep their bytes, no spool row.
• test_candidate_setup_failure_records_failed_row: a ladder-load failure spools exactly one failed row with real input pins and no cloned phase.

Full driver + adoption suites (172), logbook contract/backfill/archive/CLI + untouched US driver suites (204), and ruff all pass locally after the fix.

🤖 Generated with Claude Code

[vahid-ahmadi]

The exception structure is the part I most expected to be subtly wrong, and it isn't. I verified the two claims rather than taking them:

• GateBatteryBlockedError(RuntimeError) — confirmed at gate_battery.py:718. So the earlier except ValueError arm genuinely cannot shadow it, and arm order between the two is irrelevant.
• Both ValueError sub-paths record exactly once: the named-edge rung abort records discarded then returns, the unrecognized case records via _record_failed_exception then re-raises. A re-raise can't fall through to a later arm on the same try, so sibling arms are the right shape and len(rows) == 1 holds on every path.

Seeding a valid placeholder input_pins_digest/identity_digest before anything can fail is what makes preflight failures recordable at all — that's the detail that makes "discarded intermediate builds stay visible" true rather than aspirational. Hoisting the source digests out of the checkpoint-only path so failure rows carry real input pins is the same instinct.

Three things.

1. local_artifact_reference can still emit a host-absolute path.

The docstring says "without host-absolute paths" and the PR body says "never host-absolute paths (rows are exported publicly)", but the final fallback is:

return f"local://{resolved.as_posix().lstrip('/')}"

which is the absolute path with the leading slash removed. The repo-root and home branches cover a dev laptop and a CI runner. A build machine writing to a dedicated mount — /mnt/scratch/uk-builds/..., /data/populace/... — falls through to the third branch and exports local://mnt/scratch/uk-builds/..., which is exactly the host layout the contract says it won't carry. And that machine is the one producing the rows worth exporting.

Either the fallback should refuse (a reference outside both roots is arguably a configuration error for an exportable row), or the docstring and changelog should say "repo- and home-relative where possible" instead of "never".

2. _json_ready silently drops None-valued keys from receipts.

if isinstance(value, Mapping):
    return {
        _json_ready(key): _json_ready(item)
        for key, item in value.items()
        if item is not None
    }

write_error_receipt puts gate_verdicts through this, so a verdict carrying an explicit null loses the key entirely — and in an immutable audit receipt, "explicitly null" and "never set" stop being distinguishable. For compaction in a general JSON writer that's fine; for the artifact that is the build-attempt memory, it quietly erases a distinction someone reading a failure six months later would want.

Worth either keeping nulls in receipts or saying in the docstring that null-valued keys are dropped by design, so the next person doesn't read an absent key as absent evidence.

3. Two copies of the adoption helpers, and the shared one is the untested copy.

logbook_adoption.py is a clean extraction, but the US driver keeps its local copy because "its contract suite pins those seams by monkeypatch". So the same logic now exists twice, and the copy with the contract suite holding it in place is the one that isn't shared. A fix or hardening applied to the UK path won't reach the US path, and vice versa — with the drift running in the direction of the untested module.

Naming it as a follow-up is right, and I'm not asking you to migrate the US driver here. But the follow-up is load-bearing rather than tidying, and it'd be worth an issue with that framing rather than a sentence in this PR body.

On the two judgment calls you flagged

Rung abort as discarded rather than failed reads correct to me — a named-edge abort is a deliberate non-result, not a failure, and the receipt keeps its own aborted_with_receipt disposition regardless.

Dry runs recording nothing on success is obviously right. On failure I'd think about it once more: a dry run that fails is a real signal about configuration, and it's the cheapest possible build attempt to have a record of. Not a strong view — a dry run isn't a build attempt, so excluding it is defensible — but of the two calls it's the one where I could imagine wanting the row later.

[juaristi22]

Thanks @vahid-ahmadi — all three addressed in df5b8fa, and your verification of the exception structure is appreciated. Point by point:

1. local_artifact_reference fallback. You're right, and it turned out to be inherited: the US original (build_us_multispine_pool.py:3667) has the identical fallback and the identical "never embed an absolute host path" docstring claim — the lie was copied faithfully along with the code. I took your second option: the docstring, changelog, and PR body now state the real ladder (checkout → home → absolute-minus-leading-slash) and why the fallback doesn't refuse — recording must never turn a completed build into a failure over reference formatting, and a refusal would do exactly that on a dedicated-mount build host. The docstring also tells operators producing exportable rows to keep outputs under the checkout or home so the fallback never fires. Worth noting the home branch is what carries the username-leak concern, and it's covered; the residual leak is non-home server layout. Reconciling the US docstring's same false claim is folded into #689 rather than touched here.

2. _json_ready dropping nulls. This one was not inherited — the US writer preserves None values and rejects non-string mapping keys; the drop was introduced during extraction. Fixed by re-aligning to the US behavior (nulls survive, non-string keys raise), with a regression test asserting an explicitly-null reason field survives into the receipt. Your framing was exactly right: this was the first concrete instance of the two-copy drift, in the direction you predicted, caught within days of the split. It's now cited as evidence in the migration issue.

3. Follow-up as an issue. Filed as #689 with the load-bearing framing: two copies, the deep contract suite pinning the unshared one, drift running toward the shared module — with the _json_ready divergence as proof the risk is real. Scope includes folding the US _json_ready's dataclass/set handling into the shared module and reconciling both docstrings; acceptance gate is test_us_multispine_pool_tool.py passing unchanged. The changelog and PR body now point at #689 instead of a loose sentence.

4. Dry-run failures. Keeping them unrecorded for now, with your point noted as real. The tie-breakers: recording dry-run failures would make broken-config dry-runs require valid chain configuration (prev_row_digest threading through exactly the cheap planning loops dry-run exists for), and the failure signal already lands where the operator is looking — the exit code and traceback of an interactive command. The exclusion is one attempt is None branch per driver, so if migration work surfaces a case where we miss those rows, flipping it is a small, isolated change. Happy to revisit if you come to hold the view more strongly.

🤖 Generated with Claude Code

[vahid-ahmadi]

df5b8fa6 closes all three. Verified:

• _json_ready no longer drops null-valued keys, and now refuses non-string mapping keys — matching the US writer rather than diverging from it. The regression test asserting an explicitly-null reason survives into the receipt is the right pin.
• The reference ladder is stated honestly now (checkout → home → absolute-minus-leading-slash), with the operator guidance to keep exportable outputs under the checkout or home.
• #689 filed with the load-bearing framing.

On the fallback: I agree with the call, and your reason is better than my suggestion. Refusing would turn a completed build into a failure over reference formatting, on exactly the dedicated-mount host most likely to be doing real work — that trade is clearly wrong, and the honest docstring is the right resolution. Good catch that the username-leak case is the one already covered and the residual is non-home server layout.

The _json_ready provenance is the part worth recording: not inherited, introduced during extraction, in the direction of the shared copy. That's the two-copy risk producing a concrete divergence within days of the split, which makes #689 an argument from evidence rather than from principle. Worth keeping that sentence in the issue.

On dry-run failures — your tie-breaker settles it for me. Requiring valid chain configuration in exactly the cheap planning loops dry-run exists for would defeat the point, and the failure signal genuinely does land in front of an operator running an interactive command. I don't hold the view more strongly than that; leave it.

Nothing further — clear from my side.