Magpie

Conserve the log. Derive the rest.

A replayable memory kernel for AI systems.
Signed history. Explicit evidence. Policy-defined conclusions.

Quick start   ·   Architecture   ·   Standing   ·   Limits   ·   Roadmap   ·   Documentation

---

Magpie keeps a signed, append-only history of what was recorded. Claims, evidence, provenance audits, search indexes, and policy conclusions are derived from that history. The historical record remains intact when derived state is rebuilt.

A remembered claim should retain the distinction between what was said, what was checked, and what may follow.

Important

Experimental kernel. Magpie is not a finished memory product. Successful verification establishes only the predicate checked against the supplied inputs; it does not establish truth, key ownership, freshness, global completeness, or permission to act. See the current boundary and audit disposition.

01 / RECORD History you can inspect An Ed25519-signed hash chain, canonical event bytes, explicit checkpoint expectations, and bounded local SQLite persistence.

02 / REPLAY State you can regenerate Complete supplied-history verification before replay. Search and claim projections are rebuildable views over the record.

03 / RESOLVE Conclusions with boundaries Explicit standing policies, immutable supplied content, provenance checks, deterministic audit explanations, and detached native-v2 standing receipts.

Release coordinates: latest GitHub source release v0.3.0 · tracked release contract v0.1.0 · workspace packages v0.1.0. main may be ahead of the latest release. Details ↓

Why Magpie exists

An agent recalls a claim from last week. The text survived—but did its qualifications?

What the agent finds	What still needs to be established
A search result	Whether it is relevant evidence for this claim.
Bytes matching a digest	Where those bytes came from and what they support.
Two agreeing reports	Whether they share an origin and qualify under the selected rule.
A successful deterministic check	The exact proposition that procedure checked.
Yesterday's policy conclusion	Which inputs and policy produced it, and whether it answers today's question.

Magpie keeps these distinctions explicit. The signed log is historical authority relative to the verifying key supplied by the caller. Derived views explain the record and its policy consequences; they cannot rewrite it.

This is an infrastructure problem shared by scientific evidence synthesis, machine reasoning, and information integrity: preserve the distinctions between an assertion, its provenance, evidence bearing on it, and the conclusions an explicitly selected method permits.

The vocabulary: seven distinct questions

Term	Question
Observation	What did the recorded event say?
Availability	Were the exact bytes needed for the computation supplied?
Verification	Do those bytes satisfy the named checking procedure?
Provenance	How is an artifact connected to a recorded acquisition or derivation?
Origin admission	Is a contribution admitted into this exact origin comparison?
Eligibility	May this evidence participate in the selected policy rule?
Governed standing	What conclusion does that explicitly selected policy produce?

Quick start

From a repository checkout, run the workspace tests and the deterministic standing tour:

cargo test --workspace --locked
cargo run --locked --example tour -p magpie-claims

The frozen tour creates and verifies one signed history, resolves policies v0–v2, deletes the derived state, and replays the history to check byte-identical regeneration. Policies v3 and v4 are implemented and tested separately; they are outside the frozen tour output.

Keep a ledger with the magpie command

cargo install --locked --path crates/magpie-cli
magpie init ~/ledger                     # prints the verifying key; record it elsewhere
export MAGPIE_STORE=~/ledger
magpie claim "Theorem 2 holds" --domain OperationalObservation
magpie evidence "numerics agree to 1e-15" --kind ExecutionEvidence --file run.log
magpie link supports ev-2 claim-1 --rationale "numerical check"
magpie why claim-1 --policy v2           # the policy's own explanation
magpie verify                            # signatures, portable profile, checkpoint

Every read verifies the whole log first. There is no default policy. The log is JSONL, so tools/verify_chain.py can check it independently. The crate docs explain why it doesn't use the SQLite L0 yet, and how its rollback checkpoint works.

The same CLI can produce and independently re-check the coordinate-complete detached standing receipt implemented for native policy v2:

magpie standing-receipt claim-1 \
  --history ~/ledger/log.jsonl \
  --verifying-key "$MAGPIE_VERIFYING_KEY" \
  --profile magpie-standing-receipt-history-v2-v0 \
  --policy magpie-claims-standing-v2 \
  --expectation none > receipt.json

magpie check-standing-receipt claim-1 \
  --history ~/ledger/log.jsonl \
  --receipt receipt.json \
  --verifying-key "$MAGPIE_VERIFYING_KEY" \
  --profile magpie-standing-receipt-history-v2-v0 \
  --policy magpie-claims-standing-v2 \
  --expectation none > checked.json

cmp receipt.json checked.json

These receipt commands use explicit history bytes and coordinates rather than ambient store/config/checkpoint state. Successful output is the exact canonical receipt byte sequence, with no trailing newline.

Verify the golden history with the independent Python conformer

Use the Python environment and dependencies described in the development checks, then run:

python tools/verify_chain.py \
  crates/magpie-log/testdata/golden-v1.jsonl \
  ea4a6c63e29c520abef5507b132ec5f9954776aebebe7b92421eea691446d22c

The verifier checks canonical encoding, sequence numbers, previous-hash links, stored hashes, signatures, genesis binding, and payload validity. The verifying key is supplied explicitly; the log does not authenticate its own trust root.

How Magpie works

flowchart TB
    H["Signed append-only history"] --> V["Complete supplied-history verification"]
    V --> R["Deterministic replay"]
    R --> E["Episodic search"]
    R --> C["Claims, evidence, and anchors"]
    C --> S["Selected standing policy"]
    I["Exact supplied artifacts and bundles"] --> S
    S --> G["Governed standing and audit explanation"]
    G --> D["Detached native-v2 standing receipt"]

Loading

The replay path verifies one complete supplied record snapshot before publishing its authoritative replay result. The same retained events then feed the derived projections.

Policies that need external artifacts or foreign bundles receive those exact bytes through an immutable content closure. Resolution performs no ambient filesystem or network lookup to fill in missing evidence. Provenance, origin admission, and eligibility are evaluated where the selected rule requires them.

Note

The diagram shows computation boundaries. A path through the diagram does not, by itself, establish evidence eligibility or promote standing.

Workspace

Crate	Owns	Produces
magpie-log	Canonical encoding, signatures, hash chain, readers/writers, SQLite L0, complete verification, checkpoints, and replay.	The historical record and verification of supplied histories.
magpie-claims	Typed claims and evidence, immutable supplied content, provenance and origin checks, policies v0–v4, detached standing receipt v0.	Governed standing, one-way audit explanations, and a privately constructed coordinate-complete native-v2 receipt.
magpie-episodic	Rebuildable SQLite projection, FTS5, deterministic log-order search.	Searchable derived state with no write-back path to history.
magpie-cli	The magpie command: store setup, signed writes, verified reads, rollback checkpoints, explicit detached-receipt production and checking.	A claim ledger from the shell, magpie-desk-export-v0, and exact canonical standing-receipt bytes.

What works today

Portable verification

3 languages	432 frozen cases	7 result fields
Rust · Python · Go	19 ACCEPT · 413 REJECT	Exact differential agreement

The selected ADR 0010 signature profile and portable input language have complete conformers in all three languages.

• Rust: the public exact-byte FileStore::verify_portable_history path drives the authoritative portable-history conformer.
• Python: tools/verify_chain.py is the readable independent reference conformer; the earlier tools/portable_verifier.py remains comparison evidence.
• Go: a standalone conformer shares no Rust or Python verifier implementation. Its approved filippo.io/edwards25519 v1.2.0 arithmetic dependency is vendored for offline builds.

The differential runner checks the frozen manifest and exact case bytes before comparing verdict, class, line, record_index, event_count, tip, and ordered_recomputed_hashes. Bounded, non-normative Rust fuzz/metamorphic assurance additionally exercises the production conformer path.

This is finite conformance evidence for one exact corpus and profile. It is not a proof of verifier correctness, universal input equivalence, key trust, currentness, or release-environment portability. The frozen corpus is an oracle, not a verifier.

Detached standing receipts

Magpie now implements one concrete coordinate-complete detached result boundary: native policy-v2 standing over an explicitly supplied portable history. The frozen detached standing receipt v0 contract is implemented in magpie-claims and exposed by the read-only standing-receipt and check-standing-receipt CLI paths.

The producing context binds the exact supplied-history identity, independently supplied external verifying key, explicit history expectation, claim selector, receipt profile, and policy selection. Complete portable verification releases one opaque verified history vector; expectation, selector, replay, and native v2 resolution all derive from that same vector. The receipt then binds that context and the complete outcome through domain-separated context_sha256 and result_sha256 identities.

Detached bytes do not become authority by parsing successfully. Checking receives the retained inputs independently, re-verifies and rederives the result, reconstructs the canonical receipt, and requires exact byte-for-byte equality. Whitespace changes, field reordering, a trailing newline, altered coordinates, or a fabricated stronger outcome therefore fail acceptance rather than being normalized away.

The CLI preserves that boundary: it preflights the complete external key before acquiring history, reads the selected history exactly once, rejects --store, ignores ambient Magpie store/config/checkpoint state, and writes only the library-owned canonical receipt bytes on success. This receipt establishes only the named historical derivation; it does not establish truth, currentness, latest history, permission to act, source authenticity beyond the selected verification boundary, or global completeness.

Persistence

Backend	Intended role	Boundary
SqliteL0Store	Supported local persistent L0.	Separate creation and verified reopen; finite resource limits; stale-writer rejection and atomic successor append.
FileStore	JSONL compatibility, development, inspection, and portable verification.	Does not provide the supported durability or concurrent-writer boundary.
MemStore	In-memory tests and embedding.	No persistent-storage guarantee.

Supported SQLite reopen checks L0 ownership, schema, storage profile, limits, and the complete retained history before publishing a writer or verified result. Append revalidates the persisted history inside the transaction that may add its successor. A stale writer must reopen; it is not silently refreshed or rebased.

Read the persistence contract →

Writing authority

LogWriter owns the signing key and is the low-level append capability. LogReader and derived projections cannot append. The optional Deadbolt anchor path has its own reviewed writer boundary.

An ordinary governed claim/evidence writer and EpistemicGate are not implemented. Recording an event does not automatically admit it into an epistemic process.

Standing

Governed standing is the result of an explicitly chosen policy. Raw historical status remains available for audit; merely recording a status does not make it a governed conclusion. ADR 0003 defines the model.

Policy	Rule introduced	Maximum new effect
v0	Candidate explanations, ceilings, and blockers.	Explanation without promotion.
v1	Exact same-replay Deadbolt occurrence and inclusion.	Settled for the exact occurrence/inclusion proposition.
v2	Exact deterministic direct support.	Supported, never Settled.
v3	Narrow external-report corroboration.	Supported, never Settled.
v4	Subject-bound deterministic direct refutation.	Refuted under conservative precedence.

Policies have stable identities. The caller chooses; there is no resolved_standing_latest.

Warning

v3 retains compatibility semantics. Distinct admitted origin groups are not proof of statistical, causal, or organisational independence. The current path does not independently authenticate the claimant's right to assign an origin group. ADR 0007 defines an accepted authority-bound successor; that runtime is not implemented.

Why v4 owns its subject

The implemented v4 rule keeps both the subject bytes and expected digest on the claim. Evidence may identify and bind the route to that claim, but cannot substitute unrelated bytes and present their mismatch as refutation.

Parse and binding failures remain failures. Only the exact eligible negative path may produce Refuted, and multiple eligible candidates do not amplify the result: the direct rule applies at most once.

When eligible counterevidence meets inherited Supported or Settled, v4 preserves that status and exposes an explicit contradiction blocker. Consumers must read the qualification alongside the status; broader contradiction policy remains future work.

The rejected substitution: evidence-selected bytes

An earlier design allowed evidence to supply an unrelated witness_hex. A digest mismatch could then appear to refute the claim while establishing only that the evidence-selected bytes differed from the expected digest. That design was rejected. Claim-owned subjects keep the implemented rule bound to the proposition it actually checks.

Read the v4 contract →

Rules Magpie refuses to blur

Principle	Consequence
History is append-only.	Corrections are later signed events. Currentness and supersession are separate derived questions.
Verification has a scope.	Verified bytes do not establish provenance, independent origin, or truth.
Failure is not falsity.	Missing bytes, malformed metadata, bad signatures, failed bindings, and incomplete audits report their own failures.
Repetition is not strength.	Repeated anchors and audit outputs gain no authority through multiplicity. v3 collapses same-origin multiplicity; v4 applies its direct rule at most once.
Audit output is explanatory.	Authority-bearing audit types have no public caller-supplied deserialization route back into resolution.
Policy is explicit.	A new policy version does not silently replace the caller's selected semantics.

Verified history is not current history

A verified prefix establishes the validity of the exact supplied history, under the supplied key and limits. A longer valid history may exist elsewhere. Verification does not establish freshness, latest state, or global completeness.

open_containing_checkpoint_v0 additionally checks whether that history contains an exact caller-supplied checkpoint. Success does not authenticate the checkpoint or establish secure retention, canonicality, or rollback resistance. ADR 0009 →

A derived result must name what produced it

ADR 0008 requires governed detached results to identify their complete producing inputs: history, policy, exact supplied artifacts, subjects, inherited results, and applicable authority inputs. Equal-looking outputs can have different derivations.

That doctrine does not retrofit every existing API. Legacy public results remain coordinate-poor. One deliberately narrow boundary now satisfies it: detached native-v2 standing receipts over explicit supplied history. Other public result families are not thereby upgraded, generalized, or made current. See the detached receipt contract and current boundary.

Deadbolt integration

Deadbolt acts. Magpie remembers. The optional connection is a protocol boundary, not a Rust crate dependency. Magpie can be used without Deadbolt.

A SegmentAnchored event records one sealed foreign bundle's identity:

Identity field
Kind and run	bundle_kind · run_id
Witness	witness_root · witness_algorithm
Encoding	canonicalization_profile

Magpie verifies inclusion of the anchor event and its exact fields in the signed history. The foreign verifier remains responsible for the bundle's contents. New bundle kinds can use new bundle_kind values without a new Magpie event variant.

Protocol decision · Anchor contract · Portable base

Current boundary

Implemented today

• History: canonical signed events and golden vectors; append-only hash chain; complete verification before replay.
• Storage: bounded local SQLite L0; verified reopen; explicit prefix/checkpoint semantics; stale-writer detection and atomic successor append.
• Portability: complete Rust, Python, and Go conformers; frozen 432-case corpus; exact seven-field differential and bounded Rust assurance.
• Derived state: rebuildable search and claims; typed evidence and edges; immutable supplied content; provenance, origin-admission, and contribution audits.
• Resolution: explicit policies v0–v4; deterministic support; compatibility corroboration; subject-bound direct refutation; one-way audit explanations.
• Portable output: coordinate-complete detached native-v2 standing receipts with explicit history identity, external key, expectation, selector, profile and policy; trusted rederivation plus exact-byte detached checking; read-only CLI transport with no ambient store semantics.
• Integration: optional Deadbolt bundle anchoring.

Explicitly not implemented

Area	Remaining scope
Trust and retention	Secure retained checkpoints, stronger rollback resistance, production key custody and rotation.
Broader portable outputs	Detached result profiles beyond the implemented native-v2 standing receipt, including other policy/result families and any future generalized portable-output framework.
Admission	Authority-bound origin-group admission under ADR 0007; an ordinary governed claim/evidence writer; EpistemicGate.
Acquisition	Acquisition loading, content-addressed storage, filesystem and network ingestion.
Knowledge lifecycle	Broader contradiction policy and contradiction debt; runtime currentness, invalidation, and supersession under ADR 0006; broader propagation of source standing.
Agent access	A read-only librarian or research navigator.
Qualification	Broader platform and release-environment qualification beyond the frozen three-way differential corpus.

These are capability boundaries, not an automatically authorized roadmap. Accepted doctrine, implemented runtime, audit disposition, and release readiness are separate states.

What Magpie is not

Magpie is not a chatbot, autonomous research agent, vector database, mutable knowledge graph, general truth engine, statistical-independence oracle, automatic contradiction resolver, crawler, public ingestion service, or production key-management system. It does not silently choose the newest policy.

Audit disposition · Pre-alpha convergence · Preserved audits

Roadmap and research horizon

Direction: qualify the existing kernel before expanding the conclusions it may produce. The stages below are dependency-oriented candidate work, not a release schedule or permission to implement. Some investigation can proceed in parallel, but any new authority-bearing mechanism requires its own governing decision, reviewed contract, hostile tests, and owner approval. An accepted ADR, proposed design, passing CI, or open PR is not evidence that a runtime capability or release is complete.

Horizon	Candidate work	Required boundary or gate
1 · Public pre-alpha assurance	Complete the selected C4 evidence programme—including the SQLite L0 qualification in #158 and dependency/native qualification in #159—then consider the C5 falsification and release decision.	Follow the frozen assurance profile; independently review exact candidate evidence. Neither PR nor green CI grants release authority.
2 · Corroboration authority	Explore an additive authority-bound successor to claimant-label origin grouping under ADR 0007.	Authenticate the right to assign an exact origin group under externally selected trust coordinates. Preserve existing v3 compatibility semantics. Even authorized grouping does not prove causal or statistical independence.
3 · Governed admission and access	Resolve the admission readiness questions before considering an ordinary governed claim/evidence writer or EpistemicGate. Separately explore the proposed read-only librarian.	No unreviewed proposal, retrieved material, or query output may gain history-writing authority, evidence eligibility, or standing by crossing an interface.
4 · Knowledge lifecycle	Investigate derived currentness, targeted withdrawal, supersession, and explicit contradiction handling, consistent with ADR 0006 and other accepted lifecycle decisions.	Resolve at a specified verified snapshot and selected policy; retain previous results reproducibly. A newer, withdrawn, or conflicting assertion does not rewrite history or automatically establish falsity.
5 · Artifacts and interoperability	Consider bounded acquisition, content-addressed artifacts, and loss-aware provenance export, potentially using W3C PROV-O or Workflow Run RO-Crate at an adapter boundary.	Imported and exported metadata preserves scope and qualifications. External formats and systems may carry information, never silently confer Magpie authority.
6 · Experimental epistemics	Study dependent evidence, missing observations, source-selection bias, contextual relevance, and calibrated uncertainty through isolated evaluations. Cochrane's GRADE guidance is one methodological reference for transparent, question-specific assessment.	Use explicit ground truth where available, adversarial controls, and negative-result criteria. Research findings do not automatically become general standing policies or source-reputation scores.

Research boundary

A useful initial test is whether copied or strategically coordinated reports are mistakenly treated as independent corroboration. Other test cases include a validly signed but false assertion, missing or retracted scientific evidence, and a later contradiction against a previously supported claim. Measure correct boundary enforcement, abstention, replayability, and explanation fidelity, not merely the number of claims promoted. Learning-based assessments, probabilistic estimates, and domain-specific evidence grades remain attributed inputs to research until their semantics and failure cases justify any narrower governed use.

The epistemic invariant ledger is an orientation aid, not a source of new doctrine. The underlying separations remain: history is not evidence; evidence is not standing; standing is not truth; provenance is not trust. This roadmap amends no ADR or frozen profile, closes no audit finding, selects no implementation, and authorizes no release.

Documentation

Choose a starting point; expand a reference shelf when you need the exact rules.

If you want to…	Start here
Understand the signed record	FORMAT
Run the deterministic example	Standing tour
Understand storage and checkpoints	L0 persistence
Inspect the portable verifier contract	Input language · Frozen corpus
Produce or check a detached v2 standing result	Detached standing receipt v0
Understand evidence and policy	Standing ceilings · Provenance and origin admission
Review development history	Tickets · Working rules

Architecture decisions · ADR 0001–0010

Decision	Subject
0001	Deadbolt protocol connection.
0002	Governed claim memory and raw-status quarantine.
0003	Canonical standing model.
0004	Derived lifecycle facets.
0005	Withdrawal without claiming falsity.
0006	Currentness and supersession doctrine.
0007	Authority-bound origin assignment; successor runtime is unimplemented.
0008	Complete immutable producing inputs for governed detached results.
0009	Supplied-history verification and explicit checkpoint expectations.
0010	Accepted, owner-ratified Ed25519 subprofile implemented by the portable conformers.

Accepted decisions describe doctrine. Their individual runtime and evidence status must be read separately; acceptance does not implement a mechanism or establish release readiness.

Verification and standing · implementation contracts

• Portable input language
• Rust frontend contract v0
• Rust history conformer contract v0
• Portable verifier corpus v1
• Standing ceilings
• Provenance and origin admission
• Policy v3: external-report corroboration
• Policy v4: claim-inline direct refutation
• Detached standing receipt v0

The corpus is owner-approved, merged, exact-byte, frozen oracle material. Finite agreement does not itself prove verifier correctness or close a finding; consult the separate owner-reviewed disposition record.

Agent and query boundaries · design references

• MCP boundary
• MCP librarian
• Verified librarian query
• Provenance response
• MCP capability boundary
• AgentProposer boundary

These documents separate reading, proposing, querying, and authority. They do not imply that a librarian or MCP runtime is present.

Admission research · questions and proposed contracts

• Boundary questions
• Threat model
• Readiness checklist
• Scenario analysis
• Minimal semantics
• Record boundary
• Evidence admission
• Pipeline separation
• Epistemic boundary map
• Invariant ledger

Status matters: exploratory questions and proposed contracts are not runtime behaviour or implementation authorization.

Development

The CI workflow records the current validation setup, including Python dependencies. The Go conformer guide covers both POSIX and PowerShell usage. These are source-checkout workflows; this README does not claim registry availability.

Rust · formatting, tests, linting

cargo fmt --all --check
cargo test --workspace --locked
cargo test --doc --locked
cargo clippy --workspace --all-targets --locked -- -D warnings

The default Rust test graph includes bounded portable-verifier smoke assurance. The larger deterministic campaign is opt-in:

cargo test -p magpie-log --locked --lib assurance_a_extended -- --ignored

Conformance · corpus integrity, Python, Go, and differential checks

python tools/check_verifier_corpus_manifest.py
python tools/check_release_metadata.py
python -m unittest tools.test_portable_verifier tools.test_verify_chain tools.test_portable_verifier_differential -v
python tools/check_portable_verifier_python.py
go -C tools/go-verify-chain test -mod=vendor -count=1 ./...
go -C tools/go-verify-chain vet -mod=vendor ./...
go -C tools/go-verify-chain build -mod=vendor -o go-verify-chain .
./tools/go-verify-chain/go-verify-chain --check-manifest fixtures/verifier-language-v1/manifest.json
python tools/check_portable_verifier_differential.py --repeat 2

The Go commands above use a POSIX shell. The Go guide provides PowerShell equivalents and defines the direct executable's governed 0 / 1 / 2 exit contract. go run is not that governed interface.

Focused hostile and integration tests also cover provenance, origin admission, v3/v4 standing, raw-status quarantine, same-replay binding, failure ordering, non-amplification, and canonical audit vectors.

Changes should remain small, typed, replayable, and explicit about what grants authority. Contributing → · Reporting a vulnerability →

Release status

Coordinate	Current boundary
GitHub source release	v0.3.0. Earlier releases are listed in the changelog.
Tracked formal release contract	v0.1.0. The v0.2.0 and v0.3.0 source releases have no equivalent tracked release-contract document.
Workspace package version	0.1.0, which is what magpie --version prints. No registry publication or package availability is claimed.
Development	main may be ahead of the latest source release. Experimental status remains.

---

Conserve the log. Derive the rest.
_{Magpie · Apache-2.0 · Experimental}

Back to top ↑