ExitDrill

Run structural recovery drills for leaving SaaS systems.

Status: technical alpha · synthetic data only · offline verifier

A vendor export can contain every row and still lose the structure an organization needs: relationships, attachment bytes, permissions, or audit history. ExitDrill compares a separately captured baseline with a normalized export and keeps those dimensions visible instead of collapsing them into one portability score.

The current release proves a narrower claim: declared structural losses can be detected and summarized in a replayable aggregate receipt. It does not prove a completed migration or an operational exit.

Three-minute demo

Requires Python 3.12+ and uv.

make install
make demo-compare

The command runs the same baseline against two invented CRM exports:

• the clean fixture returns structurally_restorable;
• the adversarial fixture keeps the entity row count unchanged while changing identity, relationship, attachment, permission, and audit evidence;
• that second fixture returns not_structurally_restorable; and
• the final comparison reports the observed increase in missing or invalid evidence without inventing an overall score.

The command ends with a four-line human summary. The clean and lossy aggregate reports are written to examples/synthetic-crm/out/report.html and examples/synthetic-crm-lossy/out/report.html. Receipts, comparisons, and generated reports are ignored by version control.

What it checks

Dimension	Question
Entities	Are the expected identities and declared critical values present?
Relationships	Can the expected links still be reconstructed?
Attachment bytes	Are the referenced bytes present and unchanged?
Permissions	Are the declared grants represented?
Audit history	Are the expected events represented?

Each dimension has its own coverage declaration and result. Missing evidence is not averaged away.

How the evidence flows

1. An operator records a baseline before requesting the export.
2. A source-specific process normalizes the export into ExitDrill's fixed input contract.
3. The evaluator compares the two inputs and loads the normalized graph into a foreign-key-enforced SQLite reference model.
4. ExitDrill writes an aggregate receipt that can be validated, replayed, and rendered offline.

The neutral SQLite load tests representability, not whether people can work in another production system.

Run the CLI directly

exitdrill validate \
  examples/synthetic-crm/baseline.json \
  examples/synthetic-crm/export.json

exitdrill drill \
  examples/synthetic-crm/baseline.json \
  examples/synthetic-crm/export.json \
  --attachment-root examples/synthetic-crm/export-files \
  --out receipt.json

exitdrill verify receipt.json \
  --baseline examples/synthetic-crm/baseline.json \
  --export examples/synthetic-crm/export.json \
  --attachment-root examples/synthetic-crm/export-files

exitdrill report receipt.json --out report.html

To compare two same-scope receipts:

exitdrill compare reference-receipt.json candidate-receipt.json

The caller supplies the order. ExitDrill rejects comparisons whose drill, baseline, contract, coverage, or expected counts differ.

Receipts and trust

• Receipts contain aggregates and input digests, not record fields or attachment contents. On all three published paths this is a merge gate rather than a promise. tests/test_disclosure.py covers the synthetic demo and tests/test_canary_disclosure.py covers the Directus and CiviCRM canaries. Each derives every record value from the committed fixtures, proves the derived values still occur in those fixtures before trusting them, and fails if one appears in an aggregate artifact. They compare whole values, so they would not catch a leaked fragment of one.
• Replay verification recomputes the result from the baseline, export, and attachment bytes.
• Receipts are unsigned and contain no trusted time. A checksum detects internal inconsistency; it does not authenticate the operator or inputs.
• The evaluator does not execute mapping commands, SQL, JQ, plugins, URLs, or model-generated transforms.

Real-process synthetic canaries

The repository contains two deliberately narrow integration exercises:

• Directus 11.17.4 source canary verifies and normalizes one local synthetic API-response capture profile. Its equal-count adversarial derivative produces six observed loss signals.
• CiviCRM 6.16.2 target canary loaded that fixed source profile into an isolated local target once and recorded one independent read-back. Five target-interface probes pass, while the structural evaluation still reports six source-to-target gaps.

These are exact-profile observations. They do not establish a general Directus export, a Directus-to-CiviCRM connector, vendor-wide portability, UI usability, accessibility conformance, or production readiness. The example READMEs contain the setup, evidence inventory, known failures, and claim limits.

The nine committed browser-workflow, accessibility, and keyboard observations are pinned captures from that one lab run, not something the offline check re-executes. What the offline check does verify, every time: the structural evaluator against the committed evidence, and that each committed browser-*.json still binds to the exact literal its capture script declares as its output for a successful run -- so an edit to a script's declared output that isn't matched by a recapture is caught, without needing a live CiviCRM, Playwright, or Docker. It cannot confirm the small set of fields only a live page can produce (axe-core's rule counts, its version, and its violation list, plus the measured keyboard tab-count); see scripts/check_browser_capture_bindings.mjs for exactly which fields that is.

Run their committed offline acceptance checks with:

make demo-directus-canary
make demo-civicrm-target-canary

Result states

State	Meaning
structurally_restorable	Complete declared coverage with no observed structural loss
structurally_restorable_with_findings	No observed loss, with additional exported items
not_structurally_restorable	Expected evidence is missing, invalid, corrupt, or cannot be loaded
indeterminate	Baseline coverage is partial or unavailable

None of these states means portable, exit-ready, operationally equivalent, or legally compliant.

Current boundary and next milestone

ExitDrill is ready for outside technical evaluation using its invented fixtures. It is not ready for production or sensitive data.

Feature scope is paused at this boundary. The next milestone is an outside person running the synthetic demo without help and explaining whether the receipt answers their exit question. Until that happens, the project will not add another connector, evidence family, or data category.

Real, production-derived, or merely deidentified exports remain prohibited until the data-governance gate is satisfied.

Development

make verify
make package

The merge gate runs Ruff, strict mypy, pytest, and at least 90% branch coverage. The release workflow is dispatch-only: it verifies a signed annotated tag against trusted main, rebuilds and re-verifies at that exact commit, and can publish a GitHub Release. No tag or release exists yet, and no package-registry publication is configured.

ExitDrill is built AI-assisted, within a portfolio that shares a common quality standard: merge-blocking gates guard the core safety properties, audit artifacts are committed rather than claimed, and the accountable maintainer reviews and owns every change.

Documentation

• Architecture
• Architecture decision index
• Data governance
• Threat model
• Responsible-technology audit
• Roadmap and metrics ledger
• Internationalization declaration
• Release posture
• Incident response

Detailed source and target evidence stays with the corresponding example rather than being duplicated here.

Standards Conformance

Declared per the portfolio standards set. Every standard is either applied or recorded as N/A with a reason; there are no silent skips.

Standard	State
Code Quality	Applies: Single root pyproject.toml, uv.lock, Ruff lint and format, strict mypy over src, tests, and scripts, pytest with at least 90% branch coverage, pre-commit hooks, and make verify as the merge gate.
Security & Supply-Chain	Applies: SHA-pinned actions, scoped workflow permissions, Semgrep, gitleaks, strict pip-audit plus npm audit, zizmor, Dependabot, and private vulnerability reporting per SECURITY.md.
CI/CD	Applies: ci.yml runs the same make targets a contributor runs locally; the release workflow is dispatch-only, verifies a signed annotated tag against trusted main, and separates verification from publication authority. No tag or release exists yet. See docs/RELEASE.md for the full release posture.
Release & Versioning	Applies: .github/workflows/release.yml runs only on maintainer dispatch, verifies an SSH-signed annotated tag against trusted main, and hands publication to a separate job that never checks out code. No tag or release exists yet and no package registry is configured.
Observability	Applies (Tier C scope). Offline single-run CLI; evidence is deterministic exit codes, replayable receipts, and rendered reports. The out-of-scope decision for tracing and SLO surfaces is recorded in docs/ROADMAP.md.
Performance	N/A (offline single-run CLI with no hosted route and no served page; the HTML report is written to a local path on demand and pulls no subresources, so there is no delivery surface to budget).
Accessibility	Applies (scoped). The offline HTML report is static, script-free, and escaped. The CiviCRM canary records one sanitized automated accessibility scan and one keyboard observation without claiming WCAG conformance; human assistive-technology review remains open.
Internationalization	N/A (single-operator, English-only CLI with no public or bilingual delivery obligation; declared in docs/I18N.md).
AI Evaluation	N/A (no LLM or model component in the evaluator or its trust path; excluding model-generated transforms is a load-bearing invariant in AGENTS.md).
Documentation	Applies: README, architecture, ADR log, CHANGELOG, SECURITY.md, CONTRIBUTING.md, CITATION.cff, threat model, data governance, and responsible-tech audit are committed and current.
Quality & Metrics	Applies: The metrics ledger and milestone gates live in docs/ROADMAP.md; make verify is the single merge gate locally and in CI.
AI Development Measurement	Applies: No tool-usage counter is collected and none gates a merge. The committed ledger in docs/ROADMAP.md is the record, and its gates are what a change must clear regardless of how it was authored.
Incident Response	Applies: docs/INCIDENT-RESPONSE.md sets the process. No incident has occurred. Reports go through GitHub private vulnerability reporting per SECURITY.md, and a postmortem will be committed under docs/incidents/ when there is one to write.
Data Governance	Applies: docs/DATA-GOVERNANCE.md permits invented synthetic fixtures only; any non-synthetic input stays prohibited until an explicit gate is satisfied, and that gate is recorded as open in docs/ROADMAP.md.
Responsible-Tech Framework	Applies: docs/RESPONSIBLE-TECH-AUDITS.md covers public-interest value, privacy, equity, transparency, and accountability; the synthetic-only boundary is enforced through docs/DATA-GOVERNANCE.md.

License

ExitDrill is available under the Apache License 2.0.

Support

This is independent work, published so it can be read and checked rather than taken on trust. If your organization is planning a platform migration or an exit and wants help running one, see consulting and workshops.