Stateful differential testing for Model Context Protocol implementations.
Many MCP failures only appear across a sequence: initialize, overlap requests,
cancel one, reconnect an SSE stream, or recover from an expired session.
mcp-statecheck models these interactions as canonical actions and records the
wire behavior as deterministic, redacted traces.
Note
Version 0.1.0 provides an installable quick-check CLI, deterministic replay
of package-controlled failures, offline reports, a package-owned SDK matrix,
a composite Action, and cross-platform acceptance. Arbitrary-target replay
and the deep profile remain intentionally outside this release.
From a fresh checkout with uv installed:
git clone https://github.com/lxingy3/mcp-statecheck.git
cd mcp-statecheck
uv sync --locked
uv run mcp-statecheck check --output artifacts/quickstart.json --junit artifacts/quickstart.xml --sarif artifacts/quickstart.sarif --html artifacts/quickstart.html --stdio -- uv run python -m mcp_statecheck._controlled_peer --stdio --mode sdk-smoke
Check passed: wrote artifacts/quickstart.jsonThe released CLI can also be installed without a checkout:
uv tool install mcp-statecheck==0.1.0
mcp-statecheck --version
mcp-statecheck 0.1.0The same quick profile can target only a server you explicitly name:
mcp-statecheck check --stdio -- python server.py
mcp-statecheck check --url http://127.0.0.1:3000/mcp
mcp-statecheck check --header-env Authorization=MCP_TOKEN --url https://example.test/mcp--header-env HEADER=ENVIRONMENT_VARIABLE reads the complete header value
from the environment. The value is never placed in CLI arguments and is
redacted before traces or reports are stored. The bounded quick profile
initializes one session, sends ping, probes tools/list without calling any
tool, verifies response shapes, and confirms transport cleanup.
| Area | Current implementation |
|---|---|
| Protocol state | Lifecycle, capability negotiation, pending requests, cancellation, streams, and logical sessions |
| Request identity | Internal action IDs remain separate from MCP request IDs, including duplicate concurrent IDs |
| stdio | JSON Lines subprocess transport with deadlines, bounded stderr, and child cleanup |
| Streamable HTTP | Session headers, JSON and SSE responses, reconnect cursors, status handling, and cleanup |
| Traces | Versioned JSON, deterministic writes, recursive secret redaction, and stable session aliases |
| Installable CLI | Explicit-target checks, allowlisted replay, offline reports, and separate locked legacy and modern SDK matrices with 0/1/2 exit codes |
| Reports | Deterministic JSON, JUnit XML, SARIF 2.1.0, and a script-free single-file HTML trace explorer |
| Automation | A JSON-argv composite Action plus scheduled macOS runs of the full SDK matrix and clean-package acceptance |
| M1 controlled peers | Five original controlled scenarios exercised over real stdio or localhost HTTP connections |
| M2 controlled corpus | Five seeded RuleBasedStateMachine failures with stable signatures, shrinking, saved-trace reload, and 10-run replay |
| M3 real SDK clients | Four isolated Python and TypeScript SDK runners across two released protocol revisions and both transports, with 16/16 cells checked against saved traces |
| M5 application servers | Release-bound, allowlisted recipes drive pinned Filesystem and Git reference servers through verified state transitions and reproduce one normalized trace across ten fresh processes |
| M6.1 modern SDK clients | A separate four-cell 2026-07-28 profile covers Python mcp 2.1.1 and TypeScript client 2.0.0 over stdio and Streamable HTTP |
| M6.2 Tasks wire profile | Three generated controlled defects across both transports, with independent state checks, minimized traces, 10-run replay, and ten conforming baselines |
flowchart LR
A["Canonical actions"] --> B["State reducer"]
A --> C["stdio or Streamable HTTP"]
C --> D["Controlled MCP peer"]
D --> E["Wire observations"]
B --> F["Versioned trace"]
E --> F
F --> G["Hypothesis shrink<br/>(five M2 controlled failures)"]
G --> H["Saved-trace reload<br/>and 10-run replay"]
F --> I["JSON · JUnit · SARIF · offline HTML"]
The checked-in M1 acceptance report records the pinned Python runtime, exact passing test count, and all five M1 fixture checks within a 180-second hard deadline. CI runs the locked project on Ubuntu and Windows.
Install uv, then run:
uv sync --locked
uv run python scripts/run_m1_acceptance.py
uv buildThe acceptance command runs the full test suite and writes one wire trace per
controlled fixture to artifacts/m1/.
| Fixture | Transport | Observation preserved by M1 |
|---|---|---|
http-error-as-timeout |
Streamable HTTP | An HTTP 503 remains distinct from a timeout |
duplicate-concurrent-request-id |
stdio | Two pending calls retain separate logical identities despite sharing one MCP request ID |
second-sse-resume-token-loss |
Streamable HTTP | Each reconnect sends the newest event ID |
request-before-initialized |
stdio | A request issued before initialization completes remains visible in the trace |
late-response-after-cancellation |
stdio | A cancelled result is cross-correlated with the later call |
For example, the HTTP error fixture records the protocol result and cleanup state separately:
{
"fixture_id": "http-error-as-timeout",
"transport": "streamable-http",
"normalized_events": [
{
"kind": "http_error",
"status": 503
}
],
"cleanup": {
"client_closed": true,
"listener_closed": true
}
}M1 preserves the observations required for detection. M2 now closes that loop for all five controlled defects:
$ uv run python scripts/run_m2_slice.py --fixture http-error-as-timeout
M2 slice passed: minimized and replayed 10 times; wrote artifacts/m2/http-error-as-timeout.json
$ uv run python scripts/run_m2_slice.py
M2 slice passed: minimized and replayed 10 times; wrote artifacts/m2/request-before-initialized.json
$ uv run python scripts/run_m2_slice.py --fixture duplicate-concurrent-request-id
M2 slice passed: minimized and replayed 10 times; wrote artifacts/m2/duplicate-concurrent-request-id.json
$ uv run python scripts/run_m2_slice.py --fixture late-response-after-cancellation
M2 slice passed: minimized and replayed 10 times; wrote artifacts/m2/late-response-after-cancellation.json
$ uv run python scripts/run_m2_slice.py --fixture second-sse-resume-token-loss
M2 slice passed: minimized and replayed 10 times; wrote artifacts/m2/second-sse-resume-token-loss.jsonHypothesis reduces the generated sequence to initialize followed by
tools/list for the lifecycle case, and to two overlapping tools/call
requests sharing one ID for the request-identity case. The cancellation case
uses fixture canaries to detect a full response cross-swap after call-a is
cancelled and call-b is pending. That oracle is enabled only for this
controlled fixture, and it verifies the cancellation request ID and both sides
of the swap without assuming a response order. A late response alone is not
classified as a failure. Each saved trace is loaded from disk and replayed
against ten fresh peers; every run produces the expected stable signature.
The HTTP slice proves that a real 503 reached the transport before a controlled client adapter misclassifies it as a timeout. The SSE slice keeps the expected latest cursor in the canonical action while the controlled adapter omits it on the second reconnect, after the required initialized notification; the peer captures the actual cursor, session, and protocol headers. Each HTTP candidate and replay uses a fresh localhost peer, verifies client and listener cleanup, and verifies session deletion when a session was established.
The installed replay command accepts only checked-in artifacts with the
versioned controlled-fixture target recipe:
$ mcp-statecheck replay artifacts/m2/request-before-initialized.json
Replay reproduced mcp-statecheck:v1:62237ccbaf578ac57aedd89b1b34c81c01913f6541dc3d3d414fbc180fd25dc2 in 10/10 attemptsReproducing the protocol or differential failure exits 1. Invalid artifacts,
unknown recipe versions or fixture IDs, startup failures, and cleanup failures
exit 2. The recipe schema has exactly three fields: version, kind, and
fixture_id. It cannot store or execute a command, URL, environment variable,
or working directory.
The client matrix covers four pinned SDK runners, both released protocol
revisions in the benchmark, and both stdio and Streamable HTTP. It requires
uv and Node.js 24.14.1; the command installs the exact isolated Python and
npm dependencies:
| Runner | SDK | 2025-06-18 | 2025-11-25 |
|---|---|---|---|
python-v1 |
mcp 1.28.1 |
stdio / HTTP | stdio / HTTP |
python-v2 |
mcp 2.0.0 |
stdio / HTTP | stdio / HTTP |
typescript-v1 |
@modelcontextprotocol/sdk 1.30.0 |
stdio / HTTP | stdio / HTTP |
typescript-v2 |
@modelcontextprotocol/client 2.0.0 |
stdio / HTTP | stdio / HTTP |
$ uv run python scripts/run_m3_client_matrix.py --check --output artifacts/m3
Matrix passed: 16/16 locked SDK transport cells match artifactsThe installed command uses the bundled canonical benchmark by default:
$ mcp-statecheck matrix --output artifacts/matrix
Matrix passed: wrote 16 locked SDK transport tracesEach runner performs initialization, ping, tool discovery, and one echo tool
call against a controlled peer. For each protocol revision, all four runners
produce the same normalized events. The HTTP traces additionally prove
post-initialize session and protocol header preservation, valid content
negotiation, session deletion, and listener cleanup. The check also forces one
Python and one TypeScript SDK call per transport to hit the outer timeout and
verifies cleanup.
MCP 2026-07-28 is a separate protocol era, not another version cell in the
legacy matrix. The M6.1 profile pins Python mcp 2.1.1 and TypeScript
@modelcontextprotocol/client 2.0.0, then runs both clients over stdio and
Streamable HTTP.
$ uv run mcp-statecheck matrix --profile modern --check --output artifacts/m6/matrix
Matrix passed: 4/4 locked SDK transport cells match artifactsEach cell sends server/discover, tools/list, and one echo tool call. The
oracle checks namespaced per-request metadata; the required HTTP protocol,
method, and tool-name routing headers; the absence of protocol sessions and
standalone GET streams; and normalized differential equality. Four additional
hang probes force each SDK and transport through the outer deadline and prove
adapter, child-process, and listener cleanup. Three complete runs must remain
byte-identical before the checked evidence is accepted.
The default legacy profile and its 16 saved traces remain unchanged. This is a
bounded valid-client profile: Tasks, subscriptions, modern cancellation
sequences, custom x-mcp-header parameters, arbitrary targets, and the deep
profile remain outside this slice.
The tasks command exercises the io.modelcontextprotocol/tasks extension
against package-controlled peers. It generates action sequences with Hypothesis,
shrinks a detected defect, and reloads the saved artifact for ten fresh replays.
Both stdio and Streamable HTTP use task handles returned by the server.
$ uv run mcp-statecheck tasks --fixture task-result-shape --transport streamable-http --output artifacts/tasks-failure.json --html artifacts/tasks-failure.html
Tasks fixture reproduced: tasks.invalid_result; 2 actions; replay 10/10; wrote artifacts/tasks-failure.jsonThe command exits with code 1 when it reproduces the selected defect, following
the same failure convention as check and replay. It accepts no external target.
The version 2 replay recipe selects a known packaged fixture, never an executable
command embedded in an artifact.
| Controlled defect | Minimum requests | Observable failure |
|---|---|---|
task-terminal-regression |
3 | A completed task returns to working with no expiry |
task-input-key-reuse |
4 | An input key is reused for a different request |
task-result-shape |
2 | A completed task omits its final result |
Acceptance also checks five conforming scenarios on both transports: completion, input/update, delayed cancellation, protocol failure, and a completed tool error. Cancellation acknowledgements do not imply an immediate state change; the oracle allows the eventual-consistency behavior specified by the extension.
uv run python scripts/run_m6_tasks_acceptance.py --checkThis profile uses wire clients. The pinned TypeScript client rejects parts of the
modern Tasks API, while Python requires a custom client extension; those API
probes are documented in the Tasks research note.
The profile does not claim a Tasks SDK matrix, subscription coverage, concurrent
poll ordering, or arbitrary-server deep testing.
Saved traces are the single JSON source of truth. Reports validate schema v1, redact again at the output boundary, and never contact a network:
$ mcp-statecheck report artifacts/m2/request-before-initialized.json --json report.json --junit report.xml --sarif report.sarif --html report.html
Report wrote 4 files: report.json, report.xml, report.sarif, report.htmlBecause that artifact contains a detected failure, the command writes every
requested report and exits 1. Malformed input, an I/O/configuration problem,
or a saved infrastructure error exits 2; a passing artifact exits 0.
The repository-local composite Action accepts a JSON array rather than a shell command string:
- uses: lxingy3/mcp-statecheck@v0.1.0
env:
MCP_TOKEN: ${{ secrets.MCP_TOKEN }}
with:
arguments: >-
["check", "--url", "https://example.test/mcp",
"--header-env", "Authorization=MCP_TOKEN",
"--output", "artifacts/statecheck.json",
"--junit", "artifacts/statecheck.xml"]The clean-package acceptance command builds both distributions, installs them outside the source tree, and verifies their import origins. It exercises the wheel's actual console script against real stdio and localhost Streamable HTTP peers, replays all five controlled failures ten times from an untrusted working directory, parses all four outputs for each transport, proves HTTP session deletion and listener/process cleanup, probes the sdist-installed matrix assets, runs the wheel-installed 16-cell legacy and four-cell modern SDK matrices from an empty directory, compares every trace with the checked-in goldens, verifies both profiles' package resources remain byte-identical, and checks that a runtime authorization secret does not reach artifacts or process output:
uv run python scripts/run_m4_acceptance.pyM5 starts with one deliberately narrow external canary. The repository pins
@modelcontextprotocol/server-everything and its full npm dependency graph,
installs it with lifecycle scripts disabled, then runs the existing
2025-11-25 stdio quick check against ten fresh direct Node processes:
uv run python scripts/run_m5_external_canary.py --checkThe ten complete artifacts must be byte-identical, each protocol exchange must
pass initialize, ping, and tools/list, and each direct server process must
exit with status 0 and be reaped. The external process receives an isolated
home, temporary directory, npm cache, and a minimal environment that excludes
host credentials. CI repeats the same locked check on Ubuntu and Windows.
The Everything server is an external protocol canary, not a real-world application server. Passing this slice establishes the external-target acceptance path.
The next M5 slice pins two official reference servers and their complete dependency graphs:
uv run python scripts/run_m5_filesystem_acceptance.py --check
uv run python scripts/run_m5_git_acceptance.py --check@modelcontextprotocol/server-filesystem@2026.7.10
reports its allowed directory, writes and edits one file, reads and lists the
result, rejects write/edit/list operations against a sibling sentinel, and
then serves the edited file again.
mcp-server-git==2026.8.18
observes an untracked file, stages its exact path, reports the staged diff,
creates a deterministic commit, reads the two-entry history, observes the
clean worktree, creates a branch, and rejects a sibling-repository mutation.
The Git server always starts with an explicit --repository boundary.
Host-side checks verify exact file bytes, directory contents, commit metadata,
tree contents, history length, worktree state, and Git refs instead of trusting
tool response text alone.
Each benchmark includes a strict three-field recipe manifest. Version 2 uses
the application-state kind and an immutable recipe ID that binds the target
name, release, transport, and fixed scenario. Extra fields, duplicate keys,
non-finite numbers, unknown IDs, and target-version mismatches are rejected
before external tool discovery. A recipe cannot supply executable commands,
argument arrays, URLs, environments, paths, selectable package coordinates, or
tool calls; the runner maps the allowlisted ID to package-owned actions and
fresh temporary paths. Recipe metadata is recorded in the trace; the acceptance
report also records the manifest SHA-256.
Each target runs in ten fresh direct processes with an isolated home, cache,
temporary directory, and credential-free environment. All ten normalized
traces must be byte-identical; every process must exit with status 0 and be
reaped. Pull-request CI repeats both locked checks on Ubuntu and Windows, and
the scheduled workflow covers macOS.
These checks validate the configured server allowlist and repository boundary for the tested sibling mutations. They do not provide an operating-system sandbox: the pinned reference-server code still runs with the current user's host permissions. Arbitrary targets, HTTP transports, symlink/ACL attacks, and destructive Git history operations remain outside this slice.
The Git executable is supplied by the host rather than the Python lock. Its version is recorded in the acceptance report. The runner validates dirty and clean state semantically, stores stable markers instead of Git's changeable long-status prose, and retains the staged diff and log as upgrade-review evidence.
Application recipes belong to these checkout-only acceptance runners. They are
not accepted by the installed mcp-statecheck replay command and are not a
general external-target replay interface. The installed command remains limited
to version 1 controlled-fixture recipes and package-owned peers.
The official
MCP conformance framework
remains the source for fixed specification scenarios. mcp-statecheck is
being built for generated stateful sequences, normalized cross-implementation
comparison, shrinking, and deterministic replay. It complements conformance
rather than replacing it.
Schema lockfiles, API compatibility checks, and protocols other than MCP are outside the v0.1 scope.
The v0.1 model and legacy matrix target MCP 2025-06-18 and 2025-11-25.
M6.1 adds 2026-07-28 through a separate stateless action profile because that
revision removes protocol-level sessions, the initialize handshake, and SSE
resumption. It is not treated as a version-string-only legacy matrix cell.
The installed CLI exposes the bounded check, replay, report, and matrix
commands.
The matrix copies only its allowlisted adapter and runner inputs into a
temporary workspace before preparing isolated SDK environments; it never
installs into package resources. Replay accepts only the five package-controlled
fixture recipes and starts only the package-owned peer; artifacts cannot select
an arbitrary executable or network target. The deep profile is deferred until
generated sequences can target user-selected servers without weakening these
execution boundaries.
| Milestone | Status | Scope |
|---|---|---|
| M0 | Complete | Reproducible Python 3.12 environment, lockfile, design baseline, and cross-platform CI |
| M1 | Complete | Canonical model, wire transports, trace recorder, and controlled peers |
| M2 | Complete (5/5 fixtures) | Hypothesis state machine, invariants, differential oracle, signatures, shrinking, and replay |
| M3 | Complete | 16/16 real SDK client cells across stdio and Streamable HTTP, with exact differential traces and cleanup probes |
| M4 | Complete | Quick-check CLI, controlled replay, reports, Action, clean-package acceptance, documentation, and the v0.1 gate |
| M5 | In progress (M5.3 complete) | Pinned external canary plus versioned Filesystem/Git application-state recipes; upstream feedback requires a reproducible finding |
| M6 | In progress (M6.2 complete) | Modern SDK baseline plus generated Tasks wire testing; subscriptions and arbitrary-server deep coverage remain planned |
The exact v0.1 benchmark, limitations, and acceptance evidence are recorded in the v0.1.0 release notes.
- Design baseline
- MCP landscape snapshot
- M1 acceptance report
- M2 HTTP classification failure
- M2 lifecycle failure
- M2 duplicate request ID failure
- M2 cancellation correlation failure
- M2 SSE resume failure
- M3 real SDK client traces
- M3 benchmark pins
- M4 v0.1 clean-package acceptance
- M5 external canary trace
- M5 external canary acceptance
- M5 Filesystem acceptance
- M5 Filesystem trace
- M5 Filesystem recipe
- M5 Git acceptance
- M5 Git trace
- M5 Git recipe
- M6.1 modern profile notes
- M6.1 clean-package acceptance
- M6.1 acceptance
- M6.1 real SDK client traces
- M6.1 benchmark pins
- M6.2 Tasks acceptance
- Tasks extension and SDK compatibility notes
- v0.1.0 release notes
Apache-2.0.