Skip to content

Check skill availability with a malformed MCP document - #9

Merged
jonathanhefner merged 1 commit into
mainfrom
agent/invalid-mcp-fixture
Sep 18, 2026
Merged

jonathanhefner merged 1 commit into
mainfrom
agent/invalid-mcp-fixture

Conversation

@jonathanhefner

@jonathanhefner jonathanhefner commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

A malformed mcp.json should leave the plugin's valid skills available. Add a separate fixture with a valid manifest, syntactically invalid MCP document, and one healthy skill to exercise this whole-component failure boundary.

The existing guide records its marker into the same JSON report. The new skills.recovery.invalid-mcp-document result measures valid-skill availability; missing evidence remains not_verified. The suite now installs four plugins and uses one guide invocation. No additional server or setup process is required.

Validation:

  • All 68 tests pass; build and committed-distributable checks pass.
  • Independent review found no actionable issues.
  • CI passes on Linux and Windows with Node 22 and 24.
  • Native Codex and Copilot runs of unchanged packages at 1c053db both pass the new case: each client diagnoses the malformed MCP JSON while advertising and loading the fixture's valid skill.
  • Codex: 9 observations, 20 pass. Copilot: 8 observations, 16 pass, 2 known expansion failures, and 2 not_verified from unavailable recovery MCP evidence.
  • Independent audit confirms package bytes, native-to-recorded evidence equality, and exact deterministic reevaluation. Codex records each observation immediately; Copilot batches four Core MCP calls before serial recording and makes an extra list_agents call. No failed calls, retries, or report repairs occurred.

Add a separate valid-manifest fixture with syntactically invalid
`mcp.json` and one healthy skill. Whole-document invalidity requires its
own package because it prevents the existing recovery fixture from
exercising individual server-entry handling.

Collect its marker through the existing guide and record one
availability result under section 7.2.2. Missing evidence stays
`not_verified`; the result makes no MCP rejection or diagnostic claim.
@jonathanhefner
jonathanhefner marked this pull request as ready for review September 18, 2026 23:25
@jonathanhefner
jonathanhefner merged commit 45a0329 into main Sep 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant