Skip to content

Check plugin data writability through native observations - #4

Merged
jonathanhefner merged 1 commit into
mainfrom
agent/plugin-data-writability
Sep 18, 2026
Merged

jonathanhefner merged 1 commit into
mainfrom
agent/plugin-data-writability

Conversation

@jonathanhefner

@jonathanhefner jonathanhefner commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

Add mcp.stdio.data.writable to the core suite. The existing absolute-path check does not establish that the plugin subprocess can write to its data directory.

Each call to the default server’s observe tool exclusively creates a temporary file in PLUGIN_DATA, writes and closes it, and removes the file it created. The observation preserves filesystem errors for deterministic evaluation. Repeating the check on each call supplies current evidence when a client retains the MCP process across runs.

Cleanup failures appear as warnings in JSON and human summaries without changing a successful writability result. The tool description and annotations disclose the write, and the existing guide records the expanded observation through the same workflow.

Validation:

  • All 53 local tests pass, including actual writes, failed writes, cleanup, preservation of existing files, repeated observations, and saved warning rendering.
  • Syntax and committed bundle reproducibility checks pass.
  • The abbreviated README example parses as JSON, its counts match the displayed results, and all four documented jq queries produce the expected outcomes.
  • Independent implementation and documentation review completed.
  • CI passes on Linux and Windows with Node 22 and 24.

Native validation used unchanged copies of commit 577aa38, the ordinary guide request, and each client’s native plugin loading mechanism:

Client Saved report New writability check
Codex CLI 0.154.0-alpha.6.2 15 pass, 0 fail, 0 not_verified Pass
GitHub Copilot CLI 1.0.83 13 pass, 2 fail, 0 not_verified Pass

Both runs saved all seven observations unchanged, matched deterministic evaluation, and removed the temporary probe file. Independent review checked the traces, recorded evidence, and package integrity. Programmatic acceptance of mcp.stdio.data.writable passed for both reports.

Copilot retained its previously observed argument and environment expansion failures: $APC_VALUE expanded in an argument, while ${PLUGIN_DATA} remained literal in a configured environment value. It also batched the four observation calls before recording them, contrary to the guide’s instruction to record each observation immediately. Every recording command completed sequentially, and all returned evidence matched the saved report.

Comment thread plugins/agent-plugins-conformance-core/skills/conformance-guide/SKILL.md Outdated
Comment thread plugins/agent-plugins-conformance-core/README.md Outdated
Comment thread plugins/agent-plugins-conformance-core/README.md Outdated
Comment thread plugins/agent-plugins-conformance-core/README.md Outdated
Comment thread plugins/agent-plugins-conformance-core/README.md Outdated
The existing data-path check does not establish that a plugin subprocess
can write there. Have each default `observe` call exclusively create,
write, close, and remove a temporary file in `PLUGIN_DATA`, preserving
operation facts for deterministic evaluation.

Keep cleanup warnings separate from writability outcomes and render
saved warnings in human summaries. Repeat the write on each observation
so clients that retain MCP processes still collect current evidence.
@jonathanhefner
jonathanhefner force-pushed the agent/plugin-data-writability branch from 577aa38 to 20ddeb4 Compare September 18, 2026 01:16
@jonathanhefner
jonathanhefner marked this pull request as ready for review September 18, 2026 01:27
@jonathanhefner
jonathanhefner merged commit 03fb497 into main Sep 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant