Add Agent Plugins Conformance — Core

[jonathanhefner]

Add Agent Plugins Conformance — Core, an installable plugin at plugins/agent-plugins-conformance-core, for selected Agent Plugins 1.0.0 client checks.

The user asks the agent to run the conformance-guide skill and save a JSON report to an absolute path. The guide collects skill markers and MCP launch observations, and its reporting script evaluates the evidence and writes the report directly to that destination.

Behavior

• Cases cover skill discovery, MCP tool availability, working directories, argument preservation, configured environment values, and selected placeholder expansion rules.
• Each run initializes its report, creating parent directories as needed. Sequential recording updates preserve the latest complete observation for each component and replace the file atomically.
• Results include hierarchical case IDs, labels, diagnostics, specification references, and preserved observations. Insufficient evidence yields not_verified; consumers choose the results they require.
• A separate summarizer displays saved outcomes. The README includes a prompt example, an evaluator-verified JSON excerpt, complete acceptance queries, and instructions usable from an installed or extracted plugin directory.
• The plugin requires Node.js 22 or newer and agent command execution. Its MCP bundle includes runtime dependencies. Software uses Apache-2.0; documentation and other non-software materials use CC-BY-4.0.

Validation

• All 40 tests pass, covering schema validation, packaged MCP tools, recording and file errors, evidence evaluation, and summary output. Build checks reproduce committed distributables and notices.
• The JSON example matches evaluator output, and all four documented jq queries were executed against it.
• Independent review covered the implementation, tests, documentation, and examples.
• CI passes on Linux and Windows with Node 22 and 24.

Native client exercise

GitHub Copilot CLI 1.0.83 loaded an unchanged copy of commit 3601a9e through native --plugin-dir support on 2026-09-17. The plugin tree is 2c43d53ac8d6d73074a2b7b62c341ebd99e96862. The run used an isolated workspace and client configuration, with the prompt supplying the skill request and report destination.

All three skills loaded and all four MCP tools returned observations. Initialization used exactly {"action":"start"}, followed by six successful sequential recording calls. The agent recorded the two probe skills and four MCP observations, but omitted the guide observation. The saved evidence matched the submitted observations, and the report matched deterministic evaluation. All 18 plugin files remained unchanged. Separate summarization preserved the report bytes.

The report contained 11 pass, 2 fail, 1 not verified. Skill discovery remained not_verified because the guide observation was missing. The two failures were:

Case	Observed behavior
mcp.stdio.args.preservation-and-expansion	$APC_VALUE expanded in an argument although it is not a specified placeholder.
mcp.stdio.env.expansion	${PLUGIN_DATA} remained literal inside a configured environment value.

The client batched the four independent observation calls before recording them sequentially. Its final prose incorrectly claimed that every expected observation had been recorded; the JSON accurately preserved the missing guide evidence. This run verifies report delivery and evaluation, while also recording incomplete execution of the collection procedure. Every report write completed before the next began. The reporter requires sequential writes to a destination.

The PR remains draft. Coverage is limited to the documented fixtures, and skill discovery relies on the collecting agent's account of how the client exposed each skill.