Repository navigation
Extract the conformance entry point from core fixtures - #5
Merged
Merged
Conversation
Keep the collection procedure and deterministic reporting in the independent `agent-plugins-conformance` plugin, exposed through `run-conformance`. Reporting can initialize even when the core fixture is unavailable, and fixture coverage can grow behind one skill invocation and JSON destination. Leave the core probes and two discovery witnesses in `agent-plugins-conformance-core`. Evaluate core discovery from those witnesses alone, preserve the report contract, and package each plugin with its own runtime dependencies and licenses.
jonathanhefner
marked this pull request as ready for review
September 18, 2026 18:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The collection guide currently lives inside the core fixture, so losing that fixture can also remove the reporting entry point. Extract it into
agent-plugins-conformance, with therun-conformanceskill, recording and summarizing scripts, and deterministic evaluator packaged together.Install the primary plugin alongside
agent-plugins-conformance-coreand invoke one skill with the JSON destination. Core retains its two marker skills and four MCP probes; only the core markers contribute to its discovery result. Recording semantics and result IDs are preserved. Documentation separates running and consuming the report from fixture coverage.Validation:
Native validation used unchanged packages from
f85f288and the documented skill request:Both complete runs collected all six observations unchanged, recorded each before requesting the next, and produced reports equal to deterministic evaluation. The primary-only run initialized its report and accurately identified unavailable core components. Installed packages remained unchanged.
Copilot reports the same two client expansion failures observed before this change:
$APC_VALUEexpands in arguments, and${PLUGIN_DATA}remains literal inAPC_EXPANSION.