Repository navigation
Add combined recovery fixture coverage - #6
Merged
Merged
Conversation
Exercise continued availability of a valid skill and stdio MCP server alongside recoverable manifest, skill, and server errors in one plugin. Combining the invalid inputs bounds installation cost while keeping the guide independent of fixture loading. Extend `run-conformance` and deterministic reporting with two distinct availability witnesses. Missing evidence remains `not_verified`; positive observations make no claim about diagnostics or rejection of invalid siblings. Preserve core evidence, recording, and summary behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
agent-plugins-conformance-recoveryto check whether a valid skill and stdio MCP server remain available alongside recoverable invalid configuration. The fixture combines an unknown manifest field, non-objectextensions, a skill missingdescription, and a server entry missingcommandin one additional plugin.The existing
run-conformanceskill collects both recovery witnesses into the same JSON report as core. Two new cases measure continued availability; missing observations remainnot_verified. These cases do not establish whether invalid entries were skipped or diagnostics were emitted.Validation:
0.155.0-alpha.2.6: all three unchanged plugins installed, eight observations collected, all 17 cases passed. The guide initialized once and recorded each observation before collecting the next.1.0.83on Node24.18.0: seven observations, 14 passed, two failed, onenot_verified. The recovery skill was available; Copilot rejected the recovery MCP configuration because the invalid entry lackscommand, so the valid server was unavailable. The two failures are the same core argument/environment expansion differences observed before this PR.Independent trace review confirmed that both native runs used the README prompt and exact commit
d067c50eaff8f6df35af70816ad8f1148f0e55c8. Saved observations faithfully match the client evidence, and both reports match fresh deterministic evaluation. Copilot collected four core MCP observations together before recording them, contrary to the guide's instruction to record immediately; its recording commands were sequential. That deviation is preserved in the validation evidence.