Repository navigation
Add optional Streamable HTTP conformance checks - #10
Merged
Merged
Conversation
jonathanhefner
force-pushed
the
agent/optional-http
branch
2 times, most recently
from
September 20, 2026 17:15
f5c3b89 to
902cf39
Compare
jonathanhefner
commented
Sep 20, 2026
| This plugin exercises valid stdio configurations; invalid configuration handling, failure isolation, and remote MCP transports are outside its coverage. | ||
| | `mcp.streamable-http.tool-availability` | The client exposes the HTTP tool and it returns a valid observation. A completed discovery or call attempt with no observation fails this check only when the reporter confirms that the local fixture is healthy. | | ||
| | `mcp.streamable-http.url.literal-route-and-query` | The tool request reaches `/conformance/mcp` with the single decoded query pair `value=$APC_HTTP_VALUE`, preserving the placeholder-like text literally. | | ||
| | `mcp.streamable-http.headers.literal-value` | The tool request includes `x-apc-fixture` with the literal value `${PLUGIN_ROOT}\|${PLUGIN_DATA}\|fixture value with spaces`. | |
Member
Author
There was a problem hiding this comment.
with the literal value
${PLUGIN_ROOT}\|${PLUGIN_DATA}\|fixture value with spaces
That description seems overly specific.
Comment on lines
+74
to
+89
| Every HTTP observation contains the native tool's `evidence` payload, or `null` when a completed discovery or call attempt produced no observation. The reporter checks the local fixture's health for each HTTP recording and adds `serverHealthCheck`, either `"passed"` or `"failed"`. For example, this report excerpt records an unsuccessful native attempt while the fixture was healthy: | ||
|
|
||
| ```json | ||
| { | ||
| "observations": [ | ||
| { | ||
| "kind": "mcp-streamable-http", | ||
| "server": "http", | ||
| "evidence": null, | ||
| "serverHealthCheck": "passed" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| Valid native evidence makes `mcp.streamable-http.tool-availability` pass, even if the health check fails. With `evidence: null`, availability fails when the health check passes and remains `not_verified` otherwise. The URL and header checks require native evidence. |
Member
Author
There was a problem hiding this comment.
Why are we going into this detail here? Is there a rationale, or did a subagent turn this README into an information dumping ground?
If it is the latter, do not let it happen again.
| Install these plugins through the client's Agent Plugins support: | ||
| Node.js 22 or newer must be available as `node` on the client's executable search path, and the agent must be able to execute commands. The packaged plugins require no build or dependency installation. Each directory below is an installable plugin; the repository root is the development project. | ||
|
|
||
| To include the optional Streamable HTTP checks, follow the [Core fixture's HTTP setup](plugins/agent-plugins-conformance-core/#optional-http-setup) **before the client loads the plugins**. This starts a bundled local server in a separate terminal; CI can launch the same command. Without the server, HTTP checks remain `not_verified` and the agent continues collecting the other checks. |
Member
Author
There was a problem hiding this comment.
I think this paragraph might be better positioned after the table below.
Ship a standalone HTTP fixture with core so users and CI can opt into HTTP checks by starting it before client loading. Observe configured URL and header values through the native MCP tool while leaving other checks usable without the server. Keep each HTTP attempt in observations, with native evidence or null when collection produces none. The reporter adds serverHealthCheck after independently checking the local HTTP server. A passed health check makes unsuccessful native collection fail availability; missing evidence alone remains not verified. Preserve the latest observation for each component and reevaluate saved evidence without network access.
jonathanhefner
force-pushed
the
agent/optional-http
branch
from
September 20, 2026 17:47
902cf39 to
93c1124
Compare
jonathanhefner
marked this pull request as ready for review
September 20, 2026 17:50
34 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add optional Streamable HTTP coverage with a standalone server bundled in the core fixture. Users or CI start it before the client loads the plugins; the guide collects its native MCP observation and the reporter checks literal URL/query and header delivery. Other checks continue when the server is absent.
Each HTTP entry in
observationscontains the nativeevidencepayload ornullafter an unsuccessful collection attempt. On every HTTP recording, the reporter checks the server's identifying health endpoint and addsserverHealthCheck: "passed" | "failed". Native evidence passes availability; without it, a passed health check makes availability fail and a failed health check leaves itnot_verified. Missing observations alone never produce failure. Recording replaces the previous observation for that component, and subsequent evaluation uses saved evidence without network access.Validation:
not_verified.not_verified, withevidence: nullandserverHealthCheck: "failed"recorded. Native evidence transfer and deterministic reevaluation match exactly; recording is sequential. The present run also recorded a failed reporter health check (cause not established); successful native evidence correctly kept HTTP availability passing. The separate real-endpoint test covers a passed reporter health check.