You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 577aa38
Browse filesBrowse the repository at this point in the historyBrowse files
Check plugin data writability through native observations
The existing data-path check does not establish that a plugin subprocess
can write there. Have each default `observe` call exclusively create,
write, close, and remove a temporary file in `PLUGIN_DATA`, preserving
operation facts for deterministic evaluation.
Keep cleanup warnings separate from writability outcomes and render
saved warnings in human summaries. Repeat the write on each observation
so clients that retain MCP processes still collect current evidence.
Copy file name to clipboardExpand all lines: README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ Plugins for collecting evidence about client support for [Agent Plugins](https:/
6
6
7
7
| Plugin | Coverage |
8
8
| --- | --- |
9
-
|[Agent Plugins Conformance — Core](plugins/agent-plugins-conformance-core/)| Skill discovery, MCP tool availability, working directories, and selected subprocess environment and placeholder rules |
9
+
|[Agent Plugins Conformance — Core](plugins/agent-plugins-conformance-core/)| Skill discovery, MCP tool availability, working directories, plugin data writability, and selected subprocess environment and placeholder rules |
10
10
11
11
Install **`plugins/agent-plugins-conformance-core`** through the client's Agent Plugins support. Each directory under `plugins/` is an installable plugin; the repository root is the development project.
Copy file name to clipboardExpand all lines: plugins/agent-plugins-conformance-core/README.md
+11-6Lines changed: 11 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Agent Plugins Conformance — Core
2
2
3
-
A portable [Agent Plugins](https://agent-plugins.org/) plugin for collecting evidence about skill discovery, MCP tool availability, working directories, and selected subprocess environment and placeholder rules. It provides three [Agent Skills](https://agentskills.io/specification) and four stdio MCP servers. A deterministic reporter evaluates their observations against fixed fixtures.
3
+
A portable [Agent Plugins](https://agent-plugins.org/) plugin for collecting evidence about skill discovery, MCP tool availability, working directories, plugin data writability, and selected subprocess environment and placeholder rules. It provides three [Agent Skills](https://agentskills.io/specification) and four stdio MCP servers. A deterministic reporter evaluates their observations against fixed fixtures.
4
4
5
5
Results describe the observed run and these covered cases; they are not whole-client certification.
6
6
@@ -52,16 +52,18 @@ The following example is abbreviated: observations, notes, and some results are
52
52
}
53
53
],
54
54
"summary": {
55
-
"pass": 6,
55
+
"pass": 7,
56
56
"fail": 1,
57
57
"not_verified": 7,
58
-
"total": 14
58
+
"total": 15
59
59
}
60
60
}
61
61
```
62
62
63
63
Each result identifies its case through `id` and `label`, explains the outcome in `detail`, and cites the relevant specification sections in `specSections`.
64
64
65
+
An optional `warning` reports a probe cleanup failure, including the probe file's path and error. It does not change the check's status.
66
+
65
67
The three statuses mean:
66
68
67
69
| Status | Meaning |
@@ -129,7 +131,7 @@ jq '
129
131
130
132
### Summarize a saved report
131
133
132
-
The summarizer displays saved outcomes as a table with explanations for checks that failed or were not verified. From the plugin directory, run:
134
+
The summarizer displays saved outcomes as a table with explanations for checks that failed or were not verified, followed by any warnings. From the plugin directory, run:
@@ -141,7 +143,7 @@ Reporting and summarizing errors are command failures. Producing a valid report
141
143
142
144
## Coverage
143
145
144
-
The environmentand argument checks use the `default` server's observation; each of the other servers supplies tool-availability and working-directory evidence.
146
+
The environment, argument, and plugin data writability checks use the `default` server's observation; each of the other servers supplies tool-availability and working-directory evidence.
145
147
146
148
| Case IDs | What the check establishes |
147
149
| --- | --- |
@@ -151,9 +153,12 @@ The environment and argument checks use the `default` server's observation; each
151
153
|`mcp.stdio.cwd.plugin-relative`, `mcp.stdio.cwd.plugin-root`| Relative and root-placeholder `cwd` resolve to `probe-workdir`. |
152
154
|`mcp.stdio.cwd.plugin-data`| Data-placeholder `cwd` matches the data path resolved by the probe. |
153
155
|`mcp.stdio.env.plugin-root`| Supplied `PLUGIN_ROOT` matches the independently determined package root. |
154
-
|`mcp.stdio.env.plugin-data-absolute`| Supplied `PLUGIN_DATA` is an absolute path. This check does not exercise storage dedication, writability, or persistence across updates. |
156
+
|`mcp.stdio.env.plugin-data-absolute`| Supplied `PLUGIN_DATA` is an absolute path. This check does not exercise writability, storage dedication, or persistence across updates. |
157
+
|`mcp.stdio.data.writable`| The default server can create, write, and close a small temporary file directly in the supplied `PLUGIN_DATA` directory. |
155
158
|`mcp.stdio.env.configured-value`| The configured value reaches the subprocess. The fixture does not arrange a conflicting inherited value, so it does not establish override behavior. |
156
159
|`mcp.stdio.args.preservation-and-expansion`| Argument boundaries, including spaces and an empty argument, are preserved; recognized placeholders expand and unknown placeholder-like text stays literal. |
157
160
|`mcp.stdio.env.expansion`| Repeated recognized placeholders expand in environment values and unknown placeholder-like text stays literal. |
158
161
162
+
Each default observation attempts a fresh write using a unique, exclusively created file, then removes that file. Missing or relative `PLUGIN_DATA` leaves writability `not_verified`; an unusable absolute path fails. Cleanup failures appear as warnings in JSON and the requested human summary without changing a successful write result.
163
+
159
164
This plugin exercises valid stdio configurations; invalid configuration handling, failure isolation, and remote MCP transports are outside its coverage.
Copy file name to clipboardExpand all lines: plugins/agent-plugins-conformance-core/skills/conformance-guide/SKILL.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -52,7 +52,7 @@ Replace `observation` with the complete observation just obtained. The script re
52
52
53
53
1. Record the guide observation above only if the client exposed this skill and you loaded its body through the client's normal skill mechanism. A client-provided skill catalog followed by reading its advertised resource is a valid mechanism. Locating a known skill by filesystem search or receiving its body in a prompt does not establish client discovery.
54
54
2. Find and load `conformance-alpha` and `conformance-beta` through that same client mechanism. Record the observation each loaded body supplies, one at a time. Do not infer their markers or claim discovery from files located independently of the client.
55
-
3. Find each server's `observe` tool: `default`, `relative`, `root`, and `data`. Tool names may be client-namespaced. Call each available tool with `{}`. Take its observation from `structuredContent` (some clients show `structured_content`), or parse the JSON in the tool's text content. Immediately record that complete object. It begins with `{"kind":"mcp-stdio","server":...}`; exclude the MCP result wrapper containing `content` or `structuredContent`. Preserve the returned paths and values exactly.
55
+
3. Find each server's `observe` tool: `default`, `relative`, `root`, and `data`. Tool names may be client-namespaced. Call each available tool with `{}`. The default tool also tests data-directory writability by creating, writing, and removing its own temporary file. Take each observation from `structuredContent` (some clients show `structured_content`), or parse the JSON in the tool's text content. Immediately record that complete object. It begins with `{"kind":"mcp-stdio","server":...}`; exclude the MCP result wrapper containing `content` or `structuredContent`. Preserve the returned paths and values exactly.
56
56
4. Omit observations for unavailable skills, missing tools, or calls that did not return evidence. Keep track of those collection limitations for your final response. Missing evidence remains `not_verified`; it does not by itself establish failure.
57
57
58
58
A recording error means that observation was not successfully saved. Address an input or command error using the actual evidence, or explicitly identify the unsaved observation in your final response. Distinguish a failed write from unavailable probe evidence. Retry an individual recording with `record`; invoking `start` again discards previously collected evidence.
@@ -31,9 +32,11 @@ const server = new Server({ name: `agent-plugins-conformance-${serverName}`, ver
31
32
{capabilities: {tools: {}}});
32
33
constobserveTool={
33
34
name: 'observe',
34
-
description: 'Return this process launch evidence. Record the observation object from structuredContent (or parsed JSON text) unchanged with the conformance guide reporter; exclude the MCP result wrapper. Reads only fixture variables.',
35
+
description: 'Return this process launch evidence, including only fixture environment variables.'+
36
+
(serverName==='default' ? ' Each call also creates, writes, closes, and removes a uniquely named temporary file directly in an absolute PLUGIN_DATA directory, recording any operation or cleanup error.' : '')+
37
+
' Record the observation object from structuredContent (or parsed JSON text) unchanged with the conformance guide reporter; exclude the MCP result wrapper.',
0 commit comments