Skip to content

Commit 577aa38

Browse files
Check plugin data writability through native observations
The existing data-path check does not establish that a plugin subprocess can write there. Have each default `observe` call exclusively create, write, close, and remove a temporary file in `PLUGIN_DATA`, preserving operation facts for deterministic evaluation. Keep cleanup warnings separate from writability outcomes and render saved warnings in human summaries. Repeat the write on each observation so clients that retain MCP processes still collect current evidence.
1 parent b251355 commit 577aa38

15 files changed

Lines changed: 484 additions & 38 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Plugins for collecting evidence about client support for [Agent Plugins](https:/
66

77
| Plugin | Coverage |
88
| --- | --- |
9-
| [Agent Plugins Conformance — Core](plugins/agent-plugins-conformance-core/) | Skill discovery, MCP tool availability, working directories, and selected subprocess environment and placeholder rules |
9+
| [Agent Plugins Conformance — Core](plugins/agent-plugins-conformance-core/) | Skill discovery, MCP tool availability, working directories, plugin data writability, and selected subprocess environment and placeholder rules |
1010

1111
Install **`plugins/agent-plugins-conformance-core`** through the client's Agent Plugins support. Each directory under `plugins/` is an installable plugin; the repository root is the development project.
1212

‎plugins/agent-plugins-conformance-core/README.md‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Agent Plugins Conformance — Core
22

3-
A portable [Agent Plugins](https://agent-plugins.org/) plugin for collecting evidence about skill discovery, MCP tool availability, working directories, and selected subprocess environment and placeholder rules. It provides three [Agent Skills](https://agentskills.io/specification) and four stdio MCP servers. A deterministic reporter evaluates their observations against fixed fixtures.
3+
A portable [Agent Plugins](https://agent-plugins.org/) plugin for collecting evidence about skill discovery, MCP tool availability, working directories, plugin data writability, and selected subprocess environment and placeholder rules. It provides three [Agent Skills](https://agentskills.io/specification) and four stdio MCP servers. A deterministic reporter evaluates their observations against fixed fixtures.
44

55
Results describe the observed run and these covered cases; they are not whole-client certification.
66

@@ -52,16 +52,18 @@ The following example is abbreviated: observations, notes, and some results are
5252
}
5353
],
5454
"summary": {
55-
"pass": 6,
55+
"pass": 7,
5656
"fail": 1,
5757
"not_verified": 7,
58-
"total": 14
58+
"total": 15
5959
}
6060
}
6161
```
6262

6363
Each result identifies its case through `id` and `label`, explains the outcome in `detail`, and cites the relevant specification sections in `specSections`.
6464

65+
An optional `warning` reports a probe cleanup failure, including the probe file's path and error. It does not change the check's status.
66+
6567
The three statuses mean:
6668

6769
| Status | Meaning |
@@ -129,7 +131,7 @@ jq '
129131

130132
### Summarize a saved report
131133

132-
The summarizer displays saved outcomes as a table with explanations for checks that failed or were not verified. From the plugin directory, run:
134+
The summarizer displays saved outcomes as a table with explanations for checks that failed or were not verified, followed by any warnings. From the plugin directory, run:
133135

134136
```sh
135137
node skills/conformance-guide/scripts/summarize.mjs '/absolute/path/to/report.json'
@@ -141,7 +143,7 @@ Reporting and summarizing errors are command failures. Producing a valid report
141143

142144
## Coverage
143145

144-
The environment and argument checks use the `default` server's observation; each of the other servers supplies tool-availability and working-directory evidence.
146+
The environment, argument, and plugin data writability checks use the `default` server's observation; each of the other servers supplies tool-availability and working-directory evidence.
145147

146148
| Case IDs | What the check establishes |
147149
| --- | --- |
@@ -151,9 +153,12 @@ The environment and argument checks use the `default` server's observation; each
151153
| `mcp.stdio.cwd.plugin-relative`, `mcp.stdio.cwd.plugin-root` | Relative and root-placeholder `cwd` resolve to `probe-workdir`. |
152154
| `mcp.stdio.cwd.plugin-data` | Data-placeholder `cwd` matches the data path resolved by the probe. |
153155
| `mcp.stdio.env.plugin-root` | Supplied `PLUGIN_ROOT` matches the independently determined package root. |
154-
| `mcp.stdio.env.plugin-data-absolute` | Supplied `PLUGIN_DATA` is an absolute path. This check does not exercise storage dedication, writability, or persistence across updates. |
156+
| `mcp.stdio.env.plugin-data-absolute` | Supplied `PLUGIN_DATA` is an absolute path. This check does not exercise writability, storage dedication, or persistence across updates. |
157+
| `mcp.stdio.data.writable` | The default server can create, write, and close a small temporary file directly in the supplied `PLUGIN_DATA` directory. |
155158
| `mcp.stdio.env.configured-value` | The configured value reaches the subprocess. The fixture does not arrange a conflicting inherited value, so it does not establish override behavior. |
156159
| `mcp.stdio.args.preservation-and-expansion` | Argument boundaries, including spaces and an empty argument, are preserved; recognized placeholders expand and unknown placeholder-like text stays literal. |
157160
| `mcp.stdio.env.expansion` | Repeated recognized placeholders expand in environment values and unknown placeholder-like text stays literal. |
158161

162+
Each default observation attempts a fresh write using a unique, exclusively created file, then removes that file. Missing or relative `PLUGIN_DATA` leaves writability `not_verified`; an unusable absolute path fails. Cleanup failures appear as warnings in JSON and the requested human summary without changing a successful write result.
163+
159164
This plugin exercises valid stdio configurations; invalid configuration handling, failure isolation, and remote MCP transports are outside its coverage.

‎plugins/agent-plugins-conformance-core/dist/probe.mjs‎

Lines changed: 44 additions & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎plugins/agent-plugins-conformance-core/skills/conformance-guide/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,7 @@ Replace `observation` with the complete observation just obtained. The script re
5252

5353
1. Record the guide observation above only if the client exposed this skill and you loaded its body through the client's normal skill mechanism. A client-provided skill catalog followed by reading its advertised resource is a valid mechanism. Locating a known skill by filesystem search or receiving its body in a prompt does not establish client discovery.
5454
2. Find and load `conformance-alpha` and `conformance-beta` through that same client mechanism. Record the observation each loaded body supplies, one at a time. Do not infer their markers or claim discovery from files located independently of the client.
55-
3. Find each server's `observe` tool: `default`, `relative`, `root`, and `data`. Tool names may be client-namespaced. Call each available tool with `{}`. Take its observation from `structuredContent` (some clients show `structured_content`), or parse the JSON in the tool's text content. Immediately record that complete object. It begins with `{"kind":"mcp-stdio","server":...}`; exclude the MCP result wrapper containing `content` or `structuredContent`. Preserve the returned paths and values exactly.
55+
3. Find each server's `observe` tool: `default`, `relative`, `root`, and `data`. Tool names may be client-namespaced. Call each available tool with `{}`. The default tool also tests data-directory writability by creating, writing, and removing its own temporary file. Take each observation from `structuredContent` (some clients show `structured_content`), or parse the JSON in the tool's text content. Immediately record that complete object. It begins with `{"kind":"mcp-stdio","server":...}`; exclude the MCP result wrapper containing `content` or `structuredContent`. Preserve the returned paths and values exactly.
5656
4. Omit observations for unavailable skills, missing tools, or calls that did not return evidence. Keep track of those collection limitations for your final response. Missing evidence remains `not_verified`; it does not by itself establish failure.
5757

5858
A recording error means that observation was not successfully saved. Address an input or command error using the actual evidence, or explicitly identify the unsaved observation in your final response. Distinguish a failed write from unavailable probe evidence. Retry an individual recording with `record`; invoking `start` again discards previously collected evidence.

‎plugins/agent-plugins-conformance-core/src/cases.mjs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ export const CASES = Object.freeze([
1313
]),
1414
['mcp.stdio.env.plugin-root', 'Plugin root environment', ['9.1']],
1515
['mcp.stdio.env.plugin-data-absolute', 'Plugin data environment', ['9.1']],
16+
['mcp.stdio.data.writable', 'Plugin data writability', ['9.1']],
1617
['mcp.stdio.env.configured-value', 'Configured environment', ['9.1']],
1718
['mcp.stdio.args.preservation-and-expansion', 'Argument preservation and expansion', ['7.2.1', '9.2']],
1819
['mcp.stdio.env.expansion', 'Environment expansion', ['9.2']],
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
import fs from 'node:fs';
2+
import { randomUUID } from 'node:crypto';
3+
import { isAbsolute, join } from 'node:path';
4+
5+
const errorFacts = (error) => ({ code: error.code ?? null, message: error.message });
6+
7+
export function observeDataWrite(data) {
8+
if (!data || !isAbsolute(data)) return null;
9+
const observation = {
10+
path: join(data, `.agent-plugins-conformance-${randomUUID()}.tmp`),
11+
error: null,
12+
cleanupError: null,
13+
};
14+
let descriptor;
15+
let operation = 'create';
16+
try {
17+
descriptor = fs.openSync(observation.path, 'wx');
18+
operation = 'write';
19+
fs.writeFileSync(descriptor, 'Agent Plugins conformance write probe\n');
20+
} catch (error) {
21+
observation.error = { operation, ...errorFacts(error) };
22+
} finally {
23+
// Only a successful exclusive create grants ownership of this file.
24+
if (descriptor !== undefined) {
25+
try {
26+
fs.closeSync(descriptor);
27+
} catch (error) {
28+
observation.error ??= { operation: 'close', ...errorFacts(error) };
29+
}
30+
try {
31+
fs.unlinkSync(observation.path);
32+
} catch (error) {
33+
observation.cleanupError = errorFacts(error);
34+
}
35+
}
36+
}
37+
return observation;
38+
}

‎plugins/agent-plugins-conformance-core/src/probe.mjs‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import { isAbsolute } from 'node:path';
44
import { Server } from '@modelcontextprotocol/sdk/server/index.js';
55
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
66
import { CallToolRequestSchema, ListToolsRequestSchema } from '@modelcontextprotocol/sdk/types.js';
7+
import { observeDataWrite } from './data-write.mjs';
78

89
const serverName = process.argv[2];
910
if (!['default', 'relative', 'root', 'data'].includes(serverName)) {
@@ -31,9 +32,11 @@ const server = new Server({ name: `agent-plugins-conformance-${serverName}`, ver
3132
{ capabilities: { tools: {} } });
3233
const observeTool = {
3334
name: 'observe',
34-
description: 'Return this process launch evidence. Record the observation object from structuredContent (or parsed JSON text) unchanged with the conformance guide reporter; exclude the MCP result wrapper. Reads only fixture variables.',
35+
description: 'Return this process launch evidence, including only fixture environment variables.' +
36+
(serverName === 'default' ? ' Each call also creates, writes, closes, and removes a uniquely named temporary file directly in an absolute PLUGIN_DATA directory, recording any operation or cleanup error.' : '') +
37+
' Record the observation object from structuredContent (or parsed JSON text) unchanged with the conformance guide reporter; exclude the MCP result wrapper.',
3538
inputSchema: { type: 'object', properties: {}, additionalProperties: false },
36-
annotations: { readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: false },
39+
annotations: { readOnlyHint: serverName !== 'default', destructiveHint: false, idempotentHint: serverName !== 'default', openWorldHint: false },
3740
};
3841
server.setRequestHandler(ListToolsRequestSchema, async () => ({ tools: [observeTool] }));
3942
server.setRequestHandler(CallToolRequestSchema, async ({ params }) => {
@@ -44,6 +47,7 @@ server.setRequestHandler(CallToolRequestSchema, async ({ params }) => {
4447
throw new Error('observe requires an empty object');
4548
}
4649
const observation = { kind: 'mcp-stdio', server: serverName, evidence: { version: 1, ...launch } };
50+
if (serverName === 'default') observation.evidence.dataWrite = observeDataWrite(process.env.PLUGIN_DATA);
4751
return { content: [{ type: 'text', text: JSON.stringify(observation) }], structuredContent: observation };
4852
}
4953
throw new Error(`Unknown tool: ${params.name}`);

‎plugins/agent-plugins-conformance-core/src/report-format.mjs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ export function validateSavedReport(report) {
3030
const at = `report.results[${index}]`;
3131
object(result, at);
3232
for (const key of ['id', 'label', 'detail']) string(result[key], `${at}.${key}`);
33+
if (Object.hasOwn(result, 'warning')) string(result.warning, `${at}.warning`);
3334
if (ids.has(result.id)) invalid(`${at}.id`, 'duplicate result ID');
3435
ids.add(result.id);
3536
if (!STATUSES.includes(result.status)) invalid(`${at}.status`, `expected one of: ${STATUSES.join(', ')}`);
@@ -79,5 +80,12 @@ export function formatReport(report) {
7980
lines.push(` ${escapeText(label)} (${escapeText(id)})`, ` ${escapeText(detail)}`);
8081
}
8182
}
83+
const warnings = report.results.filter((result) => result.warning !== undefined);
84+
if (warnings.length) {
85+
lines.push('', 'Warnings');
86+
for (const { id, label, warning } of warnings) {
87+
lines.push(` ${escapeText(label)} (${escapeText(id)})`, ` ${escapeText(warning)}`);
88+
}
89+
}
8290
return lines.join('\n');
8391
}

0 commit comments

Comments
 (0)