You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ddf63e9
Browse filesBrowse the repository at this point in the historyBrowse files
authored
fix(docker): close two driver:docker channels that leaked grading material to the agent (#179)
* feat(sandbox): 1/3 — delete staged task.yaml in-container after load (anti-cheat)
Fix A: make /work/input a writable mount (drop :ro; grant input dir
writable) and delete the staged task.yaml immediately after load_task in
the in-container entry point, gated on IN_CONTAINER_ENV. The staged file
is the post-override TaskDefinition with success_criteria, and the agent
runs in the same container, so leaving it readable hands over the grading
answer key. It is read exactly once; grading reads criteria from memory.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sandbox): 2/3 — allowlist tmpfs-mask of auto-mounted plugin trees
Fix B: default-deny mask over an auto-mounted Claude-plugin root. The
whole root stays :ro-mounted so the plugin loads, but every child dir
outside the keep-set (.claude-plugin + manifest-declared skill dirs) is
masked with an empty tmpfs. Colocated eval material — sibling task YAMLs,
reference solutions, tests/, node_modules/ — is masked by default so an
unknown layout can never leak. New pure helper eval_material.mask_dirs;
shared skill-dir resolver manifest_skill_dirs (renamed public in
agents/_skills.py) is the SSOT for "what is a skill dir".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(lint): 3/3 — CE065 flags eval material inside a skill dir
Fix C: a static lint rule (pytest class, CE055 template) that flags the one
residual the Fix B allowlist cannot close — a task_id: YAML or a resolved
reference.directory colocated INSIDE a skill dir reached through an in-repo
agent.plugins[].path / sandbox.template_sources[].path. Masking a skill dir
would hide the skill, so such material stays readable and leaks the grading
answer key under driver: docker. Reuses the shared manifest_skill_dirs
resolver (SSOT, import-identity asserted) with positive+negative sensors.
Documents both docker anti-cheat fixes in docs/DOCKER_ISOLATION.md and the
CE065 entry in CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: also delete context.json in-container (source_yaml leaked criteria)
Code review found Fix A closed only /work/input/task.yaml but left the identical
success_criteria readable in /work/input/context.json via source_yaml (top-level
AND inside every config_lineage entry / ConfigLineageEntry.source_yaml) — in a
dir this change makes world-readable+writable. The agent could cat context.json
to recover its grading answer key. Delete both staged files after they are read
into memory (_scrub_staged_task_yaml -> _scrub_staged_inputs); correct the
docstring and docs/DOCKER_ISOLATION.md (the 'context.json carries no criteria'
claim was false).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(harness): log deferred Low findings from anti-cheat review
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(anti-cheat): apply review Low findings L1/L2/L3
L1 — CE065 now flags ANY task_id: YAML or reference dir left readable under a
plugin root (not just inside a skill dir): it reuses the runtime mask_dirs (SSOT),
walks the readable surface, and catches a loose task_id: YAML *file* at the plugin
root too (a tmpfs masks a dir, not a single file). Renamed the class to
TestCE065EvalMaterialReadableUnderPluginRoot; added a loose-root-file sensor.
L2 — docker_runner resolves a relative agent.plugins[].path against the task-file
dir (new _resolve_mount_path), not CWD, matching reference/template/CE065
resolution so the static rule and the runtime inspect the same tree.
L3 — eval_material.mask_dirs no longer descends a kept path, so a degenerate
manifest skills: '.' masks nothing (fail-safe) instead of hiding the whole skill
surface; added a test.
Updated CLAUDE.md CE065 entry and docs. make verify green (the one failure is a
pre-existing flaky wall-clock datetime test in test_agent_telemetry, passes on
rerun, unrelated to these files).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(anti-cheat): apply multi-model review findings M1/M2/M3 + doc drift
Multi-model review (Opus + Fable + Sonnet) found no Critical/High and no leak
bypass; addressed the 3 Mediums and the doc drift:
M2 — nested plugin roots (plugin B under plugin A) produced a duplicate Docker
mount destination: A's mask emitted --tmpfs <A>/B while B's own mount bound
<A>/B:ro. Defer mask emission until all binds are known and drop any mask whose
path is also a bind (the bind wins so B loads and masks its own children).
Extracted the auto-mount block into _append_auto_mounts to stay under the
per-method lint limits. Added a nested-plugin test.
M1 — added direct tests for _resolve_mount_path (relative->task-file dir,
absolute unchanged, task_file None->cwd, env-var expansion); the L2 fix had no
coverage.
M3 — warn when a plugin root's mask stands down (manifest skills: '.') so the
anti-cheat mask never voids silently. Added a test.
Doc drift (L-a/L-b/L-c): corrected the now-false 'task.yaml readable at
/work/input' / 'hiding criteria is unsolved' claims in orchestrator.py,
fs_permissions.py and CLAUDE.md (Fix A deletes task.yaml + context.json after
load); widened DOCKER_ISOLATION.md's residual note to include the loose-root-file
case; marked the harness-candidates entry RESOLVED (nothing deferred).
make verify green (5758 passed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Bai Li <bai.li@uipath.com>
Copy file name to clipboardExpand all lines: docs/DOCKER_ISOLATION.md
+39Lines changed: 39 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -339,6 +339,45 @@ If the task declares a `reference:` block, a throwaway **copy** of its directory
339
339
340
340
See [Reference Solutions](TASK_DEFINITION_GUIDE.md#reference-solutions).
341
341
342
+
### Two more passive-read anti-cheat blocks
343
+
344
+
These are `driver: docker` only. `driver: tempdir` shares the host uid and has no
345
+
filesystem isolation, so neither applies there (nor can — there is nothing to
346
+
mask). Both are defense-in-depth passive-read blocks, consistent with the
347
+
reference window's posture above; neither contains an adversarial agent.
348
+
349
+
- **The staged grading inputs are deleted after load.** The host stages the
350
+
post-override `TaskDefinition` (with `success_criteria`) at `/work/input/task.yaml`
351
+
**and** a `context.json` whose `source_yaml` is the raw task text — criteria
352
+
verbatim, both at the top level and inside every `config_lineage` entry — for the
353
+
in-container orchestrator to load once at startup. The agent runs in the same
354
+
container, so leaving *either* readable would hand it the grading answer key
355
+
(deleting only `task.yaml` leaves the identical criteria one file over in
356
+
`context.json`). The in-container entry point deletes **both** immediately after
357
+
they are consumed — `context.json` is parsed into memory in the command body and
358
+
`task.yaml`by `load_task`, both before the delete (gated on
359
+
`CODER_EVAL_IN_CONTAINER`). They are read exactly once — grading reads criteria
360
+
from the in-memory task, never from disk. The `/work/input` mount is therefore
361
+
read-write (a `:ro` mount rejects `rm` with EROFS). `prior.json` is kept: it is
362
+
read later on the regrade path, and a regrade runs no agent so it is not a leak.
363
+
364
+
- **Auto-mounted plugin trees are default-deny masked.** An `agent.plugins[].path`
365
+
(or a `TemplateDirSource.path` that is itself a plugin root) is auto-mounted at
366
+
its host path `:ro` so the plugin loads. Eval material colocated under that tree
367
+
as siblings of the skills dir — sibling task YAMLs, reference solutions, test
368
+
fixtures — would otherwise be readable. So the runner keeps the whole root
369
+
mounted but layers an empty `--tmpfs` over every child dir OUTSIDE the keep-set
370
+
(`.claude-plugin` + the manifest-declared skill dirs). Everything that is not
371
+
the plugin surface is masked by default, so an unknown or new eval layout can
372
+
never leak; `tests/`, `node_modules/`, and reference solutions are masked for
373
+
free. A root agent cannot `umount` a tmpfs (`CAP_SYS_ADMIN` is not in Docker's
374
+
default set), so this mask is *stronger* than the mode-000 reference window. Two
375
+
residuals the mask cannot cover — an eval def or reference COLOCATED inside a
376
+
skill dir (masking it would hide the skill), and a `task_id:` YAML **file** loose
377
+
at the plugin root (a tmpfs masks a directory, not a single file) — are caught by
378
+
lint rule CE068 (keep eval material out of skill dirs and off the plugin root;
379
+
put it under a sibling `tests/`).
380
+
342
381
Inside the container, the entrypoint invokes `coder-eval _run-task-internal` (hidden subcommand), which loads the staged YAML + context, runs the standard in-process Orchestrator (driver auto-coerced back to `tempdir`), and writes `task.json` to the output mount. Host reads it and feeds the existing aggregation pipeline.
343
382
344
383
A `result_kind` discriminator on `CriterionResult` ensures `ClassificationCriterionResult` subclasses survive the JSON round-trip — without it, host-side aggregation would silently lose `observed_label`/`expected_label`.
0 commit comments