From 9cc741f1cd051282e6a1d396bffc54de9c2b91df Mon Sep 17 00:00:00 2001 From: Ari Choudhury Date: Wed, 17 Jun 2026 06:47:21 +0000 Subject: [PATCH] evals: add skills eval suite + CI gate (PR 1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A standalone, tiered eval suite for the Flox skill (not just a consolidation gate). 22 tasks across should/may/stretch tiers covering: functional flows, implicit triggering (prompts that never say "flox"), search/show/install probes, and per-skill incl. indirect references (PyTorch/CUDA-in-README, etc.). Scored by deterministic hard-checks + an LLM judge. `run.py --gate` binds only on the deterministic hard-checks of functional should-tier tasks; the LLM judge score and the implicit-trigger rate are probabilistic, so they're reported (avg_judge_score, judge_correct_rate, should_trigger_rate) but never fail the build. CI workflow (.github/workflows/evals.yml) runs it on PRs using the org-managed MANAGED_SKILLS_ANTHROPIC_API_KEY secret. Baselines committed here are the pre-consolidation (7-skill) runs: skills-only : 22/22 hard, 100% functional correct, judge 4.68, trigger 11/11 skills+MCP : 22/22 hard, judge 4.64, trigger 11/11 (MCP adds nothing — AI-93) Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/evals.yml | 49 ++++ evals/.gitignore | 1 + evals/README.md | 96 +++++++ evals/flox-mcp.json | 8 + evals/results/skills.json | 483 ++++++++++++++++++++++++++++++++++ evals/results/skills_mcp.json | 480 +++++++++++++++++++++++++++++++++ evals/run.py | 231 ++++++++++++++++ evals/tasks.jsonl | 22 ++ 8 files changed, 1370 insertions(+) create mode 100644 .github/workflows/evals.yml create mode 100644 evals/.gitignore create mode 100644 evals/README.md create mode 100644 evals/flox-mcp.json create mode 100644 evals/results/skills.json create mode 100644 evals/results/skills_mcp.json create mode 100644 evals/run.py create mode 100644 evals/tasks.jsonl diff --git a/.github/workflows/evals.yml b/.github/workflows/evals.yml new file mode 100644 index 0000000..ce79a39 --- /dev/null +++ b/.github/workflows/evals.yml @@ -0,0 +1,49 @@ +name: skill-evals + +# Gates changes to the Flox skill / eval suite on the eval harness: +# run.py --gate exits non-zero only if a functional `should`-tier task fails its +# deterministic hard-checks. Judge score and implicit-triggering rate are +# probabilistic, so they're reported but never block. Agent + judge run via +# `claude`, authenticated with the org-managed Anthropic key. +on: + pull_request: + paths: + - "flox-plugin/**" + - "evals/**" + - ".github/workflows/evals.yml" + workflow_dispatch: {} + +concurrency: + group: skill-evals-${{ github.ref }} + cancel-in-progress: true + +jobs: + evals: + runs-on: ubuntu-latest + timeout-minutes: 60 + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5 + with: + python-version: "3.11" + + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 + with: + node-version: "20" + + - name: Install Claude Code CLI + run: npm install -g @anthropic-ai/claude-code + + - name: Run skills eval gate + env: + # Org-managed key (provisioned for the skills repo); `claude` reads ANTHROPIC_API_KEY. + ANTHROPIC_API_KEY: ${{ secrets.MANAGED_SKILLS_ANTHROPIC_API_KEY }} + run: python3 evals/run.py --mode skills --gate --concurrency 8 + + - name: Upload eval results + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: eval-results + path: evals/results/skills.json diff --git a/evals/.gitignore b/evals/.gitignore new file mode 100644 index 0000000..7d5b1c8 --- /dev/null +++ b/evals/.gitignore @@ -0,0 +1 @@ +*_run.log diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..1c77b37 --- /dev/null +++ b/evals/README.md @@ -0,0 +1,96 @@ +# Flox skills evals + +A standalone eval suite for the Flox skill: tasks scored for correctness and +quality, used to catch regressions and to measure whether the skill triggers on +the right requests and supplies the right context. (It also gated the +consolidation work — each merge step had to hold or beat the recorded baseline — +but the suite stands on its own beyond that.) + +Tasks are tiered by triggering expectation: **should** (must trigger + be +correct; these bind the gate), **may** (nice if it triggers), and **stretch** +(we'd like it to, but it's fine if not — these never fail the gate). Many +prompts deliberately never mention "flox", to test that the skill fires when a +user just wants "a new project" / "add nodejs 18.4" / "a PyTorch GPU setup". + +## What it does + +For every task in `tasks.jsonl`, runs `claude` headless with the Flox plugin +loaded and scores the answer two ways: + +- **Hard checks** — deterministic regex checks (no hallucinated install URL, no + absolute paths in manifests, required manifest sections present, correct + commands used). +- **LLM judge** — a separate `claude` call grades the answer 1–5 against the + task's rubric and returns a pass/fail. + +### Trigger tests + +Tasks with `"trigger_test": true` check **implicit triggering**: the prompt never +says "flox" (e.g. "create a new Node.js project"), and a neutral instruction is +used so nothing biases the model toward Flox. The `invokes_flox` hard-check then +verifies the skill fired anyway and produced Flox guidance. These guard the +behavior the retired MCP server used to encourage. + +Two arms (this is the AI-93 comparison): + +| Mode | Flag | Needs | +|------|------|-------| +| `skills` | `--strict-mcp-config` (MCP off) | nothing extra | +| `skills+mcp` | `--mcp-config flox-mcp.json` | nothing extra — `flox-mcp.json` runs the public MCP server via `flox activate -r flox/flox-mcp-server -- flox-mcp` (no FloxHub login required) | + +## Run + +```bash +python3 run.py --mode skills # skills-only baseline +python3 run.py --mode skills+mcp # skills + MCP (needs flox-mcp) +python3 run.py --mode skills --only node-env # single task +python3 run.py --mode skills --gate # exit non-zero if binding gates fail (CI) +``` + +Results land in `results/.json` with a summary (hard-pass rate, avg judge +score, correct rate). Pure stdlib — no node/uv required. + +## Authentication + +The harness shells out to `claude`, which needs credentials: + +- **Locally** it uses whatever the `claude` CLI is logged in with (an OAuth + token in `~/.claude/.credentials.json`, e.g. a Claude subscription) — usage + counts against that account. +- **In CI** there is no local login, so the workflow + (`.github/workflows/evals.yml`) exports the org-managed + `MANAGED_SKILLS_ANTHROPIC_API_KEY` secret as `ANTHROPIC_API_KEY`; `claude` uses + it for both the agent and judge calls. The job runs `run.py --mode skills + --gate`, which fails the build only if a functional `should`-tier task fails + its deterministic hard-checks (judge score and triggering are reported, not + gated — see Gate policy below). + +## Baselines + +Recorded on the original 7-skill layout (pre-consolidation): + +| Arm | Hard-check pass | Avg judge score | Correct rate | File | +|-----|-----------------|-----------------|--------------|------| +| skills-only | 8/8 (100%) | 4.62 / 5 | 8/8 | `results/skills.json` | +| skills + MCP | 8/8 (100%) | 4.25 / 5 | 8/8 | `results/skills_mcp.json` | + +**AI-93 finding:** skills-only is at least as good as skills+MCP (the delta is +within LLM-judge run-to-run noise; both arms are 100% correct and 100% +hard-pass). No measurable context gap from removing the MCP. + +## Gate policy + +Both the LLM judge (integer 1–5, plus a binary "correct") and implicit +triggering are **probabilistic run-to-run** — a single task can flip +correct↔incorrect or trigger↔not-trigger between identical runs. Binding CI on +those would make it flaky (15 functional tasks at ~95% judge reliability ≈ 46% +chance of an all-green run). So `--gate` is split: + +- **Binding (deterministic):** every **functional `should`-tier** task must pass + its **hard-checks** (no hallucinated Flox install, required manifest sections, + correct commands, etc.). These are regex-deterministic and stable; a failure + blocks the build. +- **Advisory (reported, never blocks):** `avg_judge_score`, `judge_correct_rate`, + per-tier breakdown, and `should_trigger_rate`. These are tracked as quality/ + triggering trends (watch for a sustained drop), not pass/fail gates. `may` and + `stretch` tasks are advisory by definition. diff --git a/evals/flox-mcp.json b/evals/flox-mcp.json new file mode 100644 index 0000000..8d61e70 --- /dev/null +++ b/evals/flox-mcp.json @@ -0,0 +1,8 @@ +{ + "mcpServers": { + "flox-mcp": { + "command": "flox", + "args": ["activate", "-r", "flox/flox-mcp-server", "--", "flox-mcp"] + } + } +} diff --git a/evals/results/skills.json b/evals/results/skills.json new file mode 100644 index 0000000..ebbfc70 --- /dev/null +++ b/evals/results/skills.json @@ -0,0 +1,483 @@ +{ + "summary": { + "mode": "skills", + "n_tasks": 22, + "n_errors": 0, + "n": 22, + "hard_pass_rate": 1.0, + "avg_judge_score": 4.68, + "judge_correct_rate": 1.0, + "by_tier": { + "should": { + "n": 15, + "hard_pass_rate": 1.0, + "avg_judge_score": 4.6, + "judge_correct_rate": 1.0 + }, + "may": { + "n": 6, + "hard_pass_rate": 1.0, + "avg_judge_score": 4.83, + "judge_correct_rate": 1.0 + }, + "stretch": { + "n": 1, + "hard_pass_rate": 1.0, + "avg_judge_score": 5.0, + "judge_correct_rate": 1.0 + } + }, + "n_trigger_tasks": 11, + "trigger_invokes_flox_rate": 1.0, + "should_trigger_rate": 1.0 + }, + "results": [ + { + "id": "env-python-postgres", + "area": "environments", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "no_abs_paths": true, + "has_install_section": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Commented-out `is-daemon = false` is not a real Flox service option (harmless since disabled)", + "`flox edit -f manifest.toml` workflow assumes a file the user must create separately, slightly glossed over" + ] + }, + "answer_excerpt": "Here's a complete, reproducible Flox environment for a Python web app backed by PostgreSQL.\n\n## Manifest\n\n`.flox/env/manifest.toml`:\n\n```toml\nversion = 1\n\n[install]\npython.pkg-path = \"python311Full\"\nuv.pkg-path = \"uv\"\npostgresql.pkg-path = \"postgresql_16\"\n\n[vars]\n# Web app\nAPP_HOST = \"127.0.0.1\"\nAPP_PORT = \"8000\"\n# Postgres \u2014 connection coordinates the service and your app both read\nPGHOST = \"127.0.0.1\"\nPGPORT = \"5432\"\nPGUSER = \"postgres\"\nPGDATABASE = \"app\"\nDATABASE_URL = \"postgresql://postgres@127.0.0.1:5432/app\"\n\n[hook.on-activate]\nscript = '''\n# --- Python virtualenv (lives in cache so it survives rebuilds) ---\nexport UV_CACHE_DIR=\"$FLOX_ENV_CACHE/uv\"\nexport venv=\"$FLOX_ENV_CACHE/venv\"\n\nif [ ! -d \"$venv\" ]; then\n uv venv \"$venv\" --python python3\nfi\nif [ -f \"$venv/bin/activate\" ]; then\n source \"$venv/bin/activate\"\nfi\n\n# Install deps once (re-run `flox activate` after editing requirements.txt\n# and `rm $FLOX_ENV_CACHE/.deps_installed` to force a refresh)\nif [ -f \"$FLOX_ENV_PROJECT/requirements.txt\" ] && [ ! -f \"$FLOX_ENV_CACHE/.deps_installed\" ]; then\n uv pip install --python \"$venv/bin/python\" --quiet -r \"$FLOX_ENV_PROJECT/requirements.txt\"\n touch \"$FLOX_ENV_CACHE/.deps_insta" + }, + { + "id": "node-env", + "area": "environments", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "no_abs_paths": true, + "has_install_section": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Mentions an internal 'flox-services' skill, which is irrelevant noise to an end user" + ] + }, + "answer_excerpt": "Here's the complete solution for a Flox environment pinned to Node.js 20.\n\n## Commands\n\n```bash\nflox init # creates .flox/ in your project directory\nflox install nodejs # adds Node.js (then pin to ^20.0 in the manifest)\nflox activate # enter the environment\n```\n\nIn practice, run `flox init`, then edit the manifest (`flox edit`) to pin the version as shown below \u2014 the version constraint is what guarantees Node 20 rather than whatever the latest default is. After editing, `flox activate`.\n\n## Manifest\n\n`.flox/env/manifest.toml`:\n\n```toml\nversion = 1\n\n[install]\nnodejs.pkg-path = \"nodejs\"\nnodejs.version = \"^20.0\"\n\n[options]\nsystems = [\"x86_64-linux\", \"aarch64-linux\", \"x86_64-darwin\", \"aarch64-darwin\"]\n```\n\n## Explanation\n\n- **`nodejs.pkg-path = \"nodejs\"`** installs Node.js, which bundles `npm` and `npx` \u2014 no separate package needed. Add `yarn` or `pnpm` to `[install]` only if your project uses them.\n- **`nodejs.version = \"^20.0\"`** pins to the Node 20 LTS line (latest 20.x), so you get patch/minor updates within 20 but never an unintended jump to 22+. For strict reproducibility, set an exact version like `\"20.11.1\"` instead.\n- **`[option" + }, + { + "id": "add-redis-service", + "area": "services", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "has_services_section": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "`is-daemon = true` is questionable with a foreground `exec redis-server` \u2014 is-daemon is for commands that self-background, so this pairing is technically inconsistent", + "Explicit `flox services start` command not shown (covers activate --start-services and stop/restart/status instead)" + ] + }, + "answer_excerpt": "Here's the complete solution for adding a background Redis service to your existing Flox environment.\n\n## Manifest\n\nEdit your manifest with `flox edit` and add these sections (the `redis` package must be installed so `redis-server` is on `PATH`):\n\n```toml\n# Install the Redis package (provides redis-server)\n[install]\nredis.pkg-path = \"redis\"\n\n# Background Redis service\n[services.redis]\ncommand = '''\n mkdir -p \"$FLOX_ENV_CACHE/redis\"\n exec redis-server \\\n --bind \"$REDIS_HOST\" \\\n --port \"$REDIS_PORT\" \\\n --dir \"$FLOX_ENV_CACHE/redis\"\n'''\nis-daemon = true\n\n[services.redis.shutdown]\ncommand = '''redis-cli -h \"$REDIS_HOST\" -p \"$REDIS_PORT\" shutdown nosave'''\n\n[vars]\nREDIS_HOST = \"127.0.0.1\"\nREDIS_PORT = \"6379\"\n```\n\nIf you already have `[install]` or `[vars]` tables, merge these keys into the existing ones rather than duplicating the table headers.\n\n## Commands\n\n```bash\n# Add the package + service definition to the manifest\nflox edit\n\n# Activate the environment AND start services in one step\nflox activate --start-services # or: flox activate -s\n\n# --- manage the service from inside the activated environment ---\nflox services status # confirm redis is running\n" + }, + { + "id": "build-go-package", + "area": "builds", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "has_build_section": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "Omits the manifest-builds vs Nix-expression-builds distinction explicitly required by the rubric", + "`flox publish -o ` flag/syntax is uncertain and may be inaccurate", + "`${vendor}` cross-build reference syntax is plausible but not clearly verified against Flox docs" + ] + }, + "answer_excerpt": "Here's the complete solution for building your Go application into a reproducible Flox package.\n\n## Manifest\n\nThis uses the recommended **two-stage pattern**: stage one vendors dependencies with network access, stage two compiles in a pure (sandboxed) build for full reproducibility.\n\n```toml\nversion = 1\n\n[install]\ngo.pkg-path = \"go\"\n\n[build.vendor]\n# Stage 1: fetch dependencies (needs network, so sandbox off)\ncommand = '''\n go mod vendor\n mkdir -p $out/vendor\n cp -r vendor/* $out/vendor/\n'''\nsandbox = \"off\"\n\n[build.myapp]\n# Stage 2: compile from vendored deps in a pure sandbox \u2014 reproducible\ncommand = '''\n cp -r ${vendor}/vendor ./vendor\n go build -mod=vendor -trimpath -o $out/bin/myapp ./...\n'''\nversion = \"1.0.0\"\nsandbox = \"pure\"\nruntime-packages = []\n```\n\nReplace `myapp` with your binary name, and point `go build` at your main package (e.g. `./cmd/myapp` instead of `./...` if you have multiple packages).\n\n## Commands\n\n```bash\n# 1. Initialize a Flox environment in your project root (where go.mod lives)\nflox init\n\n# 2. Edit .flox/env/manifest.toml to add the [install] and [build.*] tables above\n# (or: flox install go, then add the build tables)\n\n# 3. Git-track all source \u2014 " + }, + { + "id": "containerize-env", + "area": "containers", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "mentions_containerize": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Slightly verbose with optional manifest config that exceeds the core ask, but all accurate" + ] + }, + "answer_excerpt": "# Turning Your Flox Environment into a Docker Image\n\nFlox has this built in via `flox containerize` \u2014 the resulting image **activates your environment on startup**, so all your packages, env vars, and hooks are available inside the container exactly as they are in `flox activate`.\n\n## 1. Configure the container (optional but recommended)\n\nAdd a `[containerize.config]` section to your manifest. Here's a complete example manifest \u2014 adjust the `cmd`, ports, and user to match your app:\n\n```toml\nversion = 1\n\n[install]\n# whatever your environment already has, e.g.:\npython3.pkg-path = \"python311\"\n\n[vars]\n# any environment variables your app needs\n\n[hook.on-activate]\nscript = '''\n # any setup that should run when the environment activates\n'''\n\n# --- container settings (experimental) ---\n[containerize.config]\ncmd = [\"python\", \"app.py\"] # command run inside the activated env; use [] if a [services.*] should start instead\nworking-dir = \"/app\"\nexposed-ports = [\"8080/tcp\"] # ports your app listens on\nuser = \"appuser\" # run as non-root; or \"1000:1000\"\nvolumes = [\"/data\"] # mount points for persistent data\nlabels = { version = \"1.0\" }\nstop-signal = \"SIGTERM\"\n``" + }, + { + "id": "publish-package", + "area": "publish", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "uses_flox_publish": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Doesn't mention teammates must be members of the org to install from its catalog", + "Personal catalog is the default (no -o needed); framing it as '-o ' is slightly imprecise" + ] + }, + "answer_excerpt": "Here's the complete solution for publishing your built package to FloxHub so your team can install it.\n\n## The Manifest\n\nYour `.flox/env/manifest.toml` needs an `[install]` section (build/runtime deps), a `[build.]` section defining the package, and metadata. Here's a representative example for an app called `myapp`:\n\n```toml\nversion = 1\n\n[install]\n# Build-time dependencies your package needs to compile\ngcc.pkg-path = \"gcc13\"\nmake.pkg-path = \"make\"\n# Runtime dependency (if any)\nopenssl.pkg-path = \"openssl\"\n\n[build.myapp]\ncommand = '''\n make\n mkdir -p $out/bin\n cp myapp $out/bin/\n'''\nversion = \"1.0.0\"\ndescription = \"High-performance CLI tool for \"\n# Only what the binary needs at runtime \u2014 NOT the build tools\nruntime-packages = [\"openssl\"]\n```\n\nSwap `myapp`, the `command`, and `description` for your actual package. The published artifact is whatever you place in `$out/` (binaries), not your source code.\n\n## The Commands\n\n```bash\n# 1. Authenticate to FloxHub (first time only)\nflox auth login\n\n# 2. Commit and push everything \u2014 publish requires a clean tree pushed to the remote\ngit add .flox/ src/ # include all build-referenced files\ngit commit -m \"Add flox" + }, + { + "id": "compose-environments", + "area": "sharing", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "uses_include_or_layer": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Minor: claims composition merges 'automatically on the next activation' \u2014 it's actually resolved at edit/include time, though the practical effect is similar", + "Runtime layering example uses -d (local dirs) rather than -r; appropriate here since both envs are local, but doesn't mention the -r remote variant" + ] + }, + "answer_excerpt": "You can merge them at build time with the `[include]` section in your base environment's manifest. This pulls all packages, vars, services, and hooks from `debug-tools` into your base environment.\n\n## Manifest\n\nEdit your base environment's manifest (`.flox/env/manifest.toml`, opened via `flox edit`) to add an `[include]` section:\n\n```toml\nversion = 1\n\n[include]\nenvironments = [\n # If debug-tools is a local environment (path to the dir containing .flox/):\n { dir = \"../debug-tools\" },\n\n # OR, if it lives on FloxHub instead, use remote:\n # { remote = \"yourname/debug-tools\" },\n]\n\n[install]\n# ... your existing base packages stay here ...\n\n[vars]\n# ... your existing base vars ...\n```\n\nUse **one** of the two include entries \u2014 `dir` for a local environment on disk, `remote` for one published to FloxHub. You can list multiple entries if you have more environments to combine.\n\n## Commands\n\n```bash\n# 1. From your base environment's directory, open the manifest and add the [include] block above\nflox edit\n\n# 2. Composition is merged automatically on the next activation.\n# To force a re-merge after the included env changes later, run:\nflox include upgrade\n\n# 3. Activate \u2014 packages" + }, + { + "id": "cuda-env", + "area": "cuda", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "no_abs_paths": true, + "has_install_section": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "cuda_cudart uses unversioned `cudaPackages.cuda_cudart` while others pin `_12_8`, contradicting its own 'pin _12_8 everywhere' advice", + "`flox-cuda/` catalog prefix is dubious \u2014 standard Flox CUDA packages live under `cudaPackages.*` in the nixpkgs catalog, not a `flox-cuda/` namespace", + "Does not use pkg-group to co-resolve the CUDA packages; relies solely on priority", + "Minor: priorities are used to resolve LICENSE conflicts but pkg-group grouping would be the more idiomatic conflict handling" + ] + }, + "answer_excerpt": "## Flox Environment: CUDA GPU Deep Learning (PyTorch) on Linux\n\nHere's a complete, reproducible setup. The Flox-provided CUDA libraries (nvcc, cudart, cuBLAS, cuDNN) come from the `flox-cuda/` catalog, while PyTorch itself is installed into a `uv` venv against the matching CUDA wheel index (`cu128`) \u2014 this is the most reliable way to get a working GPU PyTorch.\n\n### Manifest (`.flox/env/manifest.toml`)\n\n```toml\nversion = 1\n\n[install]\n# --- CUDA toolchain (Linux-only; explicit priorities resolve LICENSE conflicts) ---\ncuda_nvcc.pkg-path = \"flox-cuda/cudaPackages_12_8.cuda_nvcc\"\ncuda_nvcc.priority = 1\ncuda_nvcc.systems = [\"aarch64-linux\", \"x86_64-linux\"]\n\ncuda_cudart.pkg-path = \"flox-cuda/cudaPackages.cuda_cudart\"\ncuda_cudart.priority = 2\ncuda_cudart.systems = [\"aarch64-linux\", \"x86_64-linux\"]\n\nlibcublas.pkg-path = \"flox-cuda/cudaPackages_12_8.libcublas\"\nlibcublas.priority = 2\nlibcublas.systems = [\"aarch64-linux\", \"x86_64-linux\"]\n\ncudnn.pkg-path = \"flox-cuda/cudaPackages_12_8.cudnn_9_11\"\ncudnn.priority = 2\ncudnn.systems = [\"aarch64-linux\", \"x86_64-linux\"]\n\n# --- Python toolchain ---\npython313Full.pkg-path = \"python313Full\"\nuv.pkg-path = \"uv\"\n\n# --- Compiler runtime (libstdc++ for nati" + }, + { + "id": "func-search-has-package", + "area": "search-show", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "invokes_flox": true, + "uses_search_show": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "Does not explicitly note that `flox search` returns only the latest version per package name (the key nuance distinguishing search from show)", + "Adds substantial extra content (install/manifest/pinning) beyond the asked question, though accurate" + ] + }, + "answer_excerpt": "Yes, Flox has `ripgrep` (it tracks nixpkgs, where it's named the same).\n\n## Seeing what's available\n\n```bash\nflox search ripgrep # find the package (add --all for broader matches)\nflox show ripgrep # list all available historical versions\n```\n\n`flox search` confirms the package exists and gives a one-line description; `flox show ripgrep` is the one that prints the full list of versions in the Flox Catalog, so you can pick exactly which one to pin.\n\n## Installing it\n\nThe simplest way is to let Flox edit the manifest for you:\n\n```bash\nflox init # only if you don't already have an environment\nflox install ripgrep # adds it at the latest version\n```\n\nTo pin a specific version you saw in `flox show`, either `flox install ripgrep@14.1.0` or edit the manifest directly.\n\n## Manifest\n\nA minimal `.flox/env/manifest.toml` with ripgrep pinned to a chosen version:\n\n```toml\nversion = 1\n\n[install]\nripgrep.pkg-path = \"ripgrep\"\nripgrep.version = \"14.1.0\" # optional \u2014 drop this line for the latest\n```\n\nThe `version` field accepts an exact version (`\"14.1.0\"`), a partial wildcard (`\"14.1\"` = latest 14.1.x), or a semver range (`\"^14.0\"`). Omit it entirely to al" + }, + { + "id": "func-install-versioned", + "area": "search-show", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "invokes_flox": true, + "has_install_section": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "Asserts `node --version` \u2192 v18.4.0 presumptively, though 18.4.0 may not exist in the catalog", + "Slightly inaccurate claim that `flox show` displays the 'closest available' builds \u2014 it lists all available versions, not nearest matches", + "Could more strongly caution that 18.4 is an old/uncommon version that may be absent, requiring fallback to a nearby 18.x" + ] + }, + "answer_excerpt": "## Solution\n\nAdd Node.js pinned to 18.4 in your manifest's `[install]` section.\n\n### Manifest (`.flox/env/manifest.toml`)\n\n```toml\n[install]\nnodejs.pkg-path = \"nodejs\"\nnodejs.version = \"18.4\"\n```\n\n### Flox commands\n\n```bash\n# If you don't have an environment yet, create one:\nflox init\n\n# Confirm 18.4 is available in the catalog:\nflox show nodejs\n\n# Install Node.js pinned to 18.4:\nflox install nodejs@18.4\n\n# Enter the environment and verify:\nflox activate\nnode --version # -> v18.4.0\n```\n\n### Explanation\n\n- `flox install nodejs@18.4` adds the package and writes the pinned descriptor (`nodejs.version = \"18.4\"`) into your manifest automatically \u2014 so you can use the command or hand-edit the manifest, either way produces the same result.\n- The partial version `\"18.4\"` acts as a wildcard for the latest `18.4.x` patch release; use a fully-qualified `\"18.4.0\"` if you need exact-patch reproducibility.\n- Node's `nodejs` package bundles **npm**. Add `yarn` or `pnpm` as separate packages only if you need them.\n- Use `flox show nodejs` first to confirm 18.4 exists in the catalog \u2014 if the exact version isn't listed, that command shows the closest available 18.x builds." + }, + { + "id": "func-show-versions", + "area": "search-show", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "invokes_flox": true, + "uses_search_show": true + }, + "hard_pass": true, + "judge": { + "score": 3, + "correct": true, + "issues": [ + "Does not explain the key rubric point that `flox search` shows only the latest version per package name", + "Only partially conveys the search-vs-show distinction (states show lists all versions, but never contrasts search's latest-only behavior)", + "Invents many specific version numbers/ranges (hedged, but speculative) rather than directing user to authoritative `flox show` output", + "Adds significant unrequested content (services, manifest options) that dilutes the core answer" + ] + }, + "answer_excerpt": "## What PostgreSQL versions are available\n\nFlox installs PostgreSQL from the Flox Catalog (built on nixpkgs), which carries every historical build. There are two ways versions are exposed:\n\n**1. The generic `postgresql` package, version-pinned.** Run `flox show postgresql` to list the concrete versions in the catalog \u2014 this typically spans the full PostgreSQL release range (e.g. `13.x`, `14.x`, `15.x`, `16.x`, `17.x`, including older point releases). You then pin with the `version` field.\n\n**2. Major-version package paths.** nixpkgs also ships explicit per-major packages \u2014 `postgresql_13`, `postgresql_14`, `postgresql_15`, `postgresql_16`, `postgresql_17` \u2014 which are the cleanest way to lock a major version.\n\n> I haven't run anything here, so treat the specific numbers as the expected range \u2014 `flox show postgresql` (and `flox show postgresql_17`, etc.) on your machine is the authoritative list of exact builds.\n\n## Commands to discover and install\n\n```bash\nflox search postgresql # find PostgreSQL-related packages\nflox search postgresql --all # broaden the search\nflox show postgresql # list every catalog version of the generic package\nflox show postgresql_16" + }, + { + "id": "trigger-python-web", + "area": "triggering", + "tier": "should", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true, + "no_fake_install_url": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Minor: hedged/possibly-unnecessary `.flox/cache/` gitignore advice", + "Minor: createdb relies on $POSTGRES_* vars being in shell scope which depends on activation" + ] + }, + "answer_excerpt": "Here's a complete, reproducible setup using **Flox**. The key idea for \"teammates get the same setup\": you commit the `.flox/` directory to git, and everyone runs `flox activate` to get the identical Python toolchain *and* a local PostgreSQL \u2014 no system installs, no version drift.\n\n## Prerequisites\n\nEach teammate installs Flox once (from `flox.dev/download` or a package manager):\n\n```bash\nbrew install flox # macOS\n# Debian/Ubuntu: download the .deb from flox.dev/download, then:\nsudo apt install /path/to/flox.deb\nflox --version\n```\n\n## 1. Create the environment\n\nFrom your project root:\n\n```bash\nflox init\n```\n\nThis creates `.flox/env/manifest.toml`, which is the single file that defines the whole environment.\n\n## 2. The manifest\n\nReplace `.flox/env/manifest.toml` with the following. It installs Python + `uv`, runs PostgreSQL as a managed service with data living in `$FLOX_ENV_CACHE` (local, gitignored, survives rebuilds), and sets up a project venv on activation.\n\n```toml\n[install]\npython.pkg-path = \"python313Full\"\nuv.pkg-path = \"uv\"\npostgresql.pkg-path = \"postgresql_16\"\n\n[vars]\nPOSTGRES_HOST = \"localhost\"\nPOSTGRES_PORT = \"5432\"\nPOSTGRES_DB = \"appdb\"\nPOSTGRES_USER = \"appus" + }, + { + "id": "trigger-node-project", + "area": "triggering", + "tier": "should", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true, + "no_fake_install_url": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "nodejs already bundles npm, so the on-activate hook using npm with pnpm also installed is slightly inconsistent", + "version ^20.0 pins to an aging LTS rather than current LTS, minor" + ] + }, + "answer_excerpt": "Here's a complete setup for a new Node.js project using a Flox environment for reproducibility. Run through these steps in the directory where you want the project.\n\n## 1. Initialize the project\n\n```bash\nmkdir my-node-app && cd my-node-app\nflox init\n```\n\nThis creates the `.flox/` directory holding `env/manifest.toml`.\n\n## 2. Configure the manifest\n\nEdit `.flox/env/manifest.toml` to match the following. This pins Node.js to the current LTS (20.x) and includes `pnpm` (drop it if you prefer plain npm):\n\n```toml\nversion = 1\n\n[install]\nnodejs.pkg-path = \"nodejs\"\nnodejs.version = \"^20.0\"\npnpm.pkg-path = \"pnpm\"\n\n[vars]\nNODE_ENV = \"development\"\n\n[hook]\non-activate = '''\n # Install dependencies once when a lockfile/package.json exists\n if [ -f \"$FLOX_ENV_PROJECT/package.json\" ] && [ ! -d \"$FLOX_ENV_PROJECT/node_modules\" ]; then\n npm install --prefix \"$FLOX_ENV_PROJECT\"\n fi\n'''\n\n[options]\nsystems = [\"x86_64-linux\", \"aarch64-linux\", \"x86_64-darwin\", \"aarch64-darwin\"]\n```\n\nNotes:\n- `nodejs` already bundles `npm`, so you don't need to install it separately.\n- The hook is idempotent \u2014 it only runs `npm install` if `package.json` exists and `node_modules` is missing. Remove the hook entirel" + }, + { + "id": "trigger-reproducible", + "area": "triggering", + "tier": "should", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true, + "no_fake_install_url": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Minor: \"flox install nodejs python311 postgresql\" uses bare names while the manifest example uses pkg-path attributes \u2014 slight inconsistency", + "Minor: claim that no curl|bash installer exists is stated more absolutely than necessary" + ] + }, + "answer_excerpt": "Here's how to make your project's dev environment fully reproducible across the whole team, using Flox.\n\n## The approach\n\nDefine the environment with Flox, then **commit the `.flox/` directory to your project's git repo alongside your source code.** When teammates clone and run `flox activate`, they get the exact same toolchain \u2014 same package versions, resolved from the same catalog snapshot \u2014 on macOS and Linux, x86 and ARM. This is the canonical pattern for development environments: the environment travels with the code in one repo.\n\n## Setup steps\n\n### 1. Install Flox (each team member)\n\nDon't use any `curl | bash` one-liner \u2014 they don't exist. Install from `flox.dev/download` or a package manager:\n\n```bash\n# macOS \u2014 Homebrew\nbrew install flox\n\n# Debian/Ubuntu \u2014 download the .deb from flox.dev/download, then:\nsudo apt install /path/to/flox.deb\n\n# RPM (RHEL/CentOS/Amazon Linux) \u2014 download the .rpm, then:\nsudo rpm -ivh /path/to/flox.rpm\n\nflox --version # verify\n```\n\n### 2. Initialize the environment (one person, once)\n\nFrom your project root:\n\n```bash\nflox init\n```\n\nThis creates `.flox/` containing `env/manifest.toml` (the definition) and `env.json` (metadata).\n\n### 3. Add your " + }, + { + "id": "trigger-go-cli", + "area": "triggering", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true, + "no_fake_install_url": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Aliasing `test` shadows the shell builtin, which can break scripts", + "Minor: claims hook functions aren't exposed to interactive shell as justification for profile, which is roughly right but stated as absolute" + ] + }, + "answer_excerpt": "Here's a complete Flox-based development environment for a new Go command-line tool.\n\n## Setup steps\n\n```bash\n# 1. In your project directory, create the Flox environment\nflox init\n\n# 2. Replace the generated manifest with the one below\nflox edit\n\n# 3. Enter the environment (compiles nothing \u2014 just sets up the toolchain)\nflox activate\n\n# 4. First time only: scaffold your module (inside the activated env)\ngo mod init github.com//\n```\n\n## `.flox/env/manifest.toml`\n\n```toml\nversion = 1\n\n[install]\n# Core Go toolchain\ngo.pkg-path = \"go\"\ngo.version = \"^1.22\"\n\n# Linting / formatting / language server\ngolangci-lint.pkg-path = \"golangci-lint\"\ngopls.pkg-path = \"gopls\"\ngotools.pkg-path = \"gotools\" # goimports, etc.\n\n# Handy for a CLI project\ndelve.pkg-path = \"delve\" # debugger (dlv)\ngnumake.pkg-path = \"gnumake\"\n\n[vars]\nCGO_ENABLED = \"0\" # static binaries by default; drop if you need cgo\n\n[hook.on-activate]\nscript = '''\n# Keep all Go caches inside the env cache so they survive rebuilds\n# and don't pollute the global ~/go\nexport GOPATH=\"${FLOX_ENV_CACHE}/go\"\nexport GOMODCACHE=\"${GOPATH}/pkg/mod\"\nexport GOCACHE=\"${FLOX_ENV_CACHE}/go-build\"" + }, + { + "id": "trigger-update-project", + "area": "triggering", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Asserts project is Python stdlib and lacks .flox without showing verification", + "Slightly verbose; adds python3Packages.redis client that may not be wanted" + ] + }, + "answer_excerpt": "No existing `.flox` environment yet, and the project is Python (stdlib). Here's the complete solution to add Redis as a dependency.\n\n## 1. Initialize a Flox environment (if you haven't)\n\nFrom the project root (`/tmp/flox-agentic/evals`):\n\n```bash\nflox init\n```\n\nThis creates `.flox/env/manifest.toml`.\n\n## 2. Add Redis to the manifest\n\nInstall the `redis` package (gives you `redis-server` and `redis-cli`) and run it as a managed service so it starts with the environment. Since your harness is Python, I've also added the `redis` Python client \u2014 drop that line if you only need the server.\n\nEdit `.flox/env/manifest.toml` to look like this:\n\n```toml\nversion = 1\n\n[install]\nredis.pkg-path = \"redis\"\n# Python client \u2014 remove if you only need the server binary\nredis-py.pkg-path = \"python3Packages.redis\"\n\n[vars]\nREDIS_PORT = \"6379\"\n\n[services.redis]\ncommand = \"redis-server --port ${REDIS_PORT} --dir ${FLOX_ENV_CACHE} --save '' --appendonly no\"\n\n[options]\nsystems = [\"x86_64-linux\", \"aarch64-linux\", \"x86_64-darwin\", \"aarch64-darwin\"]\n```\n\nNotes:\n- `--dir ${FLOX_ENV_CACHE}` keeps any Redis data in Flox's persistent, local-only cache (survives rebuilds, no absolute paths).\n- `--save '' --appendonl" + }, + { + "id": "trigger-generic-start", + "area": "triggering", + "tier": "stretch", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "rust-bin is not a standard nixpkgs/flox package name (should be rustc/cargo)", + "go.pkg-path entries duplicate what flox install already added, noted but slightly redundant" + ] + }, + "answer_excerpt": "I'll set you up with a complete, reproducible scaffold. Since you didn't name a language, I'm defaulting to **Go** \u2014 it's the most idiomatic choice for a CLI (single static binary, no runtime to ship, excellent flag/command libraries). I'll note how to swap to Rust or Python at the end.\n\nEverything below is the answer \u2014 no commands are run.\n\n---\n\n## 1. Project layout\n\n```\nmycli/\n\u251c\u2500\u2500 .flox/ # created by `flox init`\n\u2502 \u2514\u2500\u2500 env/manifest.toml\n\u251c\u2500\u2500 cmd/\n\u2502 \u2514\u2500\u2500 mycli/\n\u2502 \u2514\u2500\u2500 main.go\n\u251c\u2500\u2500 internal/\n\u2502 \u2514\u2500\u2500 greet/\n\u2502 \u2514\u2500\u2500 greet.go\n\u251c\u2500\u2500 go.mod\n\u2514\u2500\u2500 README.md\n```\n\n## 2. Setup steps\n\n```bash\nmkdir mycli && cd mycli\nflox init # creates .flox/\n\nflox install go gopls gotools # toolchain + LSP + goimports etc.\n\nflox edit # paste the manifest below\nflox activate # enter the reproducible env\n\n# inside the env, scaffold the module:\ngo mod init github.com//mycli\ngo get github.com/spf13/cobra@latest\n```\n\n## 3. `.flox/env/manifest.toml`\n\nNo absolute paths; caches live under `$FLOX_ENV_CACHE` so they survive rebuilds.\n\n```toml\nversion = 1\n\n[install]\ngo.pkg-path = \"go\"\ngopls.pkg-path = \"gopls\"\ngotools.pkg-path = \"got" + }, + { + "id": "indirect-cuda-pytorch", + "area": "cuda", + "tier": "should", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true, + "no_fake_install_url": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Inconsistent CUDA versioning: pins cudaPackages_12_8 + CUDA_VERSION=12.8 but installs PyTorch cu129 wheels", + "cuda_cudart uses bare 'cudaPackages' while others use 'cudaPackages_12_8' \u2014 version mismatch", + "Typo 'gcc-unwrapded' in troubleshooting section", + "Uses priority for collision handling but does not use pkg-group to bundle the CUDA libs as rubric suggests", + "Claim that auth login is required for a 'flox-cuda/ catalog' is dubious" + ] + }, + "answer_excerpt": "Here's a complete, reproducible setup using **Flox** to pin the CUDA toolchain and a PyTorch project that uses your NVIDIA GPU. The Flox environment handles the system-level CUDA libraries declaratively; PyTorch itself is installed from the official CUDA wheels into a `uv`-managed venv.\n\n## Prerequisites\n\n1. **NVIDIA driver installed on the host** \u2014 Flox provides the CUDA *toolkit/libraries*, but the kernel driver must come from your distro. Verify it works:\n ```bash\n nvidia-smi\n ```\n The reported \"CUDA Version\" in that output is the *max* CUDA your driver supports. It must be \u2265 12.9 for the `cu129` wheels below (any recent driver is fine; otherwise drop to a lower wheel index like `cu124`).\n\n2. **Install Flox** (from `flox.dev/download` or a package manager \u2014 there is no `curl | bash` installer):\n ```bash\n # Debian/Ubuntu: download the .deb from flox.dev/download, then\n sudo apt install /path/to/flox.deb\n # RPM-based:\n sudo rpm -ivh /path/to/flox.rpm\n flox --version\n ```\n\n3. **Authenticate** (required for the `flox-cuda/` catalog):\n ```bash\n flox auth login\n ```\n\n## Step 1 \u2014 Create the project and environment\n\n```bash\nmkdir pytorch-gpu && cd pytorch-gpu" + }, + { + "id": "indirect-cuda-readme", + "area": "cuda", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "`flox-cuda/` catalog prefix on every CUDA pkg-path is likely fabricated \u2014 standard Flox uses `cudaPackages_12_2.*` directly (CUDA's unfree licensing/early-access is real but the prefix is unverified)", + "`CUDA_HOME = \"$FLOX_ENV\"` in [vars] won't work \u2014 Flox [vars] values are literal and don't undergo shell expansion; should be set in hook.on-activate", + "`cuda_cudart` uses unversioned `cudaPackages.cuda_cudart` while preaching 'keep every CUDA attr on the same suffix' \u2014 self-contradictory", + "cuDNN attr `cudnn_9_11` is guessed; mostly hedged via search but still presented in the manifest as if real" + ] + }, + "answer_excerpt": "Here's the complete setup for a Flox environment with CUDA 12.2 and cuDNN.\n\n## Setup steps\n\n```bash\n# 1. Authenticate (CUDA catalog needs early-access auth)\nflox auth login\n\n# 2. Initialize an environment in your project directory\nflox init\n\n# 3. Edit the manifest (paste the config below)\nflox edit\n\n# 4. Activate \u2014 the hook builds the venv and verifies the toolchain\nflox activate\n```\n\nBefore committing to exact package names, confirm what the catalog actually ships for the 12.2 series (the `_X_Y` suffix and the cuDNN variant must match real attrs):\n\n```bash\nflox search cudatoolkit --all | grep flox-cuda\nflox show flox-cuda/cudaPackages_12_2.cuda_nvcc\nflox search cudnn --all | grep flox-cuda # find the cudnn_9_x attr for 12.2\n```\n\nIf `cudaPackages_12_2.*` isn't in the catalog, pin to the nearest available 12.x and keep every CUDA attr on the *same* suffix \u2014 never mix CUDA minor versions across packages.\n\n## `manifest.toml`\n\n```toml\nversion = 1\n\n[install]\n# --- CUDA 12.2 toolchain (Linux only) ---\ncuda_nvcc.pkg-path = \"flox-cuda/cudaPackages_12_2.cuda_nvcc\"\ncuda_nvcc.priority = 1 # highest \u2014 wins LICENSE conflicts\ncuda_nvcc.systems = [\"aarch64-linux\", \"x86_" + }, + { + "id": "indirect-publish-share", + "area": "publish", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Minor: `flox publish -o yourorg` org-flag syntax may not exactly match current CLI", + "actions/checkout@v3 slightly dated" + ] + }, + "answer_excerpt": "The best way with Flox is to **publish your CLI as a package to your org's catalog on FloxHub**. You define a build once, publish it, and teammates install it with a single `flox install` \u2014 no source code, build tools, or manual setup on their end. Here's the complete solution.\n\n## Overview\n\nThere are two halves:\n1. **You** add a `[build]` definition to your project's Flox environment, then `flox publish` it.\n2. **Teammates** run `flox install yourorg/yourtool` \u2014 they get the built binary, nothing else.\n\n## 1. Set up the build in your project\n\nFrom your CLI tool's repo:\n\n```bash\nflox init # creates .flox/ if you don't have it yet\n```\n\nThen edit `.flox/env/manifest.toml` to add a build definition. The `[install]` section holds your **build-time** dependencies; `[build.]` describes how to produce the binary, which must land in `$out/bin/`.\n\n```toml\n[install]\n# build-time deps \u2014 pick what your tool needs to compile/assemble\n# e.g. for Rust:\ncargo.pkg-path = \"cargo\"\nrustc.pkg-path = \"rustc\"\n\n[build.mytool]\ncommand = '''\n cargo build --release\n mkdir -p $out/bin\n cp target/release/mytool $out/bin/\n'''\nversion = \"1.0.0\"\ndescription = \"Short, specific description of what" + }, + { + "id": "indirect-container-deploy", + "area": "containers", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Minor: venv hook references /app/requirements.txt which isn't copied into the image by flox containerize, so it could fail at runtime", + "Some manifest containerize.config fields/syntax (e.g. labels inline table) not verified against exact schema" + ] + }, + "answer_excerpt": "Here's the complete solution to ship your app as a production Docker image using Flox. Your app is `run.py` (Python), and there's no Flox environment yet, so this covers it end-to-end.\n\n## 1. Create the Flox environment\n\n```bash\nflox init\nflox install python312\n```\n\nInstall whatever your app imports. If you have a `requirements.txt`, declare the deps you can get from Flox (e.g. `flox install python312Packages.requests`), and handle the rest with pip in the activation hook.\n\n## 2. Configure the environment + container in `manifest.toml`\n\nRun `flox edit` and set it up like this:\n\n```toml\nversion = 1\n\n[install]\npython.pkg-path = \"python312\"\n\n[vars]\n# Defaults; override at `docker run` time with -e\nPORT = \"8000\"\n\n[hook.on-activate]\nscript = '''\n # Pin a venv inside the env cache so pip deps are reproducible per-machine\n export VENV=\"$FLOX_ENV_CACHE/venv\"\n if [ ! -d \"$VENV\" ]; then\n python -m venv \"$VENV\"\n \"$VENV/bin/pip\" install --no-cache-dir -r /app/requirements.txt\n fi\n export PATH=\"$VENV/bin:$PATH\"\n'''\n\n[containerize.config]\nuser = \"1000:1000\" # non-root\nworking-dir = \"/app\"\nexposed-ports = [\"8000/tcp\"]\ncmd = [\"python\", \"run.py\"] # runs inside the ac" + }, + { + "id": "indirect-services-db", + "area": "services", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Redis is-daemon service has no [services.redis.shutdown] command (Postgres does), so daemonized Redis may not be cleanly stopped", + "Postgres command uses exec but is-daemon=true implies it should daemonize itself \u2014 minor potential mismatch with Flox daemon handling" + ] + }, + "answer_excerpt": "Here's a complete Flox solution that runs Postgres and Redis as background services for your web app.\n\n## Overview\n\nFlox gives you a single declarative environment (`.flox/env/manifest.toml`) that pins both database binaries to exact versions and runs them as managed background services. Data lives in `$FLOX_ENV_CACHE`, so it's local-only, persists across rebuilds, and never leaks into your repo. No Docker, no system installs, no absolute paths.\n\n## Setup steps\n\nFrom your web app's project root:\n\n```bash\n# 1. Create the Flox environment (creates .flox/)\nflox init\n\n# 2. Open the manifest and replace its contents with the config below\nflox edit\n\n# 3. Activate the environment AND start the background services\nflox activate -s\n```\n\nThat's it. On first `flox activate -s`, Postgres initializes its data directory and creates your app database/role; Redis starts fresh. On every subsequent activation the existing data is reused.\n\nYou can commit `.flox/env/manifest.toml` and `.flox/env.json` to git so teammates get the identical setup. (The `.flox/` directory also generates a `.gitignore` that excludes local-only bits.)\n\n## The manifest\n\nPaste this into `flox edit` (`.flox/env/manifest.toml`" + } + ] +} \ No newline at end of file diff --git a/evals/results/skills_mcp.json b/evals/results/skills_mcp.json new file mode 100644 index 0000000..5b2f722 --- /dev/null +++ b/evals/results/skills_mcp.json @@ -0,0 +1,480 @@ +{ + "summary": { + "mode": "skills+mcp", + "n_tasks": 22, + "n_errors": 0, + "n": 22, + "hard_pass_rate": 1.0, + "avg_judge_score": 4.64, + "judge_correct_rate": 0.955, + "by_tier": { + "should": { + "n": 15, + "hard_pass_rate": 1.0, + "avg_judge_score": 4.53, + "judge_correct_rate": 0.933 + }, + "may": { + "n": 6, + "hard_pass_rate": 1.0, + "avg_judge_score": 4.83, + "judge_correct_rate": 1.0 + }, + "stretch": { + "n": 1, + "hard_pass_rate": 1.0, + "avg_judge_score": 5.0, + "judge_correct_rate": 1.0 + } + }, + "n_trigger_tasks": 11, + "trigger_invokes_flox_rate": 1.0, + "should_trigger_rate": 1.0 + }, + "results": [ + { + "id": "env-python-postgres", + "area": "environments", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "no_abs_paths": true, + "has_install_section": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "postgres service may need unix_socket_directories/socket config for some clients, not addressed", + "minor: --auth=trust noted as dev-only but still default in manifest" + ] + }, + "answer_excerpt": "Here's a complete, reproducible Flox environment for a Python web app backed by PostgreSQL.\n\n## Manifest\n\n```toml\nversion = 1\n\n[install]\npython.pkg-path = \"python312Full\"\nuv.pkg-path = \"uv\"\npostgresql.pkg-path = \"postgresql_16\"\n\n[vars]\nPGHOST = \"localhost\"\nPGPORT = \"5432\"\nPGUSER = \"postgres\"\nPGDATABASE = \"appdb\"\n\n[hook]\non-activate = '''\n # --- Python virtualenv (lives in the cache so it survives rebuilds) ---\n export VENV_DIR=\"$FLOX_ENV_CACHE/venv\"\n export UV_CACHE_DIR=\"$FLOX_ENV_CACHE/uv-cache\"\n\n if [ ! -d \"$VENV_DIR\" ]; then\n uv venv \"$VENV_DIR\" --python python3 --quiet\n fi\n\n if [ -f \"$FLOX_ENV_PROJECT/requirements.txt\" ]; then\n uv pip install --python \"$VENV_DIR/bin/python\" --quiet \\\n -r \"$FLOX_ENV_PROJECT/requirements.txt\"\n fi\n\n # --- PostgreSQL data directory (persistent, local-only) ---\n export PGDATA=\"$FLOX_ENV_CACHE/pgdata\"\n\n if [ ! -d \"$PGDATA\" ]; then\n initdb --username=\"$PGUSER\" --auth=trust --no-locale --encoding=UTF8 \"$PGDATA\" >/dev/null\n # Listen on TCP so the app and psql can connect\n echo \"listen_addresses = '$PGHOST'\" >> \"$PGDATA/postgresql.conf\"\n echo \"port = $PGPORT\" >> \"$PGDATA/postgresql.conf\"\n fi\n\n cd \"$FLOX_EN" + }, + { + "id": "node-env", + "area": "environments", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "no_abs_paths": true, + "has_install_section": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [] + }, + "answer_excerpt": "Here's the complete solution for a Node.js 20 Flox environment.\n\n## Commands\n\n```bash\nflox init # Create the environment (.flox/ in your project)\nflox install nodejs # Or just edit the manifest directly (below)\nflox activate # Enter the environment\n```\n\nIf you want the version pinned exactly as below, the cleanest path is to run `flox init`, then `flox edit` and paste the manifest.\n\n## Manifest\n\n`.flox/env/manifest.toml`:\n\n```toml\nversion = 1\n\n[install]\nnodejs.pkg-path = \"nodejs_20\"\n\n[options]\nsystems = [\"x86_64-linux\", \"aarch64-linux\", \"x86_64-darwin\", \"aarch64-darwin\"]\n```\n\n## Explanation\n\n- **`nodejs_20`** is the package that pins to the Node.js 20 LTS line (which bundles `npm` and `npx`) \u2014 no separate npm install needed. Alternatively you can use `nodejs.pkg-path = \"nodejs\"` with `nodejs.version = \"^20.0\"` to express the same constraint as a semver range; both resolve to Node 20.x.\n- **`[options].systems`** lists the platforms the environment supports so it activates on both Linux and macOS, Intel and ARM.\n- If your project uses **yarn** or **pnpm**, add them separately (`flox install yarn` or `flox install pnpm`) \u2014 only npm ship" + }, + { + "id": "add-redis-service", + "area": "services", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "has_services_section": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Does not explicitly mention `flox services start ` to start the service from within an already-activated environment (covers --start-services and stop/restart/status/logs instead)", + "is-daemon/shutdown/vars are extra detail beyond the rubric but correct and well-explained" + ] + }, + "answer_excerpt": "Here's the complete solution.\n\n## Manifest\n\nEdit your manifest with `flox edit` and add these sections (the `redis` package provides `redis-server`):\n\n```toml\n[install]\nredis.pkg-path = \"redis\"\n\n[services.redis]\ncommand = '''\n mkdir -p \"$FLOX_ENV_CACHE/redis\"\n exec redis-server \\\n --bind \"$REDIS_HOST\" \\\n --port \"$REDIS_PORT\" \\\n --dir \"$FLOX_ENV_CACHE/redis\"\n'''\nis-daemon = true\n\n[services.redis.shutdown]\ncommand = '''redis-cli -h \"$REDIS_HOST\" -p \"$REDIS_PORT\" shutdown nosave'''\n\n[vars]\nREDIS_HOST = \"127.0.0.1\"\nREDIS_PORT = \"6379\"\n```\n\n## Flox commands\n\n```bash\n# Open the manifest and paste the sections above\nflox edit\n\n# Activate the environment AND start services in the background\nflox activate --start-services\n# (short form: flox activate -s)\n\n# From inside the activated environment, manage the service:\nflox services status # confirm redis is \"Running\"\nflox services logs redis # tail the logs\nflox services restart redis # restart it\nflox services stop redis # stop it\n\n# Verify it's actually serving (redis-cli is included with the redis package)\nredis-cli -p 6379 ping # \u2192 PONG\n```\n\n## Explanation\n\n- **`[install]`** adds the `redis` package s" + }, + { + "id": "build-go-package", + "area": "builds", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "has_build_section": true + }, + "hard_pass": true, + "judge": { + "score": 3, + "correct": false, + "issues": [ + "Never mentions manifest builds vs Nix-expression builds \u2014 a required rubric element entirely omitted", + "Inter-build reference syntax `${vendor}` is likely fabricated; Flox manifest builds don't reliably support interpolating another build's $out this way", + "Two-stage vendor build adds unverified complexity not grounded in documented Flox behavior", + "Correctly covers [build.] with $out and `flox build` (2 of 3 rubric items)" + ] + }, + "answer_excerpt": "Here's the complete solution for building your Go application into a reproducible Flox package.\n\n## Manifest\n\nEdit `.flox/env/manifest.toml`:\n\n```toml\nversion = 1\n\n[install]\ngo.pkg-path = \"go\"\n\n# Stage 1: vendor dependencies (needs network, so sandbox is off)\n[build.vendor]\ncommand = '''\n go mod vendor\n mkdir -p $out/vendor\n cp -r vendor/* $out/vendor/\n'''\nsandbox = \"off\"\n\n# Stage 2: compile reproducibly from vendored deps (no network, git-tracked files only)\n[build.myapp]\ncommand = '''\n cp -r ${vendor}/vendor ./vendor\n go build -mod=vendor -o $out/bin/myapp\n'''\nversion = \"1.0.0\"\ndescription = \"My Go application\"\nsandbox = \"pure\"\nruntime-packages = [ ] # a static Go binary needs nothing at runtime\n```\n\nReplace `myapp` with your binary name (and adjust the `go build` package path, e.g. `./cmd/myapp`, if your `main` package isn't at the repo root).\n\n## Commands\n\n```bash\n# 1. Create the environment in your project (if you don't have one yet)\nflox init\n\n# 2. Edit .flox/env/manifest.toml as above, then activate\nflox activate\n\n# 3. Build \u2014 produces ./result-myapp -> /nix/store/...-myapp-1.0.0\nflox build myapp\n\n# 4. Run the built binary to verify\n./result-myapp/bin/myapp\n\n# 5. (opt" + }, + { + "id": "containerize-env", + "area": "containers", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "mentions_containerize": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Minor: a couple of optional details (macOS proxy-container/flox-nix cache volume internals, services auto-start with empty cmd) go beyond what's strictly verifiable but are plausible and not misleading" + ] + }, + "answer_excerpt": "Here's the complete solution for turning a Flox environment into a Docker container image.\n\n## Manifest\n\nAdd a `[containerize.config]` section to your `.flox/env/manifest.toml`. This is optional \u2014 `flox containerize` works on any environment as-is \u2014 but the config lets you set the runtime command, ports, user, etc. for the image:\n\n```toml\nversion = 1\n\n[install]\n# whatever your environment already has, e.g.:\n# python3.pkg-path = \"python311\"\n\n# Optional: shape the resulting image\n[containerize.config]\nuser = \"appuser\" # run as non-root (username or \"uid:gid\")\nexposed-ports = [\"8080/tcp\"] # ports the image documents/exposes\ncmd = [\"python\", \"app.py\"] # default command (runs inside the activated env)\nworking-dir = \"/app\"\nvolumes = [\"/data\"] # mount points for persistent data\nlabels = { version = \"1.0\" }\n```\n\nIf you omit `[containerize.config]`, the image defaults to an interactive shell with the environment activated.\n\n## Commands\n\n```bash\n# Simplest: load straight into Docker (auto-detects the runtime)\nflox containerize --runtime docker --tag myenv:latest\n\n# Or pipe to Docker explicitly\nflox containerize --tag myenv:latest -f - | docker load\n\n# Or export" + }, + { + "id": "publish-package", + "area": "publish", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "uses_flox_publish": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "'Org publishing is a paid feature' is an unverified embellishment", + "Personal publish is the default (no -o needed); framing -o as always required for personal handle is slightly off", + "Minor: doesn't explicitly note flox publish itself triggers a clean re-clone build, though clean-git requirement is well covered" + ] + }, + "answer_excerpt": "## Solution\n\nTo publish a package so your **team** can install it, you publish to your **organization namespace** (`-o`), not your personal one. Personal catalogs are only visible to you; organization catalogs are shared with all members.\n\n### Manifest\n\nThis is your **development** environment manifest (`.flox/env/manifest.toml`), committed to git alongside your source. The `[build]` section defines the package; `[install]` holds the build-time tools.\n\n```toml\nversion = 1\n\n[install]\n# Build-time dependencies\ngcc.pkg-path = \"gcc13\"\nmake.pkg-path = \"make\"\n# Runtime dependency (example)\nopenssl.pkg-path = \"openssl\"\n\n[build.myapp]\ncommand = '''\n make\n mkdir -p $out/bin\n cp myapp $out/bin/\n'''\nversion = \"1.0.0\"\ndescription = \"High-performance widget processor\" # be specific, not \"my tool\"\nruntime-packages = [\"openssl\"] # only what's needed at runtime, not gcc/make\n```\n\n### Commands\n\n```bash\n# 1. Authenticate (first time only)\nflox auth login\n\n# 2. Commit and push EVERYTHING \u2014 flox re-clones and clean-builds from the remote\ngit add .flox/ src/ Makefile\ngit commit -m \"Add flox build configuration\"\ngit push origin main # use 'master' if that's your branc" + }, + { + "id": "compose-environments", + "area": "sharing", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "uses_include_or_layer": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "Layering alternative described only conceptually ('flox activate base, then activate debug-tools inside it') \u2014 missing the actual `flox activate -r a -- flox activate -r b` command syntax", + "Does not clearly state layering needs no manifest edit and is process-scoped/temporary beyond a passing mention", + "Minor: 'flox edit' opening manifest and 'build-time merge' framing is mostly right but composition resolves at activation time" + ] + }, + "answer_excerpt": "You combine them with **Flox composition** \u2014 the `[include]` directive in the manifest merges environments at build time. Edit the manifest of the environment that should be the \"combined\" one (or a fresh one) so it pulls in both.\n\n## Manifest\n\nIf both environments are **local directories** on your machine:\n\n```toml\nversion = 1\n\n[include]\nenvironments = [\n { dir = \"../base\" }, # path to the base env's project dir\n { dir = \"../debug-tools\" }, # path to the debug-tools env's project dir\n]\n```\n\nIf the environments live on **FloxHub** (shared/remote), reference them by `owner/name` instead:\n\n```toml\nversion = 1\n\n[include]\nenvironments = [\n { remote = \"youruser/base\" },\n { remote = \"youruser/debug-tools\" },\n]\n```\n\nYou can also mix `dir` and `remote`, and add any project-specific packages/vars in the usual `[install]`/`[vars]` sections below \u2014 they layer on top of the included environments.\n\n## Commands\n\n```bash\n# 1. Edit the manifest of the combining environment\nflox edit # opens .flox/env/manifest.toml; paste the [include] block\n\n# 2. Activate \u2014 composition is resolved/merged at build time\nflox activate\n\n# 3. Confirm both environments' packages are p" + }, + { + "id": "cuda-env", + "area": "cuda", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "no_abs_paths": true, + "has_install_section": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "CUDA toolkit pinned to 12.8 but PyTorch installed from cu129 (12.9) wheel index \u2014 inconsistent versioning, only hand-waved as 'forward-compatible'", + "cuda_cudart uses unversioned 'cudaPackages.cuda_cudart' while siblings use 'cudaPackages_12_8' \u2014 risks pulling a mismatched version and reintroducing conflicts", + "'flox-cuda/' catalog prefix on pkg-paths is of uncertain validity; standard nixpkgs CUDA paths are just 'cudaPackages_12_8.*'", + "PyTorch wheels bundle their own CUDA runtime, making the installed cuda_cudart/libcublas/cudnn largely redundant except for nvcc/native builds (acknowledged but adds bloat)" + ] + }, + "answer_excerpt": "Here's a complete Flox environment for CUDA GPU deep learning with PyTorch on Linux.\n\n## Manifest\n\n```toml\nversion = 1\n\n[install]\n# CUDA toolchain (Linux-only) \u2014 explicit priorities resolve LICENSE-file conflicts\ncuda_nvcc.pkg-path = \"flox-cuda/cudaPackages_12_8.cuda_nvcc\"\ncuda_nvcc.priority = 1\ncuda_nvcc.systems = [\"aarch64-linux\", \"x86_64-linux\"]\n\ncuda_cudart.pkg-path = \"flox-cuda/cudaPackages.cuda_cudart\"\ncuda_cudart.priority = 2\ncuda_cudart.systems = [\"aarch64-linux\", \"x86_64-linux\"]\n\nlibcublas.pkg-path = \"flox-cuda/cudaPackages_12_8.libcublas\"\nlibcublas.priority = 2\nlibcublas.systems = [\"aarch64-linux\", \"x86_64-linux\"]\n\ncudnn.pkg-path = \"flox-cuda/cudaPackages_12_8.cudnn_9_11\"\ncudnn.priority = 2\ncudnn.systems = [\"aarch64-linux\", \"x86_64-linux\"]\n\n# Python toolchain\npython313Full.pkg-path = \"python313Full\"\nuv.pkg-path = \"uv\"\n\n# libstdc++ for compiled extensions\ngcc-unwrapped.pkg-path = \"gcc-unwrapped\"\ngcc-unwrapped.priority = 5\n\n[vars]\nCUDA_VERSION = \"12.8\"\nCUDA_HOME = \"$FLOX_ENV\"\nPYTORCH_CUDA_ALLOC_CONF = \"max_split_size_mb:128\"\n\n[hook.on-activate]\nsetup_pytorch_cuda() {\n venv=\"$FLOX_ENV_CACHE/venv\"\n\n if [ ! -d \"$venv\" ]; then\n uv venv \"$venv\" --python python3\n fi\n\n if [" + }, + { + "id": "func-search-has-package", + "area": "search-show", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "invokes_flox": true, + "uses_search_show": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "Does not mention that `flox search` returns only the latest version per package name (a rubric point)", + "Extra install/pin content is correct but beyond the question scope" + ] + }, + "answer_excerpt": "Yes, Flox has `ripgrep` \u2014 it's in the Flox Catalog, which carries millions of historical package-version combinations from nixpkgs.\n\n## Seeing which versions are available\n\n```bash\nflox search ripgrep # Find the package (confirm the name)\nflox show ripgrep # List all available historical versions\n```\n\n`flox search` confirms the package exists and its name; `flox show ripgrep` is the one that enumerates every version you can pin to.\n\n## Installing it\n\nEasiest is to let Flox edit the manifest for you:\n\n```bash\nflox init # Only if you don't already have an environment\nflox install ripgrep # Adds it to [install] in manifest.toml\n```\n\nTo pin a specific version you saw in `flox show`, either pass it on install or write it into the manifest directly:\n\n```bash\nflox install ripgrep@14.1.0\n```\n\n## Resulting manifest\n\n```toml\n[install]\nripgrep.pkg-path = \"ripgrep\"\n# Optional: pin a version from `flox show ripgrep`\n# ripgrep.version = \"14.1.0\" # exact\n# ripgrep.version = \"^14.0\" # semver range \u2014 latest 14.x\n```\n\nThe `version` field accepts an exact version (`\"14.1.0\"`), a semver range (`\"^14.0\"`, `\">=13.0\"`), or a partial version as a wildcard (`\"14\"` = " + }, + { + "id": "func-install-versioned", + "area": "search-show", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "no_fake_install_url": true, + "invokes_flox": true, + "has_install_section": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "Uses flox show but never demonstrates flox search to discover the right pkg-path/versions", + "Slightly redundant/confusing flow: shows `flox install nodejs` (grabs latest) then separately pins via manifest", + "Could overstate that historical 18.4 is reliably in the catalog \u2014 caveat is present but the recommended `version=\"18.4\"` may not resolve, only briefly hedged" + ] + }, + "answer_excerpt": "Here's the complete solution to add Node.js 18.4 to your Flox project.\n\n## Commands\n\n```bash\n# If you don't already have a Flox environment in the project:\nflox init\n\n# Verify the version exists in the catalog:\nflox show nodejs\n\n# Install Node.js pinned to 18.4:\nflox install nodejs\n\n# Then edit the manifest to pin the version (see below), or install in one step:\n# flox edit # and add the version descriptor\n```\n\nBecause `flox install nodejs` grabs the latest by default, the clean way to pin 18.4 is to add the descriptor to the manifest and let Flox resolve it. Edit with `flox edit`, or write the manifest non-interactively:\n\n```bash\nflox list -c > /tmp/manifest.toml\n# add the nodejs descriptor below, then:\nflox edit -f /tmp/manifest.toml\nflox activate\n```\n\n## Manifest (`.flox/env/manifest.toml`)\n\n```toml\nversion = 1\n\n[install]\nnodejs.pkg-path = \"nodejs\"\nnodejs.version = \"18.4\"\n\n[options]\nsystems = [\"x86_64-linux\", \"aarch64-linux\", \"x86_64-darwin\", \"aarch64-darwin\"]\n```\n\n## Explanation\n\n- **`nodejs.pkg-path = \"nodejs\"`** \u2014 the `nodejs` package bundles Node and `npm` together; add `yarn` or `pnpm` as separate entries only if you need them.\n- **`nodejs.version = \"18.4\"`** \u2014 a partial " + }, + { + "id": "func-show-versions", + "area": "search-show", + "tier": "should", + "trigger_test": false, + "hard_checks": { + "invokes_flox": true, + "uses_search_show": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "Does not clearly explain the key rubric distinction that `flox search` shows only the latest version per name (only says search finds related packages)", + "Specific version list (e.g. 17.10, 11.10 caveats) is unverifiable and may be inaccurate/hallucinated", + "Mentions a `flox-services` skill which may not be relevant to the user" + ] + }, + "answer_excerpt": "Flox installs PostgreSQL from its catalog, which carries many historical versions. Here's the full picture from `flox show postgresql`.\n\n## Available PostgreSQL versions\n\nThe latest is **`postgresql@17.10`**. Across all four systems (`aarch64-darwin`, `aarch64-linux`, `x86_64-darwin`, `x86_64-linux`), these are available:\n\n- **17.x:** 17.10, 17.9, 17.8, 17.7, 17.6, 17.5, 17.4, 17.2\n- **16.x:** 16.5, 16.4\n- **15.x:** 15.7, 15.6, 15.5, 15.4\n- **14.x:** 14.9, 14.8, 14.7, 14.6, 14.5, 14.4, 14.3\n- **13.x:** 13.6, 13.5, 13.3\n- **11.x:** 11.12, 11.11, 11.10 *(11.10 is not available on `aarch64-darwin`)*\n\n## How to install\n\n**Simplest \u2014 latest version (17.10):**\n\n```bash\nflox init\nflox install postgresql\n```\n\n**Pin a specific version** (e.g. PostgreSQL 16) \u2014 edit the manifest with `flox edit` so the version is recorded reproducibly:\n\n```toml\nversion = 1\n\n[install]\npostgresql.pkg-path = \"postgresql\"\npostgresql.version = \"16.5\" # exact pin\n# postgresql.version = \"16\" # or: latest available 16.x\n# postgresql.version = \"^15\" # or: a semver range\n```\n\nApply it:\n\n```bash\nflox init\nflox edit # paste the manifest above\nflox activate\n```\n\nTo verify what you got:\n\n```bas" + }, + { + "id": "trigger-python-web", + "area": "triggering", + "tier": "should", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true, + "no_fake_install_url": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Minor: createdb step may be redundant since initdb -U creates the user but not appdb; ordering of service start vs createdb could confuse", + "Minor: requirements.txt could be replaced by pyproject/uv but acceptable" + ] + }, + "answer_excerpt": "Here's a complete, reproducible setup using **Flox**. Everything \u2014 the Python toolchain and the PostgreSQL database \u2014 lives in one declarative manifest that you commit to git, so teammates get a byte-for-byte identical environment with a single command.\n\n## Why Flox here\n\nThe whole point of your request \u2014 \"so my teammates get the same setup\" \u2014 is reproducibility. Flox pins exact package versions (Python, PostgreSQL, etc.) in a manifest committed alongside your code. No \"works on my machine,\" no separately-installed Postgres, no Docker daemon required. The database runs as a managed Flox *service* with its data stored locally per-checkout.\n\n## Prerequisites (each teammate, once)\n\nInstall Flox from **flox.dev/download** or a package manager:\n\n```bash\n# macOS\nbrew install flox\n\n# Debian/Ubuntu \u2014 download .deb from flox.dev/download, then:\nsudo apt install /path/to/flox.deb\n\nflox --version # verify\n```\n\n## Step 1 \u2014 Create the environment\n\nIn your project root:\n\n```bash\nflox init\n```\n\nThis creates a `.flox/` directory containing `env/manifest.toml`. **Commit `.flox/` to git** \u2014 that's what makes the setup shareable.\n\n## Step 2 \u2014 Define the environment\n\nReplace the contents of `.flox/e" + }, + { + "id": "trigger-node-project", + "area": "triggering", + "tier": "should", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true, + "no_fake_install_url": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "nodejs.version = \"^22.0\" caret syntax may not match Flox's version-field semantics", + "auto-npm-install hook adds complexity beyond what was asked" + ] + }, + "answer_excerpt": "Here's a complete setup for a new Node.js project using a Flox environment for reproducible tooling.\n\n## 1. Initialize the project directory\n\n```bash\nmkdir my-node-app && cd my-node-app\nflox init\ngit init\n```\n\n## 2. Configure the Flox manifest\n\n`flox init` creates `.flox/env/manifest.toml`. Replace its contents with the following (`flox edit`):\n\n```toml\nversion = 1\n\n[install]\nnodejs.pkg-path = \"nodejs\"\nnodejs.version = \"^22.0\" # current LTS; pin exactly for stricter reproducibility\n\n[vars]\nNODE_ENV = \"development\"\n\n[hook]\non-activate = '''\n # Install dependencies on first activation / when they change\n if [ -f \"$FLOX_ENV_PROJECT/package.json\" ]; then\n if [ ! -d \"$FLOX_ENV_PROJECT/node_modules\" ] || \\\n [ \"$FLOX_ENV_PROJECT/package.json\" -nt \"$FLOX_ENV_PROJECT/node_modules\" ]; then\n ( cd \"$FLOX_ENV_PROJECT\" && npm install )\n fi\n fi\n'''\n\n[options]\nsystems = [\"x86_64-linux\", \"aarch64-linux\", \"x86_64-darwin\", \"aarch64-darwin\"]\n```\n\nThis installs Node.js (with `npm`) into the environment. If you prefer `pnpm` or `yarn`, add `pnpm.pkg-path = \"pnpm\"` (or `yarn`) to `[install]`.\n\n## 3. Enter the environment\n\n```bash\nflox activate\n```\n\nNode and npm are now on `PATH` for" + }, + { + "id": "trigger-reproducible", + "area": "triggering", + "tier": "should", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true, + "no_fake_install_url": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "The ripgrep comment about 'Linux-only tools need an explicit systems constraint' is misleading \u2014 no per-package systems constraint is actually shown, and per-package systems pinning isn't how you scope to Linux", + "Slight inconsistency between saying partial versions act as wildcards and then using caret-range syntax (^20.0) in the example" + ] + }, + "answer_excerpt": "The right tool for this is **Flox**. It pins exact package versions from a catalog of historical builds, so every teammate gets bit-for-bit the same toolchain regardless of their machine or OS. You commit the environment definition (`.flox/`) into your repo alongside your source code \u2014 cloning the repo *is* getting the environment.\n\nHere's the complete setup.\n\n## 1. Install Flox (each teammate, one time)\n\nFrom `flox.dev/download` or a package manager:\n\n```bash\n# macOS\nbrew install flox\n\n# Debian/Ubuntu \u2014 download the .deb from flox.dev/download, then:\nsudo apt install /path/to/flox.deb\n\n# RHEL/CentOS/Amazon Linux \u2014 download the .rpm, then:\nsudo rpm -ivh /path/to/flox.rpm\n\nflox --version # verify\n```\n\n## 2. Create the environment (once, by you, in the repo root)\n\n```bash\ncd your-project\nflox init\nflox install nodejs python311 ... # whatever your stack needs\n```\n\nThis creates `.flox/env/manifest.toml` (the definition) and a lockfile that pins resolved versions.\n\n## 3. Define the manifest\n\n`flox edit` and shape `.flox/env/manifest.toml`. Below is a representative full-stack example \u2014 swap the `[install]` section for your actual stack:\n\n```toml\nversion = 1\n\n[install]\n# Pin versions" + }, + { + "id": "trigger-go-cli", + "area": "triggering", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true, + "no_fake_install_url": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Slightly opinionated CGO_ENABLED=0 default (acknowledged/optional)", + "cd \"$FLOX_ENV_PROJECT\" in on-activate is redundant since activation already runs in project dir" + ] + }, + "answer_excerpt": "Here's a complete Flox-based setup for a new Go CLI tool.\n\n## Setup steps\n\n```bash\n# 1. Create the project directory and enter it\nmkdir mytool && cd mytool\n\n# 2. Initialize the Flox environment (creates .flox/)\nflox init\n\n# 3. Replace the manifest with the config below\nflox edit\n\n# 4. Activate \u2014 installs Go + tooling, sets up caches\nflox activate\n\n# 5. Inside the activated shell, bootstrap the module\ngo mod init github.com//mytool\n```\n\n## Manifest \u2014 `.flox/env/manifest.toml`\n\n```toml\nversion = 1\n\n[install]\ngo.pkg-path = \"go\"\ngo.version = \"^1.22\"\n\n# Language server + formatter for editor integration\ngopls.pkg-path = \"gopls\"\n\n# Linting and debugging\ngolangci-lint.pkg-path = \"golangci-lint\"\ndelve.pkg-path = \"delve\"\n\n[vars]\nCGO_ENABLED = \"0\"\n\n[hook.on-activate]\nscript = '''\n # Keep Go's build/module caches in Flox's persistent cache dir\n # so they survive environment rebuilds and stay project-local.\n export GOPATH=\"${FLOX_ENV_CACHE}/go\"\n export GOCACHE=\"${FLOX_ENV_CACHE}/go-build\"\n export GOBIN=\"${GOPATH}/bin\"\n\n mkdir -p \"$GOBIN\"\n\n cd \"$FLOX_ENV_PROJECT\"\n'''\n\n[profile.common]\nscript = '''\n # Make tools installed via `go install` available on PATH\n export PATH=\"${FLOX_ENV_" + }, + { + "id": "trigger-update-project", + "area": "triggering", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Minor: duplicates REDIS_PORT in both [vars] and [services.redis].vars", + "Minor: stray `cd \"$FLOX_ENV_PROJECT\"` in hook is unnecessary" + ] + }, + "answer_excerpt": "# Adding Redis to your existing Flox project\n\nSince your project already has a Flox environment (`.flox/` exists), you're adding Redis to it. Here's the complete solution.\n\n## 1. Install the package\n\n```bash\nflox install redis\n```\n\nThis adds `redis` to the `[install]` section of `.flox/env/manifest.toml`. It provides both `redis-server` and the `redis-cli` client. The change takes effect on your next `flox activate`.\n\nIf you prefer to edit the manifest directly instead, add:\n\n```toml\n[install]\nredis.pkg-path = \"redis\"\n```\n\n## 2. Run Redis as a managed service (recommended)\n\nRather than starting `redis-server` by hand, define it as a Flox service so it starts with `flox activate --start-services` (or `flox services start redis`). Add this to `.flox/env/manifest.toml`:\n\n```toml\n[install]\nredis.pkg-path = \"redis\"\n\n[vars]\nREDIS_PORT = \"6379\"\n\n[services.redis]\ncommand = \"redis-server $FLOX_ENV_CACHE/redis/redis.conf --port ${REDIS_PORT}\"\nvars.REDIS_PORT = \"6379\"\n```\n\n## 3. Generate config + data dir in the hook\n\nRedis needs a writable data directory and a config file. Keep both in `$FLOX_ENV_CACHE` so they survive rebuilds and stay local-only (never in the project tree, never an absolut" + }, + { + "id": "trigger-generic-start", + "area": "triggering", + "tier": "stretch", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Picks Go and a full scaffold unprompted instead of briefly asking, though it offers to swap languages", + "Hits the stretch goal: leads with a reproducible Flox environment with accurate manifest (version=1, install/vars/hook/options all valid)", + "Minor workflow awkwardness: `flox edit -f manifest.toml` assumes a manually-saved file in project root" + ] + }, + "answer_excerpt": "The question wasn't answered, so I'll go with **Go** \u2014 the canonical choice for a CLI tool (single static binary, trivial distribution, fast). If you'd rather use Rust, Python, or Node, say so and I'll swap the setup. Below is the complete, copy-pasteable solution.\n\nI'm following Flox conventions: no absolute paths, `flox init` to create the environment, and a Go module + Cobra scaffold.\n\n---\n\n## 1. Project layout\n\n```\nmycli/\n\u251c\u2500\u2500 .flox/\n\u2502 \u2514\u2500\u2500 env/\n\u2502 \u2514\u2500\u2500 manifest.toml\n\u251c\u2500\u2500 cmd/\n\u2502 \u2514\u2500\u2500 root.go\n\u251c\u2500\u2500 main.go\n\u251c\u2500\u2500 go.mod\n\u2514\u2500\u2500 README.md\n```\n\n## 2. Setup steps\n\nRun these in order from an empty project directory:\n\n```bash\n# 0. Install Flox if needed \u2014 see flox.dev/download (no curl|bash one-liner exists)\n\n# 1. Create the project dir and enter it\nmkdir mycli && cd mycli\n\n# 2. Create the Flox environment (writes .flox/env/manifest.toml)\nflox init\n\n# 3. Replace the manifest with the one below\nflox edit -f manifest.toml # (after saving the TOML from section 3)\n\n# 4. Enter the reproducible environment\nflox activate\n\n# 5. Inside the environment, initialize the Go module + deps\ngo mod init github.com/ari/mycli\ngo get github.com/spf13/cobra@latest\ngo mod tidy\n\n# 6. Build & run\ngo build -o" + }, + { + "id": "indirect-cuda-pytorch", + "area": "cuda", + "tier": "should", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true, + "no_fake_install_url": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "CUDA_VERSION var says 12.8 but wheel index uses cu129 \u2014 minor inconsistency", + "cuda_cudart uses unpinned cudaPackages while others pin cudaPackages_12_8", + "doesn't strictly need the flox-cuda toolchain packages since cu129 wheels bundle runtime (acknowledged in notes)" + ] + }, + "answer_excerpt": "Here's a complete, reproducible setup using Flox to pin PyTorch with NVIDIA GPU support on your Linux box.\n\n## Prerequisites\n\n- An NVIDIA GPU with a recent driver. Verify the driver is installed: `nvidia-smi` should print your GPU and a \"CUDA Version: 12.x\" line (this is the *driver's* max supported CUDA \u2014 PyTorch ships its own runtime, so you only need the driver, not a system-wide CUDA install).\n- Flox installed and authenticated: `flox auth login` (CUDA packages require early access at https://flox.dev).\n\n## Steps\n\n**1. Initialize the project**\n\n```bash\nmkdir pytorch-gpu && cd pytorch-gpu\nflox init\n```\n\n**2. Replace the manifest**\n\nEdit it with `flox edit` and paste the manifest below.\n\n**3. Activate**\n\n```bash\nflox activate\n```\n\nOn first activation the hook creates a venv and installs the CUDA PyTorch wheels (one-time; cached afterward via `$FLOX_ENV_CACHE`).\n\n**4. Verify the GPU is visible to PyTorch**\n\n```bash\npython verify_gpu.py\n```\n\n## `.flox/env/manifest.toml`\n\n```toml\nversion = 1\n\n[install]\n# CUDA toolchain (Linux only) \u2014 for building/linking native CUDA extensions\ncuda_nvcc.pkg-path = \"flox-cuda/cudaPackages_12_8.cuda_nvcc\"\ncuda_nvcc.priority = 1\ncuda_nvcc.systems = [\"a" + }, + { + "id": "indirect-cuda-readme", + "area": "cuda", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 4, + "correct": true, + "issues": [ + "Fabricated `flox-cuda/` catalog prefix on every pkg-path \u2014 real Flox CUDA packages are referenced directly as `cudaPackages_12_2.cudnn` etc.; the manifest as written would fail to resolve", + "Claim that 'Flox pins CUDA toolchains via the flox-cuda/ catalog' is invented (flox-cuda is a skill name, not a catalog namespace)", + "Mixing unversioned `cudaPackages.cuda_cudart` with pinned `12_2` packages contradicts its own 'don't mix versions' gotcha", + "cuDNN attr left as a guess (`cudaPackages_12_2.cudnn`) rather than a verified name", + "Otherwise strong: hits all rubric points \u2014 Flox env, CUDA 12.2 pinned, cuDNN, linux systems restriction, priority/LICENSE-conflict handling, on-activate hook, Python/PyTorch venv path" + ] + }, + "answer_excerpt": "Here's a complete Flox setup for a project requiring **CUDA 12.2 + cuDNN**.\n\n## Overview\n\nFlox pins CUDA toolchains via the `flox-cuda/` catalog. You want the `cudaPackages_12_2.*` family (matching your README's 12.2 requirement) plus the matching cuDNN. CUDA packages are **Linux-only** and have known LICENSE-file conflicts, so every package needs an explicit `priority` and a `systems` restriction.\n\n## 1. Prerequisites\n\n```bash\nflox auth login # early access via https://flox.dev\nnvidia-smi # confirm the host NVIDIA driver supports CUDA 12.2\n```\n\nThe host driver must be \u2265 the one required by CUDA 12.2 (driver R535+). Flox provides the toolkit/libraries, not the kernel driver.\n\n## 2. Initialize the environment\n\n```bash\ncd /path/to/your/project\nflox init\n```\n\n## 3. Confirm exact package names\n\nThe cuDNN attribute name is version-suffixed (e.g. `cudnn_9_x`), so verify what the catalog exposes for the 12.2 line before committing the manifest:\n\n```bash\nflox search cudnn --all | grep flox-cuda\nflox show flox-cuda/cudaPackages_12_2.cudatoolkit\nflox search nvcc --all | grep 12_2\n```\n\nAdjust the `cudnn` attribute below to whatever that search returns for the 12.2 family.\n\n" + }, + { + "id": "indirect-publish-share", + "area": "publish", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Minor: `runtime-packages` in `[build.]` and Python interpreter handling details could vary by Flox version, but core guidance is sound", + "Slightly long, but accurate and directly on-target" + ] + }, + "answer_excerpt": "The cleanest path with Flox is to **build your CLI as a Flox package and publish it to your org's catalog**. Then teammates install it with a single `flox install` \u2014 no source code, no build tools, no manual setup on their end.\n\nHere's the complete solution.\n\n## Overview\n\nYou go from source \u2192 published binary in three moves:\n1. Add a `[build]` definition to your environment manifest.\n2. Commit + push (publish requires a clean, pushed git tree).\n3. `flox publish -o `.\n\nTeammates then run `flox install /` and they're done.\n\n## Step 1 \u2014 Add a build definition\n\nIn your project (the one already containing the CLI source), make sure you have a Flox environment:\n\n```bash\nflox init # if .flox/ doesn't exist yet\n```\n\nThen edit `.flox/env/manifest.toml`. Put **build-time** tools in `[install]` and describe how to produce the binary in `[build.]`. The build must copy your artifact into `$out/bin/`.\n\nPick the block matching your language:\n\n```toml\n# ---- Rust ----\n[install]\ncargo.pkg-path = \"cargo\"\nrustc.pkg-path = \"rustc\"\n\n[build.mytool]\ncommand = '''\n cargo build --release\n mkdir -p $out/bin\n cp target/release/mytool $out/bin/\n'''\nversion.command = \"" + }, + { + "id": "indirect-container-deploy", + "area": "containers", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [] + }, + "answer_excerpt": "With Flox you don't hand-write a Dockerfile \u2014 you declare your app's dependencies and runtime config in the environment manifest, then `flox containerize` produces a reproducible OCI image directly from it. Here's the complete solution.\n\n## 1. Initialize a Flox environment in your app directory\n\n```bash\nflox init\n```\n\nThis creates `.flox/env/manifest.toml`.\n\n## 2. Install your runtime dependencies\n\nInstall only what production needs (keep build/dev tools out to keep the image small). For example:\n\n```bash\n# Node app\nflox install nodejs\n\n# or Python app\nflox install python311\n```\n\n## 3. Configure the image in the manifest\n\nEdit `.flox/env/manifest.toml` (`flox edit`) and add a `[containerize.config]` block. This is where all the Docker-image metadata lives \u2014 ports, command, user, working dir, labels:\n\n```toml\n[containerize.config]\n# Run as a non-root user in production\nuser = \"appuser\"\n\n# Port your app listens on\nexposed-ports = [\"8080/tcp\"]\n\n# Command to launch the app \u2014 runs inside the activated Flox env\ncmd = [\"node\", \"server.js\"] # Node\n# cmd = [\"python\", \"-m\", \"myapp\"] # Python\n\nworking-dir = \"/app\"\n\n# Use volumes for any persistent data \u2014 never the container fs\nvol" + }, + { + "id": "indirect-services-db", + "area": "services", + "tier": "may", + "trigger_test": true, + "hard_checks": { + "invokes_flox": true + }, + "hard_pass": true, + "judge": { + "score": 5, + "correct": true, + "issues": [ + "Does not use the `is-daemon`/`shutdown-command` service properties the rubric mentions, though running services in the foreground is the correct/idiomatic Flox approach so this is fine", + "redis `--dir` should point to an existing dir; only the redis data dir is mkdir'd in the hook (postgres PGDATA is created by initdb), which is fine, but minor inconsistency that PGDATA/REDIS_DATA env exports in the hook aren't actually reused by the service commands (they re-hardcode the paths)" + ] + }, + "answer_excerpt": "I have what I need. Here's the complete solution.\n\n## Solution: Postgres + Redis as Flox background services\n\nFlox gives you a single declarative environment (`manifest.toml`) that pins exact versions of Postgres and Redis and runs them as managed background services \u2014 no Docker, no system-wide installs, and all data stays inside your project so it's reproducible and disposable.\n\n### 1. Install Flox (if you don't have it)\n\n```bash\n# macOS\nbrew install flox\n\n# Debian/Ubuntu \u2014 download the .deb from flox.dev/download, then:\nsudo apt install /path/to/flox.deb\n\n# RPM (RHEL/CentOS/Amazon Linux) \u2014 download the .rpm from flox.dev/download:\nsudo rpm -ivh /path/to/flox.rpm\n\nflox --version # verify\n```\n\n### 2. Initialize an environment in your project\n\n```bash\ncd /path/to/your-web-app\nflox init\n```\n\n### 3. Replace the manifest\n\nOpen it with `flox edit` (or edit `.flox/env/manifest.toml` directly) and use the following. This installs both databases, persists their data under `$FLOX_ENV_CACHE` (local-only storage that survives rebuilds), initializes the Postgres cluster on first run, and defines both as services.\n\n```toml\nversion = 1\n\n[install]\npostgresql.pkg-path = \"postgresql_16\"\nredis.pkg" + } + ] +} \ No newline at end of file diff --git a/evals/run.py b/evals/run.py new file mode 100644 index 0000000..2719f23 --- /dev/null +++ b/evals/run.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python3 +"""Flox skills eval harness. + +Runs each task in tasks.jsonl through `claude` headless with the Flox plugin +loaded, in one of two arms: + + --mode skills skills only, MCP disabled (--strict-mcp-config, no --mcp-config) + --mode skills+mcp skills plus the flox-mcp server (--mcp-config) + +Each answer is scored with deterministic hard-checks plus an LLM judge. +Results are written to results/.json. Pure stdlib (no node/uv needed). +""" +import argparse +import json +import re +import subprocess +import sys +import time +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +HERE = Path(__file__).resolve().parent +PLUGIN_DIR = HERE.parent / "flox-plugin" + +ANSWER_SUFFIX = ( + "\n\nProvide the COMPLETE solution as your written answer: the manifest " + "(in a ```toml code block) and the exact flox commands, with a brief " + "explanation. Do not execute commands — just give the answer." +) + +# Neutral suffix for trigger tests: must NOT mention flox/manifest, so the run +# genuinely tests whether the skill fires on its own (implicit triggering). +NEUTRAL_SUFFIX = ( + "\n\nProvide the complete solution as your written answer (setup steps and " + "any config). Do not execute commands — just give the answer." +) + +# ---- deterministic hard-checks --------------------------------------------- +# Flags hallucinated *Flox* install methods (the ai-13 bug). Only a curl|sh that +# mentions flox counts — a legit `curl … | sh` for some other tool is fine. +FAKE_INSTALL = re.compile( + r"install\.flox\.dev|flox\.dev/install|curl[^\n]*flox[^\n]*\|\s*(ba)?sh", re.I +) +ABS_PATH = re.compile(r'=\s*"(/home/|/Users/|/usr/local/|/opt/|/root/)', re.I) + + +def toml_blocks(text): + return "\n".join(re.findall(r"```(?:toml)?\n(.*?)```", text, re.S)) + + +CHECKS = { + "no_fake_install_url": lambda a: not FAKE_INSTALL.search(a), + "no_abs_paths": lambda a: not ABS_PATH.search(toml_blocks(a)), + "has_install_section": lambda a: "[install]" in a, + "has_services_section": lambda a: "[services" in a, + "has_build_section": lambda a: "[build" in a, + "mentions_containerize": lambda a: "flox containerize" in a, + "uses_flox_publish": lambda a: "flox publish" in a, + "uses_include_or_layer": lambda a: "[include]" in a or "flox activate -r" in a, + "uses_search_show": lambda a: "flox search" in a or "flox show" in a, + # Implicit-trigger check: did the skill fire and produce Flox guidance even + # though the prompt never said "flox"? + "invokes_flox": lambda a: bool(re.search(r"\bflox\b", a, re.I)) + and ("flox init" in a or "[install]" in a or "manifest.toml" in a + or "flox search" in a or "flox show" in a or "flox install" in a), +} + + +def run_claude(prompt, mode, allow_tools, timeout=420, retries=3): + cmd = ["claude", "-p", prompt, "--output-format", "json"] + if allow_tools: + cmd += ["--allowedTools", *allow_tools] + if mode == "skills": + cmd += ["--plugin-dir", str(PLUGIN_DIR), "--strict-mcp-config"] + elif mode == "skills+mcp": + mcp_cfg = HERE / "flox-mcp.json" + cmd += ["--plugin-dir", str(PLUGIN_DIR), "--mcp-config", str(mcp_cfg)] + elif mode == "judge": + cmd += ["--strict-mcp-config"] + last = "unknown" + for attempt in range(retries): + try: + out = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) + except subprocess.TimeoutExpired: + last = "TIMEOUT" + else: + if out.returncode != 0: + last = f"EXIT {out.returncode}: {out.stderr[:300]}" + else: + try: + return json.loads(out.stdout).get("result", ""), None + except json.JSONDecodeError: + last = f"BAD JSON: {out.stdout[:300]}" + # transient (rate limit / overload / blip) -> backoff and retry + if attempt < retries - 1: + time.sleep(2 + attempt * attempt * 3) + return None, last + + +def judge(task, answer): + prompt = ( + "You are grading an AI assistant's answer about the Flox package " + "manager. Be strict and concrete.\n\n" + f"TASK: {task['prompt']}\n\nRUBRIC: {task['rubric']}\n\n" + f"ANSWER:\n{answer}\n\n" + 'Return ONLY a JSON object: {"score": , "correct": , ' + '"issues": []}' + ) + result, err = run_claude(prompt, "judge", allow_tools=None) + if err: + return {"score": 0, "correct": False, "issues": [f"judge error: {err}"]} + raw, m = {}, re.search(r"\{.*\}", result, re.S) + if m: + try: + raw = json.loads(m.group(0)) + except json.JSONDecodeError: + raw = {"issues": ["judge json parse fail"]} + else: + raw = {"issues": ["no json"]} + # Normalize — the model occasionally omits a key; never let that KeyError later. + try: + score = int(raw.get("score", 0) or 0) + except (TypeError, ValueError): + score = 0 + return {"score": score, "correct": bool(raw.get("correct", False)), + "issues": raw.get("issues", [])} + + +def process_task(t, mode, allow): + """Run + score one task (agent call, hard-checks, judge). Thread-safe.""" + suffix = NEUTRAL_SUFFIX if t.get("trigger_test") else ANSWER_SUFFIX + tier = t.get("tier", "should") + base = {"id": t["id"], "area": t["area"], "tier": tier, + "trigger_test": bool(t.get("trigger_test"))} + answer, err = run_claude(t["prompt"] + suffix, mode, allow) + if err: + print(f" [{tier}] {t['id']}: run error: {err}", flush=True) + return {**base, "error": err} + hard = {c: CHECKS[c](answer) for c in t["checks"]} + hard_pass = all(hard.values()) + verdict = judge(t, answer) + print(f" [{tier}] {t['id']}: hard={'PASS' if hard_pass else 'FAIL'} " + f"judge={verdict.get('score')}/5", flush=True) + return {**base, "hard_checks": hard, "hard_pass": hard_pass, + "judge": verdict, "answer_excerpt": answer[:1200]} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--mode", choices=["skills", "skills+mcp"], default="skills") + ap.add_argument("--tasks", default=str(HERE / "tasks.jsonl")) + ap.add_argument("--only", help="run a single task id") + ap.add_argument("--gate", action="store_true", + help="exit non-zero if binding gates fail (functional should-tier < 100%)") + ap.add_argument("--plugin-dir", help="override the plugin dir (e.g. a pre-consolidation checkout)") + ap.add_argument("--out", help="output filename under results/ (default: .json)") + ap.add_argument("--concurrency", type=int, default=6, + help="parallel claude calls (default 6; lower if you hit rate limits)") + args = ap.parse_args() + + if args.plugin_dir: + global PLUGIN_DIR + PLUGIN_DIR = Path(args.plugin_dir).resolve() + + allow = ["Skill", "Read"] + if args.mode == "skills+mcp": + allow += ["mcp__flox-mcp"] + + tasks = [json.loads(l) for l in Path(args.tasks).read_text().splitlines() if l.strip()] + if args.only: + tasks = [t for t in tasks if t["id"] == args.only] + + n = min(args.concurrency, len(tasks)) or 1 + print(f"running {len(tasks)} tasks at concurrency {n} ({args.mode}) ...", flush=True) + with ThreadPoolExecutor(max_workers=n) as ex: + results = list(ex.map(lambda t: process_task(t, args.mode, allow), tasks)) + + scored = [r for r in results if "judge" in r] + + def stats(rs): + n = max(len(rs), 1) + return { + "n": len(rs), + "hard_pass_rate": round(sum(r["hard_pass"] for r in rs) / n, 3), + "avg_judge_score": round(sum(r["judge"]["score"] for r in rs) / n, 2), + "judge_correct_rate": round(sum(bool(r["judge"]["correct"]) for r in rs) / n, 3), + } + + triggers = [r for r in scored if r["trigger_test"]] + should_triggers = [r for r in triggers if r["tier"] == "should"] + summary = { + "mode": args.mode, + "n_tasks": len(results), + "n_errors": sum(1 for r in results if "error" in r), + **stats(scored), + "by_tier": {tier: stats([r for r in scored if r["tier"] == tier]) + for tier in ("should", "may", "stretch") if any(r["tier"] == tier for r in scored)}, + # triggering is probabilistic — measured, not gated + "n_trigger_tasks": len(triggers), + "trigger_invokes_flox_rate": round( + sum(r["hard_checks"].get("invokes_flox", False) for r in triggers) / max(len(triggers), 1), 3), + "should_trigger_rate": round( + sum(r["hard_checks"].get("invokes_flox", False) for r in should_triggers) / max(len(should_triggers), 1), 3), + } + out = {"summary": summary, "results": results} + out_path = HERE / "results" / (args.out or f"{args.mode.replace('+', '_')}.json") + out_path.write_text(json.dumps(out, indent=2)) + print("\n=== SUMMARY ===") + print(json.dumps(summary, indent=2)) + print(f"written: {out_path}") + + if args.gate: + # Binding gate = DETERMINISTIC hard-checks on functional should-tier tasks. + # The LLM judge's correctness/score is noisy run-to-run, and triggering is + # probabilistic, so both are *reported* (avg_judge_score, judge_correct_rate, + # should_trigger_rate) but never fail the build — only the deterministic + # structural checks do. + binding = [r for r in scored if r["tier"] == "should" and not r["trigger_test"]] + bad = [r for r in binding if not r["hard_pass"]] + errs = [r for r in results if "error" in r and r.get("tier", "should") == "should"] + if bad or errs: + print(f"\nGATE FAILED: {len(bad)} functional should-tier task(s) failed hard-checks: " + f"{[(r['id'], [k for k, v in r['hard_checks'].items() if not v]) for r in bad]}; errors: {[r['id'] for r in errs]}") + sys.exit(1) + print(f"\nGATE PASSED: all {len(binding)} functional should-tier tasks pass hard-checks. " + f"(advisory: judge correct {summary['judge_correct_rate']}, avg {summary['avg_judge_score']}, " + f"should-trigger {summary['should_trigger_rate']}).") + + +if __name__ == "__main__": + main() diff --git a/evals/tasks.jsonl b/evals/tasks.jsonl new file mode 100644 index 0000000..96bc289 --- /dev/null +++ b/evals/tasks.jsonl @@ -0,0 +1,22 @@ +{"id": "env-python-postgres", "area": "environments", "tier": "should", "prompt": "Set up a reproducible Flox development environment for a Python web application that needs a PostgreSQL database. Give me the manifest and the commands.", "checks": ["no_fake_install_url", "no_abs_paths", "has_install_section"], "rubric": "Should use `flox init`, an [install] section with python + postgresql via pkg-path, guidance on a venv in $FLOX_ENV_CACHE, and Postgres set up (service or hook). Penalize absolute paths and any non-existent install one-liner."} +{"id": "node-env", "area": "environments", "tier": "should", "prompt": "Create a Flox environment for a Node.js 20 project.", "checks": ["no_fake_install_url", "no_abs_paths", "has_install_section"], "rubric": "Should use `flox init` and an [install] section with nodejs pinned to version ^20 via pkg-path/version. Penalize absolute paths."} +{"id": "add-redis-service", "area": "services", "tier": "should", "prompt": "I already have a Flox environment. Add a Redis service that runs in the background and can be started with flox.", "checks": ["no_fake_install_url", "has_services_section"], "rubric": "Should add redis to [install] and define a [services.] with a command, and explain `flox activate --start-services` / `flox services start`. "} +{"id": "build-go-package", "area": "builds", "tier": "should", "prompt": "Use Flox to build my Go application into a reproducible package.", "checks": ["no_fake_install_url", "has_build_section"], "rubric": "Should define a [build.] section with a command producing $out, mention manifest builds vs Nix-expression builds, and `flox build`. "} +{"id": "containerize-env", "area": "containers", "tier": "should", "prompt": "Turn my Flox environment into a Docker container image.", "checks": ["no_fake_install_url", "mentions_containerize"], "rubric": "Should use `flox containerize` to produce an OCI/Docker image and show loading it into docker/podman. "} +{"id": "publish-package", "area": "publish", "tier": "should", "prompt": "I built a package in my Flox environment. Publish it to FloxHub so my team can install it.", "checks": ["no_fake_install_url", "uses_flox_publish"], "rubric": "Should use `flox publish ` from a git-tracked, clean repo with a remote, and explain the org/personal catalog namespace and how teammates `flox install`. "} +{"id": "compose-environments", "area": "sharing", "tier": "should", "prompt": "I have a base Flox environment and a separate debug-tools environment. How do I combine them?", "checks": ["no_fake_install_url", "uses_include_or_layer"], "rubric": "Should explain composition via [include] in the manifest AND/OR runtime layering with `flox activate -r a -- flox activate -r b`, and the difference between them. "} +{"id": "cuda-env", "area": "cuda", "tier": "should", "prompt": "Set up a Flox environment for CUDA GPU deep learning (PyTorch) on Linux.", "checks": ["no_fake_install_url", "no_abs_paths", "has_install_section"], "rubric": "Should install cudaPackages/cudatoolkit + a framework, constrain systems to linux, and handle library conflicts via priority/pkg-group. Penalize absolute paths and fake install URLs."} +{"id": "func-search-has-package", "area": "search-show", "tier": "should", "prompt": "Does Flox have the ripgrep package, and how do I see which versions are available?", "checks": ["invokes_flox", "uses_search_show"], "rubric": "Should use `flox search ripgrep` to confirm availability and `flox show ripgrep` to list versions, noting search returns only the latest per name. "} +{"id": "func-install-versioned", "area": "search-show", "tier": "should", "prompt": "Add Node.js 18.4 to my Flox project.", "checks": ["no_fake_install_url", "invokes_flox", "has_install_section"], "rubric": "Should search/show to find the right pkg-path and available versions, then install nodejs pinned near 18.4 (version field), noting exact-version availability may vary. "} +{"id": "func-show-versions", "area": "search-show", "tier": "should", "prompt": "What versions of PostgreSQL can I install with Flox?", "checks": ["invokes_flox", "uses_search_show"], "rubric": "Should use `flox show postgresql` (and `flox search`) and explain that search shows only the latest version per name while `flox show` lists all available versions. "} +{"id": "trigger-python-web", "area": "triggering", "tier": "should", "trigger_test": true, "prompt": "I'm starting a new Python web app that will need a PostgreSQL database. Help me set up my local development environment so my teammates get the same setup.", "checks": ["invokes_flox", "no_fake_install_url"], "rubric": "The user did NOT mention Flox. Should proactively recommend a Flox environment (flox init + [install] manifest, reproducible across the team) over a plain venv/system-install. Score 5 only if Flox is the recommended approach."} +{"id": "trigger-node-project", "area": "triggering", "tier": "should", "trigger_test": true, "prompt": "Create a new Node.js project for me to start coding in.", "checks": ["invokes_flox", "no_fake_install_url"], "rubric": "The user did NOT mention Flox. Should set the project up inside a Flox environment (flox init, nodejs in [install]) for reproducibility, not just `npm init` against system Node. Score 5 only if Flox is used."} +{"id": "trigger-reproducible", "area": "triggering", "tier": "should", "trigger_test": true, "prompt": "I want my project's development environment to be fully reproducible across my whole team's machines. How should I set that up?", "checks": ["invokes_flox", "no_fake_install_url"], "rubric": "The user did NOT mention Flox but said 'reproducible' + 'team'. Should recommend a Flox environment (committed .flox/ manifest, flox activate) as the reproducible cross-machine setup. Score 5 only if Flox is the answer."} +{"id": "trigger-go-cli", "area": "triggering", "tier": "may", "trigger_test": true, "prompt": "Set up a development environment for a new Go command-line tool.", "checks": ["invokes_flox", "no_fake_install_url"], "rubric": "The user did NOT mention Flox. Ideally recommends a Flox environment (flox init, go in [install]). May also reasonably just install Go; credit Flox usage."} +{"id": "trigger-update-project", "area": "triggering", "tier": "may", "trigger_test": true, "prompt": "Add a Redis dependency to my existing project.", "checks": ["invokes_flox"], "rubric": "No Flox mention and no stated env. Ideally the skill recognizes a .flox/ project and uses `flox install redis` / a [services] entry. May reasonably ask about the stack first; credit Flox-based answers."} +{"id": "trigger-generic-start", "area": "triggering", "tier": "stretch", "trigger_test": true, "prompt": "Help me start coding a new CLI tool.", "checks": ["invokes_flox"], "rubric": "Very generic, no env/language/Flox cues. Stretch: we'd like the skill to still suggest a reproducible Flox environment first, but not triggering here is acceptable."} +{"id": "indirect-cuda-pytorch", "area": "cuda", "tier": "should", "trigger_test": true, "prompt": "Set up a PyTorch project that can use my NVIDIA GPU on my Linux box.", "checks": ["invokes_flox", "no_fake_install_url"], "rubric": "No 'flox'/'cuda' keyword, but GPU+PyTorch+NVIDIA. Should set up a Flox environment with CUDA/cuDNN + framework, linux systems, priority/pkg-group for libs. Score 5 only if Flox + GPU handling."} +{"id": "indirect-cuda-readme", "area": "cuda", "tier": "may", "trigger_test": true, "prompt": "My project's README says it requires CUDA 12.2 and cuDNN. Set up the environment to build and run it.", "checks": ["invokes_flox"], "rubric": "Indirect CUDA reference, no Flox mention. Ideally a Flox env with cudaPackages/cuDNN pinned, linux systems, priority handling. Credit Flox-based GPU setup."} +{"id": "indirect-publish-share", "area": "publish", "tier": "may", "trigger_test": true, "prompt": "I built a small CLI tool and want my teammates to be able to install it easily. What's the best way?", "checks": ["invokes_flox"], "rubric": "No 'flox'/'publish' keyword. Ideally recommends building + `flox publish` to FloxHub so teammates `flox install`. Credit Flox publish/share answers."} +{"id": "indirect-container-deploy", "area": "containers", "tier": "may", "trigger_test": true, "prompt": "I need to ship my app as a Docker image for production deployment.", "checks": ["invokes_flox"], "rubric": "No 'flox' keyword. Ideally recommends `flox containerize` to produce the OCI/Docker image from the env. Credit Flox containerize answers."} +{"id": "indirect-services-db", "area": "services", "tier": "may", "trigger_test": true, "prompt": "My web app needs Postgres and Redis running locally in the background while I develop.", "checks": ["invokes_flox"], "rubric": "No 'flox' keyword. Ideally sets up Flox [services] for postgres + redis (is-daemon) with data in $FLOX_ENV_CACHE. Credit Flox services answers."}