Repository navigation
PR 1 — evals: skills eval suite + CI gate - #9
Conversation
|
It would be nice to already have a CI here. |
c4ee91a to
a8cfa52
Compare
|
Added CI: |
a8cfa52 to
ee014f4
Compare
|
Update: pulled the GitHub Actions workflow back out for now to keep this focused on the harness + baselines (the progressive-disclosure refactor in the stack is the priority). The |
| @@ -0,0 +1,80 @@ | |||
| # Flox skills evals | |||
|
|
|||
| Eval harness that gates the skills consolidation. Each step of the merge must | |||
There was a problem hiding this comment.
Generalize, after consolidation, this will be stale. It's just solid evals.
billlevine
left a comment
billlevine
left a comment
There was a problem hiding this comment.
All of these prompts specifically reference flox. I would like to see a richer set of evals including those that do not reference "Flox" at all. Assume the user may not even know the flox skill is installed, but just wants a new project, update a project, etc. May be very generic into touching some key words like reproducible, etc.
I would like to see a prompt or two for each basic functionality point touched in the skill. "Does flox have xxx software" and "add nodejs 18.4 to the project" both would touch search/show and likely install. etc.
This may be difficult and not hit 100%, but that's fine. We'll get a baseline with MCP.
Then for each consolidated skill update the evals and run before and after consolidation to test the functionality folded in. e.g. publish specific. cuda specific (maybe even indirect references like having pytorch or cuda references in readme(s)). This is so we can properly gauge if we are triggering on the right things and getting the context we need. Currently each independent skill has many keywords in the front matter to allow for rich triggering which we will loose with consolidation. The question is how much and can we mitigate some of it... somehow. I would expect dozens of evals by the time all the skills are consolidated many focused on specific areas and a "should trigger", "may trigger", and stretch prompts that may not but we would like them to.
This sets us up well for an auto-research run.
ee014f4 to
4b24740
Compare
felix8
changed the title
|
Expanded the suite per your review, and ran it before vs after consolidation (+ an MCP baseline). 22-task tiered suite (should/may/stretch).
Key finding: collapsing the 7 keyword-rich skills into one did not cost triggering — every no-"flox" prompt fired in both setups. Quality comparable; MCP added nothing. (Triggering is probabilistic — an isolated prompt flaked on one run — so the gate reports Next is your AI-249 idea: extend from prompt-level to project-level conversion evals, run the consolidated skill, then add |
| {"id": "containerize-env", "area": "containers", "tier": "should", "prompt": "Turn my Flox environment into a Docker container image.", "checks": ["no_fake_install_url", "mentions_containerize"], "rubric": "Should use `flox containerize` to produce an OCI/Docker image and show loading it into docker/podman. "} | ||
| {"id": "publish-package", "area": "publish", "tier": "should", "prompt": "I built a package in my Flox environment. Publish it to FloxHub so my team can install it.", "checks": ["no_fake_install_url", "uses_flox_publish"], "rubric": "Should use `flox publish <build>` from a git-tracked, clean repo with a remote, and explain the org/personal catalog namespace and how teammates `flox install`. "} | ||
| {"id": "compose-environments", "area": "sharing", "tier": "should", "prompt": "I have a base Flox environment and a separate debug-tools environment. How do I combine them?", "checks": ["no_fake_install_url", "uses_include_or_layer"], "rubric": "Should explain composition via [include] in the manifest AND/OR runtime layering with `flox activate -r a -- flox activate -r b`, and the difference between them. "} | ||
| {"id": "cuda-env", "area": "cuda", "tier": "should", "prompt": "Set up a Flox environment for CUDA GPU deep learning (PyTorch) on Linux.", "checks": ["no_fake_install_url", "no_abs_paths", "has_install_section"], "rubric": "Should install cudaPackages/cudatoolkit + a framework, constrain systems to linux, and handle library conflicts via priority/pkg-group. Penalize absolute paths and fake install URLs."} |
There was a problem hiding this comment.
This should use the flox-cuda catalog. May need an update to the skill file.
billlevine
left a comment
billlevine
left a comment
There was a problem hiding this comment.
A couple little nits, but looks good!
e65699f to
494afcb
Compare
A standalone, tiered eval suite for the Flox skill (not just a consolidation gate). 22 tasks across should/may/stretch tiers covering: functional flows, implicit triggering (prompts that never say "flox"), search/show/install probes, and per-skill incl. indirect references (PyTorch/CUDA-in-README, etc.). Scored by deterministic hard-checks + an LLM judge. `run.py --gate` binds only on the deterministic hard-checks of functional should-tier tasks; the LLM judge score and the implicit-trigger rate are probabilistic, so they're reported (avg_judge_score, judge_correct_rate, should_trigger_rate) but never fail the build. CI workflow (.github/workflows/evals.yml) runs it on PRs using the org-managed MANAGED_SKILLS_ANTHROPIC_API_KEY secret. Baselines committed here are the pre-consolidation (7-skill) runs: skills-only : 22/22 hard, 100% functional correct, judge 4.68, trigger 11/11 skills+MCP : 22/22 hard, judge 4.64, trigger 11/11 (MCP adds nothing — AI-93) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
494afcb to
9cc741f
Compare
Standalone, tiered eval suite for the Flox skill (no longer framed as just a consolidation gate, per @billlevine).
22 tasks, tiered should / may / stretch, covering:
Gate:
run.py --gatebinds on the deterministic hard-checks of functional should-tier tasks. The LLM judge and implicit-trigger rate are probabilistic, so they're reported (avg_judge_score,judge_correct_rate,should_trigger_rate) but never fail CI. Workflow.github/workflows/evals.ymlruns it using the org-managedMANAGED_SKILLS_ANTHROPIC_API_KEY.Baselines here = pre-consolidation (7 skills). See the comment for the before/after.
🤖 Generated with Claude Code