ade-bench-plugin

A Claude Code plugin that generates ADE-Bench benchmark tasks from your own dbt project.

What it does

Instead of relying on ADE-Bench's built-in sample projects, this plugin lets you benchmark AI agents against your own dbt models and data. It provides two skills:

• plan-tasks (recommended) — Interactive pair-planning. Explores your project with you, reasons about what makes good benchmarks, and builds a task plan collaboratively before generating anything.
• create-task — Automated pipeline. Scans your project, matches models against a pattern catalog, and generates tasks. Faster, but less tailored.

Prerequisites

• Claude Code installed
• git, uv, and Docker available on your machine (for running the generated tasks)
• A dbt project with either:
  • A DuckDB database file, or
  • A Snowflake account with credentials configured

The plugin will install ADE-Bench for you on first use — plan-tasks and create-task detect when it's missing and offer to set it up at ~/.ade-bench. You can also install it explicitly:

/ade-bench:setup

Install

From a local clone (recommended while iterating)

git clone https://github.com/typedef-ai/ade-bench-plugin
claude plugin marketplace add ./ade-bench-plugin
claude plugin install ade-bench@ade-bench-marketplace

marketplace add accepts a path to any directory containing .claude-plugin/marketplace.json — the one in this repo registers the plugin as ade-bench@ade-bench-marketplace.

Verify:

claude plugin list

From GitHub directly

claude plugin marketplace add typedef-ai/ade-bench-plugin
claude plugin install ade-bench@ade-bench-marketplace

Session-only (no install)

claude --plugin-dir /path/to/ade-bench-plugin

Loads the plugin for that one session without registering it globally — useful for one-off testing.

Usage

Interactive planning (recommended)

/ade-bench:plan-tasks /path/to/my-dbt-project

With custom instructions:

/ade-bench:plan-tasks /path/to/my-dbt-project focus on the revenue pipeline, I want hard tasks that test Snowflake skills

The skill will:

1. Do a quick project scan, then ask you intake questions (how many tasks, difficulty, focus area)
2. Enter a planning loop — explore models, update a task plan file, ask you questions
3. Present the converged plan for approval
4. Generate all task files after you approve

Automated generation

/ade-bench:create-task /path/to/my-dbt-project

With options:

/ade-bench:create-task /path/to/my-dbt-project --output-dir ./my-benchmarks --db-name my_warehouse --db-path /path/to/duckdb/files

How it works

ADE-Bench tasks follow a setup/solve/verify pattern:

1. A setup patch introduces a realistic bug into a working dbt model (e.g., swapping LEFT JOIN for INNER JOIN)
2. An agent prompt describes the symptom without revealing the cause (e.g., "some customer records are missing from the mart")
3. The agent investigates and attempts a fix
4. dbt tests and table comparison against seed CSVs determine pass/fail

Both skills automate steps 1-2. The difference is how they choose bugs:

	plan-tasks	create-task
Approach	Reads models, understands business logic, reasons about what would be meaningful	Scans for SQL tokens, matches against pattern catalog
User input	Conversational — intake questions, iterative refinement	Pick from a ranked list
Task quality	Higher — bugs are semantically motivated	Template-level — correct but more generic
Speed	Slower (5-15 min with interaction)	Faster (2-5 min)

Bug categories

Difficulty	Examples
Easy	Missing column, wrong alias, removed WHERE filter
Medium	Wrong join type, incorrect aggregation, broken CASE expression
Hard	Semantic aggregation error, broken incremental logic, upstream bug manifesting downstream

Snowflake-specific bugs are also supported: IFF() logic flips, QUALIFY filter errors, FLATTEN scope issues, TRY_TO_* removal, BOOLOR/BOOLAND_AGG swaps, and more.

Running generated tasks

After generating tasks, use ADE-Bench to run them:

# Generate seeds (DuckDB only, first run)
ade run my_project001 --db duckdb --project-type dbt --agent sage --seed \
  --tasks-dir ./my-benchmarks/tasks

# Validate with sage agent
ade run my_project001 --db duckdb --project-type dbt --agent sage \
  --tasks-dir ./my-benchmarks/tasks

# Test with a real agent
ade run my_project001 --db duckdb --project-type dbt --agent claude \
  --tasks-dir ./my-benchmarks/tasks

For Snowflake projects:

# Requires SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER, SNOWFLAKE_PASSWORD,
# SNOWFLAKE_ROLE, SNOWFLAKE_WAREHOUSE environment variables

# Validate with sage agent
ade run my_project001 --db snowflake --project-type dbt --agent sage \
  --tasks-dir ./my-benchmarks/tasks

# Test with a real agent
ade run my_project001 --db snowflake --project-type dbt --agent claude \
  --tasks-dir ./my-benchmarks/tasks

Viewing results

After a run completes, ADE-Bench writes an experiments/<timestamp>/ directory containing results.json. Open it as a styled HTML dashboard:

ade view run                       # opens the most recent run in your browser
ade view run <run-id>              # opens a specific run
ade view run -o /path/to/experiments  # use a non-default experiments directory
ade view runs                      # list recent runs

Adding custom knowledge

The plan-tasks skill uses prose knowledge files as reasoning material. The create-task skill uses structured YAML patterns for matching.

For plan-tasks (prose)

Add .md files to knowledge/:

cp knowledge/_template.md knowledge/bigquery.md

Describe bug archetypes in prose — when they're impactful, how difficulty varies, how common they are. The skill uses these as inspiration, not a rigid ruleset.

For create-task (structured YAML)

Add .yaml files to patterns/:

cp patterns/_template.yaml patterns/my_patterns.yaml

Define patterns with applies_when tokens, db_types, mutation, and symptom_template.

Project structure

ade-bench-plugin/
├── .claude-plugin/
│   ├── plugin.json              # Plugin manifest
│   └── marketplace.json         # Local marketplace pointer (enables `marketplace add`)
├── commands/
│   ├── plan-tasks.md            # Slash-command wrapper → invokes plan-tasks skill
│   ├── create-task.md           # Slash-command wrapper → invokes create-task skill
│   └── setup.md                 # Slash-command wrapper → invokes setup skill
├── agents/
│   └── discover-project.md      # Read-only scanner (used by create-task)
├── knowledge/
│   ├── base.md                  # ANSI SQL bug archetypes (prose, used by plan-tasks)
│   ├── snowflake.md             # Snowflake-specific archetypes (prose)
│   └── _template.md             # Template for custom knowledge
├── patterns/
│   ├── base.yaml                # ANSI SQL patterns (structured, used by create-task)
│   ├── snowflake.yaml           # Snowflake-specific patterns (structured)
│   └── _template.yaml           # Template for custom patterns
├── skills/
│   ├── create-task/
│   │   └── SKILL.md             # Automated task generation
│   ├── plan-tasks/
│   │   └── SKILL.md             # Interactive task planning
│   └── setup/
│       └── SKILL.md             # Install/verify ade-bench (auto-invoked when needed)
└── README.md

License

MIT