Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -248,3 +248,6 @@ scripts/
!/docs/pipelines/prompts/meta.json
# Raw campaign evidence belongs in workspace/, not in public documentation.
/docs/pipelines/evidence/

# Symbiotic Security - Security Rules (auto-generated)
.github/instructions/symbiotic_security_rules.instructions.md
73 changes: 0 additions & 73 deletions THIRD_PARTY_NOTICES.md

This file was deleted.

1 change: 1 addition & 0 deletions docs/pipelines/pr_diff.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ Pass each option with `--pipeline-opt key=value`.

| Key | Default | What it does |
|---|---|---|
| `pr` | none | Target a single PR by number (`--pr 1234`). Bypasses `limit`/`since`/`until` and the draft/merge-state filters; quality filters still apply |
| `limit` | `50` | Maximum merged PRs to list. You get at most this many tasks |
| `since` / `until` | none | ISO dates (`2026-01-31`) bounding the merge date |
| `max_files_per_pr` | `5` | Skip PRs that change more files than this |
Expand Down
32 changes: 20 additions & 12 deletions docs/pipelines/pr_runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ fail before the fix and pass after it.
| Input | A GitHub or GitLab repository with merged PRs that add tests |
| Task | Resolve the issue a merged PR fixed, starting from its base commit |
| Reward | Graded `f2p_rate × p2p_rate`, plus strict `resolved` and `command_resolved` flags |
| Needs an LLM | Yes, for the one-time repository [bootstrap](../concepts/glossary.mdx#bootstrap) (cached) |
| Needs an LLM | Yes, for [bootstrap](../concepts/glossary.mdx#bootstrap) at each distinct PR base commit (cached) |
| Needs Docker | Yes, for bootstrap, validation and running tasks |
| Hosts | GitHub (needs `gh` on `PATH`) and GitLab |
| Languages | Any repository the bootstrap can build whose tests run under pytest, unittest, Django, `go test`, `cargo test`, `cargo nextest run`, Jest, Mocha or Vitest. The reference dataset has 63 Python and 37 Go tasks |
Expand All @@ -41,9 +41,12 @@ repo2rlenv generate \
--out ./tasks/click-pr-runtime
```

The first run bootstraps the repository: an LLM agent builds a Docker image in
which the test suite runs, capped by `--max-spend-usd` (default 5.0). The image is
cached, so later runs go straight to mining. `limit` counts PRs listed, not tasks
The pipeline discovers PRs before bootstrapping. For each eligible candidate, an
LLM agent builds a Docker image at the PR's exact base commit, capped by
`--max-spend-usd` (default 5.0) per bootstrap run. Images and test commands are
cached per base commit; PRs sharing a base reuse the same environment. Different
bases can require separate paid bootstrap runs. `--ref` does not override a PR's
base commit. `limit` counts PRs listed, not tasks
emitted; PRs without a test that flips from failing to passing are skipped, so
expect fewer tasks than `limit`. Each task lands in
`./tasks/click-pr-runtime/pallets__click-<pr>/`. Run the oracle, which should
Expand All @@ -70,20 +73,21 @@ flowchart TD
F --> G["Harbor task: bootstrap image,<br/>hidden tests, graded verifier"]
```

1. **Bootstrap** the repository once. An LLM agent iterates in Docker until the
test suite runs, and records the image and test commands. Results are cached.
See [Bootstrap](../reference/BOOTSTRAP.md).
2. **List merged PRs**, newest first, exactly as [`pr_diff`](pr_diff.md) does.
3. **Filter on metadata.** Skip drafts, unmerged PRs, PRs with no files, and titles
1. **List merged PRs**, newest first, exactly as [`pr_diff`](pr_diff.md) does.
2. **Filter on metadata.** Skip drafts, unmerged PRs, PRs with no files, and titles
that mark chores rather than fixes: backports, cherry-picks, reverts, version
bumps, releases, changelogs and branch merges or syncs. Optionally require a
minimum description length (`min_problem_statement_words`).
4. **Split the diff** into a source patch and a test patch by file path. Skip the
3. **Split the diff** into a source patch and a test patch by file path. Skip the
PR if either is empty.
5. **Filter on structure.** Skip PRs whose source changes are all under `.github/`
4. **Filter on structure.** Skip PRs whose source changes are all under `.github/`
(`skip_ci_only`), whose test patch adds no test function or class
(`require_new_test_funcs`), or that touch more than `max_source_files_per_pr`
source files. `lite_filter` adds SWE-bench Lite-style constraints.
source files. `lite_filter` adds SWE-bench Lite-style constraints.
5. **Bootstrap at the PR base commit**, before test synthesis or validation. An
LLM agent records a working image and test commands for that exact SHA. Both
are reused only for matching bases. An explicitly supplied bootstrap result
must match the PR base or generation fails. See [Bootstrap](../reference/BOOTSTRAP.md).
6. **Validate twice** in one container started from the bootstrap image. Reset to
the base commit, apply the test patch and run the tests that patch touches; then
reset, apply the fix and the test patch, and run them again. Keep the PR if at
Expand All @@ -110,13 +114,17 @@ Pass each option with `--pipeline-opt key=value`.

| Key | Default | What it does |
|---|---|---|
| `pr` | none | Target a single PR by number (`--pr 1234`). Bypasses `limit`/`since`/`until` and the draft/merge-state filters; quality filters still apply |
| `limit` | `50` | Maximum merged PRs to list. You get at most this many tasks |
| `since` / `until` | none | ISO dates (`2026-01-31`) bounding the merge date |
| `skip_drafts` | `true` | Skip draft PRs |
| `require_fail_to_pass` | `true` | Skip PRs with fewer than `min_fail_to_pass` F2P tests after validation |
| `min_fail_to_pass` | `1` | Minimum number of F2P tests |
| `validation_timeout_sec` | `600` | Timeout for each validation test run |
| `skip_validation` | `false` | Emit candidates without running tests. Tasks then have no F2P/P2P lists, a pass/fail exit-code reward and, with nothing measured, a 300 s per-run limit. For debugging |
| `synthesize_missing_tests` | `true` | Ask the configured LLM for a regression test when the PR contains no test changes or validation finds too few fail-to-pass tests; emit only after validation verifies the configured F2P minimum |
| `max_test_synthesis_spend_usd` | `1.5` | Maximum estimated LLM spend for missing-test synthesis during this run; `0` disables the spend cap, while attempts remain bounded |
| `max_test_synthesis_attempts` | `5` | Maximum generated-test attempts per run, including retries after validation feedback and providers that don't report an estimated cost |
| `require_new_test_funcs` | `true` | Require the test patch to add at least one test function or class |
| `skip_ci_only` | `true` | Skip PRs whose source changes are all under `.github/` |
| `max_source_files_per_pr` | `50` | Skip PRs that touch more source files than this |
Expand Down
20 changes: 17 additions & 3 deletions docs/reference/BOOTSTRAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Different repos have wildly different setup needs:
- Java projects use Maven or Gradle + a specific JDK
- C/C++ projects use CMake / Make / Bazel / Meson + system deps

Hand-writing a Dockerfile per repo is unworkable when the goal is "any repo." So the **bootstrap phase** does it for you: an LLM agent reads the repo, runs shell commands inside a sandboxed container, iterates until the build + tests succeed, then commits the result as a Docker image. The image is cached locally and reused for every subsequent task generated from that repo.
Hand-writing a Dockerfile per repo is unworkable when the goal is "any repo." So the **bootstrap phase** does it for you: an LLM agent reads the repo, runs shell commands inside a sandboxed container, iterates until the build + tests succeed, then commits the result as a Docker image. The image is cached locally and reused for tasks using the same repository commit and bootstrap options.

You only need bootstrap for pipelines that actually run code (`pr_runtime`, `commit_runtime`, `cve_patches`, `code_instruct`, `equivalence_tests`). The text-only `pr_diff` pipeline skips it entirely.

Expand Down Expand Up @@ -183,7 +183,12 @@ The agent loop is skipped; we just build your Dockerfile, commit the result, and

## How bootstrap works internally

Bootstrap is a separate phase, run **once per `(repo, commit)` and cached**. Inline iteration inside the per-task synthesis loop would be wasteful and slow.
Bootstrap runs **once per `(repo, commit, bootstrap options)` and is cached**.
For `pr_runtime`, PR discovery and cheap filters run first; bootstrap uses each
eligible PR's exact base SHA, not the repository's current HEAD or `--ref`.
Test synthesis, validation and task emission all use that base's image and test
commands. PRs sharing a base reuse the environment; different bases can require
separate paid runs, each bounded by the configured bootstrap budget.

```mermaid
flowchart TD
Expand All @@ -208,10 +213,19 @@ flowchart TD
end
```

You pay the LLM agent cost **once** per `(repo, commit)`; every later task generated from that repo reuses the cached image for free.
You pay the LLM agent cost **once** per `(repo, commit, bootstrap options)`;
later tasks using that same environment reuse the cached image for free.

The agent is implemented as a ReAct-style Thought / Action / Input loop with tools for shell execution, file reading, and a structured `SAVE_SETUP` call that ends the session by recording the rebuild + test commands the rest of the system uses. Per-language hints (Python / Node / Go / Rust / Java / C-C++) get injected into the agent's system prompt so it doesn't have to rediscover ecosystem conventions on every run.

### Parseable test output

Runtime pipelines decide which tests fail before a fix and pass after it from one result line per test. A command that exits 0 but prints only a summary, or uses a custom reporter, can never yield a task. So `SAVE_SETUP` runs the saved `test_cmds` the way the runtime pipelines will (verbosity flags added) and passes the output through the log parsers. If no per-test result is found, the save is rejected and the agent sees the output tail plus the supported formats. A rejection counts against `max_iterations`.

Supported output: pytest, unittest / Django, `go test`, cargo test / nextest, jest, mocha (`spec`), vitest (`verbose`), and TAP. TAP covers QUnit's CLI (its default reporter), tape, node-tap and any runner given `--reporter tap`. A repo whose own script hides the reporter, such as moment's `scripts/test.js`, passes once the agent builds the tests and calls `qunit` on the built files. TAP is detected from the command (`qunit`, `tape`, `--reporter tap`) or from a `TAP version` header, never from `ok` lines alone, so Go's per-package `ok` lines are not misread.

The number of parsed tests is stored as `tests_parsed` in `bootstrap.json`. A cached bootstrap with `tests_parsed == 0` is ignored and rebuilt, and `pr_runtime` rejects such an environment before test synthesis or validation. Caches written before this field existed load as `None` and are used as before. Set `BootstrapSpec.require_parseable_tests = False` to turn the bootstrap gate off.

## Cache strategy

Bootstrap is expensive (multi-minute, real LLM cost). Reuse aggressively.
Expand Down
Loading