Fact-check a vulnerability report against the code it accuses. Deterministic, offline, no model, no guessing about who wrote it.
$ refute report.md --repo ./curl
REFUTED (3)
x symbol ossl_quic_reassemble_frame line 12
no call or definition of ossl_quic_reassemble_frame( appears anywhere in the tree
x path lib/vtls/openssl_quic.c line 26
no file named openssl_quic.c exists anywhere in the tree
x symbol quic_validate_frame_length line 27
no call or definition of quic_validate_frame_length( appears anywhere in the tree
summary
9 claims checked - 0 verified, 3 refuted, 6 unverifiable
grounding: 0% of the 3 decidable claims match the code
3 claim(s) contradict this source tree.
Ask the reporter to reconcile them before spending review time.
That run took 0.9 seconds across curl's 4,453 files. Reading the report carefully enough to reach the same conclusion by hand takes about an hour.
In January 2026 curl shut down its bug bounty programme. Daniel Stenberg's reason was not that the bounty failed — it was that roughly 20% of incoming security submissions were fabricated, each one costing about an hour of maintainer time to disprove, while the share of genuine vulnerabilities fell to around 5%. He described it as open source being DDoSed.
curl is not alone. The Jazzband Python collective shut down. Godot maintainers describe triage as draining and demoralising.
A good fabricated report is confident, well-formatted and specific. It names a file, a function, a line number, and quotes code. That specificity is what makes it expensive: you have to go and look. And it is exactly what makes it checkable — because those specifics are either true of your code or they are not.
| Tool | Covers | Doesn't cover |
|---|---|---|
| anti-slop, Open Slop, GitHub's PR screening | Pull requests | Security reports — different surface entirely |
| ProjectDiscovery Triage | Sandbox reproduction of web app findings | Proprietary; needs a runnable target, not a source tree |
| honeyslop | Catching scanners with planted decoy code | Reports about your real code |
Nothing takes an inbound report and checks its claims against your actual source tree. That is the whole of what refute does.
refute pulls out every statement in a report that can be proved false and checks each one:
| Claim | Verified when | Refuted when |
|---|---|---|
path — lib/vtls/openssl.c |
the file exists | no file with that name exists anywhere in the tree |
line — urlapi.c:1423 |
the file is long enough | the file has fewer lines |
symbol — parse_hostname() |
it is called or defined somewhere | it appears nowhere in the tree, and the report ties it to this project |
symbol in file — "foo() in bar.c" |
it is in that file | it is in the tree, but not in that file |
| excerpt — code quoted as yours | the excerpt appears in the file | it does not |
| version — "affects 8.11.0" | a matching release tag exists | the project tags releases and this one isn't among them |
| CVE id | — | the year is impossible |
Anything else comes back unverifiable, which is a real answer and not a polite way of saying "suspicious".
pip install refute # or: pipx install refute / uv tool install refutePython 3.9+. Zero dependencies — it is standard library only, on purpose: a tool you run on hostile input should not drag in a supply chain.
refute report.md --repo ./curl # check the working tree
refute report.md --repo ./curl --ref v8.11.0 # check a release, no checkout needed
refute report.md --repo . --reply # draft a reply for the thread
refute report.md --repo . --json # machine-readable, for automation
gh api /repos/o/r/issues/42 --jq .body | refute - --repo .Exit codes: 0 nothing contradicted, 1 something was refuted, 2 refute
could not run. So it drops straight into a triage pipeline.
Run it on your own report before you send it. A refuted claim in your draft is
a typo, a stale checkout, or a wrong version — all of which are much cheaper to
fix before submission than to argue about afterwards. Reports that survive
refute get taken seriously faster.
These are the rules the code is actually held to, not aspirations.
1. A REFUTED verdict is an accusation, so it must be defensible.
Whenever evidence is absent or ambiguous, the verdict is UNVERIFIABLE.
A reporter writing "I reproduced this with urllib3's HTTPSConnectionPool()"
has not claimed that symbol is in your tree, so refute does not refute it —
a bare symbol counts as a claim about your code only when the report ties it
there, by naming one of your files in the same paragraph or introducing it as
"the function X".
refute would rather miss a fabricated claim than falsely accuse an honest
reporter. Extraction is strict, verification is lenient, and both biases point
the same way: a path in the wrong directory still verifies, a symbol written
with a space before the paren still matches, whitespace never decides a
snippet.
2. refute never says who or what wrote a report.
It has no evidence about that. "AI detection" is unreliable and unfair, and
guessing would make the output unfalsifiable — the exact sin the tool exists to
catch. The output says this claim does not match the code, which is checkable
by anyone with the repository. The --reply draft is written the same way.
3. Deterministic and offline. No model, no network. Same report plus same tree gives the same verdict on any machine, which is what lets you paste a finding into a thread and defend it.
4. It is held to its own standard.
CI runs refute against examples/truthful-report.md (must pass) and
examples/fabricated-report.md (must fail) on every commit. While writing the
test fixtures the author confidently quoted a curl function and a symbol called
Curl_ssl_getsock — both from memory, both wrong, and refute caught both. The
failure mode is genuinely easy to fall into, which is rather the point.
- refute checks whether a report describes real code. It does not check whether the vulnerability is real. A report can be perfectly grounded and still wrong about the security consequences — that still needs you.
- A clean result means "worth a human read", never "confirmed".
- Symbol search is textual, not a compiler. A symbol behind heavy macro
generation may not be found; that yields
REFUTEDon a symbol claim, so check the reason line before quoting it. - Version checks need release tags. Shallow clones have none, and refute says so rather than guessing.
Bug reports about false REFUTED verdicts are the most valuable thing you can
send. They are correctness bugs and are treated as such — please include the
report text and the tree it was checked against. See CONTRIBUTING.md.
git clone https://github.com/rubenmaas/refute && cd refute
PYTHONPATH=src python -m unittest discover -s tests -t . -vNo dev dependencies to install. The suite runs on a clean checkout.
refute is free and MIT licensed, and it will stay that way.
If it caught a fabricated report before you spent an hour on it, that hour is what sponsorship pays for. Maintainer triage time is the resource this whole problem is consuming — funding the tool that protects it is a reasonable trade. See Sponsor.
MIT. See LICENSE.