Skip to content

feat(full-search): read pages through Firecrawl Local first, at no credit, where it is installed - #87

Merged
gutyoh merged 4 commits into
mainfrom
feat/full-search-firecrawl-local
Nov 18, 2026
Merged

gutyoh merged 4 commits into
mainfrom
feat/full-search-firecrawl-local

Conversation

@gutyoh

@gutyoh gutyoh commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Problem

/full-search reads every page through Firecrawl Cloud first, and with a key configured each page costs a credit. Firecrawl Local (firecrawl-local, a single Rust binary) fetches and cleans pages on the machine itself, with the same cleaning code as the Firecrawl API. A page it reads costs nothing, and no provider ever sees its URL.

Change

A local provider at the head of the read chain: local firecrawl tavily brightdata direct.

It answers in the first way that works:

  1. A running firecrawl-local serve at FIRECRAWL_LOCAL_URL (default http://127.0.0.1:3002). It is used only when /health names "engine": "firecrawl-local", so a different service on that port is never sent a page. A server down refuses the connection at once; one that hangs costs at most 2 s.
  2. The binary (FIRECRAWL_LOCAL_BIN, default firecrawl-local on PATH), run once per page with --public-only and the run's timeout. It is bounded like fnox, so a hung binary is stopped.
  3. Neither: it is skipped, and is not listed among the tried providers. --via local still reports it as tried.

It is looked up only by a run that reads pages with it, or by --status. A search, or a --via to another provider, never probes it.

Security

  • Private addresses still go only to the direct fetch, so they never reach it.
  • No search key goes to it. A server started with --token takes FIRECRAWL_LOCAL_TOKEN, sent on the health check and the scrape in the stdin curl config, never on the command line.

Shared code: the cloud and local scrapes use one request body and one parser. The parser also refuses a page Firecrawl Local marks blocked.

Also

  • FIRECRAWL_API_URL points the cloud provider and its credit lookup at another Firecrawl, such as a self-hosted one.
  • --status names how Firecrawl Local answers.
  • --help now stops at the exit codes; before, it also printed set -uo pipefail.

Measurements (firecrawl-local 0.1.0, macOS arm64, headless Chromium)

The same 52 pages through each endpoint, with full-search's request:

Read p50 p90
Firecrawl Cloud, keyless 50/52 0.5 s 2.5 s
Firecrawl Local, cloud fallback on (auto) 51/52 0.6 s 2.4 s
Firecrawl Local, cloud off 40/52 0.6 s 2.1 s
  • The 52 pages span static docs, product docs, forums, GitHub, JavaScript apps, news, PDFs, bot-protected sites and package registries.
  • Firecrawl Local read Reddit and the New York Times, which refuse the cloud's addresses. The cloud read crates.io, which needs JavaScript the local run didn't finish.
  • Where both read a page, the median word overlap was 0.93. The cloud's text is larger on documentation sites because it keeps the navigation.
  • In auto, 7 of the 52 pages went to the cloud: those that refuse the machine.
  • Idle memory: 8.9 MB per server.

This PR's script, keyless, the 52 pages read once through a server and once through the binary alone:

  • Both read 51 of 52, 50 of them through local. The binary alone was faster: p50 0.5 s and p90 3.0 s, against 0.7 s and 3.6 s through a server.
  • example.com went to the cloud. Firecrawl Local's 156 characters are under MIN_CHARS, so the script reads them as a block page.
  • The 17-character dummy PDF fails everywhere, for the same reason.

19 edge cases against real servers, 19 passed:

  • a server; the binary; neither;
  • --via local with and without Firecrawl Local;
  • a private address;
  • JavaScript pages through both;
  • a PDF;
  • search with --read 2;
  • --status in each mode;
  • another HTTP service on the port;
  • a --token server with and without the token, and the token absent from ps;
  • a page that refuses the machine, with the server's cloud fallback off;
  • a stopped server.

One of these found a bug, fixed here: a --token server also wants the token on /health.

Cross-review

A second Claude session reviewed the first commit and gave it 70/100. Its HIGH finding: the binary inherited full-search's environment. With FIRECRAWL_API_KEY loaded and FIRECRAWL_CLOUD unset, firecrawl-local sent each page that refused the machine to Firecrawl Cloud with the person's key. That spent credit outside full-search's rotation and rests, and the page was still reported as ok local.

Fixed in the second commit:

  • The binary runs with every search key, FIRECRAWL_API_URL and FIRECRAWL_LOCAL_TOKEN unset, and with FIRECRAWL_CLOUD=off. The URL comes after --.
  • A page a serve fetched from the cloud isn't taken as read locally.
  • FIRECRAWL_API_URL must be https, or http on this machine. FIRECRAWL_LOCAL_TOKEN is sent only to such a URL. Trailing slashes are dropped.

Proved end to end on macOS with a fake key, FIRECRAWL_API_URL pointed at a local sink, and two pages that refuse the machine:

  • The first commit sent the sink 2 calls from the binary ("origin":"cli@local-0.1.1", Bearer fc-FAKE…).
  • This head sends none. Only full-search's own cloud provider calls, as the next step in the chain.

A mutant that survived exposed a test gap. bats never fails a test on a bare ! cmd that comes before its last line, because set -e skips negated commands. All 36 such assertions in full-search.bats now end in || false, and they all still pass.

Tests

  • tests/full-search.bats: 105 of 105, under bash 5.3 and under stock bash 3.2. There are 20 new cases.
    • The fake curl now prints its status trailer only for -w, as real curl does, and answers "connection refused" where nothing listens.
    • The binary stub logs its environment, so a leaked key fails the test.
  • Mutation controls: 23 of 23 mutants killed. An earlier round found one redundant check, now removed.
  • Full bats suite: 1494 of 1495. The one failure, the statusline forecast test, fails the same way on main and depends on the time of day.
  • just lint and just format-check are clean.

Not in this PR

  • Installing and running firecrawl-local serve on each machine. That's a host service, launchd or a systemd user unit, so it belongs in the fleet configuration. The binary alone already works, and reads as fast.

🤖 Generated with Claude Code

gutyoh and others added 3 commits November 18, 2026 19:27
…edit, where it is installed

Firecrawl Local (firecrawl-local) fetches and cleans pages on this machine
with Firecrawl's own code: no credit, and no provider sees the URL. The
read chain is now local firecrawl tavily brightdata direct.

- It answers as a running serve at FIRECRAWL_LOCAL_URL (default
  127.0.0.1:3002), used only when /health names firecrawl-local, else as
  the binary (FIRECRAWL_LOCAL_BIN) run with --public-only, else it is
  skipped unlisted. It is looked up only by a run that reads with it.
- No search key goes to it; a serve's own --token travels as
  FIRECRAWL_LOCAL_TOKEN in the stdin config, health check included.
- The cloud and local scrapes share one request body and one parser,
  which also refuses a page Firecrawl Local marks blocked.
- FIRECRAWL_API_URL points the cloud provider at another Firecrawl.
- --status names how Firecrawl Local answers; --help no longer prints
  the line after the header.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and-off; secrets go only where they are safe

Cross-review found the binary inheriting full-search's environment: with
FIRECRAWL_API_KEY loaded from fnox or mcp-keys.env and FIRECRAWL_CLOUD
unset, firecrawl-local handed each page that refused this machine to
Firecrawl Cloud with the person's key, outside the key rotation and rests,
and the page was reported read locally. Measured with a fake key and a
local sink: the old head sent it, this one sends nothing.

- The binary runs with every search key, FIRECRAWL_API_URL and
  FIRECRAWL_LOCAL_TOKEN unset and FIRECRAWL_CLOUD=off; a refused page comes
  back blocked and goes to the next provider. The URL follows `--`.
- A page a serve fetched from the cloud is not taken as read locally; the
  docs say to start serve with FIRECRAWL_CLOUD=off.
- FIRECRAWL_API_URL, where the cloud key goes, must be https or http on
  this machine; FIRECRAWL_LOCAL_TOKEN goes only to such a URL. Trailing
  slashes are dropped from both URLs.
- --help says --via names one provider for reading and search alike.
- Every negated assertion in full-search.bats now ends in `|| false`: a
  bare `! cmd` before a test's last line never fails it, as a mutant that
  survived showed. All 105 still pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…un time

portability.bats refuses a tracked private IPv4 literal; the rest of the
suite builds them with ip().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r out

On main the Windows Git Bash job already ran its three suites in series
in about 28 of its 30 minutes (status line 10, full-search 9, remote
title 8); this PR's new full-search cases pushed windows-11-arm past the
timeout. Each suite now has its own job on each OS: they run side by side,
each with the full 30 minutes. ci-status still gates on every one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gutyoh
gutyoh merged commit 7e9c7de into main Nov 18, 2026
34 of 36 checks passed
@gutyoh
gutyoh deleted the feat/full-search-firecrawl-local branch November 18, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant