Repository navigation
feat(full-search): read pages through Firecrawl Local first, at no credit, where it is installed - #87
Merged
Merged
Conversation
…edit, where it is installed Firecrawl Local (firecrawl-local) fetches and cleans pages on this machine with Firecrawl's own code: no credit, and no provider sees the URL. The read chain is now local firecrawl tavily brightdata direct. - It answers as a running serve at FIRECRAWL_LOCAL_URL (default 127.0.0.1:3002), used only when /health names firecrawl-local, else as the binary (FIRECRAWL_LOCAL_BIN) run with --public-only, else it is skipped unlisted. It is looked up only by a run that reads with it. - No search key goes to it; a serve's own --token travels as FIRECRAWL_LOCAL_TOKEN in the stdin config, health check included. - The cloud and local scrapes share one request body and one parser, which also refuses a page Firecrawl Local marks blocked. - FIRECRAWL_API_URL points the cloud provider at another Firecrawl. - --status names how Firecrawl Local answers; --help no longer prints the line after the header. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and-off; secrets go only where they are safe Cross-review found the binary inheriting full-search's environment: with FIRECRAWL_API_KEY loaded from fnox or mcp-keys.env and FIRECRAWL_CLOUD unset, firecrawl-local handed each page that refused this machine to Firecrawl Cloud with the person's key, outside the key rotation and rests, and the page was reported read locally. Measured with a fake key and a local sink: the old head sent it, this one sends nothing. - The binary runs with every search key, FIRECRAWL_API_URL and FIRECRAWL_LOCAL_TOKEN unset and FIRECRAWL_CLOUD=off; a refused page comes back blocked and goes to the next provider. The URL follows `--`. - A page a serve fetched from the cloud is not taken as read locally; the docs say to start serve with FIRECRAWL_CLOUD=off. - FIRECRAWL_API_URL, where the cloud key goes, must be https or http on this machine; FIRECRAWL_LOCAL_TOKEN goes only to such a URL. Trailing slashes are dropped from both URLs. - --help says --via names one provider for reading and search alike. - Every negated assertion in full-search.bats now ends in `|| false`: a bare `! cmd` before a test's last line never fails it, as a mutant that survived showed. All 105 still pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…un time portability.bats refuses a tracked private IPv4 literal; the rest of the suite builds them with ip(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r out On main the Windows Git Bash job already ran its three suites in series in about 28 of its 30 minutes (status line 10, full-search 9, remote title 8); this PR's new full-search cases pushed windows-11-arm past the timeout. Each suite now has its own job on each OS: they run side by side, each with the full 30 minutes. ci-status still gates on every one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
/full-searchreads every page through Firecrawl Cloud first, and with a key configured each page costs a credit. Firecrawl Local (firecrawl-local, a single Rust binary) fetches and cleans pages on the machine itself, with the same cleaning code as the Firecrawl API. A page it reads costs nothing, and no provider ever sees its URL.Change
A
localprovider at the head of the read chain:local firecrawl tavily brightdata direct.It answers in the first way that works:
firecrawl-local serveatFIRECRAWL_LOCAL_URL(defaulthttp://127.0.0.1:3002). It is used only when/healthnames"engine": "firecrawl-local", so a different service on that port is never sent a page. A server down refuses the connection at once; one that hangs costs at most 2 s.FIRECRAWL_LOCAL_BIN, defaultfirecrawl-localonPATH), run once per page with--public-onlyand the run's timeout. It is bounded like fnox, so a hung binary is stopped.--via localstill reports it as tried.It is looked up only by a run that reads pages with it, or by
--status. A search, or a--viato another provider, never probes it.Security
--tokentakesFIRECRAWL_LOCAL_TOKEN, sent on the health check and the scrape in the stdin curl config, never on the command line.Shared code: the cloud and local scrapes use one request body and one parser. The parser also refuses a page Firecrawl Local marks
blocked.Also
FIRECRAWL_API_URLpoints the cloud provider and its credit lookup at another Firecrawl, such as a self-hosted one.--statusnames how Firecrawl Local answers.--helpnow stops at the exit codes; before, it also printedset -uo pipefail.Measurements (firecrawl-local 0.1.0, macOS arm64, headless Chromium)
The same 52 pages through each endpoint, with full-search's request:
auto)auto, 7 of the 52 pages went to the cloud: those that refuse the machine.This PR's script, keyless, the 52 pages read once through a server and once through the binary alone:
local. The binary alone was faster: p50 0.5 s and p90 3.0 s, against 0.7 s and 3.6 s through a server.example.comwent to the cloud. Firecrawl Local's 156 characters are underMIN_CHARS, so the script reads them as a block page.19 edge cases against real servers, 19 passed:
--via localwith and without Firecrawl Local;--read 2;--statusin each mode;--tokenserver with and without the token, and the token absent fromps;One of these found a bug, fixed here: a
--tokenserver also wants the token on/health.Cross-review
A second Claude session reviewed the first commit and gave it 70/100. Its HIGH finding: the binary inherited full-search's environment. With
FIRECRAWL_API_KEYloaded andFIRECRAWL_CLOUDunset,firecrawl-localsent each page that refused the machine to Firecrawl Cloud with the person's key. That spent credit outside full-search's rotation and rests, and the page was still reported asok local.Fixed in the second commit:
FIRECRAWL_API_URLandFIRECRAWL_LOCAL_TOKENunset, and withFIRECRAWL_CLOUD=off. The URL comes after--.FIRECRAWL_API_URLmust behttps, orhttpon this machine.FIRECRAWL_LOCAL_TOKENis sent only to such a URL. Trailing slashes are dropped.Proved end to end on macOS with a fake key,
FIRECRAWL_API_URLpointed at a local sink, and two pages that refuse the machine:"origin":"cli@local-0.1.1",Bearer fc-FAKE…).A mutant that survived exposed a test gap. bats never fails a test on a bare
! cmdthat comes before its last line, becauseset -eskips negated commands. All 36 such assertions infull-search.batsnow end in|| false, and they all still pass.Tests
tests/full-search.bats: 105 of 105, under bash 5.3 and under stock bash 3.2. There are 20 new cases.-w, as real curl does, and answers "connection refused" where nothing listens.mainand depends on the time of day.just lintandjust format-checkare clean.Not in this PR
firecrawl-local serveon each machine. That's a host service, launchd or a systemd user unit, so it belongs in the fleet configuration. The binary alone already works, and reads as fast.🤖 Generated with Claude Code