feat(web-search): add firecrawl and duckduckgo as fallbacks - #15
Merged
Conversation
Exa's keyless MCP tier allows only 50 tool calls per day per IP, so a single research-heavy session can exhaust web search for the rest of the day. Add a provider chain that falls back to Firecrawl's keyless search endpoint and, last, DuckDuckGo Lite read through Jina's keyless reader. An empty result set ends the chain: a query with genuinely no hits must not burn every provider's quota. A quota rejection additionally opens a disk-backed circuit breaker so later searches skip the dead provider outright, which matters most for the long-lived Telegram daemon. Trips expire at the next UTC midnight but allow a probe every 30 minutes, so a false positive from an IP-shared burst does not sideline a provider for the rest of the day. Also throttle the Exa client to 2 requests per second, matching the keyless endpoint's actual limit, so a 429 reliably means the daily cap rather than a self-inflicted rate limit. The serving provider is surfaced in the tool details, the TUI call title and the Telegram tool entry, since silent degradation is otherwise invisible.
Firecrawl meters keyless search over a rolling 24h window rather than a calendar day, and reports the remaining time as `retry_after_seconds` in the 429 body instead of a Retry-After header. The breaker was therefore falling back to its next-UTC-midnight default and reopening hours early, wasting a probe cycle per search. Observed live: a keyless budget exhausted after roughly 40 searches reports ~22.9h remaining, against a default that would have expired in ~19h.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.