Repository navigation
How It Works
Every entry is embedded using bge-small-en-v1.5 via Cloudflare Workers AI — a model that converts text into 384 numbers representing its meaning. Those numbers are stored in a Vectorize index alongside the original text in D1.
When you call recall, your query is embedded the same way. Vectorize finds the stored entries whose numbers are closest to your query's numbers using cosine similarity, and returns them ranked by match score.
The practical result: you can store "users drop off at the payment step" and later query "onboarding problems" and it surfaces correctly. The keyword never appears — the meaning matches.
This is what separates Second Brain from a keyword search or tag system. You don't need to remember how you phrased something to find it again.
Recall does not rely on meaning alone. Its keyword arm finds the things meaning-based search is bad at: an exact name, a ticket number like #149, a version like v1.9, a phrase in Chinese, Japanese, or Korean, a memory captured months ago that a newer, semantically-similar note would otherwise bury. Matches are ranked by relevance, not recency, so a single old memory holding the right word is not crowded out by hundreds of newer ones that only partly match. And searching for a specific word or phrase stays fast and cheap as the brain grows, instead of getting slower and more expensive with every memory added.
Every memory's content is mirrored into a full-text index (FTS5, using a trigram tokenizer) kept in sync automatically by triggers whenever a memory is written, updated, or deleted. A trigram index matches on overlapping three-character runs, which is what makes it substring-, identifier-, and CJK-aware without any special-casing. At query time, matches are ranked by bm25, a standard relevance score; that order survives into recall's overall ranking as the tiebreak among otherwise equally-weighted candidates, rather than being discarded.
Recall always uses whichever method answers a query most cheaply and correctly, and falls back automatically the moment the index cannot be trusted for that query. A word under three characters can never match a trigram, but it no longer sends the whole query to the previous full-scan method: the index finds the memories for the longer words, and the short word ("io", "id", "k8") still counts toward the ranking. Only a query made entirely of short words uses the previous method. A query with several very common words is ranked through the index too, in two bounded passes (memories holding every word first, then the best matches for any of them), so an old memory that matches only on those words is found however large the brain grows; before 3.7.0 such a query was answered from only the newest 500 matches. If a single very common word matches more memories than the index can rank cheaply, or the bounded passes find nothing, recall falls back to the previous method for that query. And if the index is not built yet for a given brain, or a read against it fails for any reason, recall transparently falls back to the same full-scan method that has always been there.
Measured on the same brain at two sizes: a specific-word search reads on the order of a hundred rows (66-193) whether the brain holds 5,700 or 20,700 memories, flat, because the index's cost tracks how many memories match, not how many exist. The previous method read everything: 20,000-41,000 rows at the larger size. That is roughly 200-300x cheaper today, and the gap widens as the brain grows. A query that leans on a very common word costs more, and at 20,700 memories it was still about 60% cheaper than a full scan, though unlike a specific-word search, that cost grows with the brain rather than staying flat. In 3.7.0 a search with a short word reads about half as many rows as before on a 20,000-memory test brain, and the average search about 11% fewer. Saving a memory costs one small extra row write, for the per-workspace memory counter distillation relies on rather than for the search index itself, whose write was already part of the cost: seven rows instead of six.
The index is maintained without ever risking a save. If a write finds the index missing, it is recreated automatically; any other kind of index error disables syncing for it, non-destructively, until the nightly maintenance job repairs it. Either way the save itself always succeeds, and the previous full-scan method keeps working in the meantime. That nightly job also verifies the index against the real memory table, checks a rotating sample of rows for content drift, and rebuilds the index from scratch if something is actually broken.
Semantic search, the memory graph, and all API and MCP tools and response shapes are unchanged; no client needs updating. An existing brain fills its index in over successive nights; see How to Upgrade for the schedule.
Many searches are not keywords but requests: "Tell me all about Dana" or "User wants to prepare for a meeting with Dana, what should I know about her?" Words that only frame the request (user, wants, should, know, tell, show, find, help, and similar) are not used as keyword search terms when the question has other terms, so a note that merely echoes the scaffolding never outranks the note about the subject. A search made only of such words still searches for them. A rare word in the question, such as a name, always reaches the ranking even when the rest of the question is made of very common words, and a single rare word on its own ("gatewright") ranks the memory that contains it first.
Identifiers with underscores, such as DATABASE_URL and ERR_TLS_90412, are found. An identifier written directly next to Chinese, Japanese, or Korean text without a space, such as SB-024の決定, is searched as the whole identifier (sb-024), the same as when a space is typed (thanks to @oudouusa, #377).
Recall used to read every candidate note in full just to see where the query's words sit in it, and the Worker paid to parse all of that text. The database now works out where each word sits and sends back only the candidate ids and a small per-word result, never the note text. On a synthetic test brain where one note in five is 20 to 100 KB, measured Worker CPU per search fell from about 90 ms (3.6.0) to about 11 ms; brains of ordinary short notes are unchanged (about 6 ms). That figure is CPU measured on a local test setup, response parsing included and time spent waiting on Cloudflare services excluded. It has not been measured on Cloudflare itself, and nothing here claims a particular plan's CPU limit is met. Searches also no longer ask for the same candidate rows twice when they plan two passes, and no longer spend an AI call guessing topic tags: tag matching is a plain comparison against the tags you already use.
Semantic search alone has a weakness: stale memories can dominate retrieval when they're semantically similar. If you stored "Meeting Tuesday 2pm" a month ago and updated it to "Meeting moved to Wednesday 3pm" today, pure vector similarity might return the old version first.
Time-decay reranking solves this by applying exponential decay to match scores based on age.
How it works:
-
recallretrieves 3x more semantic matches than requested (e.g., top 15 for topK=5) - Each match's score is multiplied by a recency factor that decays toward a floor, not zero:
recencyFloor + (1 - recencyFloor) * e^(-age / halfLife) - Results are re-sorted by the adjusted score
- Deduplication and final topK selection happen after reranking
The half-life depends on the entry's tag: 7 days for task, 90 days for work, 180 days for context, 30 days by default.
The floor depends on volatility, importance, and status: 0.9 for canonical, durable, or importance_score >= 4 entries; 0.15 for volatile or task entries; 0.6 by default. A strong old match can never be buried entirely, only outranked by fresher ties.
The practical result: semantic search finds the right topic, time decay finds the right version. Old high-scoring matches can still surface if nothing recent is relevant, but fresh updates naturally win when the semantic scores are close.
This is especially powerful with the append workflow — update chunks get fresh timestamps, so they automatically outrank the original when you query for current information.
Recency is the primary lever, but it isn't the only one. The final ranking also factors in how often an entry has been recalled, its importance score, and its contradiction track record (see Contradiction-aware scoring below).
After the steps above pick recall's candidates, an optional second ranking step re-reads the closest ones against your question and reorders them. It uses a cross-encoder, bge-reranker-base on Workers AI, which reads the question and each memory together instead of comparing two precomputed vectors. It sits after time decay and before the diversity pass.
When it runs. RERANK_MODE is auto by default. In auto it runs only when it can change the answer: there are at least three candidates and the top two scores are within 15% of each other. When one candidate clearly leads, it is left alone. It never runs for a search that contains an identifier, because keyword search already answers those: a version (v1.9), a file name (config.yaml), a ticket number (#149), a digit next to a letter (40mg, abc123), an IP address, or a name with an underscore (snake_case). Ordinary prose does not count as an identifier: a sentence-final period, a plain number or year, and abbreviations like "e.g." do not stop it. on reranks every search with at least three candidates that has no identifier, and off never calls the model.
What it reads. Up to 30 candidates go in one request, each as a short excerpt of at most 400 characters picked around the words of your question, together with a question capped at 256 characters. The model sees no other note text.
Measured effect. On the evaluation set (about 1,700 synthetic questions, in 1,475 independent groups) the rank of the right answer improved clearly: MRR@10 rose by 0.058 (95% interval 0.049 to 0.067), most of all for searches built around a rare word (0.58 to 0.86) or a few common words (0.61 to 0.84). Questions worded completely differently from the memory improved only slightly. No kind of search we test got worse on average. In those tests it ran on about two searches in five.
Cost. On average about 0.4 extra Workers AI calls and about 0.2 neurons per search, so 1,000 searches a day use about 200 of the free plan's 10,000 daily neurons. How long the model takes on Workers AI has not been measured yet; a search waits at most 1.5 seconds for it.
It switches itself off. The model runs only after a self-check passes (repeated about weekly; the check fails if the model answers too slowly). If it errors or times out three times in a row, it is turned off for six hours until the self-check passes again. Whenever it is off, fails, or times out, search returns exactly the order it would have returned without it. Setting RERANK_MODE to off restores the previous ranking exactly. It is an admin setting in the config API; see API Reference.
The embedding model has a ~512 token limit. Anything longer gets truncated before embedding — meaning the tail of a long note is invisible to search. A single vector for a long multi-topic note also produces a diluted embedding that matches everything vaguely and nothing precisely.
Chunking solves both problems by splitting long content into overlapping segments before embedding. Each segment gets its own vector.
How it works:
- Notes under 1,600 characters are stored as a single vector — no change in behavior for short captures
- Longer notes are split at sentence or newline boundaries with 200-character overlap between chunks
- Each chunk gets a Vectorize vector pointing back to the parent entry ID
-
recallfetches 3x more results than requested, then deduplicates by parent ID — you always get the best-matching chunk per entry, never duplicates from the same note -
forgetcleans up all chunks for an entry, including update chunks
Before storing, a representative sample of the entry is embedded and checked against existing vectors (the whole note if it's 1,500 characters or fewer, otherwise 500 characters each from the start, middle, and end), and an LLM call checks whether the new note contradicts or overlaps with an existing one. Several outcomes are possible:
| Outcome | What happens |
|---|---|
| Blocked (similarity ≥ 95%) | Near-exact duplicate — nothing stored, existing entry's ID returned |
| Merged | The new note overlaps with an existing entry; the LLM combines them into one entry and re-embeds it — no new row is created |
| Replaced | The new note supersedes an existing one outright; the existing entry's content is overwritten and re-embedded |
| Contradiction resolved | The LLM detects the new note conflicts with an older one. The outdated entry is deprecated — its vectors are deleted so it drops out of recall, but the row is kept for audit — and the new note is stored tagged contradiction-resolved. If the older entry is canonical, it's protected instead: the canonical entry stays and the new note is demoted to a draft. |
| Flagged (similarity 85–95%) | Stored as a new entry tagged duplicate-candidate, with the matching entry's ID and score returned |
| Unique (similarity < 85%) | Stored normally |
A few safeguards: high-importance memories (importance_score >= 4) are protected from being silently merged or replaced — they're flagged instead so nothing critical gets overwritten without your awareness. Memories you've marked canonical get the same protection in a contradiction: the canonical version always wins, and the conflicting newcomer is kept as a draft rather than overwriting it. Transcript-sourced entries get the same protection against conflicting content from a different source: a session log saying "we decided X… actually Y" is not treated as evidence that an existing memory of X is wrong, so the transcript's version becomes a draft instead of overwriting it.
This prevents the brain from filling up with repeated or conflicting context — the same article saved twice from the bookmarklet, Claude storing similar notes across sessions, or "the meeting is Tuesday" sitting around after you've since said "actually it moved to Wednesday."
The check adds ~300ms to each capture since it requires an embed call (and sometimes an LLM call) before inserting.
Contradictions aren't just resolved once and forgotten — their outcomes feed back into how memories rank in future searches. The idea is loosely inspired by the "surprise" signal in Google's Titans research: a memory that keeps surviving challenges is more trustworthy than one that keeps getting overturned.
Every resolved contradiction records two things — a win for the memory that survived, and a loss for the one that didn't (stored in contradiction_wins / contradiction_losses columns on each entry). Each contradiction credits exactly one win and one loss.
At recall time, this history nudges a memory's effective importance:
- A memory that has survived contradictions — for example a canonical fact that repeatedly beat conflicting newcomers — gets a small ranking boost. It's battle-tested.
- A memory that has lost contradictions — for example a draft that conflicted with a canonical entry — gets a small penalty.
- The effect has diminishing returns (the fifth contradiction moves the needle far less than the first) and is bounded, so contradiction history nudges ranking without ever overpowering semantic relevance or recency.
This is derived at search time from the win/loss counts — the memory's original importance score is never overwritten, and entries that have never been in a contradiction rank exactly as they did before. The two columns are added automatically on your next deploy; no manual migration is needed.
When information changes, append adds a timestamped update to an existing entry rather than creating a near-duplicate. The original content is preserved in D1 — the full entry grows over time. The addition gets its own Vectorize vector (ID: {parentId}-update-{timestamp}) so both the original and the update are independently searchable.
forget deletes everything — original chunks and all update chunks.
High-volume tags accumulate dozens of small, repetitive notes that crowd out more useful memories in recall. A nightly cron job (limited to the 4 highest-volume tags per run) and the on-demand /digest endpoint compress a tag's compressible entries into a single synthesized digest, but only once at least 10 eligible entries have accumulated for that tag in a workspace: the LLM writes one cohesive paragraph describing the current state of that area, stored as a new synthesized entry, and the source entries are tagged rolled-up (which also drops their recall weight to 0.4×).
Not every memory is compressible. An entry is protected — kept whole and full-weight — when any of these hold:
-
Important —
importance_score >= 4. - Proven useful by recall — recalled 2+ times, or recalled once and still under 60 days old.
-
Battle-tested — it has survived a contradiction (
contradiction_wins > 0, see Contradiction-aware scoring).
This is an adaptive forgetting gate, inspired by Google's Titans research: how often a memory is actually used is a stronger signal of its value than how many entries happen to share its tag. Compression only ever folds away the genuinely low-signal notes — anything you keep returning to, or that has proven correct under conflict, stays.
Project membership is stored as an ordinary reserved tag, project:<slug>,
on each entry, plus a small registry row holding the project's name,
description, and alias list. Filtering by project reuses the exact same tag
matching every other filter already uses, expanded to an OR-group covering
the project tag and its aliases — so list, recall, digest, and graph
all understand projects without any re-indexing, new Vectorize metadata, or
migration of existing rows. See Projects for the full model.
POST /capture → resolve identity + workspace → duplicate/contradiction check → D1 insert → embed + chunk
POST /append → D1 update (preserve + timestamp) → re-embed addition chunk
POST /update → D1 overwrite → re-embed → delete old vectors
POST /share → move canonical row + edges → audit → restamp vector workspace
GET /list → identity-scoped D1 query (filterable by tag/time/workspace/author)
GET /recall → identity-scoped Vectorize search → time-decay rerank → cross-encoder rerank (auto) → MMR diversity rerank → optional graph-hop expansion → D1 hydrate → LLM insight
POST /forget → D1 delete → Vectorize cleanup
GET+POST /mcp → MCP server → same shared logic as the REST routes above
GET / → web dashboard (static HTML)
Every authenticated identity resolves to one personal workspace and the company workspaces it belongs to. That readable set is applied at D1 queries, Vectorize metadata filters, graph traversal, duplicate detection, insights, imports, integrations, and MCP tools. An admin role changes what a person may manage; it does not add another member's personal workspace to that set.
All application storage is in your own Cloudflare account. The project maintainers cannot read or log your memories. Connected integrations contact only the external services you configure.