Observations outlive their deleted source facts, and bank export drops them silently #5937

Summary

Observations can outlive the facts they were consolidated from. When that happens there are two separate problems:

1. The observations should not exist. They stay live in the bank: their text summarises facts that were deleted, and recall/reflect still serve them.
2. Bank export drops them silently. An exported-and-reimported bank has fewer observations than the source, and nothing tells the caller.

Why export excludes them

It is not because their source list is empty. In the banks we examined, every affected observation still has a non-empty source_memory_ids; what they have in common is that at least one of those ids no longer exists in memory_units (and is not in invalidated_memory_units either — the facts were hard-deleted, not invalidated).

Export resolves each source id to a (document_id, fact_index) reference so the observation can be rebuilt on import. An id that resolves to nothing has no reference, so the whole observation is skipped (engine/memories/pg/transfer.py, and the same rule in engine/transfer/export.py for store-owned paths):

source_ids = list(row["source_memory_ids"] or [])
locations = [unit_index.get(sid) for sid in source_ids]
if not source_ids or any(loc is None for loc in locations):
    # An observation with sources outside the exported documents would be
    # incoherent on import — skip it rather than emit dangling refs.
    skipped += 1
    continue

The only trace is an INFO log line (Skipped N observation(s) with sources outside the exported documents); the export result does not report it. The rule itself is reasonable given (1): these are exactly the observations that should already have been deleted.

Problem 1: observations left behind when their sources are deleted

The intended invariant: when a fact is deleted, every observation citing it is deleted too, and the surviving co-sources are re-queued for consolidation (delete_stale_observations in engine/memories/pg/writes.py: "its text is stale once even one source memory disappears"). Two earlier migrations (g7h8i9j0k1l2_backsweep_orphan_observations, c4x5y6z7a8b9_backsweep_orphan_observations_v2) swept up orphans from a path that skipped this cleanup (document re-ingest via handle_document_tracking).

We are still finding orphans created well after that fix, in two shapes:

• all sources gone — would have been caught by the backsweep migrations, had they run after;
• some sources gone — never caught by the backsweeps, which only delete when no source survives. These look healthy (they have live sources) but their text was written partly from facts that no longer exist, and consolidation never revisits them because nothing about the surviving sources changed.

In the affected banks the orphans were created over a span of several months and none recently, so the leaking path may already be closed, but we have not identified which delete path skipped the sweep. Recall does not check whether an observation's sources exist, so content from deleted documents can keep surfacing through these observations.

Detect them in a bank:

SELECT o.id,
       cardinality(o.source_memory_ids) AS n_sources,
       (SELECT count(*) FROM memory_units s
         WHERE s.id = ANY(o.source_memory_ids) AND s.bank_id = o.bank_id) AS n_live
FROM memory_units o
WHERE o.bank_id = $1 AND o.fact_type = 'observation'
  AND EXISTS (SELECT 1 FROM unnest(o.source_memory_ids) src
              WHERE NOT EXISTS (SELECT 1 FROM memory_units s WHERE s.id = src));

Problem 2: export skips silently

Even with (1) fixed, export should not drop data without saying so — a user auditing an export against the API finds observations missing and has no way to know whether they were dropped on purpose.

Proposed

1. Find the delete path(s) that remove memory_units without calling the stale-observation sweep, and route them through it.
2. A backsweep that also handles the some sources gone case, using the same rule as the live sweep (delete the observation, re-queue surviving sources), with a bound and time budget for deployments with many schemas.
3. Report skipped observations in the export result (count, and ids), not just a log line.