From 1da6b777ec6a977d3112df41c71d31cd8933b3d9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Francisco=20Miguel=20P=C3=A9rez=20Canales?= Date: Sun, 8 Nov 2026 13:31:07 +0100 Subject: [PATCH] Docs: la profundidad de la cola de muertos no es la medida; la completitud si La cola de muertos tiene 18.816 mensajes, y mirarla no dice si se perdio trabajo. Esta seccion dice cual es la medida que si lo dice, con la de hoy hecha. De los primeros 500 inspeccionados, 496 son compute_embeddings_batch venidos de protea.embeddings.batch con edad de una semana, y 4 son job_dispatch de hace un mes. Ninguno de esos numeros dice nada sobre si falta un vector. Lo que lo dice es la completitud por configuracion, que son cuatro cuentas por indice: tres de las cuatro configuraciones con filas estan completas a 684.048, que es la poblacion entera de secuencias, y la cuarta le falta exactamente las 161 que pasan de 8.259 residuos y tienen su propio arreglo. Asi que esos 18.816 son copias de lotes que SI entraron, con toda probabilidad del relanzamiento con skip_existing despues de las tres pasadas de finales de octubre que fallaron o se cancelaron con el contador casi a cero. La conclusion va escrita del lado que importa, que es el contrario: un error de base manda el lote a muertos mientras su job sigue RUNNING sin error, y el contador del padre puede avanzar con una reentrega, asi que un job puede reportar bien sobre trabajo que nunca aterrizo. La profundidad de la cola se ve igual en los dos casos y solo la cuenta de filas contra la poblacion los distingue. Con los dos limites del resumen dichos, para que nadie lo lea de mas: la edad es un bucket, asi que 1w no separa el 11-01 del 11-05, y el endpoint inspecciona una muestra. Ninguno de los dos cambia la conclusion. La idea de medirlo por completitud en vez de por la cola es de la sesion del sobremesa. --- docs/source/runbooks/dlq-triage.rst | 58 +++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/docs/source/runbooks/dlq-triage.rst b/docs/source/runbooks/dlq-triage.rst index 563cd41b..cdbe89f1 100644 --- a/docs/source/runbooks/dlq-triage.rst +++ b/docs/source/runbooks/dlq-triage.rst @@ -16,6 +16,64 @@ argument. Messages land in the DLQ when: The DLQ never drains automatically. It accumulates until an operator inspects and resolves the root cause. +The depth is not the measurement +-------------------------------- + +**A large DLQ is not evidence that work was lost, and the depth cannot tell you +whether it was.** Measured on 2026-11-08, when the queue held **18,816** +messages: of the first 500 peeked through ``GET /v1/admin/dlq/summary``, 496 +were ``compute_embeddings_batch`` dead-lettered from ``protea.embeddings.batch`` +in the ``1w`` age bucket, and 4 were ``job_dispatch`` from ``protea.jobs`` a +month old. Nothing in those numbers says whether a vector is missing. + +What settles it is **completeness per configuration**, which costs one indexed +count each: + +.. code-block:: sql + + SELECT left(c.id::text, 8), coalesce(c.display_name, c.model_name), + count(e.sequence_id) AS rows + FROM embedding_config c + LEFT JOIN sequence_embedding e ON e.embedding_config_id = c.id + GROUP BY c.id, c.display_name, c.model_name + HAVING count(e.sequence_id) > 0 + ORDER BY count(e.sequence_id) DESC; + +On that date it answered: + +.. code-block:: text + + d4b4c057 learned-code:mean:sfjepa-world-holes-topk:cut227:e7ba37da 684.048 + 34446081 learned-code:mean:sfjepa-world-holes-topk:cut211:e7ba37da 684.048 + e7ba37da ankh_base 684.048 + f3a7fe16 Ankh-base, full length, mean plus max 683.887 + +684,048 is not a round number taken from somewhere: it is the whole sequence +population, 629,895 sequences carrying one accession plus 54,153 shared ones. +**Three of the four configurations are complete**, and the fourth is short by +exactly the 161 sequences over 8,259 residues that the eager Ankh route cannot +compute on the node's card, which has its own fix. + +So those 18,816 messages are copies of batches that DID get in, most plausibly +from a relaunch with ``skip_existing`` after the three ``compute_embeddings`` +runs of 2026-10-28 and 2026-10-29 that failed or were cancelled with their +counters near zero. They occupy a queue and nothing else. + +Read the conclusion the right way round. The reason to check completeness is +that the opposite answer is the dangerous one: a database error dead-letters a +batch while its job stays RUNNING with no error, and the parent's counter can +advance on a redelivered write, so **a job can report success over work that +never landed**. The queue depth looks the same in both cases. Only the row count +against the population distinguishes them. + +Two limits of the summary, so the next reader does not over-read it: the age is +a BUCKET, so ``1w`` does not separate 2026-11-01 from 2026-11-05, and the +endpoint peeks a sample rather than the whole queue. Neither changes the +conclusion, and neither is worth inspecting 18,816 messages to refine. + +Purging is a separate decision and belongs to the researcher: +``POST /v1/admin/dlq/purge``, never as a cleanup step. + Symptoms --------