Skip to content

redis: make RealTime tail anchoring resilient to errors - #48

Merged
tyt2y3 merged 2 commits into
mainfrom
fix/realtime-anchor-error-paths
Aug 12, 2026
Merged

tyt2y3 merged 2 commits into
mainfrom
fix/realtime-anchor-error-paths

Conversation

@tyt2y3

@tyt2y3 tyt2y3 commented Aug 11, 2026

Copy link
Copy Markdown
Member

Follow-up to #47, fixing two problems in the error paths of the per-shard tail anchoring.

Problems

1. Cluster mode broke at startup. Cluster::run assigns all shards to the first node and relies on MOVED errors from the first read to redistribute them (move_shards). The anchoring XREVRANGE runs before the first XREAD and its errors were propagated raw via ?, so a MOVED reply became RedisErr::Moved, which the node loop treats as fatal. A RealTime + Latest (the default) consumer against a Redis cluster died with ConsumerDied whenever its keys did not all live on the first connected node — the common case with several streams. Ask/ClusterDown/MasterDown were similarly fatal here despite being retried on the XREAD path.

2. A transient error at first read silently re-introduced the starvation bug. first_read was cleared before the fallible anchoring loop. If an XREVRANGE failed with an error the node loop retries (Io → reconnect, TryAgain → sleep), the retry skipped the anchoring block entirely, and every still-unanchored shard fell back to sending $ on each XREAD — exactly the quiet-stream message loss #47 fixed, for the consumer's whole lifetime, with nothing surfaced.

Fix

  • Anchoring errors now go through the same four-way classification as the XREAD reply: MOVED → move_shards (cluster reassigns; the destination node re-anchors since AddShard re-arms its first_read), Io → reconnect, cluster errors → retry, anything else → send_error so the consumer sees the actual error instead of a generic ConsumerDied.
  • first_read is cleared only after every shard is anchored. Error paths return with it still true, and a retry resumes where it left off — already-anchored shards are skipped via id.is_some(), so the loop is idempotent.
  • A malformed id from XREVRANGE is also routed through send_error for consistency.

Tests

New unit tests for parse_stream_id (valid/edge/invalid ids), and five new scenarios in realtime-multi.rs on top of the three from #47:

  • quiet_history_not_starved_nor_replayed — a quiet stream with pre-existing history anchors at its own tail: no replay, and the message landing in the starvation window is not lost.
  • absent_streams_all_delivered — streams that don't exist yet anchor at 0-0; the very first entries are delivered.
  • multiple_rounds — the anchor keeps advancing after first delivery across repeated starvation cycles.
  • earliest_gets_everything — Earliest is unaffected by anchoring: full history plus starvation-window messages.
  • wrong_type_key_errors — a stream key occupied by a plain string (WRONGTYPE on XREVRANGE) surfaces the actual backend error to the consumer. Verified this fails on main (generic ConsumerDied) and passes with the fix. Requires the redis crate's blocking client as a dev-dependency for setup.

All pass locally on both runtime-tokio and runtime-smol against Redis 8. The MOVED and Io-retry paths aren't covered by CI — they need a Redis cluster / fault injection; verified by code inspection against the XREAD path, which handles the identical errors.

🤖 Generated with Claude Code

tyt2y3 added 2 commits August 11, 2026 18:18
The tail anchoring introduced in #47 cleared `first_read` before issuing
the XREVRANGE calls and propagated their errors raw, with two consequences:

* A transient error (Io, TryAgain) at the very first read is retried by
  the node loop, but with `first_read` already false the anchoring was
  skipped on retry, silently reverting un-anchored shards to `$` -- the
  very starvation bug #47 fixed.

* In cluster mode, all shards start on the first node and rely on MOVED
  errors from the first read to be redistributed. XREVRANGE now hits the
  MOVED before the XREAD ever runs, and the raw error killed the node, so
  a RealTime + Latest consumer died at startup whenever its keys did not
  all live on the first node.

Anchoring now applies the same error policy as the XREAD reply: MOVED
triggers move_shards, Io reconnects, cluster errors are retried, and
anything fatal is forwarded to the consumer via send_error instead of
a generic ConsumerDied. `first_read` is only cleared once every shard is
anchored; a retry resumes where it left off, as anchored shards are
skipped via `id.is_some()`.
@tyt2y3
tyt2y3 merged commit bb29c82 into main Aug 12, 2026
10 checks passed
@tyt2y3
tyt2y3 deleted the fix/realtime-anchor-error-paths branch August 12, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant