perf(cache)!: share concurrent inner reads and serve an outage from the local tier

CHANGELOG.md

@@ -46,6 +46,38 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ### Changed
 
+- **A Redis outage is served from the local tier by default.** `CacheOptions.ConnectionMonitorEnabled` and
+  `UseLocalOnlyWhenDisconnected` now default to `true`: while the inner tier is disconnected, reads are served from
+  the local tier and writes kept there, capped at `LocalMaxExpirationDisconnected` (30 seconds by default). The new
+  `ClearLocalOnReconnect`, also `true` by default, expires the local entries of a Redis Pub/Sub topic when its
+  connection is restored, since the invalidations sent while it was down never arrived. The topic expires them through
+  their change tokens, so it works on any local tier, a custom `IMemoryCacheFactory` one included. A forced reconnect counts too: swapping the connection
+  can drop publications although it never read as down. A Pub/Sub topic also subscribes only after a delay, at first
+  and after a reconnect, so once its subscription is in place it expires the entries cached before then; a cache with
+  `ClearLocalOnReconnect` off keeps them. Redis Streams replay what was sent meanwhile, so their local tier is kept.
+  The monitor setting is app-wide, so the broadcast providers' connection monitor turns on too. **Breaking** for an app that relied on a disconnected tier answering
+  misses: set `UseLocalOnlyWhenDisconnected` to `false`, or `ConnectionMonitorEnabled` to `false` for the previous
+  behavior. **Breaking** too for a custom `IMultilayerCacheOptions` implementation, which must add
+  `ClearLocalOnReconnect`.
+
+- **`RedisConnector.IsConnected` is `false` only when a connection exists and reports itself down.** Before the
+  first connect, while it is pending and after it faulted, it is now `true`, so a command goes through and opens or
+  retries the connection; it used to be `false`, and the operations that check it first, L2 reads and every
+  `ISetCache` call, did nothing until a write had connected. Observable to code that reads `IRedisConnector.IsConnected`
+  directly, such as a status endpoint: use `RedisHealthCheck` to learn whether the server is reachable.
+
+- **Concurrent single-key reads share one inner read.** On a local miss, concurrent single-key `GetAsync`,
+  `GetCacheEntryAsync` and hash reads of one key, and the read `GetOrAddAsync` makes before its lock, wait for a
+  single inner-tier read and share its result or failure. Reads share only when they read the same value type with
+  the same `LocalExpiration` and `LocalExpirationDisconnected`, since the shared read keeps its hit for those
+  lifetimes; reads that differ in any of them read separately. Multi-key reads still read every missing key
+  themselves. A caller that cancels stops waiting; the shared read is cancelled only once every caller waiting on it
+  has cancelled.
+
+- **`GetOrAddAsync` keeps a value the inner tier refused.** When the inner write fails, the generated value is kept
+  in the local tier for `LocalMaxExpirationDisconnected`, so the callers waiting on the local lock reuse it instead of
+  each running the generator again.
+
 - **A string key reads the local tier by its text.** On .NET 9 and later, `GetAsync`, `GetItemAsync`,
   `ContainsAsync`, `GetCacheEntryAsync` and `GetOrAddAsync` with a `CacheKey` look the in-memory tier up as the span
   reads do, and `Cache<T>` and `HashCache<T>` compose their strategy's key on the stack instead of through
@@ -125,6 +157,35 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ### Fixed
 
+- **Events dropped at a full dispatcher channel expire what the topic keeps.** A Redis Pub/Sub topic whose channel
+  was full in `Wait` mode discarded the event, and the drop modes (`DropOldest`, `DropNewest`, `DropWrite`) discarded
+  one silently on either transport; the invalidation it carried was lost. Each drop now expires the topic's local
+  entries, coalesced to one expiry at a time. A Redis Streams reader that has to wait for the dispatcher has fallen
+  behind the stream, so its next read checks for trimmed entries.
+
+- **A refresh is broadcast as `CacheRefreshed`.** `CacheEventPublisher.CacheRefreshedAsync`, which `RefreshAsync`
+  raises, sent the `CacheRemoved` type. The library's own receivers treat both alike, so only a subscriber that
+  filters by event type sees the difference.
+
+- **A Redis Streams topic that loses entries before this node reads them expires what it kept.** After a reconnect
+  the consumer checks whether entries past its last delivery were trimmed (`MaxLength`, or the maintainer's trim),
+  and a consumer group or stream removed while in use counts as a loss too. The check and the read that follows run
+  in one script, so a trim cannot land between them; where scripts are refused, they run apart. The local entries on that topic then
+  expire, in every cache that uses it, whatever `ClearLocalOnReconnect` says. They used to be kept, so a change
+  whose invalidation was trimmed was served stale until the entry expired. Below Redis 7.0, which does not count
+  the entries a group read, any trim past this node's last delivery counts as a loss, even one that removed only
+  entries this node had read. The topic signals the loss through the new
+  `IEventSubject<T>.Invalidate(MissedEventsReason)`: `Lost` for a Streams loss, which every cache honors, and
+  `SubscriptionGap` for the Pub/Sub delay above. **Breaking** for a custom subject passed to the `RedisPubSubTopic` or
+  `RedisStreamsTopic` constructor: implement `Invalidate` to expire what its observers keep. The library's change
+  tokens implement the new public `IMissedEventsObserver`, so such a subject calls `OnEventsMissed(reason)` on each
+  observer that implements it.
+
+- **Multi-key reads keep a usable local copy.** `GetAsync` and `GetCacheEntriesAsync` over several keys stored each
+  inner-tier hit locally as a key-value pair rather than as the value, so no later read found it and every one went
+  back to the inner tier. A hit is now kept as a single-key read keeps it, for `LocalMaxExpirationDisconnected` while
+  the inner tier is disconnected.
+
 - **The key prefix no longer depends on `RedisCacheOptions.KeyPrefix` being set.** A connector whose `IDatabase` is
   wrapped with `WithKeyPrefix` stores every key under that prefix, and two places need it: the stream maintainer, to
   scan for and strip it, and the cluster slot check, to hash the key the server receives. Both relied on `KeyPrefix`

docs/concepts.md

@@ -138,10 +138,35 @@ expire relatively quickly so that as soon as connectivity is restored the proces
 re-populates from a fresh L2 rather than serving stale data for the full
 `LocalMaxExpiration` window.
 
-`UseLocalOnlyWhenDisconnected` changes the disconnected behavior further: when
-true, an unhealthy L2 means reads are served from L1 only and no L2 round-trip
-is attempted. This is appropriate when you would rather serve potentially stale
-data than accumulate timeout latency on every cache miss during a Redis outage.
+`UseLocalOnlyWhenDisconnected`, on by default, changes the disconnected behavior
+further: while L2 is unhealthy, an L1 hit is served and writes are kept in L1
+only, capped at `LocalMaxExpirationDisconnected`. That serves potentially stale
+data rather than losing the local copy during a Redis outage. Set it to `false` to
+drop the local entry and answer a miss instead. An L1 miss still asks L2: the
+monitor also tracks the broadcast connection, so "unhealthy" can mean only that one
+is down, and with the fail-fast backlog policy a read on a dead data connection
+fails at once rather than waiting for its timeout.
+
+`ClearLocalOnReconnect`, also on by default, expires a `RedisPubSub` topic's
+L1 entries when its connection is restored, through their change tokens, so on
+any local tier. Invalidations published while the connection was down never
+reached this node, so an entry cached before the outage could otherwise outlive a
+change made during it. `RedisStreams` needs no clear: the consumer group resumes
+after its last delivery and replays what the outage held back. When entries past
+it were trimmed first, or the group or stream was removed while in use, the topic
+expires every local entry it invalidates, whatever `ClearLocalOnReconnect` says.
+Below Redis 7.0, which does not count the entries a group read, any trim past this
+node's last delivery counts as a loss, even one that removed only entries it had
+read. Concurrent single-key reads of one key share a single L2
+read when they read the same value type with the same `LocalExpiration` and
+`LocalExpirationDisconnected`, so the refill after a clear costs one read per key
+per node for each such combination; reads that differ in any of them, and
+multi-key reads, read L2 themselves.
+
+A `GetOrAddAsync` miss runs its generator under the local lock, and the callers
+waiting on the lock read the stored value once it is written. When L2 refuses the
+write, the generated value is kept in L1 for `LocalMaxExpirationDisconnected`, so
+those waiters reuse it instead of each running the generator again.
 
 Why split into two tiers at all? L1 eliminates Redis latency and bandwidth on
 the hot path — for a busy service, removing the network hop from cache reads can
@@ -152,11 +177,11 @@ tradeoff is that each node's L1 can drift from L2 when another node writes a new
 value. Topics solve that problem, and their relationship to layers is explained
 in the next section.
 
-It is worth noting that all three of these disconnected-scenario options
-(`LocalMaxExpiration`, `LocalMaxExpirationDisconnected`,
-`UseLocalOnlyWhenDisconnected`) require the connection monitor to be active —
-either by setting `ConnectionMonitorEnabled` to `true` on `CacheOptions`, or by
-setting it on `InMemoryRedisCacheOptions` directly. Without the monitor the cache
+It is worth noting that these disconnected-scenario options
+(`LocalMaxExpirationDisconnected`, `UseLocalOnlyWhenDisconnected`,
+`ClearLocalOnReconnect`) require the connection monitor to be active;
+`LocalMaxExpiration` caps every L1 entry and does not. It is on by default (`CacheOptions.ConnectionMonitorEnabled`),
+and `InMemoryRedisCacheOptions.ConnectionMonitorEnabled` overrides it per provider. Without the monitor the cache
 cannot distinguish "connected" from "disconnected" and the disconnected behavior
 never triggers.
 

docs/recipes/second-redis-connection.md

@@ -80,7 +80,7 @@ services.AddNamedCaching("SecondaryRedis", configuration, chain,
 
 **A separate section.** When the second server needs different *cache* options too, pass `sectionName`: `AddNamedCaching("cold", configuration, chain, sectionName: "CachingCold")` binds everything from `CachingCold`, with the connection at `CachingCold:Connections:cold`. The two sections must agree on `KeyCasing`, because `AddCaching` seeds the process-wide `CacheKey.DefaultCasing`; a stack that differs is refused, naming both values, whether the casing came from its section or from a `Configure<CacheOptions>` in `configureServices`.
 
-**Warm-up.** Set `WarmUpOnStart` on the primary connection (the stack inherits it). A connection is otherwise opened by the first *write* through `ICache` or `IHashCache`; the operations that check `IsConnected` first (every `ISetCache` call, L2 reads) do nothing while it is closed and never open it themselves. With warm-up, the stack's hosted services, started with the host, open the second connection at startup, and both servers log a handshake as the app starts.
+**Warm-up.** Set `WarmUpOnStart` on the primary connection (the stack inherits it). A connection is otherwise opened by the first command through `ICache`, `IHashCache` or `ISetCache`, read or write: a connector reports itself connected until it has a multiplexer that says otherwise, so the operations that check `IsConnected` first go through and open it. With warm-up, the stack's hosted services, started with the host, open the second connection at startup, and both servers log a handshake as the app starts.
 
 **Names.** The name is the DI key and the connection section, so it is case-sensitive as a key. `Redis` is refused, since it names the primary connection, and so is a second `AddNamedCaching` with the same name.
 

docs/reference/interfaces.md

@@ -868,11 +868,11 @@ public interface IConnectionState
 }
 ```
 
-A non-blocking snapshot of whether the backing store is reachable, plus the transitions as events. `IsConnected` never blocks and never throws. Implemented by `RedisCacheBase`, so `Redis`-provider caches expose it; the multilayer caches and `NullCache` do not, and neither does `Cache<T>`, which holds its underlying `ICache` privately.
+A non-blocking snapshot of whether commands should go to the backing store, plus the transitions as events. `IsConnected` never blocks and never throws. For `RedisConnector` it is `false` only once a connection exists and reports itself down: before the first connect, while it is pending, and after it faulted, it is `true`, so a command goes through and opens or retries the connection. Implemented by `RedisCacheBase`, so `Redis`-provider caches expose it; the multilayer caches and `NullCache` do not, and neither does `Cache<T>`, which holds its underlying `ICache` privately.
 
 What it is *not*: a way to explain a negative result. It is a cached snapshot refreshed on connection events and a timer, it says nothing about whether any particular command succeeded, and it is `true` both where there is nothing to disconnect from and where `ConnectionMonitorEnabled` is off. A `false` from `SetAsync` or [`TryAddAsync`](#icache) can perfectly well coincide with `IsConnected == true` — a serialization failure or a rejected command does that — so reading it afterwards does not recover why the call failed.
 
-**Use this when:** you are reporting or reacting to cache *health* — a readiness probe, a metric, a log line, or backing off writes while a tier is known down. Subscribe to `OnConnectionFailed` / `OnConnectionRestored` for the transitions rather than polling.
+**Use this when:** you are reacting to a tier known to be down — a metric, a log line, or backing off writes. Subscribe to `OnConnectionFailed` / `OnConnectionRestored` for the transitions rather than polling. For a readiness probe, use `RedisHealthCheck`, which asks the server: `IsConnected` is `true` while a first connection is still failing.
 
 ## Telemetry seam