* fix(csc): apply invalidation batches in one shard pass
ClientSideCacheInvalidationBatchWindow applied only a fraction of the
invalidations it received under load. apply() looped per key calling
deleteByRedisKeyCollectingHot, and that walks all 16 shards taking each
write lock, so one worker doing 16*N contended acquisitions fell
permanently behind the arriving stream. The rest stayed in the spill
buffer, and the cache went on serving values the server had invalidated
while reporting a ~100% hit rate.
Measured on a 20k/sec invalidation stream: 7,729 of 68,846 enqueued
invalidations applied (15.5%), with nothing dropped by policy
(discarded=0, epochSkip=0).
Invert the loops: take each shard's lock once per batch and handle every
key of the batch under it, so a batch costs 16 acquisitions instead of
16 per key. Per-key semantics (fetch-order guard, recency horizon,
target collection) are unchanged; only the locking granularity differs.
Invalidations applied goes to 100%.
* docs(csc): note that cached reads are not monotonic
The invalidation batch window already documents staleness -- a reader can
see the pre-invalidation value for up to the window. It does not document
the ordering consequence: a caller reading through a non-cached path can
observe a newer value than a concurrent in-flight fetch is about to
publish, and a later cached read then returns the older one.
This is inherent to asynchronous invalidation, not specific to the window.
At publish time the client cannot know its in-flight value is already
stale, because the only signal is a push that has not arrived yet; a guard
that refuses to publish values superseded by an observed invalidation was
tried and never fires. The window widens the exposure, because a published
value stays readable until its invalidation is APPLIED rather than
observed.
Observed roughly once per 44M reads with a 5ms window, and not at all with
the window off.
* fix(csc): keep the read bit on batched refreshes
collectHotAndDeleteBatch built the refresh target without the entry's
second-chance bit, which collectHotAndDelete copies for
stageRefreshAccess. With the invalidation batch window and refresh on,
a refreshed hot key was republished with its bit clear and was the next
eviction victim in a warm shard. Copy it, as the single-key path does.
