Problem
InMemoryCacheLayer (L1) records one time.time() float per cache hit into
self.access_patterns[key], a defaultdict(list). Nothing ever trims that list.
self.access_patterns = defaultdict(list) # unbounded
...
self.access_patterns[key].append(time.time()) # on every hit
The list is only discarded when the key itself leaves the cache. For a key that
stays resident — which is precisely what a hot key does — the history grows once
per read, forever. Its size is a function of total reads served, not of how
much data the cache holds.
This memory is invisible to the cache's own accounting. CacheStats.total_size_bytes
tracks CacheEntry.size_bytes only; the access history is not counted, so neither
the byte budget nor the entry-count budget can ever reclaim it.
Measurement
Probe: 500 resident keys, 2,000 hits each (1,000,000 reads total).
| metric |
observed |
| retained timestamps |
1,000,000 |
| access-history memory |
31,340 KiB (30.6 MiB) |
total_size_bytes reported by the cache |
1.5 KiB |
| resident entries |
500 |
30.6 MiB of retained history against 1.5 KiB of accounted payload. The gap is
unbounded in the read count: doubling traffic doubles the history and leaves
every reported metric unchanged.
Consumer
The only reader is _calculate_adaptive_ttl, which uses just three things:
frequency = len(accesses) / (accesses[-1] - accesses[0])
len, first element, last element. It does not need the full history to compute
a rate — a trailing window is sufficient, and is arguably a better estimator
because it reflects the key's current behaviour rather than a lifetime average.
Proposal
Give the per-key history a fixed upper bound:
ACCESS_HISTORY_WINDOW = 64
self.access_patterns = defaultdict(lambda: deque(maxlen=ACCESS_HISTORY_WINDOW))
deque(maxlen=...) evicts from the left in O(1) on append, so the hot path cost
is unchanged. Retention becomes a function of resident keys, which the cache
already bounds, instead of reads served, which it does not.
Expected on the probe above: 30.6 MiB → 1.35 MiB (22.7x), with resident entry
count, total_size_bytes and the hot-key adaptive TTL all unchanged.
Scope
In scope: bound the retention; tests for the bound, for window ordering, and for
adaptive TTL continuing to work correctly over a saturated window.
Not in scope: the entry-lifecycle bookkeeping leaks in the same structure — those
were fixed separately in #1297 / PR #1298 and are already on main. This issue
is only about the growth of a resident key's history.
Acceptance
- per-key retention never exceeds
ACCESS_HISTORY_WINDOW
- the retained samples are the most recent ones, in order
_calculate_adaptive_ttl returns the same TTL for a hot key before and after- resident entry count and
total_size_bytes are unaffected
Problem
InMemoryCacheLayer(L1) records onetime.time()float per cache hit intoself.access_patterns[key], adefaultdict(list). Nothing ever trims that list.The list is only discarded when the key itself leaves the cache. For a key that
stays resident — which is precisely what a hot key does — the history grows once
per read, forever. Its size is a function of total reads served, not of how
much data the cache holds.
This memory is invisible to the cache's own accounting.
CacheStats.total_size_bytestracks
CacheEntry.size_bytesonly; the access history is not counted, so neitherthe byte budget nor the entry-count budget can ever reclaim it.
Measurement
Probe: 500 resident keys, 2,000 hits each (1,000,000 reads total).
total_size_bytesreported by the cache30.6 MiB of retained history against 1.5 KiB of accounted payload. The gap is
unbounded in the read count: doubling traffic doubles the history and leaves
every reported metric unchanged.
Consumer
The only reader is
_calculate_adaptive_ttl, which uses just three things:len, first element, last element. It does not need the full history to computea rate — a trailing window is sufficient, and is arguably a better estimator
because it reflects the key's current behaviour rather than a lifetime average.
Proposal
Give the per-key history a fixed upper bound:
deque(maxlen=...)evicts from the left in O(1) on append, so the hot path costis unchanged. Retention becomes a function of resident keys, which the cache
already bounds, instead of reads served, which it does not.
Expected on the probe above: 30.6 MiB → 1.35 MiB (22.7x), with resident entry
count,
total_size_bytesand the hot-key adaptive TTL all unchanged.Scope
In scope: bound the retention; tests for the bound, for window ordering, and for
adaptive TTL continuing to work correctly over a saturated window.
Not in scope: the entry-lifecycle bookkeeping leaks in the same structure — those
were fixed separately in #1297 / PR #1298 and are already on
main. This issueis only about the growth of a resident key's history.
Acceptance
ACCESS_HISTORY_WINDOW_calculate_adaptive_ttlreturns the same TTL for a hot key before and aftertotal_size_bytesare unaffected