Surface silently-dropped writes in logs and metrics - #32
Merged
christianparpart merged 2 commits intoJul 13, 2026
Merged
Conversation
added 2 commits
July 13, 2026 20:37
When the protocol frame reader aborts a command before it is parsed — an oversized payload (PayloadTooLarge), a malformed frame, an over-long line, or a peer that hung up mid-frame (Truncated) — the connection was dropped with no trace of what happened: storage is never called, so no `storage:` line is emitted and a discarded write (e.g. an oversized cache SET) silently vanishes. This is the "reads log HIT/MISS but a dropped SET is invisible" confusion. - Add SessionContext::LogFrameDrop with a data-driven severity map: an actively-rejected frame logs at Warn (visible at the default level), a benign mid-frame close (Truncated) at Debug. - Emit it from every frame-drop site in the RESP, memcached-text, and memcached-binary handlers; the line names the protocol, error code, and context (which for an oversized payload carries the byte count and the cap). Signed-off-by: Christian Parpart <c.parpart@lastrada.net>
A value write that cannot be persisted — a full disk / I/O error, an exhausted memory budget, on-disk corruption, or read-only storage — was invisible at the default log level: the failure propagates as a StorageError to TracingStorage, but that decorator logs only at Trace and is only in the stack when the level is already Trace, so a disk-full SET silently did not happen and never surfaced. - Add WriteErrorReportingStorage, an always-present IStorage decorator that, on a persistence-class write failure, logs one Warn line (carrying the OS errno, e.g. ENOSPC) and counts it; benign conditional-write outcomes and client-side ValueTooLarge are excluded so the disk/persistence signal stays clean. - Surface the count as StorageStats::writeErrors and expose it as the Prometheus counter fastcached_write_errors_total. Signed-off-by: Christian Parpart <c.parpart@lastrada.net>
christianparpart
force-pushed
the
feature/surface-storage-write-and-frame-drop-failures
branch
from
July 13, 2026 18:39
eeac2ce to
6f18f1e
Compare
christianparpart
enabled auto-merge
July 13, 2026 18:41
christianparpart
deleted the
feature/surface-storage-write-and-frame-drop-failures
branch
July 13, 2026 18:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During an sccache-backed build, writes that never reached storage were invisible on the server: the storage trace only records operations that actually hit storage, and it is emitted at Trace and only wired into the stack when the level is already Trace. So an operator watching the logs saw reads (
HIT/MISS) but a droppedSETsimply vanished — no line, no counter. This PR makes the two classes of silent write loss observable at the default log level.Changes
Frame-reader drops. When the frame reader aborts a command before it is parsed — an oversized payload (
PayloadTooLarge), a malformed frame, an over-long line, or a peer that hung up mid-frame (Truncated) — the connection was dropped with nothing logged. AddedSessionContext::LogFrameDropwith a data-driven severity map (an actively-rejected frame logs atWarn, a benign mid-frame close atDebug) and emit it from every drop site in the RESP, memcached-text, and memcached-binary handlers. The line names the protocol, the error code, and its context (which for an oversized payload carries the attempted byte count and the cap).Storage value-write failures. A write that cannot be persisted — a full disk / I/O error, an exhausted memory budget, on-disk corruption, or read-only storage — now surfaces regardless of log level. Added
WriteErrorReportingStorage, an always-presentIStoragedecorator that logs oneWarnline per persistence-class failure (carrying the OS errno, e.g.ENOSPC) and counts it; benign conditional-write outcomes and client-sideValueTooLargeare excluded so the disk/persistence signal stays clean. The count is surfaced asStorageStats::writeErrorsand exposed as the Prometheus counterfastcached_write_errors_total.