Skip to content

Surface silently-dropped writes in logs and metrics - #32

Merged
christianparpart merged 2 commits into
masterfrom
feature/surface-storage-write-and-frame-drop-failures
Jul 13, 2026
Merged

Surface silently-dropped writes in logs and metrics#32
christianparpart merged 2 commits into
masterfrom
feature/surface-storage-write-and-frame-drop-failures

Conversation

@christianparpart

Copy link
Copy Markdown
Member

During an sccache-backed build, writes that never reached storage were invisible on the server: the storage trace only records operations that actually hit storage, and it is emitted at Trace and only wired into the stack when the level is already Trace. So an operator watching the logs saw reads (HIT/MISS) but a dropped SET simply vanished — no line, no counter. This PR makes the two classes of silent write loss observable at the default log level.

Changes

  • Frame-reader drops. When the frame reader aborts a command before it is parsed — an oversized payload (PayloadTooLarge), a malformed frame, an over-long line, or a peer that hung up mid-frame (Truncated) — the connection was dropped with nothing logged. Added SessionContext::LogFrameDrop with a data-driven severity map (an actively-rejected frame logs at Warn, a benign mid-frame close at Debug) and emit it from every drop site in the RESP, memcached-text, and memcached-binary handlers. The line names the protocol, the error code, and its context (which for an oversized payload carries the attempted byte count and the cap).

  • Storage value-write failures. A write that cannot be persisted — a full disk / I/O error, an exhausted memory budget, on-disk corruption, or read-only storage — now surfaces regardless of log level. Added WriteErrorReportingStorage, an always-present IStorage decorator that logs one Warn line per persistence-class failure (carrying the OS errno, e.g. ENOSPC) and counts it; benign conditional-write outcomes and client-side ValueTooLarge are excluded so the disk/persistence signal stays clean. The count is surfaced as StorageStats::writeErrors and exposed as the Prometheus counter fastcached_write_errors_total.

Christian Parpart added 2 commits July 13, 2026 20:37
When the protocol frame reader aborts a command before it is parsed — an
oversized payload (PayloadTooLarge), a malformed frame, an over-long line, or a
peer that hung up mid-frame (Truncated) — the connection was dropped with no
trace of what happened: storage is never called, so no `storage:` line is
emitted and a discarded write (e.g. an oversized cache SET) silently vanishes.
This is the "reads log HIT/MISS but a dropped SET is invisible" confusion.

- Add SessionContext::LogFrameDrop with a data-driven severity map: an
  actively-rejected frame logs at Warn (visible at the default level), a benign
  mid-frame close (Truncated) at Debug.
- Emit it from every frame-drop site in the RESP, memcached-text, and
  memcached-binary handlers; the line names the protocol, error code, and
  context (which for an oversized payload carries the byte count and the cap).

Signed-off-by: Christian Parpart <c.parpart@lastrada.net>
A value write that cannot be persisted — a full disk / I/O error, an exhausted
memory budget, on-disk corruption, or read-only storage — was invisible at the
default log level: the failure propagates as a StorageError to TracingStorage,
but that decorator logs only at Trace and is only in the stack when the level is
already Trace, so a disk-full SET silently did not happen and never surfaced.

- Add WriteErrorReportingStorage, an always-present IStorage decorator that, on
  a persistence-class write failure, logs one Warn line (carrying the OS errno,
  e.g. ENOSPC) and counts it; benign conditional-write outcomes and client-side
  ValueTooLarge are excluded so the disk/persistence signal stays clean.
- Surface the count as StorageStats::writeErrors and expose it as the Prometheus
  counter fastcached_write_errors_total.

Signed-off-by: Christian Parpart <c.parpart@lastrada.net>
@christianparpart
christianparpart force-pushed the feature/surface-storage-write-and-frame-drop-failures branch from eeac2ce to 6f18f1e Compare July 13, 2026 18:39
@christianparpart
christianparpart merged commit a8ac6a0 into master Jul 13, 2026
11 checks passed
@christianparpart
christianparpart deleted the feature/surface-storage-write-and-frame-drop-failures branch July 13, 2026 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant