Skip to content

memcache: don't close connections on clean protocol-level SERVER_ERROR responses - #196

Open
mapno wants to merge 2 commits into
bradfitz:masterfrom
mapno:server-error-resumable
Open

memcache: don't close connections on clean protocol-level SERVER_ERROR responses#196
mapno wants to merge 2 commits into
bradfitz:masterfrom
mapno:server-error-resumable

Conversation

@mapno

@mapno mapno commented Jul 28, 2026

Copy link
Copy Markdown

Summary

A SERVER_ERROR <msg>\r\n reply is a complete protocol line: the server keeps the connection open and in sync. It even swallows the data block on storage errors. Closing our side causes needless reconnect churn, e.g. on object too large for cache or OOM bursts in the worst possible time to add TCP handshake load, since the server is already under memory pressure.

This PR contains two changes:

  • efbfa16 parses SERVER_ERROR replies into the exported (but previously never returned) ErrServerError, wrapped with the server's message so it works with errors.Is.
  • 841f5a5 stops closing the connection on those errors.

Note: error text for these responses changes from memcache: unexpected response line... to memcache: server error: <msg>.

Why this is safe: memcached keeps the connection open on these errors

I've verified this change against memcached >1.6.31. For the ASCII protocol (the only one this client speaks), every common SERVER_ERROR is written with out_string(), which transitions to conn_new_cmd. On storage errors the server additionally swallows the in-flight data block, so both sides stay in sync.

memcached 1.6.31 source references
  • SERVER_ERROR object too large for cache / SERVER_ERROR out of memory storing object — replies, then swallows the data block and continues:
    proto_text.c#L2045-L2063
    out_of_memory(c, "SERVER_ERROR out of memory storing object");
    ...
    /* swallow the data line */
    conn_set_state(c, conn_swallow);
    c->sbytes = vlen;
  • SERVER_ERROR Out of memory during read (chunk alloc failure mid-read of a set payload) — same swallow pattern:
    memcached.c#L3240-L3250
  • SERVER_ERROR out of memory writing get response — get requests carry no data block, so the connection is already in sync; the server just replies and moves on:
    proto_text.c#L702-L713
  • out_of_memory() for ASCII clients is just out_string() (memcached.c#L1300-L1313); the conn_closing behavior lives only in the binary-protocol branch, which this client never uses.

Known exception: SERVER_ERROR out of memory reading request (failed read-buffer realloc on a huge multiget) sets close_after_write = true (memcached.c#L2494-L2496). If such a connection is pooled, the next request fails with EOF and the connection is then discarded as non-resumable. Severe OOMs where memcached can't even allocate a response object close the connection without writing any error line, so the client sees a read error, not SERVER_ERROR, and the connection is dropped as before.


This is a port of grafana#36 to upstream.

mapno added 2 commits July 28, 2026 17:10
SERVER_ERROR replies currently surface as generic 'unexpected response
line' errors. Parse them into the exported (previously unreachable)
ErrServerError, wrapped with the server's message, so callers can match
them with errors.Is.

Connection pooling behavior is unchanged: these errors still close the
connection.
…R responses

A SERVER_ERROR <msg>\r\n reply is a complete, well-formed protocol line:
the server keeps the connection open and in sync (on storage errors it
even swallows the data block). Closing our side causes needless
reconnect churn, e.g. on 'object too large for cache' or OOM bursts,
the worst possible time to add TCP handshake load.
@mapno
mapno force-pushed the server-error-resumable branch from 8d0de39 to 841f5a5 Compare July 28, 2026 15:10
@mapno
mapno marked this pull request as ready for review July 29, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant