memcache: don't close connections on clean protocol-level SERVER_ERROR responses - #196
Open
mapno wants to merge 2 commits into
Open
memcache: don't close connections on clean protocol-level SERVER_ERROR responses#196mapno wants to merge 2 commits into
mapno wants to merge 2 commits into
Conversation
SERVER_ERROR replies currently surface as generic 'unexpected response line' errors. Parse them into the exported (previously unreachable) ErrServerError, wrapped with the server's message, so callers can match them with errors.Is. Connection pooling behavior is unchanged: these errors still close the connection.
…R responses A SERVER_ERROR <msg>\r\n reply is a complete, well-formed protocol line: the server keeps the connection open and in sync (on storage errors it even swallows the data block). Closing our side causes needless reconnect churn, e.g. on 'object too large for cache' or OOM bursts, the worst possible time to add TCP handshake load.
mapno
force-pushed
the
server-error-resumable
branch
from
July 28, 2026 15:10
8d0de39 to
841f5a5
Compare
mapno
marked this pull request as ready for review
July 29, 2026 07:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
SERVER_ERROR <msg>\r\nreply is a complete protocol line: the server keeps the connection open and in sync. It even swallows the data block on storage errors. Closing our side causes needless reconnect churn, e.g. onobject too large for cacheor OOM bursts in the worst possible time to add TCP handshake load, since the server is already under memory pressure.This PR contains two changes:
SERVER_ERRORreplies into the exported (but previously never returned)ErrServerError, wrapped with the server's message so it works witherrors.Is.Note: error text for these responses changes from
memcache: unexpected response line...tomemcache: server error: <msg>.Why this is safe: memcached keeps the connection open on these errors
I've verified this change against memcached >1.6.31. For the ASCII protocol (the only one this client speaks), every common
SERVER_ERRORis written without_string(), which transitions toconn_new_cmd. On storage errors the server additionally swallows the in-flight data block, so both sides stay in sync.memcached 1.6.31 source references
SERVER_ERROR object too large for cache/SERVER_ERROR out of memory storing object— replies, then swallows the data block and continues:proto_text.c#L2045-L2063
SERVER_ERROR Out of memory during read(chunk alloc failure mid-read of a set payload) — same swallow pattern:memcached.c#L3240-L3250
SERVER_ERROR out of memory writing get response— get requests carry no data block, so the connection is already in sync; the server just replies and moves on:proto_text.c#L702-L713
out_of_memory()for ASCII clients is justout_string()(memcached.c#L1300-L1313); theconn_closingbehavior lives only in the binary-protocol branch, which this client never uses.Known exception:
SERVER_ERROR out of memory reading request(failed read-buffer realloc on a huge multiget) setsclose_after_write = true(memcached.c#L2494-L2496). If such a connection is pooled, the next request fails with EOF and the connection is then discarded as non-resumable. Severe OOMs where memcached can't even allocate a response object close the connection without writing any error line, so the client sees a read error, notSERVER_ERROR, and the connection is dropped as before.This is a port of grafana#36 to upstream.