Skip to content

chore(docker): keep memcached alive and fail open on cache delete - #771

Open
adekbadek wants to merge 2 commits into
mainfrom
chore/memcached-resilience
Open

chore(docker): keep memcached alive and fail open on cache delete#771
adekbadek wants to merge 2 commits into
mainfrom
chore/memcached-resilience

Conversation

@adekbadek

@adekbadek adekbadek commented Jul 30, 2026

Copy link
Copy Markdown
Member

All Submissions:

Changes proposed in this Pull Request:

Local dev only — the drop-in is copied into the container by run.sh and never ships to publishers.

In an isolated env, update_post_meta() followed by get_post_meta() in the same request returned the stale value while the DB held the new one. wp_cache_delete() and clean_post_cache() did not help. Two of fifteen running env containers were affected.

memcached had died, and nothing supervises it. With the backend unreachable, the drop-in treats its in-process array asymmetrically:

method runtime cache when memcached is down
set() writes $this->cache[$key] unconditionally
get() serves from $this->cache[$key]
delete() unset() only if false !== $result — so never

Memcache::delete() returns false when the server is unreachable, so the runtime entry survives and the cache degrades into one that can never be invalidated. Core invalidates post meta via delete, so update_post_meta() became a no-op against the cache. Anything writing and reading back in one request — CLI migrations, tests — got false results with no outward signal.

Changes:

  1. delete() and replace() always drop the runtime entry. The one that matters: it turns a silent correctness bug into, at worst, a slower local site.
  2. Unreachable servers are logged once per process, which also bounds connection_errors (written, never read, previously one entry per failed op).
  3. bin/watchdog-memcached.sh — a 30s TCP probe that restarts memcached. Bind-mounted via /var/scripts rather than inline in run.sh, so later changes to it need no image rebuild. No set -e, so a failed restart can't end the loop.
  4. Container healthchecks for the main, 8.2 and generated env compose files, plus a migration in n env up so the ~43 existing env files get one. Tuned to flag only a memcached the watchdog could not revive, with start_period covering a cold boot.
  5. Header note recording that this vendored copy is patched, so a re-vendor doesn't silently revert it.

run.sh is baked into the image, so the watchdog needs one ./build-image.sh to start; the drop-in applies on container restart.

How to test the changes in this Pull Request:

  1. On main, kill memcached in any env container:
    docker exec <container> sh -c '/etc/init.d/memcached stop; pkill -x memcached'
    
    Create a post, update_post_meta(), then get_post_meta() it back in one wp eval-file. Returns '' while a direct SELECT returns the value.
  2. With this branch's bin/object-cache.php in the container's wp-content/, memcached still down:
    immediately:   'probe_value'
    after delete:  'probe_value'
    direct db:     'probe_value'
    post title:    'probe-renamed'
    
  3. wp-content/debug.log reports the outage once per process: Memcached object cache: cannot reach 127.0.0.1:11211…
  4. Run MEMCACHED_WATCHDOG_INTERVAL=2 /var/scripts/watchdog-memcached.sh, then kill memcached three times with pkill -x memcached. It recovers each time and the loop survives.
  5. docker exec <container> bash -c 'echo > /dev/tcp/127.0.0.1/11211' exits non-zero with memcached down, zero with it up.
  6. n env up <name> on an env created before this branch inserts the healthcheck; re-running inserts nothing.

Other information:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your changes, as applicable? — no harness covers the container's runtime setup; verified manually per the steps above.
  • Have you successfully run tests with your changes locally?

All six steps verified on a live container; both compose files and a migrated env file parse via docker compose config.

@adekbadek
adekbadek marked this pull request as ready for review July 30, 2026 10:07
@adekbadek
adekbadek requested a review from a team as a code owner July 30, 2026 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant