Skip to content

fix(replication): prevent excessive wait times after deploy (fixes #131) #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
renecannao wants to merge 7 commits into
base: master
Choose a base branch
from
Open

fix(replication): prevent excessive wait times after deploy (fixes #131) #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c8b8dc8
fix(replication): prevent excessive wait times after deploy (fixes #131)
renecannao Jul 29, 2026
9e7bd82
feat(ci): add time-bounded deploy test for issue #131 regression
renecannao Jul 29, 2026
453271a
Fix wsrep detection for fresh Galera nodes and fix deploy-time-budget…
renecannao Jul 29, 2026
934eb17
fix: use root socket for wsrep polling, not /use
renecannao Jul 29, 2026
df96a7f
fix tests: template flavor checks must account for shell-level if/else
renecannao Jul 29, 2026
7c358b9
fix: harden wait_until_wsrep_ready connect and client resolution
renecannao Jul 30, 2026
829d68f
fix: address CodeRabbit findings on deploy-time-budget CI
renecannao Aug 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions .github/workflows/integration_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,64 @@ jobs:
./dbdeployer delete all --skip-confirm 2>/dev/null || true
pkill -9 -u "$USER" mysqld 2>/dev/null || true

# Time-bounded deploy test: verifies that "deploy replication" completes
# within MAX_DEPLOY_SECONDS and replicas serve queries immediately after
# deploy (regression test for issue #131).
deploy-time-budget:
name: Deploy time budget (8.4)
runs-on: ubuntu-22.04
env:
GO111MODULE: on
SANDBOX_BINARY: ${{ github.workspace }}/opt/mysql
MAX_DEPLOY_SECONDS: 45
steps:
- uses: actions/checkout@v4
Comment thread
coderabbitai[bot] marked this conversation as resolved.
with:
persist-credentials: false

- uses: actions/setup-go@v5
with:
go-version: '1.23'

- name: Install system libraries
run: |
sudo apt-get update
sudo apt-get install -y libaio1 libnuma1 libncurses5 libldap-2.5-0

- name: Build dbdeployer
run: go build -o dbdeployer .

- name: Cache MySQL tarball
uses: actions/cache@v4
with:
path: /tmp/mysql-tarball
key: mysql-8.4.4-linux-x86_64-v1

- name: Download MySQL 8.4.4
run: |
mkdir -p /tmp/mysql-tarball
TARBALL="mysql-8.4.4-linux-glibc2.17-x86_64.tar.xz"
curl -L -f -o "/tmp/mysql-tarball/$TARBALL" \
"https://dev.mysql.com/get/Downloads/MySQL-8.4/$TARBALL" \
|| curl -L -f -o "/tmp/mysql-tarball/$TARBALL" \
"https://downloads.mysql.com/archives/get/p/23/file/$TARBALL"
ls -lh "/tmp/mysql-tarball/$TARBALL"

- name: Unpack MySQL
run: |
mkdir -p "$SANDBOX_BINARY"
./dbdeployer unpack "/tmp/mysql-tarball/mysql-8.4.4-linux-glibc2.17-x86_64.tar.xz" \
--sandbox-binary="$SANDBOX_BINARY"

- name: Run deploy time budget test
run: ./test/deploy-time-budget.sh

- name: Cleanup
if: always()
run: |
./dbdeployer delete all --skip-confirm 2>/dev/null || true
pkill -9 -u "$USER" mysqld 2>/dev/null || true

# Test Percona Server: single + replication with data verification
percona-test:
name: Percona Server (${{ matrix.percona-version }})
Expand Down
4 changes: 2 additions & 2 deletions sandbox/templates/replication/init_slaves.gotxt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ cd "$SBDIR"
# Observed especially with MySQL 8.4+ and 9.x.
wait_until_replica_ready() {
local use_cmd=${1:-$SBDIR/use}
local max_attempts=${2:-60}
local max_attempts=${2:-20}
local sleep_sec=${3:-1}
for i in $(seq 1 $max_attempts); do
if $use_cmd -BN -e "SELECT 1;" >/dev/null 2>&1 ; then
Expand All @@ -39,7 +39,7 @@ echo "initializing {{.SlaveLabel}} {{.Node}}"
echo '{{.ChangeMasterTo}} {{.MasterHostParam}}="{{.MasterIp}}", {{.MasterPortParam}}={{.MasterPort}}, {{.MasterUserParam}}="{{.RplUser}}", {{.MasterPasswordParam}}="{{.RplPassword}}" {{.MasterAutoPosition}} {{.ChangeMasterExtra}}' | $SBDIR/{{.NodeLabel}}{{.Node}}/use -u root
$SBDIR/{{.NodeLabel}}{{.Node}}/use -u root -e '{{.StartReplica}}'
# Ensure the replica is serving queries before we return from initialization.
wait_until_replica_ready "$SBDIR/{{.NodeLabel}}{{.Node}}/use" 60 1
wait_until_replica_ready "$SBDIR/{{.NodeLabel}}{{.Node}}/use" 20 1
{{end}}
if [ -x ./post_initialization ]
then
Expand Down
4 changes: 2 additions & 2 deletions sandbox/templates/replication/init_slaves_84.gotxt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ cd "$SBDIR"
# Observed especially with MySQL 8.4+ and 9.x.
wait_until_replica_ready() {
local use_cmd=${1:-$SBDIR/use}
local max_attempts=${2:-60}
local max_attempts=${2:-20}
local sleep_sec=${3:-1}
for i in $(seq 1 $max_attempts); do
if $use_cmd -BN -e "SELECT 1;" >/dev/null 2>&1 ; then
Expand All @@ -40,7 +40,7 @@ echo "initializing {{.SlaveLabel}} {{.Node}}"
echo '{{.ChangeMasterTo}} {{.MasterHostParam}}="{{.MasterIp}}", {{.MasterPortParam}}={{.MasterPort}}, {{.MasterUserParam}}="{{.RplUser}}", {{.MasterPasswordParam}}="{{.RplPassword}}" {{.MasterAutoPosition}} {{.ChangeMasterExtra}}' | $SBDIR/{{.NodeLabel}}{{.Node}}/use -u root
$SBDIR/{{.NodeLabel}}{{.Node}}/use -u root -e '{{.StartReplica}}'
# Ensure the replica is serving queries before we return from initialization.
wait_until_replica_ready "$SBDIR/{{.NodeLabel}}{{.Node}}/use" 60 1
wait_until_replica_ready "$SBDIR/{{.NodeLabel}}{{.Node}}/use" 20 1
{{end}}
if [ -x ./post_initialization ]
then
Expand Down
65 changes: 59 additions & 6 deletions sandbox/templates/single/sb_include.gotxt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,13 +70,66 @@ function check_output
# Wait until wsrep_ready is ON for Galera/PXC nodes.
# This prevents grants and user DML from running before the node
# has joined the cluster.
# On non-Galera servers the status variable does not exist and
# this function returns immediately (no delay) once the server accepts
# connections.
# Detection and polling connect as root via socket (no password),
# because the sandbox user (msandbox) may not exist yet on fresh
# nodes (load_grants has not run).
# Connect failures are NOT treated as non-Galera: the pid file can
# appear before the server accepts connections, so we retry until
# root can connect, then check whether wsrep_ready exists.
function wait_until_wsrep_ready
{
local use_cmd=${1:-$SBDIR/use}
local max_attempts=${2:-60}
local sleep_sec=${3:-2}
local max_attempts=${1:-60}
local sleep_sec=${2:-2}
local mysql_cmd=""
local clients
if [ "$FLAVOR" = "mariadb" ]; then
clients="mariadb mysql"
else
clients="mysql mariadb"
fi
local d c
for d in "$CLIENT_BASEDIR/bin" "$BASEDIR/bin"; do
for c in $clients; do
if [ -x "$d/$c" ]; then
mysql_cmd="$d/$c"
break 2
fi
done
done
if [ -z "$mysql_cmd" ]; then
echo "WARNING: no mysql/mariadb client found under $CLIENT_BASEDIR/bin or $BASEDIR/bin" >&2
return 1
fi
# Wait until root can connect via socket. Empty query output before
# connect succeeds must not be treated as "non-Galera".
# Keep this budget separate and modest: pid-file can appear slightly
# before the server accepts connections, but not for minutes.
local connect_attempts=30
local connect_sleep=1
local connected=0
local i
for i in $(seq 1 $connect_attempts); do
if $mysql_cmd --no-defaults -S "$SOCKET_FILE" -u root -BN -e "SELECT 1" >/dev/null 2>&1; then
connected=1
break
fi
sleep $connect_sleep
done
if [ "$connected" -ne 1 ]; then
echo "WARNING: cannot connect as root via socket after $((connect_attempts * connect_sleep))s" >&2
return 1
fi
# Connected: empty SHOW STATUS means wsrep_ready does not exist (non-Galera).
local wsrep_check
wsrep_check=$($mysql_cmd --no-defaults -S "$SOCKET_FILE" -u root -BN -e "SHOW STATUS LIKE 'wsrep_ready';" 2>/dev/null)
if [ -z "$wsrep_check" ]; then
return 0
fi
Comment on lines +125 to +130

@coderabbitai coderabbitai Bot Aug 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Check the query's exit status before treating empty output as "non-Galera".

Line 127 discards the exit status of $mysql_cmd. If the query fails after a successful connect (transient socket drop, server restart), stdout is empty and line 128 wrongly returns 0, treating a real Galera node as non-Galera and skipping the wsrep_ready wait. This is the same failure pattern already flagged on this file for the previous grep-pipeline version: a command failure is indistinguishable from "no matching row" and is misclassified as success.

Distinguish command failure from an empty-but-successful result.

🐛 Proposed fix to distinguish query failure from a genuinely missing variable 
     # Connected: empty SHOW STATUS means wsrep_ready does not exist (non-Galera).
     local wsrep_check
-    wsrep_check=$($mysql_cmd --no-defaults -S "$SOCKET_FILE" -u root -BN -e "SHOW STATUS LIKE 'wsrep_ready';" 2>/dev/null)
-    if [ -z "$wsrep_check" ]; then
-        return 0
-    fi
+    if wsrep_check=$($mysql_cmd --no-defaults -S "$SOCKET_FILE" -u root -BN -e "SHOW STATUS LIKE 'wsrep_ready';" 2>/dev/null); then
+        if [ -z "$wsrep_check" ]; then
+            return 0
+        fi
+    else
+        echo "WARNING: wsrep_ready query failed after connect; retrying" >&2
+    fi
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@sandbox/templates/single/sb_include.gotxt` around lines 125 - 130, Update the
wsrep_check query handling near the local wsrep_check assignment to capture and
inspect $mysql_cmd’s exit status before interpreting empty output. Return 0 only
when the query succeeds with no result; propagate or handle a failed query as an
error so transient connection failures do not bypass the wsrep_ready wait.

for i in $(seq 1 $max_attempts); do
if $use_cmd -BN -e "SHOW STATUS LIKE 'wsrep_ready';" 2>/dev/null | grep -qi 'ON' ; then
if $mysql_cmd --no-defaults -S "$SOCKET_FILE" -u root -BN -e "SHOW STATUS LIKE 'wsrep_ready';" 2>/dev/null | grep -qiE 'wsrep_ready[[:space:]]+ON'; then
return 0
fi
sleep $sleep_sec
Expand All @@ -90,11 +143,11 @@ function wait_until_wsrep_ready
# initialization to prevent the race where the first client command on a
# just-started replica fails with "Access denied" or connection errors.
# Observed especially with MySQL 8.4+ and 9.x.
# Usage: wait_until_replica_ready "$SBDIR/s1/use" 60 1
# Usage: wait_until_replica_ready "$SBDIR/s1/use" 20 1
function wait_until_replica_ready
{
local use_cmd=${1:-$SBDIR/use}
local max_attempts=${2:-60}
local max_attempts=${2:-20}
local sleep_sec=${3:-1}
for i in $(seq 1 $max_attempts); do
if $use_cmd -BN -e "SELECT 1;" >/dev/null 2>&1 ; then
Expand Down
2 changes: 1 addition & 1 deletion sandbox/templates/single/wait_wsrep_after_start.gotxt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@ source {{.SandboxDir}}/sb_include

# Best-effort wait for wsrep cluster readiness before grants/DML.
# This script is safe to run even on non-Galera sandboxes (exits 0 quickly).
wait_until_wsrep_ready "$SBDIR/use" 60 2 || true
wait_until_wsrep_ready 60 2 || true
exit 0
Loading
Loading