libdb 5.3.34: DB_TXN_SNAPSHOT commits a write skew when the second write starts during the first commit

[xint-io]

Summary

libdb documents DB_TXN_SNAPSHOT as serializable snapshot isolation: one of two transactions in a non-serializable schedule must fail with DB_SNAPSHOT_CONFLICT (rfc/0003-ssi-serializable-snapshot-isolation.md, lines 23-28). In a write skew where the second write begins while the first transaction is inside DB_TXN->commit, libdb 5.3.34 at c4811dc commits both transactions. Both commits return 0, the stored state has no serial order, and no error, assertion or log message reports it.

Environment

Item	Value
libdb	Berkeley DB 5.3.34: (August 3, 2026), commit c4811dc871e313033993e95baa5b6525057c5911
Quoted build	release: dist/configure 'CFLAGS=-O2 -g', no --enable-diagnostic
Agreeing build	debug: dist/configure --enable-debug --enable-diagnostic --enable-test 'CFLAGS=-g -O0 -fno-omit-frame-pointer'
Host	Ubuntu 26.04 LTS, Linux 7.0.0-30-generic, x86_64, gcc 15.2.0
Access method	B-tree, default page size, default cache
Environment	DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | DB_THREAD, set_lk_detect(DB_LOCK_DEFAULT), default lock partitions, no DB_CONFIG
Isolation	every transaction begins with DB_TXN_SNAPSHOT; the databases open with DB_MULTIVERSION

All quoted output is from the release build.

Steps to reproduce

1. Build against a static libdb tree $TREE (db.h, libdb.a and the generated Makefile):

   cc -std=c11 -g -O0 -fno-omit-frame-pointer -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=700 -I"$TREE" repro.c "$TREE/libdb.a" $(sed -n 's/^LDFLAGS=[[:space:]]*//p' "$TREE/Makefile" | head -1) $(sed -n 's/^LIBS=[[:space:]]*//p' "$TREE/Makefile" | head -1) -ldl -pthread -o repro

2. Run in an empty directory:

   mkdir trigger && cd trigger && ../repro

3. Control: run ../repro --control in a second empty directory. T2 then writes and commits before T1 commits.

4. Optional: ../repro --late makes T2 write after T1's commit returns.

Two doctors, alice and bob, are on call; one must stay on call. T1 on the main thread reads bob's record and takes alice off call. T2 on a second thread reads alice's record and takes bob off call. Both reads finish before either write, and T1 writes first. T2 writes once T1 has set an atomic flag just before DB_TXN->commit. The two records are two one-page databases in one environment, so each write touches a page the other read.

Two databases for one roster is an odd shape, and this is the reason. A more natural program keeps both doctors in one database, with the two records on different pages of one B-tree. That program also commits both transactions, but it does so even in the control, where T2 writes and commits before T1 calls commit. So once the two records sit on different pages of one B-tree, no conflict is detected at all, race or no race. That looks like a separate defect; we have not analysed its root cause yet. Two one-page databases avoid it. There the controls behave as the documentation says, and only the timing of T2's write separates the trigger from the control.

/*
 * Write skew under DB_TXN_SNAPSHOT when the second write lands while the
 * first transaction is inside DB_TXN->commit.
 *
 * Two doctors, alice and bob, are both on call.  The rule is that at least
 * one of them stays on call.  Each transaction reads the other doctor's
 * record and, if that doctor is on call, takes its own doctor off call.
 * Under serializable isolation one of the two transactions must fail.
 *
 * Thread T1 (main): begin, read bob, write alice=0, commit.
 * Thread T2:        begin, read alice, write bob=0, commit.
 *
 * Both reads happen before either write.  T1's write always happens before
 * T2's write.  The argument decides only WHEN T2 writes:
 *   default      T2 writes as soon as T1 has called commit.
 *   --control    T2 writes and commits before T1 calls commit.
 *   --late       T2 writes after T1's commit has returned.
 *
 * The two records live in two one-page databases of one environment, so
 * each write touches a page the other transaction read.  Two records on
 * one page do not show this defect: T2's write then returns
 * DB_LOCK_DEADLOCK, because the page was modified after T2's snapshot,
 * which is documented behaviour.
 *
 * Build against a static libdb 5.3.34 and run in an empty directory.
 * Exit status 1 means both transactions committed.
 */
#include <pthread.h>
#include <sched.h>
#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "db.h"

enum { TRIGGER, CONTROL, LATE };

static DB_ENV *env;
static DB *alice_db, *bob_db;
static int mode = TRIGGER;
static pthread_barrier_t barrier;
static atomic_int t1_called_commit, t1_commit_returned;
static int t2_put_rc, t2_commit_rc;

static const char *
rc_name(int rc)
{
	if (rc == 0)
		return ("success");
	switch (rc) {
	case DB_SNAPSHOT_CONFLICT: return "DB_SNAPSHOT_CONFLICT";
	case DB_SNAPSHOT_UNSAFE: return "DB_SNAPSHOT_UNSAFE";
	case DB_LOCK_DEADLOCK: return "DB_LOCK_DEADLOCK";
	case DB_LOCK_NOTGRANTED: return "DB_LOCK_NOTGRANTED";
	default: return db_strerror(rc);
	}
}

static void
die(const char *what, int rc)
{
	fprintf(stderr, "%s: %s (%d)\n", what, rc_name(rc), rc);
	exit(2);
}

static int
expected_abort(int rc)
{
	return (rc == DB_LOCK_DEADLOCK || rc == DB_LOCK_NOTGRANTED ||
	    rc == DB_SNAPSHOT_CONFLICT || rc == DB_SNAPSHOT_UNSAFE);
}

static void
sync_point(void)
{
	int rc = pthread_barrier_wait(&barrier);
	if (rc != 0 && rc != PTHREAD_BARRIER_SERIAL_THREAD)
		die("pthread_barrier_wait", rc);
}

static int
read_on_call(DB *db, DB_TXN *txn)
{
	DBT key, data;
	int on_call = -1, rc;

	memset(&key, 0, sizeof(key));
	memset(&data, 0, sizeof(data));
	key.data = "on_call";
	key.size = 7;
	data.data = &on_call;
	data.ulen = sizeof(on_call);
	data.flags = DB_DBT_USERMEM;
	if ((rc = db->get(db, txn, &key, &data, 0)) != 0)
		die("DB->get", rc);
	return (on_call);
}

static int
write_on_call(DB *db, DB_TXN *txn, int on_call)
{
	DBT key, data;

	memset(&key, 0, sizeof(key));
	memset(&data, 0, sizeof(data));
	key.data = "on_call";
	key.size = 7;
	data.data = &on_call;
	data.size = sizeof(on_call);
	return (db->put(db, txn, &key, &data, 0));
}

/* Ends a transaction: commit after a successful write, abort otherwise. */
static int
finish(const char *name, DB_TXN *txn, int put_rc)
{
	int rc;

	if (put_rc != 0) {
		if ((rc = txn->abort(txn)) != 0)
			die("DB_TXN->abort", rc);
		printf("%s abort                  -> %s (%d)\n",
		    name, rc_name(rc), rc);
		if (!expected_abort(put_rc))
			die("unexpected put result", put_rc);
		return (put_rc);
	}
	rc = txn->commit(txn, 0);
	printf("%s commit                 -> %s (%d)\n",
	    name, rc_name(rc), rc);
	if (rc != 0 && !expected_abort(rc))
		die("unexpected commit result", rc);
	return (rc);
}

static void *
t2_main(void *arg)
{
	DB_TXN *txn;
	int alice_on_call, rc;

	(void)arg;
	if ((rc = env->txn_begin(env, NULL, &txn, DB_TXN_SNAPSHOT)) != 0)
		die("T2 txn_begin", rc);
	alice_on_call = read_on_call(alice_db, txn);
	printf("T2 read  alice.on_call = %d\n", alice_on_call);
	sync_point();			/* both reads done */
	sync_point();			/* T1's write done */

	if (mode == TRIGGER)
		while (!atomic_load(&t1_called_commit))
			sched_yield();
	else if (mode == LATE)
		while (!atomic_load(&t1_commit_returned))
			sched_yield();

	t2_put_rc = alice_on_call ? write_on_call(bob_db, txn, 0) : 0;
	printf("T2 put   bob.on_call = 0  -> %s (%d)\n",
	    rc_name(t2_put_rc), t2_put_rc);
	t2_commit_rc = finish("T2", txn, t2_put_rc);
	if (mode == CONTROL)
		sync_point();		/* T2 finished before T1 commits */
	return (NULL);
}

static void
open_all(u_int32_t create)
{
	int rc;

	if ((rc = db_env_create(&env, 0)) != 0)
		die("db_env_create", rc);
	if ((rc = env->set_lk_detect(env, DB_LOCK_DEFAULT)) != 0)
		die("set_lk_detect", rc);
	if ((rc = env->open(env, ".", create | DB_INIT_LOCK | DB_INIT_LOG |
	    DB_INIT_MPOOL | DB_INIT_TXN | DB_THREAD, 0600)) != 0)
		die("DB_ENV->open", rc);
	if ((rc = db_create(&alice_db, env, 0)) != 0 ||
	    (rc = alice_db->open(alice_db, NULL, "alice.db", NULL, DB_BTREE,
	    create | DB_MULTIVERSION | DB_AUTO_COMMIT | DB_THREAD, 0600)) != 0)
		die("open alice.db", rc);
	if ((rc = db_create(&bob_db, env, 0)) != 0 ||
	    (rc = bob_db->open(bob_db, NULL, "bob.db", NULL, DB_BTREE,
	    create | DB_MULTIVERSION | DB_AUTO_COMMIT | DB_THREAD, 0600)) != 0)
		die("open bob.db", rc);
}

static void
close_all(void)
{
	int rc;

	if ((rc = alice_db->close(alice_db, 0)) != 0 ||
	    (rc = bob_db->close(bob_db, 0)) != 0 ||
	    (rc = env->close(env, 0)) != 0)
		die("close", rc);
}

int
main(int argc, char **argv)
{
	pthread_t t2;
	DB_TXN *txn;
	const char *version;
	int alice, bob, bob_on_call, rc, t1_put_rc, t1_commit_rc;

	if (argc == 2 && strcmp(argv[1], "--control") == 0)
		mode = CONTROL;
	else if (argc == 2 && strcmp(argv[1], "--late") == 0)
		mode = LATE;
	else if (argc != 1) {
		fprintf(stderr, "usage: %s [--control|--late]\n", argv[0]);
		return (2);
	}
	version = db_version(NULL, NULL, NULL);
	if (version == NULL || strncmp(version, "Berkeley DB 5.3.34:",
	    strlen("Berkeley DB 5.3.34:")) != 0) {
		fprintf(stderr, "wrong libdb version: %s\n",
		    version == NULL ? "(null)" : version);
		return (2);
	}
	printf("%s\nmode = %s\n", version,
	    mode == TRIGGER ? "trigger" : mode == CONTROL ? "control" : "late");

	open_all(DB_CREATE);
	if ((rc = write_on_call(alice_db, NULL, 1)) != 0 ||
	    (rc = write_on_call(bob_db, NULL, 1)) != 0)
		die("initial put", rc);
	printf("initial  alice.on_call = 1  bob.on_call = 1\n");

	if ((rc = pthread_barrier_init(&barrier, NULL, 2)) != 0)
		die("pthread_barrier_init", rc);
	if ((rc = pthread_create(&t2, NULL, t2_main, NULL)) != 0)
		die("pthread_create", rc);

	if ((rc = env->txn_begin(env, NULL, &txn, DB_TXN_SNAPSHOT)) != 0)
		die("T1 txn_begin", rc);
	bob_on_call = read_on_call(bob_db, txn);
	printf("T1 read  bob.on_call   = %d\n", bob_on_call);
	sync_point();			/* both reads done */
	t1_put_rc = bob_on_call ? write_on_call(alice_db, txn, 0) : 0;
	printf("T1 put   alice.on_call = 0  -> %s (%d)\n",
	    rc_name(t1_put_rc), t1_put_rc);
	sync_point();			/* T1's write done */

	if (mode == CONTROL)
		sync_point();		/* wait until T2 has finished */
	atomic_store(&t1_called_commit, 1);
	t1_commit_rc = finish("T1", txn, t1_put_rc);
	atomic_store(&t1_commit_returned, 1);

	if ((rc = pthread_join(t2, NULL)) != 0)
		die("pthread_join", rc);
	if ((rc = pthread_barrier_destroy(&barrier)) != 0)
		die("pthread_barrier_destroy", rc);
	close_all();

	open_all(0);
	alice = read_on_call(alice_db, NULL);
	bob = read_on_call(bob_db, NULL);
	close_all();
	printf("stored after close: alice.on_call = %d  bob.on_call = %d\n",
	    alice, bob);

	if (t1_commit_rc == 0 && t2_commit_rc == 0 && alice == 0 && bob == 0) {
		printf("RESULT: both transactions committed and nobody is on call; "
		    "no serial order of T1 and T2 gives this state\n");
		printf("program exit status = 1\n");
		return (1);
	}
	printf("RESULT: one transaction did not commit; the stored state is "
	    "serializable\n");
	printf("program exit status = 0\n");
	return (0);
}

Expected result

rfc/0003-ssi-serializable-snapshot-isolation.md, lines 23-28:

The DB_TXN_SNAPSHOT transaction mode provides full
serializable isolation on top of MVCC snapshot isolation, using Michael
Cahill's Serializable Snapshot Isolation algorithm: detect the dangerous
read/write dependency structures that let snapshot isolation admit
non-serializable schedules, and abort the pivot transaction with
DB_SNAPSHOT_CONFLICT.

docs_src/api/c/txnbegin.md, line 78, says the same for the public API:

On top of that consistent snapshot, Berkeley DB tracks read/write anti-dependencies between concurrent snapshot transactions and, when it detects a dependency structure that could produce a non-serializable schedule, aborts one of the transactions so the committed history is equivalent to some serial order.

T1 then T2 stores alice.on_call = 0 bob.on_call = 1. T2 then T1 stores alice.on_call = 1 bob.on_call = 0. So one transaction must receive DB_SNAPSHOT_CONFLICT (-30968) or DB_SNAPSHOT_UNSAFE (-30967), and the stored state must be one of those two.

Actual result

Release build, default mode:

T1 commit                 -> success (0)
T2 commit                 -> success (0)
stored after close: alice.on_call = 0  bob.on_call = 0
RESULT: both transactions committed and nobody is on call; no serial order of T1 and T2 gives this state
program exit status = 1

The control gives T1 commit -> DB_SNAPSHOT_CONFLICT (-30968) and stored after close: alice.on_call = 1 bob.on_call = 0. The --late run gives T2 put bob.on_call = 0 -> DB_SNAPSHOT_UNSAFE (-30967). Twenty repeated default-mode runs gave the same lines; debug agrees in every mode.

Full logs: release, debug

release

session UTC = 2026-09-03T08:33:49Z
source revision = c4811dc871e313033993e95baa5b6525057c5911
tree = release
configure = 'CFLAGS=-O2 -g'
build line = cc -std=c11 -g -O0 -fno-omit-frame-pointer -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=700 -I<tree> repro.c <tree>/libdb.a $LDFLAGS $LIBS -ldl -pthread -o repro
build exit status = 0
=== trigger ===
Berkeley DB 5.3.34: (August 3, 2026)
mode = trigger
initial  alice.on_call = 1  bob.on_call = 1
T1 read  bob.on_call   = 1
T2 read  alice.on_call = 1
T1 put   alice.on_call = 0  -> success (0)
T2 put   bob.on_call = 0  -> success (0)
T1 commit                 -> success (0)
T2 commit                 -> success (0)
stored after close: alice.on_call = 0  bob.on_call = 0
RESULT: both transactions committed and nobody is on call; no serial order of T1 and T2 gives this state
program exit status = 1
shell exit status = 1
=== control: add --control ===
Berkeley DB 5.3.34: (August 3, 2026)
mode = control
initial  alice.on_call = 1  bob.on_call = 1
T1 read  bob.on_call   = 1
T2 read  alice.on_call = 1
T1 put   alice.on_call = 0  -> success (0)
T2 put   bob.on_call = 0  -> success (0)
T2 commit                 -> success (0)
T1 commit                 -> DB_SNAPSHOT_CONFLICT (-30968)
stored after close: alice.on_call = 1  bob.on_call = 0
RESULT: one transaction did not commit; the stored state is serializable
program exit status = 0
shell exit status = 0
=== late: add --late ===
Berkeley DB 5.3.34: (August 3, 2026)
mode = late
initial  alice.on_call = 1  bob.on_call = 1
T1 read  bob.on_call   = 1
T2 read  alice.on_call = 1
T1 put   alice.on_call = 0  -> success (0)
T1 commit                 -> success (0)
T2 put   bob.on_call = 0  -> DB_SNAPSHOT_UNSAFE (-30967)
T2 abort                  -> success (0)
stored after close: alice.on_call = 0  bob.on_call = 1
RESULT: one transaction did not commit; the stored state is serializable
program exit status = 0
shell exit status = 0
=== trigger x20 ===
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 

debug

session UTC = 2026-09-03T08:33:49Z
source revision = c4811dc871e313033993e95baa5b6525057c5911
tree = debug
configure = --enable-debug --enable-diagnostic --enable-test --with-tcl=/usr/lib/tcl8.6 --enable-compile-commands 'CFLAGS=-g -O0 -fno-omit-frame-pointer'
build line = cc -std=c11 -g -O0 -fno-omit-frame-pointer -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=700 -I<tree> repro.c <tree>/libdb.a $LDFLAGS $LIBS -ldl -pthread -o repro
build exit status = 0
=== trigger ===
Berkeley DB 5.3.34: (August 3, 2026)
mode = trigger
initial  alice.on_call = 1  bob.on_call = 1
T1 read  bob.on_call   = 1
T2 read  alice.on_call = 1
T1 put   alice.on_call = 0  -> success (0)
T2 put   bob.on_call = 0  -> success (0)
T1 commit                 -> success (0)
T2 commit                 -> success (0)
stored after close: alice.on_call = 0  bob.on_call = 0
RESULT: both transactions committed and nobody is on call; no serial order of T1 and T2 gives this state
program exit status = 1
shell exit status = 1
=== control: add --control ===
Berkeley DB 5.3.34: (August 3, 2026)
mode = control
initial  alice.on_call = 1  bob.on_call = 1
T1 read  bob.on_call   = 1
T2 read  alice.on_call = 1
T1 put   alice.on_call = 0  -> success (0)
T2 put   bob.on_call = 0  -> success (0)
T2 commit                 -> success (0)
T1 commit                 -> DB_SNAPSHOT_CONFLICT (-30968)
stored after close: alice.on_call = 1  bob.on_call = 0
RESULT: one transaction did not commit; the stored state is serializable
program exit status = 0
shell exit status = 0
=== late: add --late ===
Berkeley DB 5.3.34: (August 3, 2026)
mode = late
initial  alice.on_call = 1  bob.on_call = 1
T1 read  bob.on_call   = 1
T2 read  alice.on_call = 1
T1 put   alice.on_call = 0  -> success (0)
T1 commit                 -> success (0)
T2 put   bob.on_call = 0  -> DB_SNAPSHOT_UNSAFE (-30967)
T2 abort                  -> success (0)
stored after close: alice.on_call = 0  bob.on_call = 1
RESULT: one transaction did not commit; the stored state is serializable
program exit status = 0
shell exit status = 0
=== trigger x20 ===
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 
T1 commit                 -> success (0) T2 commit                 -> success (0) stored after close: alice.on_call = 0  bob.on_call = 0 program exit status = 1 

Analysis

Root cause

__txn_commit leaves an unrepresented state between the pivot check and the TXN_COMMITTED status update. During that interval, T1 still reports TXN_RUNNING although it has completed its only pivot check. The conflict recorder therefore assumes that T1 still has that check ahead.

1. T1 reads bob.db, and T2 reads alice.db. Each read leaves a SIREAD marker (src/db/db_meta.c#L1184-L1191, src/lock/lock.c#L1215-L1228). When T1 writes alice.db, __lock_get_internal detects T2 --rw--> T1. It sets TXN_DTL_WCONF on T1 and TXN_DTL_RCONF on T2 (src/lock/lock.c#L976-L1013).
2. T1 enters __txn_commit. Under TXN_SYSTEM_LOCK, its pivot check sees TXN_DTL_WCONF set and TXN_DTL_RCONF clear, so the check evaluates false. T1 then releases the mutex (src/txn/txn.c#L754-L766). No later code in __txn_commit reads the flags again.
3. T1 remains TXN_RUNNING until __txn_end stores TXN_COMMITTED (src/txn/txn.c#L1804-L1805). T2 writes bob.db during this interval and detects T1 --rw--> T2. The committed-reader guard does not fire because T1 is still TXN_RUNNING (src/lock/lock.c#L976-L992). The reader-will-abort optimization sees that T1 is running and already has TXN_DTL_WCONF, so it assumes that T1 will abort at a later pivot check. It skips the check that would reject T2 for its existing TXN_DTL_RCONF, does not set TXN_DTL_WCONF on T2, and sets only TXN_DTL_RCONF on T1 (src/lock/lock.c#L996-L1013).
4. The assumption is stale: T1 has already passed its pivot check. T1 now holds both pivot flags, but no code checks them again. T2 still holds only TXN_DTL_RCONF, so its own pivot check also evaluates false. In the reproduced run, no other error occurs and both commit calls return 0.

TXN_SYSTEM_LOCK serializes T1's pivot snapshot with each conflict recorder's flag update. It does not record that the pivot check has completed, and it does not cover the interval before the TXN_COMMITTED update. A recorder can therefore acquire the mutex after T1's check and add the missing TXN_DTL_RCONF while T1 still appears to be a running transaction that has not checked. The source comment (src/txn/txn.c#L746-L752) calls the test and decision atomic, and rfc/0003-ssi-serializable-snapshot-isolation.md, lines 61-63, calls the check race-free. Both claims extend beyond the critical section that the code implements.

Impact and scope

An application that enforces a rule across two records by reading them under DB_TXN_SNAPSHOT can store data that breaks the rule, and nothing marks it.

The two transactions must run in different threads. Each write must touch a page the other transaction read, but not one shared page. With both records on one page, T2's write returns DB_LOCK_DEADLOCK (-30993).

[gburd]

Thanks — this is an exemplary report (precise line citations, trigger + control + variant, release and debug builds agreeing, deterministic repeats). We've confirmed it and it's being worked.

Status: I verified #138, #139 and #140 directly against the source, and #140 is being treated as the highest priority (heap out-of-bounds write in release builds, since the bounds DB_ASSERT is diagnostic-only). Fixes for all five are in flight on separate branches.

Two things your reports changed beyond the individual bugs:

1. We've retracted the overclaimed docs (docs(ssi): retract overclaimed serializability guarantees #141, merged): rfc/0003 no longer asserts the commit-time pivot check is "race-free" or that markers are "bounded", and DB_TXN_SNAPSHOT's API page now carries a known-limitation note so applications don't rely on serializability alone until the fixes land with tests. Shipping a guarantee we don't fully keep was the worst part of this.
2. We're fixing the root-cause class, not just libdb 5.3.34: incomplete replication commit lock lists can violate client transaction isolation #140. That bug exists because SSI added a new lock mode (DB_LOCK_SIREAD) without auditing the pre-existing sites that enumerate modes exhaustively. We're doing that audit and adding a CI check so the class can't silently return.

We're also adding the test tiers that should have caught these: an isolation/anomaly checker that validates committed state against some serial order (including landing a write while a peer is inside commit, which is what #136 needs), and a resource-accounting soak tier (which would have caught #137 and #138 mechanically). Your --control/--late methodology is being adopted as our own bug bar.

On #136 specifically: we also noted your separate observation that two records on different pages of one B-tree detect no conflict at all, even in the control. We're treating that as potentially the more significant half and testing it explicitly.

[gburd]

Update: we've landed the isolation/anomaly test tier (#143) that reproduces this mechanically, and it produced a useful clarification of your "separate defect" observation.

The tier (test/isolation/) validates each committed history against some serial order by enumeration, and runs both shapes at both timings:

scenario	shape	timing	result
write_skew_trigger	two one-page DBs	T2 writes inside T1's commit	XFAIL — reproduces this issue (db[0,0], 2/2 committed, no serial order)
write_skew_control	two one-page DBs	T2 commits first	PASS (DB_SNAPSHOT_CONFLICT)
write_skew_late	two one-page DBs	T2 writes after commit returns	PASS (DB_SNAPSHOT_UNSAFE)
write_skew_samebtree_control	2 records, different pages of one btree	control	PASS — conflict IS detected
write_skew_samebtree_trigger	2 records, different pages of one btree	trigger	XFAIL — same violation

So the different-pages-of-one-btree shape is not a second independent defect: at control timing it correctly returns DB_SNAPSHOT_CONFLICT, and it only fails at the trigger timing — i.e. it's the same commit-window race, not a case where "no conflict is detected at all, race or no race". If you saw the control fail for that shape, it's worth comparing notes; our reading is that both records landing in one btree still register the anti-dependency, and the escape is purely the TXN_RUNNING → TXN_COMMITTED window.

That's good news for the fix: it means there's one root cause to close, not two, and the scope is the atomicity of the pivot check with respect to the commit transition.

The tier is wired into CI as an advisory job today and flips to a hard gate once this is fixed, so it can't silently regress afterwards.