libdb 5.3.34: __txn_reap_si_details leaks MVCC mutex slots during snapshot cleanup

[xint-io]

Summary

__txn_reap_si_details leaks an MVCC mutex slot during committed snapshot cleanup. After the detail's SIREAD and MVCC buffer references reach zero, the function frees the detail without releasing its MVCC mutex (src/txn/txn_region.c, lines 447-484).

The affected sequence begins when __txn_end retains the committed detail because an MVCC buffer still refers to it. The last MVCC buffer is then removed while a SIREAD marker remains, and later SIREAD cleanup invokes the reaper.

Each affected cleanup leaves one mutex slot in use. Repetition reduces the mutex region's available capacity and can make a later valid database operation return ENOMEM.

Environment

Item	Session value
Version and commit	Berkeley DB 5.3.34: (August 3, 2026); c4811dc871e313033993e95baa5b6525057c5911
release	Configure: 'CFLAGS=-O2 -g'; the selected Actual result lines came from this tree
debug	Configure: --enable-debug --enable-diagnostic 'CFLAGS=-g -O0 -fno-omit-frame-pointer'
asan	Configure: --enable-debug CFLAGS='-fsanitize=address -fno-omit-frame-pointer -g -O1' LDFLAGS=-fsanitize=address
Host	Linux; kernel 7.0.0-30-generic; x86_64; cc (Ubuntu 15.2.0-16ubuntu1) 15.2.0
Reproduction build	cc -g -O0 -fno-omit-frame-pointer; each tree's generated db.h, static libdb.a, LDFLAGS, and LIBS
Access method	DB_BTREE, opened with DB_AUTO_COMMIT | DB_MULTIVERSION
Environment and isolation	DB_CREATE | DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | DB_RECOVER; DB_TXN_SNAPSHOT; DB_LOCK_DEFAULT
Resource sizing	Defaults; DB_ENV->set_cachesize() and DB_ENV->mutex_set_max() are not called
Sanitizer scope	AddressSanitizer enabled; runtime leak detection disabled because the measured resource is a mutex-region slot, not a heap allocation; no LeakSanitizer result is claimed

Steps to reproduce

The reproduction program uses only public db.h APIs. It does not call internal functions or simulate cleanup.

In read mode, each snapshot transaction reads accounts.db and writes journal.db. The no-read control omits only that transaction read, so it creates no SIREAD marker.

After each successful snapshot transaction, the program flushes modified pages, reads all account rows, and writes one autocommit entry. It requests a checkpoint every 100 cycles.

1. Build three trees at the tested commit.

git clone https://github.com/berkeleydb/libdb.git
cd libdb
git checkout c4811dc871e313033993e95baa5b6525057c5911

mkdir build-release build-debug build-asan
(cd build-release && ../dist/configure CFLAGS='-O2 -g' && make -j12 libdb.a)
(cd build-debug && ../dist/configure --enable-debug --enable-diagnostic \
  CFLAGS='-g -O0 -fno-omit-frame-pointer' && make -j12 libdb.a)
(cd build-asan && ASAN_OPTIONS=detect_leaks=0 ../dist/configure --enable-debug \
  CFLAGS='-fsanitize=address -fno-omit-frame-pointer -g -O1' \
  LDFLAGS=-fsanitize=address && make -j8 libdb.a)

2. Save as repro.c.

/*
 * libdb 5.3.34: txn mvcc accounting grows across committed DB_TXN_SNAPSHOT
 * transactions that read one database and write another.
 *
 * usage: repro read      each cycle reads accounts.db and writes journal.db
 *        repro no-read   control: the same cycle without the account read
 *
 * The program uses <db.h> only.  It reports DB_ENV->mutex_stat() and the
 * "txn mvcc" line of DB_ENV->mutex_stat_print() after each reported cycle,
 * so the mutex accounting is visible through the public API.
 */
#include <db.h>

#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define	ACCOUNTS	512	/* rows touched during the read sweep */
#define	CYCLES		1000	/* upper bound on the campaign */
#define	VALUE_BYTES	256

static DB_ENV *env;
static DB *accounts, *journal;
static u_long mvcc_mutexes;
static int saw_mvcc_type;
static const char *oom_operation;	/* the call that returned ENOMEM */

static void
fail(const char *operation, int ret)
{
	fprintf(stderr, "FAIL operation=%s ret=%d name=%s\n",
	    operation, ret, db_strerror(ret));
	exit(2);
}

/* Transient transaction outcomes handled by starting a new transaction. */
static int
retryable(int ret)
{
	return (ret == DB_LOCK_DEADLOCK || ret == DB_SNAPSHOT_CONFLICT ||
	    ret == DB_SNAPSHOT_UNSAFE);
}

/*
 * DB_ENV->mutex_stat_print() emits one "<count>\t<type>" line per mutex
 * type in use.  Keep the MTX_TXN_MVCC line and drop the rest.  The callback
 * makes no libdb call.
 */
static void
message(const DB_ENV *unused, const char *text)
{
	const char *tab;

	(void)unused;
	if ((tab = strstr(text, "\ttxn mvcc")) != NULL) {
		mvcc_mutexes = strtoul(text, NULL, 10);
		saw_mvcc_type = 1;
	}
}

static void
pair(DBT *key, DBT *data, char *keybuf, char *databuf)
{
	memset(key, 0, sizeof(*key));
	memset(data, 0, sizeof(*data));
	key->data = keybuf;
	key->size = (u_int32_t)strlen(keybuf);
	key->ulen = 64;
	key->flags = DB_DBT_USERMEM;
	data->data = databuf;
	data->size = VALUE_BYTES;
	data->ulen = VALUE_BYTES;
	data->flags = DB_DBT_USERMEM;
}

static u_int32_t
in_use(void)
{
	DB_MUTEX_STAT *stat;
	u_int32_t value;
	int ret;

	if ((ret = env->mutex_stat(env, &stat, 0)) != 0)
		fail("DB_ENV->mutex_stat", ret);
	value = stat->st_mutex_inuse;
	free(stat);
	mvcc_mutexes = 0;
	saw_mvcc_type = 0;
	if ((ret = env->mutex_stat_print(env, 0)) != 0)
		fail("DB_ENV->mutex_stat_print", ret);
	if (!saw_mvcc_type)
		fail("DB_ENV->mutex_stat_print(txn mvcc missing)", EINVAL);
	return (value);
}

/* Reopen after the clean close and read one committed journal entry. */
static void
verify_stored(char *databuf)
{
	DBT key, data;
	char keybuf[64];
	int i, ret;

	if ((ret = db_env_create(&env, 0)) != 0)
		fail("db_env_create(reopen)", ret);
	env->set_errfile(env, stdout);
	if ((ret = env->open(env, ".", DB_CREATE | DB_INIT_LOCK |
	    DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | DB_RECOVER, 0600)) != 0)
		fail("DB_ENV->open(reopen)", ret);
	if ((ret = db_create(&journal, env, 0)) != 0)
		fail("db_create(journal.db reopen)", ret);
	if ((ret = journal->open(journal, NULL, "journal.db", NULL,
	    DB_BTREE, DB_AUTO_COMMIT | DB_MULTIVERSION, 0600)) != 0)
		fail("DB->open(journal.db reopen)", ret);

	snprintf(keybuf, sizeof(keybuf), "entry-0");
	pair(&key, &data, keybuf, databuf);
	if ((ret = journal->get(journal, NULL, &key, &data, 0)) != 0)
		fail("DB->get(entry-0 reopen)", ret);
	if (data.size != VALUE_BYTES)
		fail("DB->get(entry-0 size)", EINVAL);
	for (i = 0; i < VALUE_BYTES; ++i)
		if (databuf[i] != 'x')
			fail("DB->get(entry-0 value)", EINVAL);
	if ((ret = journal->close(journal, 0)) != 0)
		fail("DB->close(journal.db reopen)", ret);
	if ((ret = env->close(env, 0)) != 0)
		fail("DB_ENV->close(reopen)", ret);
	printf("STORED after_reopen key=entry-0 value=x*256 bytes=%lu ret=0 "
	    "name=DB_SUCCESS\n", (u_long)data.size);
}

/* An autocommit write outside any snapshot transaction. */
static int
autocommit_put(const char *keytext, char *databuf)
{
	DBT key, data;
	char keybuf[64];
	int ret;

	for (;;) {
		snprintf(keybuf, sizeof(keybuf), "%s", keytext);
		pair(&key, &data, keybuf, databuf);
		ret = journal->put(journal, NULL, &key, &data, 0);
		if (ret == ENOMEM)
			oom_operation = "DB->put(autocommit)";
		if (ret == 0 || ret == ENOMEM)
			return (ret);
		if (!retryable(ret))
			fail("DB->put(autocommit)", ret);
	}
}

/* One snapshot transaction: optionally read accounts, then write journal. */
static int
cycle_txn(int with_read, int cycle, char *databuf)
{
	DB_TXN *txn;
	DBT key, data;
	char keybuf[64];
	const char *site;
	int abort_ret, ret;

	for (;;) {
		txn = NULL;
		ret = env->txn_begin(env, NULL, &txn, DB_TXN_SNAPSHOT);
		if (ret == ENOMEM) {
			oom_operation = "DB_ENV->txn_begin";
			return (ret);
		}
		if (retryable(ret))
			continue;
		if (ret != 0)
			fail("DB_ENV->txn_begin", ret);

		if (with_read) {
			snprintf(keybuf, sizeof(keybuf), "account-%d",
			    cycle % ACCOUNTS);
			pair(&key, &data, keybuf, databuf);
			site = "DB->get(account)";
			if ((ret = accounts->get(accounts,
			    txn, &key, &data, 0)) != 0)
				goto undo;
		}
		snprintf(keybuf, sizeof(keybuf), "entry-%d", cycle);
		pair(&key, &data, keybuf, databuf);
		site = "DB->put(entry)";
		if ((ret = journal->put(journal, txn, &key, &data, 0)) != 0)
			goto undo;

		ret = txn->commit(txn, 0);
		if (ret == ENOMEM)
			oom_operation = "DB_TXN->commit";
		if (ret == 0 || ret == ENOMEM)
			return (ret);
		if (!retryable(ret))
			fail("DB_TXN->commit", ret);
		continue;

undo:		abort_ret = txn->abort(txn);
		if (abort_ret != 0)
			fail("DB_TXN->abort", abort_ret);
		if (ret == ENOMEM) {
			oom_operation = site;
			return (ret);
		}
		if (!retryable(ret))
			fail(site, ret);
	}
}

int
main(int argc, char *argv[])
{
	DBT key, data;
	char keybuf[64], databuf[VALUE_BYTES];
	const char *text;
	u_long base_mvcc;
	u_int32_t base, last;
	int major, minor, patch, with_read, ret, cycle, j, done, oom, valid;

	if (argc != 2 || (strcmp(argv[1], "read") != 0 &&
	    strcmp(argv[1], "no-read") != 0)) {
		fprintf(stderr, "usage: %s read|no-read\n", argv[0]);
		return (2);
	}
	with_read = strcmp(argv[1], "read") == 0;
	text = db_version(&major, &minor, &patch);
	printf("VERSION %s\n", text);
	if (major != 5 || minor != 3 || patch != 34)
		return (2);
	printf("MODE %s cache=default accounts=%d cycles=%d checkpoint_interval=100\n",
	    argv[1], ACCOUNTS, CYCLES);

	if ((ret = db_env_create(&env, 0)) != 0)
		fail("db_env_create", ret);
	env->set_errfile(env, stdout);
	env->set_msgcall(env, message);
	if ((ret = env->set_lk_detect(env, DB_LOCK_DEFAULT)) != 0)
		fail("DB_ENV->set_lk_detect", ret);
	if ((ret = env->open(env, ".", DB_CREATE | DB_INIT_LOCK |
	    DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | DB_RECOVER, 0600)) != 0)
		fail("DB_ENV->open", ret);

	if ((ret = db_create(&accounts, env, 0)) != 0)
		fail("db_create(accounts.db)", ret);
	if ((ret = accounts->open(accounts, NULL, "accounts.db", NULL,
	    DB_BTREE, DB_CREATE | DB_AUTO_COMMIT | DB_MULTIVERSION, 0600)) != 0)
		fail("DB->open(accounts.db)", ret);
	if ((ret = db_create(&journal, env, 0)) != 0)
		fail("db_create(journal.db)", ret);
	if ((ret = journal->open(journal, NULL, "journal.db", NULL,
	    DB_BTREE, DB_CREATE | DB_AUTO_COMMIT | DB_MULTIVERSION, 0600)) != 0)
		fail("DB->open(journal.db)", ret);

	memset(databuf, 'x', sizeof(databuf));
	for (j = 0; j < ACCOUNTS; ++j) {
		snprintf(keybuf, sizeof(keybuf), "account-%d", j);
		pair(&key, &data, keybuf, databuf);
		for (;;) {
			ret = accounts->put(accounts, NULL, &key, &data, 0);
			if (ret == 0)
				break;
			if (!retryable(ret))
				fail("DB->put(preload)", ret);
		}
	}
	last = base = in_use();
	base_mvcc = mvcc_mutexes;
	printf("PRELOADED in_use=%lu txn_mvcc=%lu\n",
	    (u_long)base, base_mvcc);

	done = oom = 0;
	for (cycle = 0; cycle < CYCLES; ++cycle) {
		if ((ret = cycle_txn(with_read, cycle, databuf)) == ENOMEM) {
			printf("ENOMEM operation=%s cycle=%d ret=%d name=%s\n",
			    oom_operation, cycle, ret, db_strerror(ret));
			oom = 1;
			break;
		}

		/* Flush writes, then apply read pressure to the cache. */
		if ((ret = env->memp_sync(env, NULL)) != 0)
			fail("DB_ENV->memp_sync", ret);
		for (j = 0; j < ACCOUNTS; ++j) {
			snprintf(keybuf, sizeof(keybuf), "account-%d",
			    (j + 97 * cycle) % ACCOUNTS);
			pair(&key, &data, keybuf, databuf);
			for (;;) {
				ret = accounts->get(accounts, NULL, &key, &data, 0);
				if (ret == 0)
					break;
				if (!retryable(ret))
					fail("DB->get(churn)", ret);
			}
		}

		/* One autocommit write, then the periodic checkpoint. */
		snprintf(keybuf, sizeof(keybuf), "tick-%d", cycle);
		if (autocommit_put(keybuf, databuf) == ENOMEM) {
			printf("ENOMEM operation=%s cycle=%d ret=%d name=%s\n",
			    oom_operation, cycle, ENOMEM, db_strerror(ENOMEM));
			oom = 1;
			break;
		}
		if ((cycle + 1) % 100 == 0 &&
		    (ret = env->txn_checkpoint(env, 0, 0, 0)) != 0)
			fail("DB_ENV->txn_checkpoint", ret);

		last = in_use();
		done = cycle + 1;
		if (cycle < 3 || (cycle + 1) % 100 == 0)
			printf("CYCLE %d in_use=%lu txn_mvcc=%lu\n",
			    cycle, (u_long)last, mvcc_mutexes);
	}

	printf("RESULT mode=%s cycles_completed=%d enomem=%d in_use=%lu->%lu "
	    "txn_mvcc=%lu->%lu\n", argv[1], done, oom, (u_long)base,
	    (u_long)last, base_mvcc, mvcc_mutexes);
	valid = with_read ?
	    oom && oom_operation != NULL &&
	    mvcc_mutexes > base_mvcc + CYCLES / 10 :
	    !oom && done == CYCLES && mvcc_mutexes <= base_mvcc;

	if ((ret = accounts->close(accounts, 0)) != 0)
		fail("DB->close(accounts.db)", ret);
	if ((ret = journal->close(journal, 0)) != 0)
		fail("DB->close(journal.db)", ret);
	if ((ret = env->close(env, 0)) != 0)
		fail("DB_ENV->close", ret);
	printf("CLOSED ret=0 name=DB_SUCCESS\n");
	verify_stored(databuf);
	if (!valid) {
		fprintf(stderr, "FAIL validation mode=%s enomem=%d "
		    "cycles_completed=%d txn_mvcc=%lu->%lu\n", argv[1], oom,
		    done, base_mvcc, mvcc_mutexes);
		return (2);
	}
	printf("EXIT status=0\n");
	return (0);
}

3. Compile against each tree.

compile_tree()
{
  name=$1
  tree=$2
  ldflags=$(sed -n 's/^LDFLAGS=[[:space:]]*//p' "$tree/Makefile" | head -1)
  libs=$(sed -n 's/^LIBS=[[:space:]]*//p' "$tree/Makefile" | head -1)
  cc -g -O0 -fno-omit-frame-pointer -I "$tree" repro.c \
    "$tree/libdb.a" $ldflags $libs -ldl -o "repro-$name"
}

compile_tree release build-release
compile_tree debug build-debug
compile_tree asan build-asan

4. Run each trigger.

run_mode()
{
  name=$1
  mode=$2
  prefix=$3
  mkdir "run-$name-$mode"
  (cd "run-$name-$mode" && $prefix LC_ALL=C timeout 120s "../repro-$name" "$mode")
}

run_mode release read env
run_mode debug read env
run_mode asan read "env ASAN_OPTIONS=detect_leaks=0"

5. Run each no-read control.

run_mode release no-read env
run_mode debug no-read env
run_mode asan no-read "env ASAN_OPTIONS=detect_leaks=0"

Expected result

td is the transaction detail. Its si_ref field counts SIREAD marker references, and its mvcc_ref field counts MVCC buffer references.

RFC 0003 identifies itself as implemented and normative (rfc/0003-ssi-serializable-snapshot-isolation.md, lines 1-8). Its M4 lifecycle note gives this final cleanup rule (rfc/0003/M4-commit-lifecycle.md, lines 33-40):

When td->si_ref == 0 && mvcc_ref == 0, free the detail via a
txn-region helper (cross-subsystem free, mirroring how mpool frees MVCC
details today).

On the alternate finalization path, __txn_remove_buffer returns td->mvcc_mtx before it frees the detail (src/txn/txn_region.c, lines 541-552).

After both reference counts reach zero, final cleanup must leave no mutex slot allocated for that detail. Repeating this cleanup must not make the public txn mvcc count grow without bound.

Actual result

The release trigger produced these selected lines. Eight intermediate cycle reports are omitted.

PRELOADED in_use=506 txn_mvcc=10
CYCLE 599 in_use=1829 txn_mvcc=602
BDB2034 unable to allocate memory for mutex; resize mutex region
ENOMEM operation=DB->put(entry) cycle=616 ret=12 name=Cannot allocate memory
RESULT mode=read cycles_completed=616 enomem=1 in_use=506->1861 txn_mvcc=10->618

The release control completed all 1000 cycles without ENOMEM; txn mvcc changed from 10 to 3.

Both auxiliary triggers reached ENOMEM while their controls completed; asan emitted no AddressSanitizer diagnostic.

Full release log

RUN_ID=issue87.0uP9wr
TREE=release
TARGET_COMMIT=c4811dc871e313033993e95baa5b6525057c5911
PROGRAM_SHA256=1ff0a42e5c3be1941828c6de32705341e94d842bec4de24852cd89c7095c35c7
CONFIGURE_FLAGS='CFLAGS=-O2 -g'
LIBDB_ARCHIVE_SHA256=7b4703bc21651b876249cf375a081b9c4bd2a28f703f8be88fe9ad3c7ed309a1
OS=Linux
KERNEL=7.0.0-30-generic
ARCH=x86_64
COMPILER=cc (Ubuntu 15.2.0-16ubuntu1) 15.2.0
API_SCOPE=public db.h; two DB_BTREE handles; DB_TXN_SNAPSHOT
ENV_FLAGS=DB_CREATE|DB_INIT_LOCK|DB_INIT_LOG|DB_INIT_MPOOL|DB_INIT_TXN|DB_RECOVER
DB_FLAGS=DB_CREATE|DB_AUTO_COMMIT|DB_MULTIVERSION
MUTEX_MAX_OVERRIDE=none
$ cc -g -O0 -fno-omit-frame-pointer -I <tree> repro.c <tree>/libdb.a  -luring -lpthread -ldl -o repro
$ mkdir trigger && cd trigger && LC_ALL=C timeout 120s ../repro read
VERSION Berkeley DB 5.3.34: (August 3, 2026)
MODE read cache=default accounts=512 cycles=1000 checkpoint_interval=100
PRELOADED in_use=506 txn_mvcc=10
CYCLE 0 in_use=506 txn_mvcc=5
CYCLE 1 in_use=509 txn_mvcc=6
CYCLE 2 in_use=511 txn_mvcc=7
CYCLE 99 in_use=742 txn_mvcc=103
CYCLE 199 in_use=960 txn_mvcc=203
CYCLE 299 in_use=1173 txn_mvcc=303
CYCLE 399 in_use=1396 txn_mvcc=403
CYCLE 499 in_use=1616 txn_mvcc=503
CYCLE 599 in_use=1829 txn_mvcc=602
BDB2034 unable to allocate memory for mutex; resize mutex region
ENOMEM operation=DB->put(entry) cycle=616 ret=12 name=Cannot allocate memory
RESULT mode=read cycles_completed=616 enomem=1 in_use=506->1861 txn_mvcc=10->618
CLOSED ret=0 name=DB_SUCCESS
STORED after_reopen key=entry-0 value=x*256 bytes=256 ret=0 name=DB_SUCCESS
EXIT status=0
PROCESS_STATUS mode=read status=0

$ mkdir control && cd control && LC_ALL=C timeout 120s ../repro no-read
VERSION Berkeley DB 5.3.34: (August 3, 2026)
MODE no-read cache=default accounts=512 cycles=1000 checkpoint_interval=100
PRELOADED in_use=506 txn_mvcc=10
CYCLE 0 in_use=504 txn_mvcc=4
CYCLE 1 in_use=506 txn_mvcc=5
CYCLE 2 in_use=506 txn_mvcc=5
CYCLE 99 in_use=531 txn_mvcc=3
CYCLE 199 in_use=549 txn_mvcc=3
CYCLE 299 in_use=559 txn_mvcc=3
CYCLE 399 in_use=580 txn_mvcc=3
CYCLE 499 in_use=599 txn_mvcc=3
CYCLE 599 in_use=621 txn_mvcc=3
CYCLE 699 in_use=630 txn_mvcc=3
CYCLE 799 in_use=630 txn_mvcc=3
CYCLE 899 in_use=630 txn_mvcc=3
CYCLE 999 in_use=630 txn_mvcc=3
RESULT mode=no-read cycles_completed=1000 enomem=0 in_use=506->630 txn_mvcc=10->3
CLOSED ret=0 name=DB_SUCCESS
STORED after_reopen key=entry-0 value=x*256 bytes=256 ret=0 name=DB_SUCCESS
EXIT status=0
PROCESS_STATUS mode=no-read status=0
VALIDATION status=PASS run_id=issue87.0uP9wr target_commit=c4811dc871e313033993e95baa5b6525057c5911 trigger_txn_mvcc=10->618 control_txn_mvcc=10->3

Full debug log

RUN_ID=issue87.0uP9wr
TREE=debug
TARGET_COMMIT=c4811dc871e313033993e95baa5b6525057c5911
PROGRAM_SHA256=1ff0a42e5c3be1941828c6de32705341e94d842bec4de24852cd89c7095c35c7
CONFIGURE_FLAGS=--enable-debug --enable-diagnostic 'CFLAGS=-g -O0 -fno-omit-frame-pointer'
LIBDB_ARCHIVE_SHA256=b1cb8bf2b244f11701a8a468de3386740cd3914895c88637e75d83741fcfa85c
OS=Linux
KERNEL=7.0.0-30-generic
ARCH=x86_64
COMPILER=cc (Ubuntu 15.2.0-16ubuntu1) 15.2.0
API_SCOPE=public db.h; two DB_BTREE handles; DB_TXN_SNAPSHOT
ENV_FLAGS=DB_CREATE|DB_INIT_LOCK|DB_INIT_LOG|DB_INIT_MPOOL|DB_INIT_TXN|DB_RECOVER
DB_FLAGS=DB_CREATE|DB_AUTO_COMMIT|DB_MULTIVERSION
MUTEX_MAX_OVERRIDE=none
$ cc -g -O0 -fno-omit-frame-pointer -I <tree> repro.c <tree>/libdb.a  -luring -lpthread -ldl -o repro
$ mkdir trigger && cd trigger && LC_ALL=C timeout 120s ../repro read
VERSION Berkeley DB 5.3.34: (August 3, 2026)
MODE read cache=default accounts=512 cycles=1000 checkpoint_interval=100
PRELOADED in_use=507 txn_mvcc=11
CYCLE 0 in_use=509 txn_mvcc=9
CYCLE 1 in_use=508 txn_mvcc=6
CYCLE 2 in_use=508 txn_mvcc=5
CYCLE 99 in_use=712 txn_mvcc=103
CYCLE 199 in_use=922 txn_mvcc=203
CYCLE 299 in_use=1134 txn_mvcc=303
CYCLE 399 in_use=1347 txn_mvcc=403
CYCLE 499 in_use=1557 txn_mvcc=503
CYCLE 599 in_use=1771 txn_mvcc=602
BDB2034 unable to allocate memory for mutex; resize mutex region
ENOMEM operation=DB->put(entry) cycle=643 ret=12 name=Cannot allocate memory
RESULT mode=read cycles_completed=643 enomem=1 in_use=507->1861 txn_mvcc=11->645
CLOSED ret=0 name=DB_SUCCESS
STORED after_reopen key=entry-0 value=x*256 bytes=256 ret=0 name=DB_SUCCESS
EXIT status=0
PROCESS_STATUS mode=read status=0

$ mkdir control && cd control && LC_ALL=C timeout 120s ../repro no-read
VERSION Berkeley DB 5.3.34: (August 3, 2026)
MODE no-read cache=default accounts=512 cycles=1000 checkpoint_interval=100
PRELOADED in_use=507 txn_mvcc=11
CYCLE 0 in_use=507 txn_mvcc=8
CYCLE 1 in_use=505 txn_mvcc=5
CYCLE 2 in_use=503 txn_mvcc=3
CYCLE 99 in_use=508 txn_mvcc=3
CYCLE 199 in_use=518 txn_mvcc=3
CYCLE 299 in_use=530 txn_mvcc=3
CYCLE 399 in_use=543 txn_mvcc=3
CYCLE 499 in_use=554 txn_mvcc=3
CYCLE 599 in_use=570 txn_mvcc=3
CYCLE 699 in_use=583 txn_mvcc=3
CYCLE 799 in_use=597 txn_mvcc=3
CYCLE 899 in_use=607 txn_mvcc=3
CYCLE 999 in_use=615 txn_mvcc=3
RESULT mode=no-read cycles_completed=1000 enomem=0 in_use=507->615 txn_mvcc=11->3
CLOSED ret=0 name=DB_SUCCESS
STORED after_reopen key=entry-0 value=x*256 bytes=256 ret=0 name=DB_SUCCESS
EXIT status=0
PROCESS_STATUS mode=no-read status=0
VALIDATION status=PASS run_id=issue87.0uP9wr target_commit=c4811dc871e313033993e95baa5b6525057c5911 trigger_txn_mvcc=11->645 control_txn_mvcc=11->3

Full asan log

RUN_ID=issue87.0uP9wr
TREE=asan
TARGET_COMMIT=c4811dc871e313033993e95baa5b6525057c5911
PROGRAM_SHA256=1ff0a42e5c3be1941828c6de32705341e94d842bec4de24852cd89c7095c35c7
CONFIGURE_FLAGS=--enable-debug 'CFLAGS=-fsanitize=address -fno-omit-frame-pointer -g -O1' LDFLAGS=-fsanitize=address
LIBDB_ARCHIVE_SHA256=dc4086b8d0e804d3270d41d34efc5a9bb9579719c684953b73a4a9f92ea3a16f
OS=Linux
KERNEL=7.0.0-30-generic
ARCH=x86_64
COMPILER=cc (Ubuntu 15.2.0-16ubuntu1) 15.2.0
API_SCOPE=public db.h; two DB_BTREE handles; DB_TXN_SNAPSHOT
ENV_FLAGS=DB_CREATE|DB_INIT_LOCK|DB_INIT_LOG|DB_INIT_MPOOL|DB_INIT_TXN|DB_RECOVER
DB_FLAGS=DB_CREATE|DB_AUTO_COMMIT|DB_MULTIVERSION
MUTEX_MAX_OVERRIDE=none
SANITIZER=address; RUNTIME_LEAK_DETECTION=disabled; REASON=measured resource is a mutex-region slot, not a heap allocation
$ cc -g -O0 -fno-omit-frame-pointer -I <tree> repro.c <tree>/libdb.a -fsanitize=address -luring -lpthread -ldl -o repro
$ mkdir trigger && cd trigger && LC_ALL=C ASAN_OPTIONS=detect_leaks=0 timeout 120s ../repro read
VERSION Berkeley DB 5.3.34: (August 3, 2026)
MODE read cache=default accounts=512 cycles=1000 checkpoint_interval=100
PRELOADED in_use=506 txn_mvcc=10
CYCLE 0 in_use=506 txn_mvcc=5
CYCLE 1 in_use=509 txn_mvcc=6
CYCLE 2 in_use=511 txn_mvcc=7
CYCLE 99 in_use=742 txn_mvcc=103
CYCLE 199 in_use=960 txn_mvcc=203
CYCLE 299 in_use=1173 txn_mvcc=303
CYCLE 399 in_use=1396 txn_mvcc=403
CYCLE 499 in_use=1616 txn_mvcc=503
CYCLE 599 in_use=1829 txn_mvcc=602
BDB2034 unable to allocate memory for mutex; resize mutex region
ENOMEM operation=DB->put(entry) cycle=616 ret=12 name=Cannot allocate memory
RESULT mode=read cycles_completed=616 enomem=1 in_use=506->1861 txn_mvcc=10->618
CLOSED ret=0 name=DB_SUCCESS
STORED after_reopen key=entry-0 value=x*256 bytes=256 ret=0 name=DB_SUCCESS
EXIT status=0
PROCESS_STATUS mode=read status=0

$ mkdir control && cd control && LC_ALL=C ASAN_OPTIONS=detect_leaks=0 timeout 120s ../repro no-read
VERSION Berkeley DB 5.3.34: (August 3, 2026)
MODE no-read cache=default accounts=512 cycles=1000 checkpoint_interval=100
PRELOADED in_use=506 txn_mvcc=10
CYCLE 0 in_use=504 txn_mvcc=4
CYCLE 1 in_use=506 txn_mvcc=5
CYCLE 2 in_use=506 txn_mvcc=5
CYCLE 99 in_use=531 txn_mvcc=3
CYCLE 199 in_use=549 txn_mvcc=3
CYCLE 299 in_use=559 txn_mvcc=3
CYCLE 399 in_use=580 txn_mvcc=3
CYCLE 499 in_use=599 txn_mvcc=3
CYCLE 599 in_use=621 txn_mvcc=3
CYCLE 699 in_use=630 txn_mvcc=3
CYCLE 799 in_use=630 txn_mvcc=3
CYCLE 899 in_use=630 txn_mvcc=3
CYCLE 999 in_use=630 txn_mvcc=3
RESULT mode=no-read cycles_completed=1000 enomem=0 in_use=506->630 txn_mvcc=10->3
CLOSED ret=0 name=DB_SUCCESS
STORED after_reopen key=entry-0 value=x*256 bytes=256 ret=0 name=DB_SUCCESS
EXIT status=0
PROCESS_STATUS mode=no-read status=0
VALIDATION status=PASS run_id=issue87.0uP9wr target_commit=c4811dc871e313033993e95baa5b6525057c5911 trigger_txn_mvcc=10->618 control_txn_mvcc=10->3

Analysis

Root cause

1. __txn_begin maps DB_TXN_SNAPSHOT to the internal TXN_SNAPSHOT_SAFE state (src/txn/txn.c, lines 233-252). A multiversion read then selects a SIREAD marker (src/db/db_meta.c, lines 1178-1194). When it grants the marker, __lock_get_internal increments td->si_ref (src/lock/lock.c, lines 1212-1229).

2. The first multiversion update calls __memp_fget, which allocates td->mvcc_mtx (src/mp/mp_fget.c, lines 244-265). Buffer ownership calls __txn_add_buffer, which increments td->mvcc_ref (src/mp/mp_mvcc.c, lines 24-50; src/txn/txn_region.c, lines 493-502). At transaction end, __txn_end sees a nonzero mvcc_ref, puts the detail on the committed-snapshot list, and leaves its mutex allocated (src/txn/txn.c, lines 1822-1871). This is required for the leak: if mvcc_ref were already zero, __txn_end would release the mutex before retaining a SIREAD-only detail.

3. If the last buffer leaves first, __txn_remove_buffer decrements td->mvcc_ref to zero while td->si_ref remains positive. It does not finalize the detail on that path (src/txn/txn_region.c, lines 529-545).

4. A checkpoint or a later snapshot transaction can call __lock_sicleanup (src/txn/txn_chkpt.c, lines 151-157; src/txn/txn.c, lines 264-279). Cleanup removes eligible markers and decrements td->si_ref (src/lock/lock.c, lines 99-173). After it releases the object locks, it calls __txn_reap_si_details (src/lock/lock.c, lines 196-228).

5. When both reference counts are zero, __txn_reap_si_details unlinks and frees the detail. It does not call __mutex_free for td->mvcc_mtx (src/txn/txn_region.c, lines 447-484). By contrast, __txn_remove_buffer calls __mutex_free before it frees the detail when it owns finalization (src/txn/txn_region.c, lines 541-552).

6. __mutex_free_int returns a mutex slot to the free list and decreases st_mutex_inuse (src/mutex/mut_alloc.c, lines 220-262). Because the reaper omits this call, each affected detail leaves its slot in use. Public statistics label MTX_TXN_MVCC as txn mvcc (src/mutex/mut_stat.c, lines 493-498). When the free list is empty and the shared region cannot extend, the next mutex allocation reports BDB2034 and returns an allocation error (src/mutex/mut_alloc.c, lines 94-132).

Impact and scope

The failing DB->put(entry) cannot complete. The C documentation says that applications "should always treat ENOMEM as a fatal error" (docs_src/guides/upgrading/upgrade_4_3_enomem.md, lines 8-12).

The affected order requires a DB_TXN_SNAPSHOT transaction to commit while an MVCC buffer and a SIREAD marker still refer to its detail. After commit, its last MVCC buffer must be removed while the marker remains, and marker cleanup must occur later.

[gburd]

Thanks — this is an exemplary report (precise line citations, trigger + control + variant, release and debug builds agreeing, deterministic repeats). We've confirmed it and it's being worked.

Status: I verified #138, #139 and #140 directly against the source, and #140 is being treated as the highest priority (heap out-of-bounds write in release builds, since the bounds DB_ASSERT is diagnostic-only). Fixes for all five are in flight on separate branches.

Two things your reports changed beyond the individual bugs:

1. We've retracted the overclaimed docs (docs(ssi): retract overclaimed serializability guarantees #141, merged): rfc/0003 no longer asserts the commit-time pivot check is "race-free" or that markers are "bounded", and DB_TXN_SNAPSHOT's API page now carries a known-limitation note so applications don't rely on serializability alone until the fixes land with tests. Shipping a guarantee we don't fully keep was the worst part of this.
2. We're fixing the root-cause class, not just libdb 5.3.34: incomplete replication commit lock lists can violate client transaction isolation #140. That bug exists because SSI added a new lock mode (DB_LOCK_SIREAD) without auditing the pre-existing sites that enumerate modes exhaustively. We're doing that audit and adding a CI check so the class can't silently return.

We're also adding the test tiers that should have caught these: an isolation/anomaly checker that validates committed state against some serial order (including landing a write while a peer is inside commit, which is what #136 needs), and a resource-accounting soak tier (which would have caught #137 and #138 mechanically). Your --control/--late methodology is being adopted as our own bug bar.

On #136 specifically: we also noted your separate observation that two records on different pages of one B-tree detect no conflict at all, even in the control. We're treating that as potentially the more significant half and testing it explicitly.