libdb 5.3.34: sequential read-only DB_TXN_SNAPSHOT transactions eventually cause DB_ENV->txn_begin to return ENOMEM

[xint-io]

Summary

libdb 5.3.34 has a Serializable Snapshot Isolation (SSI) resource-exhaustion defect in an environment kept open across transactions. SIREAD cleanup removes obsolete markers but does not reclaim the associated committed-reader lockers, so their logical mutexes remain allocated. Sequential read-only DB_TXN_SNAPSHOT transactions therefore accumulate these resources even when only one transaction is active at a time. The accumulation eventually exhausts the mutex region, and a later DB_ENV->txn_begin returns ENOMEM.

Environment

Item	Value
libdb	Berkeley DB 5.3.34: (August 3, 2026), source commit c4811dc871e313033993e95baa5b6525057c5911
Release build	release: CFLAGS=-O2 -g, no diagnostic instrumentation; source of the quoted output
Debug build	debug: --enable-debug --enable-diagnostic --enable-test --enable-compile-commands, CFLAGS=-g -O0 -fno-omit-frame-pointer; independently reproduces the trigger and control
Host	Ubuntu 26.04 LTS, Linux 7.0.0-30-generic, x86_64, gcc 15.2.0
Workload	B-tree, one record, one process, one thread, up to 2,000 sequential read-only transactions per mode
DB environment	DB_CREATE | DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN; set_lk_detect(DB_LOCK_DEFAULT)
Database and transactions	DB_CREATE | DB_AUTO_COMMIT | DB_MULTIVERSION; trigger flag DB_TXN_SNAPSHOT; control flag 0
Locker and mutex-region sizing	Default; no explicit locker limit, mutex limit, or environment memory maximum

Steps to reproduce

1. Get and build the tested source.

   git clone https://github.com/berkeleydb/libdb.git
   cd libdb
   git checkout c4811dc871e313033993e95baa5b6525057c5911
   mkdir build-issue
   cd build-issue
   CFLAGS='-O2 -g' ../dist/configure
   make -j12 libdb.a
   cd ..

2. Save this program as repro.c in the repository root.

/*
 * Reproduce unreclaimed committed-reader lockers in libdb 5.3.34.
 *
 * The trigger repeatedly reads one B-tree record in a DB_TXN_SNAPSHOT
 * transaction and commits.  The control uses a plain transaction; that flag
 * is the only behavioral difference.  Run each mode in a new directory.
 */
#include <db.h>

#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>

#define ATTEMPTS 2000
#define REPORT_EVERY 250

struct stats {
	u_int32_t lockers;
	u_int32_t locks;
};

static const char *
ret_name(int ret)
{
	if (ret == 0)
		return ("OK");
	if (ret == ENOMEM)
		return ("ENOMEM");
	return ("OTHER");
}

static int
read_stats(DB_ENV *env, struct stats *out)
{
	DB_LOCK_STAT *s;
	int ret;

	s = NULL;
	if ((ret = env->lock_stat(env, &s, 0)) != 0)
		return (ret);
	out->lockers = s->st_nlockers;
	out->locks = s->st_nlocks;
	free(s);
	return (0);
}

static void
print_stats(const char *label, int completed, const struct stats *s)
{
	printf("stats label=%s completed=%d lockers=%u locks=%u\n",
	    label, completed, s->lockers, s->locks);
}

static int
seed(DB_ENV *env, DB *db)
{
	DBT key, data;
	DB_TXN *txn;
	char k[] = "key", v[] = "value";
	int ret;

	if ((ret = env->txn_begin(env, NULL, &txn, 0)) != 0)
		return (ret);
	memset(&key, 0, sizeof(key));
	memset(&data, 0, sizeof(data));
	key.data = k;
	key.size = sizeof(k);
	data.data = v;
	data.size = sizeof(v);
	if ((ret = db->put(db, txn, &key, &data, 0)) != 0) {
		(void)txn->abort(txn);
		return (ret);
	}
	return (txn->commit(txn, 0));
}

int
main(int argc, char **argv)
{
	DB_ENV *env;
	DB *db;
	DB_TXN *txn;
	DBT key, data;
	struct stats baseline, final, previous, sample;
	char k[] = "key", value[32];
	const char *mode;
	u_int32_t txn_flags;
	int begin_ret, completed, control, db_close_ret, env_close_ret;
	int i, lock_drop_with_locker_growth, major, minor, operation_ret, patch;
	int ret;

	env = NULL;
	db = NULL;
	txn = NULL;
	if (argc != 2 || (strcmp(argv[1], "snapshot") != 0 &&
	    strcmp(argv[1], "control") != 0)) {
		fprintf(stderr, "usage: %s snapshot|control\n", argv[0]);
		return (2);
	}
	mode = argv[1];
	control = strcmp(mode, "control") == 0;
	txn_flags = control ? 0 : DB_TXN_SNAPSHOT;
	setvbuf(stdout, NULL, _IONBF, 0);

	printf("mode=%s\n", mode);
	printf("target_signature=%s\n", db_version(&major, &minor, &patch));
	if (major != 5 || minor != 3 || patch != 34)
		return (1);
	if (mkdir("DBHOME", 0700) != 0) {
		perror("mkdir DBHOME");
		return (1);
	}

	if ((ret = db_env_create(&env, 0)) != 0)
		goto setup_fail;
	env->set_errfile(env, stderr);
	env->set_errpfx(env, "libdb");
	if ((ret = env->set_lk_detect(env, DB_LOCK_DEFAULT)) != 0 ||
	    (ret = env->open(env, "DBHOME", DB_CREATE | DB_INIT_LOCK |
	    DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN, 0600)) != 0)
		goto setup_fail;
	if ((ret = db_create(&db, env, 0)) != 0 ||
	    (ret = db->open(db, NULL, "data.db", NULL, DB_BTREE,
	    DB_CREATE | DB_AUTO_COMMIT | DB_MULTIVERSION, 0600)) != 0 ||
	    (ret = seed(env, db)) != 0)
		goto setup_fail;
	if ((ret = read_stats(env, &baseline)) != 0)
		goto setup_fail;
	print_stats("baseline", 0, &baseline);

	completed = 0;
	previous = baseline;
	lock_drop_with_locker_growth = 0;
	begin_ret = 0;
	operation_ret = 0;
	for (i = 0; i < ATTEMPTS; i++) {
		txn = NULL;
		begin_ret = env->txn_begin(env, NULL, &txn, txn_flags);
		if (begin_ret != 0)
			break;
		memset(&key, 0, sizeof(key));
		memset(&data, 0, sizeof(data));
		key.data = k;
		key.size = sizeof(k);
		data.data = value;
		data.ulen = sizeof(value);
		data.flags = DB_DBT_USERMEM;
		if ((ret = db->get(db, txn, &key, &data, 0)) != 0) {
			(void)txn->abort(txn);
			operation_ret = ret;
			break;
		}
		if ((ret = txn->commit(txn, 0)) != 0) {
			operation_ret = ret;
			break;
		}
		completed++;
		if (completed % REPORT_EVERY == 0) {
			if ((ret = read_stats(env, &sample)) != 0) {
				operation_ret = ret;
				break;
			}
			if (sample.locks < previous.locks &&
			    sample.lockers > previous.lockers)
				lock_drop_with_locker_growth = 1;
			previous = sample;
			print_stats("progress", completed, &sample);
		}
	}
	if ((ret = read_stats(env, &final)) != 0) {
		operation_ret = operation_ret == 0 ? ret : operation_ret;
		memset(&final, 0, sizeof(final));
	} else
		print_stats("final", completed, &final);
	printf("lock_drop_with_locker_growth=%d\n",
	    lock_drop_with_locker_growth);
	printf("begin_ret=%d begin_name=%s begin_message=%s completed=%d\n",
	    begin_ret, ret_name(begin_ret),
	    begin_ret == 0 ? "none" : db_strerror(begin_ret), completed);
	printf("operation_ret=%d operation_name=%s operation_message=%s\n",
	    operation_ret, ret_name(operation_ret),
	    operation_ret == 0 ? "none" : db_strerror(operation_ret));

	db_close_ret = db->close(db, 0);
	db = NULL;
	env_close_ret = env->close(env, 0);
	env = NULL;
	printf("close db_ret=%d env_ret=%d\n", db_close_ret, env_close_ret);
	if (db_close_ret != 0 || env_close_ret != 0)
		return (1);

	if (control) {
		if (completed != ATTEMPTS || begin_ret != 0 || operation_ret != 0 ||
		    final.lockers != baseline.lockers ||
		    lock_drop_with_locker_growth != 0)
			return (1);
		printf("RESULT mode=%s outcome=CONTROL_OK\n", mode);
	} else {
		if (completed <= 0 || completed >= ATTEMPTS ||
		    begin_ret != ENOMEM || operation_ret != 0 ||
		    lock_drop_with_locker_growth != 1 ||
		    final.lockers <= baseline.lockers ||
		    final.lockers <= final.locks)
			return (1);
		printf("RESULT mode=%s outcome=BUG_REPRODUCED\n", mode);
	}
	return (0);

setup_fail:
	fprintf(stderr, "setup: %s (%d)\n", db_strerror(ret), ret);
	if (db != NULL)
		(void)db->close(db, 0);
	if (env != NULL)
		(void)env->close(env, 0);
	return (1);
}

3. Compile the program against the release build and its generated linker settings.

   cc -std=gnu99 -Wall -Wextra -O2 -g -Ibuild-issue \
     repro.c build-issue/libdb.a \
     $(sed -n 's/^LDFLAGS=[[:space:]]*//p' build-issue/Makefile | head -1) \
     $(sed -n 's/^LIBS=[[:space:]]*//p' build-issue/Makefile | head -1) \
     -ldl -pthread -o repro

4. Run the trigger and control in separate new directories.

   repro_path=$PWD/repro
   run_root=$(mktemp -d)
   (
     cd "$run_root" &&
     mkdir snapshot control &&
     (cd snapshot && "$repro_path" snapshot) &&
     (cd control && "$repro_path" control)
   )

The control changes only the transaction flag from DB_TXN_SNAPSHOT to 0. Neither mode calls a checkpoint or changes the default locker or mutex-region sizing.

Expected result

SSI intentionally retains a committed reader's SIREAD markers while active transactions can still use them for conflict detection. The lifecycle design gives this release rule for the associated locker (M4-commit-lifecycle.md, lines 33-40):

When a locker's markers reach zero and it is DB_LOCKER_FREED, free it (deferred to after the partition mutex is released).

Cleanup can occur after commit, so the locker count does not have to return to its baseline after every commit. When cleanup removes a locker's last marker, it must make the locker and its logical mutex reusable. Both 2,000-transaction runs must complete without ENOMEM.

Actual result

Between the 750- and 1,000-transaction samples, the current lock count decreases from 751 to 200 while the current locker count increases from 752 to 1,002. After 1,409 snapshot transactions complete, the next DB_ENV->txn_begin prints the mutex-region diagnostic and returns ENOMEM:

stats label=baseline completed=0 lockers=2 locks=1
stats label=progress completed=750 lockers=752 locks=751
stats label=progress completed=1000 lockers=1002 locks=200
libdb: BDB2034 unable to allocate memory for mutex; resize mutex region
stats label=final completed=1409 lockers=1411 locks=609
lock_drop_with_locker_growth=1
begin_ret=12 begin_name=ENOMEM begin_message=Cannot allocate memory completed=1409
operation_ret=0 operation_name=OK operation_message=none
close db_ret=0 env_ret=0
RESULT mode=snapshot outcome=BUG_REPRODUCED

The control completes all 2,000 transactions and ends with its initial locker count of 2:

stats label=final completed=2000 lockers=2 locks=1
lock_drop_with_locker_growth=0
begin_ret=0 begin_name=OK begin_message=none completed=2000
operation_ret=0 operation_name=OK operation_message=none
close db_ret=0 env_ret=0
RESULT mode=control outcome=CONTROL_OK

The debug trigger also stops after 1,409 transactions with ENOMEM, and its control completes 2,000 transactions without an API error. The stopping point depends on the configuration; 1,409 is not a portable transaction threshold.

Analysis

Root cause

At commit, __lock_sicommit detaches persistent SIREAD markers from the locker's heldby list and sets DB_LOCKER_FREED (src/lock/lock.c, lines 274-305). The flag records that locker reclamation was deferred. At transaction end, __lock_freelocker_int reads si_ref, the number of SIREAD markers that still reference the owning transaction detail. It defers locker reclamation while this count is nonzero (src/txn/txn.c, lines 1776-1802, src/lock/lock_id.c, lines 498-520). This deferral is required while the markers remain.

Later, __lock_siclean_obj removes each obsolete marker. It decrements the owning transaction detail's si_ref and the locker's nlocks (src/lock/lock.c, lines 119-170). After the sweep, __lock_sicleanup reclaims eligible transaction details but does not call the locker-reclamation path (src/lock/lock.c, lines 185-228, src/txn/txn_region.c, lines 448-483, src/lock/lock_id.c, lines 528-539). The cleanup path therefore leaves the DB_LOCKER_FREED locker allocated after its last marker is gone. The proactive-cleanup call site also notes that committed-reader lockers are not reclaimed (src/txn/txn.c, lines 250-278).

The empty locker and its logical mutex therefore remain allocated. Creating a new locker allocates another logical mutex (src/lock/lock_id.c, lines 305-316, src/lock/lock_id.c, lines 369-382). In the release run, __mutex_alloc_int, which __mutex_alloc calls, reaches its no-memory branch, emits BDB2034, and returns ENOMEM (src/mutex/mut_alloc.c, lines 20-55, src/mutex/mut_alloc.c, lines 94-132).

Impact and scope

An application that keeps one environment open and repeatedly commits read-only snapshot transactions can exhaust the default mutex region. A later snapshot transaction then cannot start because DB_ENV->txn_begin returns ENOMEM. This path uses legal public APIs and only one active transaction at a time. It does not require a forced checkpoint, an injected fault, or a concurrent schedule.

[gburd]

Thanks — this is an exemplary report (precise line citations, trigger + control + variant, release and debug builds agreeing, deterministic repeats). We've confirmed it and it's being worked.

Status: I verified #138, #139 and #140 directly against the source, and #140 is being treated as the highest priority (heap out-of-bounds write in release builds, since the bounds DB_ASSERT is diagnostic-only). Fixes for all five are in flight on separate branches.

Two things your reports changed beyond the individual bugs:

1. We've retracted the overclaimed docs (docs(ssi): retract overclaimed serializability guarantees #141, merged): rfc/0003 no longer asserts the commit-time pivot check is "race-free" or that markers are "bounded", and DB_TXN_SNAPSHOT's API page now carries a known-limitation note so applications don't rely on serializability alone until the fixes land with tests. Shipping a guarantee we don't fully keep was the worst part of this.
2. We're fixing the root-cause class, not just libdb 5.3.34: incomplete replication commit lock lists can violate client transaction isolation #140. That bug exists because SSI added a new lock mode (DB_LOCK_SIREAD) without auditing the pre-existing sites that enumerate modes exhaustively. We're doing that audit and adding a CI check so the class can't silently return.

We're also adding the test tiers that should have caught these: an isolation/anomaly checker that validates committed state against some serial order (including landing a write while a peer is inside commit, which is what #136 needs), and a resource-accounting soak tier (which would have caught #137 and #138 mechanically). Your --control/--late methodology is being adopted as our own bug bar.

On #136 specifically: we also noted your separate observation that two records on different pages of one B-tree detect no conflict at all, even in the control. We're treating that as potentially the more significant half and testing it explicitly.