test: add isolation, resource-accounting and lock-mode test tiers

.github/workflows/test-tiers.yml

@@ -0,0 +1,180 @@
+# Isolation, resource-accounting and lock-mode test tiers.
+#
+# These three tiers close a structural blind spot: the existing suite is strong
+# on crash/durability (test/sim) and on memory safety against malformed input
+# (test/fuzz), but was blind to (a) isolation semantics as a checkable property
+# and (b) long-running resource accounting.  Five external bug reports
+# (#136-#140) landed on v5.3.34 through that gap.
+#
+#   B1  test/isolation/  serializability checker -- would have caught #136
+#   B2  test/soak/       resource-accounting soak -- would have caught #137/#138
+#   B3  test/lockmatrix/ lock-mode matrix under ASan -- would have caught #140
+#
+# Gating policy follows the house style:
+#   - B1 is a HARD GATE on every push/PR.  It is fast (~1 min) and its
+#     expectations are self-correcting: the checker fails both when a new
+#     serializability violation appears AND when a scenario marked as
+#     reproducing a known issue stops violating, so a fix cannot land without
+#     updating the expectation.
+#   - B3 is advisory (continue-on-error) for now, because on current master it
+#     legitimately aborts under ASan: that IS the #140 reproduction.  Flip it to
+#     a hard gate in the same PR that fixes #140.
+#   - B2 (soak) is scheduled/nightly plus manual dispatch: it is a long run and
+#     the same expectation-flip applies once #137/#138 land.
+
+name: Test tiers (isolation / soak / lock matrix)
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+    paths:
+      - 'test/isolation/**'
+      - 'test/soak/**'
+      - 'test/lockmatrix/**'
+      - 'src/lock/**'
+      - 'src/txn/**'
+      - 'src/mp/**'
+      - 'dist/**'
+      - '.github/workflows/test-tiers.yml'
+  schedule:
+    # Nightly (04:41 UTC), offset from ci.yml's 03:17 so the runners do not
+    # contend.  This is when the soak tier runs.
+    - cron: '41 4 * * *'
+  workflow_dispatch:
+    inputs:
+      soak_n:
+        description: 'Transactions per soak workload'
+        default: '5000'
+
+concurrency:
+  group: test-tiers-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  # --------------------------------------------------------------------------
+  # Tier B1 -- isolation / anomaly checker.  Hard gate: fast and deterministic
+  # enough to run per push.
+  # --------------------------------------------------------------------------
+  isolation:
+    name: B1 isolation/anomaly checker
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install build deps
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y liburing-dev
+
+      - name: Build libdb (debug, so DB_ASSERTs are live)
+        working-directory: build_unix
+        run: |
+          ../dist/configure --enable-debug
+          make -j"$(nproc)"
+
+      - name: Run the anomaly scenarios
+        working-directory: test/isolation
+        run: ISO_TIMEOUT=600 ./run.sh
+
+      - name: Upload scenario databases on failure
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: isolation-artifacts
+          path: test/isolation/build/ISODIR.*/**
+          if-no-files-found: ignore
+
+  # --------------------------------------------------------------------------
+  # Tier B3 -- lock-mode matrix under ASan.  Advisory until #140 lands, because
+  # the ASan heap-buffer-overflow it finds in __lock_vec IS the bug report.
+  # --------------------------------------------------------------------------
+  lock-matrix:
+    name: B3 lock-mode matrix (ASan)
+    runs-on: ubuntu-latest
+    continue-on-error: true   # advisory until #140 is fixed
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install clang + build deps
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y clang llvm liburing-dev
+          clang --version
+
+      - name: Run the lock-mode matrix (builds an ASan libdb once)
+        working-directory: test/lockmatrix
+        run: |
+          # Pipe-to-tee would mask the driver's exit status behind tee's, and
+          # an ASan abort is exactly the signal we must not lose.  Capture the
+          # status explicitly and re-raise it after the summary step has run.
+          set -o pipefail
+          CC=clang LOCK_TIMEOUT=900 ./run.sh 2>&1 | tee matrix.log
+
+      - name: Summarise
+        if: always()
+        working-directory: test/lockmatrix
+        run: |
+          if grep -q 'AddressSanitizer' matrix.log; then
+            echo "::warning::Tier B3 reproduced an ASan fault in the lock list"
+            echo "Last shape attempted before the fault:"
+            grep -E '^    (PUT_READ|UPGRADE_WRITE)' matrix.log | tail -1
+            grep -m1 'SUMMARY: AddressSanitizer' matrix.log || true
+          fi
+
+      - name: Upload matrix log
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: lock-matrix-log
+          path: test/lockmatrix/matrix.log
+          if-no-files-found: ignore
+
+  # --------------------------------------------------------------------------
+  # Tier B2 -- resource-accounting soak.  Scheduled/manual only: it is a long
+  # run (thousands of sequential transactions per workload).
+  # --------------------------------------------------------------------------
+  soak:
+    name: B2 resource-accounting soak
+    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    continue-on-error: true   # advisory until #137/#138 are fixed
+    timeout-minutes: 60
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install build deps
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y liburing-dev
+
+      - name: Build libdb (debug)
+        working-directory: build_unix
+        run: |
+          ../dist/configure --enable-debug
+          make -j"$(nproc)"
+
+      - name: Soak
+        working-directory: test/soak
+        run: |
+          set -o pipefail   # do not let tee mask a failing soak
+          SOAK_N="${{ github.event.inputs.soak_n || 5000 }}" \
+          SOAK_TIMEOUT=2400 ./run.sh 2>&1 | tee soak.log
+
+      - name: Summarise growth
+        if: always()
+        working-directory: test/soak
+        run: |
+          echo "== slopes that exceeded tolerance =="
+          grep -E 'GROWING|ENOMEM' soak.log || echo "(none)"
+
+      - name: Upload growth curves
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: soak-growth-curves
+          path: test/soak/soak.log
+          if-no-files-found: ignore

.gitignore

@@ -8,6 +8,17 @@ test/tcl/tclIndex
 TESTDIR_sim_*/
 test/sim/TESTDIR_sim_*/
 
+# Test-tier build dirs and scratch env dirs (test/isolation, test/soak,
+# test/lockmatrix -- each run.sh builds into ./build and each driver creates
+# its scratch environments beside the binary)
+test/isolation/build/
+test/soak/build/
+test/lockmatrix/build/
+ISODIR.*/
+SOAKDIR.*/
+LOCKDIR/
+build_dst/**
+
 # Meson/Ninja out-of-tree build dirs
 build-meson/
 build/

dist/Makefile.in

@@ -1916,6 +1916,39 @@ fi_sweep: fi_sweep@o@ $(DEF_LIB)
 
 fi_tests: fi_sweep
 
+##################################################
+# Isolation / soak / lock-matrix test tiers -- test/isolation/, test/soak/,
+# test/lockmatrix/.
+#
+# These need no library-side hooks at all (they drive the public API only), so
+# unlike the DST and faultinject tiers there is no configure option and no
+# ADDITIONAL_OBJS.  `make tier_tests` builds all three drivers.  Each tier also
+# has a run.sh that builds standalone against an existing build_unix; see the
+# README.md in each directory.
+##################################################
+test_iso_anomaly@o@: $(testdir)/isolation/test_iso_anomaly.c
+	$(CC) $(CFLAGS) $(DEPFLAGS) $<
+test_iso_anomaly: test_iso_anomaly@o@ $(DEF_LIB)
+	$(CCLINK) -o $@ \
+	    $(LDFLAGS) test_iso_anomaly@o@ $(DEF_LIB) $(TEST_LIBS) $(LIBS)
+	$(POSTLINK) $@
+
+test_soak_resources@o@: $(testdir)/soak/test_soak_resources.c
+	$(CC) $(CFLAGS) $(DEPFLAGS) $<
+test_soak_resources: test_soak_resources@o@ $(DEF_LIB)
+	$(CCLINK) -o $@ \
+	    $(LDFLAGS) test_soak_resources@o@ $(DEF_LIB) $(TEST_LIBS) $(LIBS)
+	$(POSTLINK) $@
+
+test_lock_matrix@o@: $(testdir)/lockmatrix/test_lock_matrix.c
+	$(CC) $(CFLAGS) $(DEPFLAGS) $<
+test_lock_matrix: test_lock_matrix@o@ $(DEF_LIB)
+	$(CCLINK) -o $@ \
+	    $(LDFLAGS) test_lock_matrix@o@ $(DEF_LIB) $(TEST_LIBS) $(LIBS)
+	$(POSTLINK) $@
+
+tier_tests: test_iso_anomaly test_soak_resources test_lock_matrix
+
 ##################################################
 # Targets for example programs.
 ##################################################

meson.build

@@ -9,6 +9,7 @@
 #   ninja -C build           # parallel build -> libdb
 #   ninja -C build docs      # render docs_src/ -> docs-build/
 #   ninja -C build bench     # build the test/bench microbenchmark drivers
+#   meson test -C build --suite tiers   # isolation + lock-matrix tiers
 
 project('libdb', 'c',
   version: '5.3.34',
@@ -34,3 +35,8 @@ subdir('dist')
 if not get_option('hegel').disabled()
   subdir('test/pbt')
 endif
+
+# Isolation / soak / lock-matrix test tiers (B1/B2/B3) -- test/tiers. Always
+# entered: the drivers are build_by_default:false so a plain `ninja` is
+# unaffected, and they need no library-side hooks (public API only).
+subdir('test/tiers')

test/isolation/README.md

@@ -0,0 +1,117 @@
+# Tier B1 — isolation / anomaly checker
+
+Runs concurrent transaction schedules under `DB_TXN_SNAPSHOT` (which in this
+fork means serializable snapshot isolation) and checks the **committed** result
+against some serial order of the committed transactions.
+
+## Why this tier exists
+
+The existing suite is strong on crash/durability (`test/sim`, 41 DST
+scenarios) and on memory safety against malformed input (`test/fuzz`). Neither
+can see a write skew that *commits successfully*: nothing crashes, no page is
+corrupt, no sanitizer fires — the database just holds a state that no serial
+execution could have produced. Issue #136 is exactly that shape, and it
+shipped in v5.3.34.
+
+## The verdict is computed, not hard-coded
+
+Each scenario declares, per transaction, a `model` function: the transaction's
+semantics as a pure function over an abstract state vector. After the schedule
+runs, the harness reads the real state back out of the databases (after closing
+and reopening the environment, so the verdict is about the **durable** state),
+then enumerates every permutation of the transactions that actually committed
+and applies their models serially. If no permutation reproduces the observed
+state, the history is not serializable and the scenario fails with the schedule
+printed.
+
+The state vector has two kinds of slot:
+
+- **record slots** — one per database record; the observed value is read back
+  from the database.
+- **observation slots** — what a transaction claims it *read*. These are what
+  make the read-only anomaly checkable: there the stored state is perfectly
+  fine and only the read-only transaction's observation has no serial
+  explanation. Observation slots of a transaction that did not commit are
+  ignored.
+
+## Scenarios
+
+| Scenario | Shape | Expectation on master |
+|---|---|---|
+| `write_skew_trigger` | two one-page DBs; T2's write lands while T1 is inside `commit` | **XFAIL — reproduces #136** |
+| `write_skew_control` | two one-page DBs; T2 writes and commits before T1 commits | PASS (`DB_SNAPSHOT_CONFLICT` to T1) |
+| `write_skew_late` | two one-page DBs; T2 writes after T1's commit returned | PASS (`DB_SNAPSHOT_UNSAFE` to T2) |
+| `write_skew_samebtree_control` | two records on **different pages of one** B-tree; control timing | PASS |
+| `write_skew_samebtree_trigger` | same, trigger timing | **XFAIL — reproduces #136** |
+| `g2_antidep` | G2-item: both txns scan for markers, both insert one | PASS |
+| `read_only_anomaly` | Fekete's 3-txn pattern; the read-only txn's observation is checked | PASS |
+| `lost_update` | both txns read the counter and write read+1 | PASS |
+| `read_your_writes` | sanity: a txn must observe its own uncommitted write | PASS |
+
+`read_your_writes` exists so a *vacuously* passing checker is detectable: if
+the harness ever stops driving the engine, that scenario fails.
+
+### On the "separate defect" reported alongside #136
+
+The #136 reporter suspected a second, independent defect: two records on
+different pages of one B-tree detecting no conflict at all, *even in the
+control*. That does **not** reproduce here.
+`write_skew_samebtree_control` builds the shape explicitly (512-byte pages plus
+filler keys sorting between `alice` and `bob`, giving 33 leaf pages with
+`alice` as the minimum key and `bob` as the maximum, verified via
+`DB->stat`→`bt_leaf_pg`) and the control correctly returns
+`DB_SNAPSHOT_CONFLICT`. Only the trigger timing commits both. On this
+construction the different-pages case has the **same** root cause as #136
+proper (the commit-window race), not an extra page-granularity hole. The
+reporter did not publish their same-btree variant, so their shape may differ;
+the control is kept as a live PASS expectation precisely so a real
+page-granularity regression would surface here.
+
+## How the #136 interleaving is reached — no engine hook
+
+Landing T2's write while T1 is **inside** `DB_TXN->commit` is done entirely from
+the application side: `pthread_barrier` for the ordered phases, plus an atomic
+flag that T1 sets immediately before entering `commit` and T2 spins on. This is
+the reporter's own technique. **No engine change, no `HAVE_DST` site, zero
+production overhead.** A test-only yield point in the commit path was
+considered and not needed.
+
+That window is genuinely racy — T1's commit can finish before T2's put reaches
+the conflict check, degenerating into the benign "late" schedule. So the racy
+scenarios run multiple attempts (40 by default) and the rule is asymmetric on
+purpose: **one** violation in any attempt is a reproduction, while a pass
+requires **every** attempt to be clean. A serializability violation is a real
+counterexample; a single clean run of a racy schedule proves nothing.
+
+In practice both #136 shapes violate on the first attempt.
+
+## Running it
+
+```sh
+# Build libdb first (once):
+cd build_unix && ../dist/configure --enable-debug && make -j"$(nproc)"
+
+# All scenarios:
+cd test/isolation && ./run.sh
+
+# One scenario, with the btree-shape diagnostics:
+ISO_VERBOSE=1 ./run.sh write_skew_samebtree_control
+
+./run.sh --list        # scenario names, with expect-fail marked
+./run.sh build         # build only
+```
+
+Environment: `CC`, `LIBDB_BUILD` (default `../../build_unix`), `ISO_TIMEOUT`
+(default 300s), `ISO_SAN=1` to add ASan.
+
+## Exit status
+
+- `0` — every scenario matched its recorded expectation.
+- `1` — a scenario did not. Either a new serializability violation, **or** an
+  expect-fail scenario that stopped violating, meaning the referenced issue got
+  fixed and `expect_fail` should be cleared in the table in
+  `test_iso_anomaly.c`. The message says which.
+- `2` — harness error.
+
+When #136 lands, clear `expect_fail` on `write_skew_trigger` and
+`write_skew_samebtree_trigger`; the tier then gates the fix against regression.