Lazy secondary index fd release on startup

AGENTS.md

@@ -20,6 +20,7 @@ Always try to use available tools through the MCP servers, which should provide
 - **Pre-compile over per-call dispatch**: if a descriptor or configuration object is known at setup time, compile it into closures or lookup structures once rather than re-interpreting it on every hot-path call. The gain compounds with call frequency (e.g. building operator closures once per matcher object instead of running `Object.entries` + switch per document).
 - **WeakMap for caching derived state on user objects**: prefer `WeakMap` over Symbol properties or plain properties to attach computed state keyed on user-supplied objects — avoids mutation, works on frozen objects, and is GC-friendly. On the hot path use `cache.get()` directly rather than `has()` + `get()` to avoid a double lookup.
 - **Pass cheap prefilter results as hints to expensive scans**: when a cheap pass (e.g. `Buffer.indexOf`) already locates a candidate position, carry that result forward as a hint to the costlier depth- or context-aware pass rather than letting it re-scan from the start. Eliminates a full O(n) scan on every call.
+- **Virtualize known lengths instead of pre-allocating sparse arrays**: when startup metadata already carries collection size (e.g. secondary index entry counts), keep an explicit length field and lazily fill cache entries on demand. Avoids large sparse-array allocations and reduces startup heap pressure.
 - **Custom implementations only when benchmarks justify the complexity**: a hand-rolled solution may edge out a standard library call by 10–15 %, but the maintenance cost is rarely worth it unless the gain is substantial and measurable at realistic data sizes (e.g. a custom numeric parser vs. `JSON.parse` with try/catch).
 - **Prefer a flag on a shared helper over near-duplicate functions**: when two functions differ only in a single behavioral detail, add a boolean parameter to the shared function rather than maintaining two near-identical copies. Keep the flag's semantics explicit and limited to one axis of variation.
 - **Inline single-use hot-path helpers once the surrounding flow becomes simpler**: if a helper is only called from one place and its logic can be embedded while keeping the caller below the local complexity/return-count budget, prefer the inline version. Removes call indirection and often makes the hot path easier to read as straight-line pseudo-code.
@@ -101,7 +102,6 @@ No build step; source is plain ESM consumed directly. No linter configured.
 | `src/Clock.js`                          | Monotonic microsecond clock |
 | `src/IndexEntry.js`                     | Index record serialization |
 | `src/IndexMatcher.js`                   | O(1) discriminant-based matcher classification |
-| `src/PartitionPool.js`                  | LRU-evicting pool for open partition handles |
 | `src/Watcher.js` / `src/WatchesFile.js` | Ref-counting directory watcher and mixin |
 | `src/utils/fsUtil.js`                   | `ensureDirectory`, `scanForFiles` |
 | `src/utils/apiHelpers.js`               | API-shaping helpers for top-level and internal class APIs (argument normalization, stream-name mapping, predicate/raw/revision coercion) |

concepts/2026-07-25-storage-manifest.md

@@ -0,0 +1,131 @@
+# Storage startup manifest for fast secondary-index recovery
+
+## Problem
+
+Every time a `WritableStorage` instance opens, it must discover and restore all secondary
+indexes that were registered in the previous session.  Without a manifest the only way to
+do this is a full filesystem scan:
+
+1. **`scanFiles()`** — an O(N) `readdir` over the data/index directory.
+2. **Per-index file open** — for each `.index` file found, `openIndex()` opens the file,
+   reads its binary header (magic + metadata block) to reconstruct the matcher, reads the
+   file length to restore `index.data.length`, then closes the fd.
+
+With 5 000 secondary indexes this means at least 5 000 file-open/read/close cycles at
+startup, even before a single document is processed.  Startup time grows linearly with the
+number of indexes.
+
+## Solution: persist a manifest at graceful shutdown
+
+`WritableStorage` writes a single JSON manifest file to the index directory on every
+graceful `close()`.  On the next `open()`, if the manifest exists and its HMAC
+verifies, the storage restores all secondary indexes and partition registrations from the
+manifest **without any file I/O**, achieving nearly O(1) startup (bounded only by the time
+to read and parse the manifest).
+
+## Manifest format
+
+```
+.<storageName>.manifest.json
+```
+
+```json
+{
+  "version": 1,
+  "partitions": ["bench.0", "bench.1", "bench.100"],
+  "indexes": {
+    "stream-orders": {
+      "matcher": { "stream": "orders" },
+      "length": 4712,
+      "headerSize": 64
+    },
+    "audit-log": {
+      "matcher": "function(doc) { return doc.audit === true; }",
+      "length": 800,
+      "headerSize": 64
+    }
+  },
+  "hmac": "<sha256-hex of JSON.stringify({version, partitions, indexes})>"
+}
+```
+
+Fields per index entry:
+
+| Field        | Purpose |
+|---|---|
+| `matcher`    | Object matcher (stored as-is) or function matcher (serialised via `toString()`). |
+| `length`     | Number of entries currently persisted to the index file. |
+| `headerSize` | Byte offset of the first index entry in the file.  Allows reads to compute exact file positions without re-reading the binary header. |
+
+Partition entries are the relative file names (relative to `dataDirectory`) used when each
+partition was first registered, e.g. `"bench.0"`.
+
+## Security model
+
+A single HMAC (SHA-256, keyed with the existing `config.hmacSecret`) is computed over
+the entire manifest body (`version + partitions + indexes`).  This covers:
+
+- All matcher values, including serialised function matchers.
+- All partition file names.
+- The index lengths and header sizes.
+
+**Why one document-level HMAC is sufficient**
+
+When the manifest was written, all matchers were already trusted: they came from either
+user-supplied code in the same process, or from individual index-file metadata that was
+already HMAC-verified on a previous startup.  The manifest's document-level HMAC prevents
+an attacker who can write to the disk from injecting a malicious function matcher.
+
+If `hmacSecret` is the empty string (default), the HMAC is still computed and verified;
+it just uses an empty key, which gives the same protection level as the existing per-matcher
+HMAC approach with an empty secret.
+
+## Startup decision tree (WritableStorage.open)
+
+```
+open(callback)
+├── LOCK_RECLAIM mode?
+│   └── yes → full scan + torn-write repair (manifest ignored)
+└── no  → try loadManifest()
+    ├── manifest missing or HMAC invalid?
+    │   └── full scan (same as LOCK_RECLAIM, minus repair)
+    └── manifest valid → restoreFromManifest()
+        ├── register partitions (zero file I/O — creates objects, no open())
+        ├── for each index: create WritableIndex via manifestData option
+        │   (sets opened=true, data length, headerSize — no file open)
+        ├── register matchers in IndexMatcher
+        ├── emit 'index-created' for each index (EventStore picks up streams)
+        ├── set initialized = true
+        └── openIndexes() → open primary index, emit 'opened'
+```
+
+## Persistence timing and crash safety
+
+The manifest is written **only on graceful `close()`**, after all write buffers have
+been flushed to disk.  This makes it robust by design:
+
+- **No crash → manifest valid**: next startup takes the fast path.
+- **Crash before close**: manifest is missing or reflects the state of the *last* graceful
+  shutdown.  Next startup (with `LOCK_RECLAIM`) falls back to the full scan and
+  torn-write repair path, just as it did before this feature.  On the following graceful
+  `close()`, a fresh manifest is written.
+- **Manifest written then crash**: impossible — the process exits before or after
+  `close()` completes, never in the middle of `close()` after manifest write but before
+  other cleanup (the manifest write is the last step).
+
+Because recovery is not performance-critical and a full scan is expected in that case,
+there is no need for transactional or crash-safe manifest writes (e.g. write-then-rename).
+
+## Performance expectation
+
+| Startup path  | Complexity | Bottleneck |
+|---|---|---|
+| Manifest (happy path) | O(manifest file size) | JSON parse of ~70 bytes/index |
+| Full scan fallback    | O(N indexes × file open cost) | per-index file open + header read |
+
+For 5 000 indexes with simple object matchers the manifest file is roughly 300–400 KB.
+
+Latest local re-run (Node v24.18.0, same benchmark harness copied into dedicated worktrees for `main`, `PR #339`, and this branch) with **non-empty** secondary indexes:
+
+- **Scenario A (1 index per partition)** startup improves over `main` at higher counts (`5k: 279.8 → 236.0 ms`, `10k: 540.9 → 453.3 ms`, `20k: 971.7 → 902.5 ms`).
+- **Scenario B (100 partitions, growing indexes)** startup is mixed (`100: 8.0 → 7.2 ms`, `1k: 28.4 → 40.8 ms`, `5k: 157.5 → 145.7 ms`).

docs/performance.md

@@ -50,12 +50,11 @@ Read latency is dominated by the per-document partition seek and is largely flat
 
 ### Startup time
 
-Startup time scales linearly with the number of files on disk:
+On a clean shutdown path, startup uses the persisted manifest: partitions and secondary-index metadata are restored from one manifest read instead of opening every index file to read headers.
 
-- The store performs one `readdir` call over the data directory followed by one `open` per partition.
-- Registering secondary indexes (`openIndex` / `ensureIndex`) costs one `open` + header-read per index file.
+Secondary indexes are still registered at startup, but their file descriptors are released immediately and reopened lazily on first real index access. This keeps startup fd pressure and heap pressure lower when thousands of indexes exist.
 
-For large deployments consider lazy-loading secondary indexes on first access rather than opening all of them during startup.
+When startup enters crash-recovery mode (`LOCK_RECLAIM` after torn-write repair), the manifest fast path is bypassed and the store falls back to the full scan/rebuild path.
 
 ---
 
@@ -158,4 +157,3 @@ Interpretation:
 
 - When events are **already deserialized**, object matchers are the correct baseline and typically faster.
 - In workloads where data is still in **raw buffer form** (streaming), raw matchers avoid `JSON.parse` and can outperform a deserialize+object-match pipeline by roughly **~50%** in practice.
-

src/Index/ReadableIndex.js

@@ -67,6 +67,7 @@ class ReadableIndex extends events.EventEmitter {
     initialize(options) {
         /* @type Array<Entry> */
         this.data = [];
+        this.entryCount = 0;
         this.opened = false;
         this.fileMode = 'r';
         this.fileHandlePool = options.fileHandlePool || new FileHandlePool();
@@ -79,6 +80,9 @@ class ReadableIndex extends events.EventEmitter {
             this.metadata = Object.assign({entryClass: options.EntryClass.name, entrySize: options.EntryClass.size}, options.metadata);
         }
         this.headerSize = 0;
+        // When set, open() restores state from this object instead of reading from disk.
+        // Used by WritableStorage to skip per-index file I/O on manifest-based startup.
+        this.manifestData = options.manifestData || null;
     }
 
     /**
@@ -102,7 +106,7 @@ class ReadableIndex extends events.EventEmitter {
      * @returns {number}
      */
     get length() {
-        return this.data.length;
+        return this.entryCount;
     }
 
     /**
@@ -165,11 +169,22 @@ class ReadableIndex extends events.EventEmitter {
         if (this.opened) {
             return false;
         }
+
+        if (this.manifestData) {
+            this.opened = true;
+            this.readUntil = -1;
+            this.headerSize = this.manifestData.headerSize;
+            this.entryCount = this.manifestData.length;
+            this.manifestData = null;
+            return true;
+        }
+
         this.opened = this.getFileHandle() !== null;
 
         this.readUntil = -1;
 
         const length = this.readFileLength();
+        this.entryCount = length;
         if (length > 0) {
             this.data = new Array(length);
             // Read last item to get the index started
@@ -251,6 +266,7 @@ class ReadableIndex extends events.EventEmitter {
      */
     close() {
         this.data = [];
+        this.entryCount = 0;
         this.readUntil = -1;
         this.readBuffer.fill(0);
         this.fileHandlePool.evict(this, false);

src/Index/WritableIndex.js

@@ -118,7 +118,7 @@ class WritableIndex extends ReadableIndex {
      * @param {number} fd
      */
     onBeforeClose(fd) {
-        if (!this.writeBuffer) {
+        if (this.writeBufferCursor === 0) {
             return;
         }
         this.flush();
@@ -190,13 +190,15 @@ class WritableIndex extends ReadableIndex {
         assertEqual(entry.constructor.name, this.EntryClass.name, `Wrong entry object.`);
         assertEqual(entry.constructor.size, this.EntryClass.size, `Invalid entry size.`);
 
-        const dataLen = this.data.length;
-        assert(dataLen === 0 || this.data.at(-1).number < entry.number, 'Consistency error. Tried to add an index that should come before existing last entry.');
+        const dataLen = this.length;
+        const lastEntry = dataLen > 0 ? this.get(dataLen) : null;
+        assert(dataLen === 0 || lastEntry.number < entry.number, 'Consistency error. Tried to add an index that should come before existing last entry.');
 
         if (this.readUntil === dataLen - 1) {
             this.readUntil++;
         }
         this.data[dataLen] = entry;
+        this.entryCount = dataLen + 1;
 
         if (this.writeBufferCursor === 0) {
             this.flushTimeout = setTimeout(() => this.flush(), this.flushDelay);
@@ -232,8 +234,11 @@ class WritableIndex extends ReadableIndex {
             return;
         }
         fs.truncateSync(this.fileName, truncatePosition);
-        this.data.splice(after);
-        this.readUntil = Math.min(this.readUntil, after);
+        if (this.data.length > after) {
+            this.data.splice(after);
+        }
+        this.entryCount = after;
+        this.readUntil = Math.min(this.readUntil, after - 1);
     }
 }
 

src/Storage/ReadableStorage.js

@@ -170,7 +170,7 @@ class ReadableStorage extends events.EventEmitter {
     }
 
     /**
-     * Scan partitions and secondary index files; emit 'index-created' for each found index.
+     * Scan partitions and secondary index files; register each found index.
      * @param {function} done Called when both scans finish.
      */
     scanFiles(done) {
@@ -193,7 +193,7 @@ class ReadableStorage extends events.EventEmitter {
             const indexPattern = new RegExp(`^${escaped}\\.(.+)\\.index$`);
             scanForFiles(this.indexDirectory, indexPattern, (name) => {
                 if (!(name in this.secondaryIndexes)) {
-                    this.emit('index-created', name);
+                    this.registerFoundIndex(name);
                 }
             }, (indexErr) => {
                 // The directory could disappear between existsSync and readdir (e.g. test cleanup).
@@ -204,6 +204,16 @@ class ReadableStorage extends events.EventEmitter {
         });
     }
 
+    /**
+     * Register a secondary index discovered while scanning the index directory.
+     *
+     * @protected
+     * @param {string} name
+     */
+    registerFoundIndex(name) {
+        this.emit('index-created', name);
+    }
+
     /**
      * Only the primary index is opened eagerly; secondary indexes open on demand.
      *
@@ -440,10 +450,20 @@ class ReadableStorage extends events.EventEmitter {
         // Register the actual stored matcher (may have been reconstructed from metadata by WritableStorage.createIndex).
         this.indexMatcher.add(name, this.secondaryIndexes[name].matcher);
 
-        index.open();
+        this.afterRegisterSecondaryIndex(index);
         return index;
     }
 
+    /**
+     * Called after a secondary index has been opened and registered via openIndex().
+     * No-op in the base class; WritableStorage overrides this to check for stale entries.
+     *
+     * @protected
+     * @param {ReadableIndex} index
+     */
+    afterRegisterSecondaryIndex(index) { // eslint-disable-line no-unused-vars
+    }
+
     /**
      * Remove a secondary index from the write path and the matcher lookup table.
      *