diff --git a/CLAUDE.md b/CLAUDE.md
index 27efb0c..f3fbf8a 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -21,13 +21,20 @@ Cargo workspace (`resolver = "2"`), four crates under `crates/`:
| Crate | Role |
|-------|------|
-| `ltk_hashdb` | The `.hashdb` format: `mmap` reader (`HashDb`) + streaming writer (`HashDbWriter`), `ExtendedHashDb` overlay |
+| `ltk_hashdb` | The `.hashdb` format: `mmap` reader (`HashDb`) + streaming writer (`HashDbWriter`), `LayeredHashDb` overlay |
| `ltk_mimir_cache` | Shared cache dir, `manifest.json`, versioned publish, update lock, GC, in-process updater (`HashStore`, `HashStore::update`) |
| `ltk_mimir_gen` | Hash-discovery ("hunt") engine - guessers that resolve unknown hashes |
| `ltk_mimir_cli` | The `mimir` binary (`build` / `get` / `verify` / `stats` / `gen` / `update` / `merge` / `bundle`) |
-Docs live in `docs/`: `FORMAT.md` (byte-level spec, format version 1), `CONSUMERS.md`
-(integration API), `BENCHMARKS.md` (frame-size/compression measurements).
+Docs live in `docs/`: `DESIGN.md` (rationale, how the format works, measurements -
+the narrative the README used to carry), `FORMAT.md` (byte-level spec, format version 1),
+`CONSUMERS.md` (integration API), `BENCHMARKS.md` (frame-size/compression measurements),
+`ROADMAP.md` (planned work, in dependency order, with the additive-only constraints that
+govern it).
+
+`README.md` is the usage front door: install, quick start, the API surface with examples,
+and the CLI. Keep its examples true to the current API - they are typechecked by hand, not
+by CI.
## Conventions
diff --git a/Cargo.toml b/Cargo.toml
index ff7e1ec..9b2d690 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -4,9 +4,9 @@ members = ["crates/*"]
[workspace.package]
edition = "2021"
-license = "MIT OR Apache-2.0"
+license = "Apache-2.0"
repository = "https://github.com/LeagueToolkit/mimir"
-authors = ["League Toolkit contributors"]
+authors = ["Crauzer <0xcrauzer@proton.me>", "League Toolkit contributors"]
[workspace.dependencies]
# Internal crates (path-linked within this workspace; the published ones also
diff --git a/LICENSE-APACHE b/LICENSE
similarity index 100%
rename from LICENSE-APACHE
rename to LICENSE
diff --git a/LICENSE-MIT b/LICENSE-MIT
deleted file mode 100644
index aad79e4..0000000
--- a/LICENSE-MIT
+++ /dev/null
@@ -1,21 +0,0 @@
-MIT License
-
-Copyright (c) 2026 League Toolkit contributors
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
diff --git a/NOTICE b/NOTICE
new file mode 100644
index 0000000..fb7005c
--- /dev/null
+++ b/NOTICE
@@ -0,0 +1,16 @@
+mimir
+Copyright 2026 Crauzer <0xcrauzer@proton.me>
+
+This product includes software developed by Crauzer and the League Toolkit
+contributors.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License. You may obtain a copy of the
+License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
diff --git a/README.md b/README.md
index 890a667..32cd3b1 100644
--- a/README.md
+++ b/README.md
@@ -1,158 +1,329 @@
-# mimir
-
-A toolkit for **generating**, **storing**, and **serving** League of Legends
-hash → path tables as a compact, memory-mapped, seekable binary format (`.hashdb`).
-It reimplements and extends CommunityDragon's CDTB `hashes.py` and the
-CommunityDragon/Data txt artifacts.
-
-The `.hashdb` format is general-purpose - an mmap-backed, read-only map from integer keys
-to string values, with nothing League-specific in its layout. League Toolkit distributes
-its own tables in this format under the `.lhdb` extension (identical bytes).
-
-## Why this exists
-
-Almost every League of Legends tool - WAD unpackers, `.bin` inspectors, mod loaders,
-asset browsers - hits the same wall: the game identifies files and fields by **hash**,
-not by name. To show a human-readable path you need a **hash table** that maps each
-hash back to its original string.
-
-Today that table is the CommunityDragon `hashes.*.txt` set: **~348 MB of plain text**,
-which every tool downloads and keeps around. That approach has two costs that compound
-the moment a machine runs more than one of these tools:
-
-- **Memory.** To resolve hashes efficiently a program has to load the whole table into
- an in-memory map. Run three tools that each need the game hashes and you pay for
- **three private copies** of the same hundreds-of-megabytes table, all resident at once.
-- **Startup & distribution.** Every tool ships (or re-downloads) the same giant text
- files and spends time parsing them into a map before it can answer a single query.
-
-**mimir** replaces that text blob with a purpose-built, **read-only** binary format for
-hash storage. The design goals are, in order:
-
-1. **Usable as shipped** - no unpack or full-expansion step; a consumer `mmap`s the file
- and immediately does lookups.
-2. **Small** - the game table drops from ~348 MB of text to roughly **~50 MB** on disk.
-3. **Memory-efficient across processes** - the file is memory-mapped, so the OS page
- cache holds **one** copy that every tool on the machine shares. Resident RAM stays
- low because pages are faulted in lazily and dropped under pressure, and a lookup
- *miss* touches zero string data.
-
-## How it works
-
-A `.hashdb` file is a single logical table laid out for direct, zero-parse use over an
-`mmap`:
-
-- **Sorted key array** - the integer hashes, stored strictly ascending so a lookup is a
- **binary search straight over the mapped bytes**. A miss is decided here and never
- reads any string data.
-- **Parallel offset + length arrays** - for a found key, where its path lives in the arena
- and how long it is.
-- **String arena** - all the path strings concatenated with no separators, compressed as
- a **Zstandard Seekable Format** stream. The seek table means a hit decompresses just the
- **one small frame** that holds its path (default 16 KiB frames), not the whole table -
- so partial, on-demand reads stay cheap. Paths are packed in **lexicographic order** so
- a directory's files land in the same frames, which both compresses far better (~4× vs.
- key order on the real game table) and makes directory-local batch lookups touch fewer
- frames.
-
-The file is immutable once published; updates ship as new versioned files, and a downloaded
-file is treated as untrusted - the header is validated on open and every read bounds-checks
-its own extent. See [`docs/FORMAT.md`](docs/FORMAT.md) for the byte-level specification.
-
-Because it's memory-mapped and read-only, a lazy consumer (say, a mod loader) can open the
-table only when it first needs to resolve a hash, share the page cache with every other
-mimir-backed tool running, and drop the handle to reclaim its (already small) footprint.
-
-## Performance
-
-Real-data measurements against the CommunityDragon `hashes.*.txt` snapshot of
-2026-07-07 (~2.97 M entries across 8 tables). Full tables, methodology, and
-reproduction steps are in [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).
-
-
-
-
-
-
-The whole corpus drops from **~253 MiB of txt to ~52 MiB** of `.hashdb` - and the
-binary is usable as-shipped: `open` is a header validation plus an `mmap`, with no
-parse or expansion step before the first lookup.
-
-
-
-
-
-
-A hit decompresses exactly one small frame; batched lookups amortize that through
-the reader's frame cache. A **miss is decided by binary search over the raw key
-section and never touches string data** - ~143 ns whether the file is raw or
-compressed, which matters because hash hunting hammers misses.
-
-
-
-
-
-
-Sorting the arena by path packs each directory into the same frames, so the seekable
-arena compresses **~4× better than key order** - beating even a solid, non-seekable
-zstd stream - while making hits faster and directory-local batches frame-coherent.
-
-## Layout
+
-| Crate | Role |
-|-------|------|
-| `ltk_hashdb` | The `.hashdb` format: `mmap` reader (`HashDb`) + streaming writer |
-| `ltk_mimir_cache` | Shared cache dir, manifest, versioned publish, update lock, GC, in-process updater |
-| `ltk_mimir_gen` | Hash-discovery ("hunt") engine for still-unknown hashes |
-| `ltk_mimir_cli` | The `mimir` binary |
+Hash → path tables for League of Legends tooling, stored as a compact, memory-mapped,
+seekable binary format (`.hashdb`). It replaces CommunityDragon's ~348 MB of
+`hashes.*.txt` with a **~52 MiB binary that is usable as shipped** - no parse step before
+the first lookup, and **one copy in the page cache that every tool on the machine
+shares**.
+
+The format itself is general-purpose: a read-only map from integer keys to string values,
+with nothing League-specific in its layout. League Toolkit distributes its own tables
+under the `.lhdb` extension (identical bytes).
+
+
+
+## Install
+
+Nothing is on crates.io yet, so depend on the repository directly:
+
+```toml
+[dependencies]
+ltk_hashdb = { git = "https://github.com/LeagueToolkit/mimir" }
+
+# Only if you want the shared cache and the download-driven updater.
+# `ureq` gives you a blocking fetcher, `reqwest` an async one; both are optional.
+ltk_mimir_cache = { git = "https://github.com/LeagueToolkit/mimir", features = ["ureq"] }
+```
-## Using it
+The CLI:
-As a library - open a table and resolve a hash:
+```sh
+cargo install --git https://github.com/LeagueToolkit/mimir ltk_mimir_cli
+```
+
+## Quick start
+
+Pull the published tables into the shared cache, then resolve a hash out of it:
+
+```sh
+mimir update
+mimir get 0x1234abcd --table game
+```
+
+The same thing from Rust:
```rust
-use ltk_hashdb::HashDb;
+use ltk_mimir_cache::{HashStore, Table};
+
+let store = HashStore::discover()?;
+let db = store.open_shared(Table::Game)?; // mmap + validate header; no parse step
-let db = HashDb::open("game.hashdb")?; // mmap + validate header, lazy
if let Some(path) = db.get(0x1234_5678_9abc_def0) {
println!("{path}");
}
```
-From the CLI:
+The cache lives in the platform data directory - `%LOCALAPPDATA%\LeagueToolkit\hashes` on
+Windows, `$XDG_DATA_HOME/LeagueToolkit/hashes` on Linux, `~/Library/Application
+Support/LeagueToolkit/hashes` on macOS - and `MIMIR_DIR` overrides it.
+
+## Library API
+
+Two crates matter to consumers. `ltk_hashdb` is the format: open a file, resolve hashes.
+`ltk_mimir_cache` is everything around it: where tables live on disk, which version is
+active, and how they get updated. A tool that ships its own tables needs only the first.
+
+### Resolving a hash
+
+`get` returns a **`PathRef`**, which borrows its bytes instead of copying them - out of
+the mapping for a raw arena, out of the cached decompressed frame for a compressed one.
+It derefs to `str`, so it behaves like one:
+
+```rust
+use ltk_hashdb::HashDb;
+
+let db = HashDb::open("game.lhdb")?;
+
+if let Some(path) = db.get(hash) {
+ println!("{path}"); // Display
+ if path.ends_with(".dds") { /* … */ } // Deref
+ let owned: String = path.into_owned(); // copy only when you keep it
+}
+
+// A miss is decided by binary search over the key array and never touches the arena.
+assert!(!db.contains(0xdead_beef));
+```
+
+Hashing a path with the table's own algorithm, so you never have to know which one it is:
+
+```rust
+let hash = db.hash_path("assets/characters/ahri/ahri.bin");
+assert!(db.contains(hash));
+```
+
+### Resolving many hashes
+
+Both forms resolve hits in **arena order** so each compressed frame is decompressed once.
+Use `get_batch` when you want results back in input order, and `for_each_batch` when you
+want them streamed with no intermediate `Vec`:
+
+```rust
+// Collected, in input order.
+for (hash, path) in db.get_batch(&chunk_hashes) {
+ match path {
+ Some(path) => println!("{path}"),
+ None => println!("{hash:016x} (unknown)"),
+ }
+}
+
+// Streamed. Calls arrive in arena order, so the first argument is the input position.
+db.for_each_batch(&chunk_hashes, |i, hash, path| match path {
+ Some(path) => println!("{i}: {path}"),
+ None => println!("{i}: {hash:016x}"),
+});
+```
+
+### Layering tables and adding your own hashes
+
+`LayeredHashDb` puts a writable in-memory overlay over one or more read-only bases.
+Lookups consult the overlay first, then each base in push order; the first hit wins, and
+no base file is ever mutated. This is what WAD consumers want, since chunk resolution
+spans both `game` and `lcu`:
+
+```rust
+use ltk_mimir_cache::{HashStore, Table};
+
+let store = HashStore::discover()?;
+
+// Missing tables are reported, not fatal - the tool stays usable and their hashes miss.
+let (mut db, errors) = store.open_layered(&[Table::Game, Table::Lcu]);
+for (table, e) in &errors {
+ eprintln!("skipping {table:?}: {e}");
+}
+
+// Register a path your mod introduced; it is hashed with the first base's algorithm.
+let hash = db.insert_path("assets/mymod/custom.dds").expect("has a base");
+assert_eq!(db.get(hash).as_deref(), Some("assets/mymod/custom.dds"));
+```
+
+> [!NOTE]
+> Every base must agree on key width, hash algorithm, and casing, because lookups take a
+> hash the caller already computed and no base re-hashes it. `game` and `lcu` do; the four
+> 32-bit `bin*` tables are separate hash universes and must not be layered together.
+
+### Enumerating a table
+
+```rust
+// Streams in arena order (lexicographic path order), one decompress per frame.
+for (hash, path) in db.iter() {
+ println!("{hash:016x} {path}");
+}
+
+// Opt-in resident mode: the whole table as an owned map. Costs the full decompressed
+// size in private memory and forfeits the shared page cache, so reach for it last.
+let map = db.load_all();
+```
+
+### Updating the cache
+
+The crate ships no HTTP client of its own - you hand it a fetcher. `UreqFetch` (feature
+`ureq`) and `ReqwestFetch` (feature `reqwest`, async) cover the common case:
+
+```rust
+use ltk_mimir_cache::{HashStore, ReleaseSource, UpdateOptions, UpdateOutcome, UreqFetch};
+
+let store = HashStore::discover()?;
+let remote = UreqFetch::new(ReleaseSource::github("LeagueToolkit/mimir"));
+
+match store.update(&remote, UpdateOptions::default())? {
+ UpdateOutcome::Completed(report) => println!("installed {:?}", report.installed),
+ UpdateOutcome::Locked => println!("another process is already updating"),
+}
+```
+
+Only tables whose sha256 differs are downloaded. Installs are atomic - versioned files
+land first, the manifest pointer flips last - so a reader sees either the whole old
+version or the whole new one, and readers never take a lock.
+
+### Building your own table
+
+```rust
+use std::fs::File;
+use ltk_hashdb::{Casing, Compression, HashDbWriter, HashKind, KeyWidth};
+
+let mut writer = HashDbWriter::new(KeyWidth::U64, Compression::default())
+ .hash_kind(HashKind::Xxh64) // recorded, so readers can hash new paths
+ .casing(Casing::Insensitive); // League tables hash the lowercased path
+
+writer.insert(hash, "assets/characters/ahri/ahri.bin");
+writer.extend(pairs);
+
+let stats = writer.build(File::create("mine.hashdb")?)?;
+println!("{} entries, {} bytes", stats.entries, stats.file_len);
+```
+
+`Compression::default()` is the publishing configuration: 16 KiB frames at level 19, the
+measured size/latency knee. `Compression::None` writes a raw arena that lookups borrow
+straight out of the mapping.
+
+### API surface
+
+**`HashDb`** - one `.hashdb` file. Cheap to clone; every clone shares the mapping and the
+frame cache. `Send + Sync`.
+
+| Method | |
+|---|---|
+| `open` · `open_bytes` | mmap a file, or open an in-memory image |
+| `options()` | open-time knobs: `frame_cache_bytes(n)`, `0` disables |
+| `get` | resolve a hash → `Option` |
+| `try_get` | as `get`, but a corrupt arena errors instead of reading as a miss |
+| `get_into` | copy into a reusable `String`; holds no frame afterwards |
+| `contains` | membership, never touches the arena |
+| `get_batch` · `for_each_batch` | bulk resolve, collected or streamed |
+| `iter` · `load_all` | enumerate in arena order, or decode into an owned map |
+| `hash_path` | hash a string with this table's algorithm and casing |
+| `verify` · `is_healthy` | full integrity pass; sticky flag set by a failed read |
+| `len` · `key_width` · `hash_kind` · `casing` · `is_compressed` | shape |
+| `downgrade` | a `WeakHashDb` for registries that must not pin the table |
+
+**`LayeredHashDb`** - an overlay over N ordered bases.
+
+| Method | |
+|---|---|
+| `from_bases` · `push_base` | layer read-only tables, highest priority first |
+| `insert` · `insert_path` · `extend` | write to the overlay, shadowing every base |
+| `get` · `contains` · `get_into` | overlay first, then each base in order |
+| `get_batch` · `for_each_batch` | staged bulk resolve; each base sees only the residual |
+| `iter` | every entry, each shadowed key yielded once by the layer that answers it |
+| `bases` · `overlay_len` · `base_len` · `is_healthy` | shape |
+
+**`HashStore`** - the shared cache directory.
+
+| Method | |
+|---|---|
+| `discover` · `at` | resolve the platform cache dir, or point at your own |
+| `open_shared` | open the active version, reusing a handle this store already has |
+| `open` · `open_many` | open a fresh mapping, one table or several |
+| `open_layered` | open several into one `LayeredHashDb`, reporting per-table errors |
+| `manifest` · `path_for` | what is installed, and where |
+| `update` · `update_async` | compare → download → verify → install → GC |
+| `commit` · `gc` · `try_lock_update` | publish versions, sweep old ones, take the lock |
+
+**`PathRef`** - a resolved path. `Deref`, plus `as_str`, `is_owned`
+(whether the bytes were copied rather than borrowed), and `into_owned`.
+
+**`HashDbWriter`** - `new` → `hash_kind` / `casing` → `insert` / `extend` → `build`.
+
+## CLI
+
+```
+mimir
+
+build Build a .hashdb table from a txt hash list (lines of ``)
+get Resolve one hash from a .hashdb file or the shared cache
+update Download the latest published tables into the shared cache
+gen Run the hunt engine: discover paths for still-unknown hashes
+merge Sorted dedup merge of CDragon txt hash lists
+bundle Build all tables + manifest from CDragon txt inputs, staged for a GH release
+verify Structural + checksum validation of a .hashdb file
+stats Sizes, entry counts, compression ratio of a .hashdb file
+```
```sh
-# Build a .hashdb table from a CDragon `` txt list
+# Build a table from a CDragon txt list
mimir build --input hashes.game.txt --table game --out game.hashdb
-# Resolve one hash
+# Resolve a hash, from a file or from the shared cache
mimir get 0x1234abcd --file game.hashdb
+mimir get 0x1234abcd --table game
-# Validate a downloaded file (structure + checksum)
+# Keep the shared cache current (--url for a mirror, --dir for a private cache)
+mimir update
+mimir update --force
+
+# Inspect and validate
+mimir stats game.hashdb
mimir verify game.hashdb
```
-See [`docs/CONSUMERS.md`](docs/CONSUMERS.md) for the shared-cache and
-custom-hash-extension APIs.
+## Crates
+
+| Crate | Role |
+|-------|------|
+| `ltk_hashdb` | The `.hashdb` format: `mmap` reader (`HashDb`) + streaming writer |
+| `ltk_mimir_cache` | Shared cache dir, manifest, versioned publish, update lock, GC, in-process updater |
+| `ltk_mimir_gen` | Hash-discovery ("hunt") engine for still-unknown hashes |
+| `ltk_mimir_cli` | The `mimir` binary |
+
+## Documentation
+
+| | |
+|---|---|
+| [`docs/DESIGN.md`](docs/DESIGN.md) | Why this exists, how the format works, what it measures |
+| [`docs/FORMAT.md`](docs/FORMAT.md) | Byte-level specification of `.hashdb`, format version 1 |
+| [`docs/CONSUMERS.md`](docs/CONSUMERS.md) | Integration guide: lookup patterns, threading, custom pipelines |
+| [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) | Frame-size and compression measurements, with reproduction steps |
+| [`docs/ROADMAP.md`](docs/ROADMAP.md) | Planned work, in dependency order |
## Status
-Early development. The `.hashdb` format, reader/writer, the shared cache, release
-publishing (`mimir bundle` + a scheduled CI job that ships every table as versioned
-`.lhdb` GitHub release assets, rebuilt from the canonical CommunityDragon txt lists), the
+Early development. The format, reader/writer, shared cache, release publishing
+(`mimir bundle` plus a scheduled CI job that ships every table as versioned `.lhdb`
+release assets, rebuilt from the canonical CommunityDragon txt lists), the
download-driven `mimir update` flow, and the hunt engine - including WAD string mining
-(`mimir gen --wad`) - are in place.
+(`mimir gen --wad`) - are all in place.
-## License
+The txt lists stay canonical; the binaries are generated release artifacts, never the
+source of truth.
-Licensed under either of
+## License
-- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or
- http://www.apache.org/licenses/LICENSE-2.0)
-- MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
+Copyright 2026 Crauzer <0xcrauzer@proton.me>
-at your option.
+Licensed under the Apache License, Version 2.0 ([LICENSE](LICENSE) or
+http://www.apache.org/licenses/LICENSE-2.0). Attribution requirements are in
+[NOTICE](NOTICE).
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in the work by you, as defined in the Apache-2.0 license, shall be
-dual licensed as above, without any additional terms or conditions.
+licensed as above, without any additional terms or conditions.
diff --git a/crates/ltk_hashdb/examples/frame_cache.rs b/crates/ltk_hashdb/examples/frame_cache.rs
new file mode 100644
index 0000000..62cb998
--- /dev/null
+++ b/crates/ltk_hashdb/examples/frame_cache.rs
@@ -0,0 +1,122 @@
+//! Measures what the frame cache is for: point lookups against batched ones.
+//!
+//! A WAD extractor resolves one chunk hash at a time - `ltk_wad`'s `PathResolver` is
+//! point-shaped by construction - but the chunks of any one archive share a path prefix,
+//! so they land in a handful of arena frames. This walks that shape: take a contiguous
+//! run of paths (one "archive"), probe them in hash order (a WAD's table-of-contents
+//! order, which is unrelated to path order), and compare point lookups with the cache on,
+//! with it off, and one `get_batch` over the same keys.
+//!
+//! ```text
+//! cargo run --release -p ltk_hashdb --example frame_cache -- [chunks]
+//! ```
+
+use std::time::Instant;
+
+use ltk_hashdb::HashDb;
+
+fn main() {
+ let mut args = std::env::args_os().skip(1);
+ let Some(path) = args.next() else {
+ eprintln!("usage: frame_cache [chunks]");
+ std::process::exit(2);
+ };
+ let chunks: usize = args
+ .next()
+ .and_then(|n| n.to_str().and_then(|n| n.parse().ok()))
+ .unwrap_or(20_000);
+
+ let db = HashDb::open(&path).expect("open");
+ println!(
+ "{}: {} entries, arena {:.1} MiB raw / {:.1} MiB on disk\n",
+ path.to_string_lossy(),
+ db.len(),
+ db.arena_decompressed_size() as f64 / (1 << 20) as f64,
+ db.arena_compressed_size() as f64 / (1 << 20) as f64,
+ );
+
+ // `iter` yields in arena order, so a window of it is a run of neighbouring paths -
+ // one archive's worth. Probing order is by hash, as a WAD's TOC would give them.
+ let started = Instant::now();
+ let arena_order: Vec = db.iter().map(|(key, _)| key).collect();
+ println!(
+ "full scan: {:?} for {} entries",
+ started.elapsed(),
+ arena_order.len()
+ );
+
+ let window = arena_order.len().min(chunks);
+ let start = (arena_order.len() - window) / 2;
+ let mut keys: Vec = arena_order[start..start + window].to_vec();
+ keys.sort_unstable();
+
+ let cached = HashDb::open(&path).expect("open");
+ let uncached = HashDb::options()
+ .frame_cache_bytes(0)
+ .open(&path)
+ .expect("open");
+
+ // Warm the mapping so the first pass isn't paying for page faults the others skip.
+ let _ = uncached.get(keys[0]);
+
+ let point_uncached = time(|| {
+ for &key in &keys {
+ std::hint::black_box(uncached.get(key));
+ }
+ });
+ let point_cached = time(|| {
+ for &key in &keys {
+ std::hint::black_box(cached.get(key));
+ }
+ });
+ let batched = time(|| {
+ for entry in cached.get_batch(&keys) {
+ std::hint::black_box(entry);
+ }
+ });
+
+ let per = |d: std::time::Duration| d.as_secs_f64() * 1e9 / keys.len() as f64;
+ println!("\n{window} chunks, probed in hash order:");
+ println!(" point, no cache : {:>8.0} ns/lookup", per(point_uncached));
+ println!(" point, cached : {:>8.0} ns/lookup", per(point_cached));
+ println!(" get_batch : {:>8.0} ns/lookup", per(batched));
+ println!(
+ "\n cache speedup : {:>8.1}x point/batch ratio: {:.2}x",
+ point_uncached.as_secs_f64() / point_cached.as_secs_f64(),
+ point_cached.as_secs_f64() / batched.as_secs_f64(),
+ );
+
+ // The other end of the spectrum: keys spread over the whole table, where no cache
+ // of any size holds the working set. This is what the cache must not make slower.
+ let stride = arena_order.len() / window.max(1);
+ let mut scattered: Vec = arena_order.iter().copied().step_by(stride.max(1)).collect();
+ scattered.truncate(window);
+ scattered.sort_unstable();
+
+ let scattered_uncached = time(|| {
+ for &key in &scattered {
+ std::hint::black_box(uncached.get(key));
+ }
+ });
+ let scattered_cached = time(|| {
+ for &key in &scattered {
+ std::hint::black_box(cached.get(key));
+ }
+ });
+
+ println!("\n{} keys spread across the whole table:", scattered.len());
+ println!(
+ " point, no cache : {:>8.0} ns/lookup",
+ scattered_uncached.as_secs_f64() * 1e9 / scattered.len() as f64
+ );
+ println!(
+ " point, cached : {:>8.0} ns/lookup",
+ scattered_cached.as_secs_f64() * 1e9 / scattered.len() as f64
+ );
+}
+
+fn time(mut f: impl FnMut()) -> std::time::Duration {
+ let started = Instant::now();
+ f();
+ started.elapsed()
+}
diff --git a/crates/ltk_hashdb/src/cache.rs b/crates/ltk_hashdb/src/cache.rs
new file mode 100644
index 0000000..f05b214
--- /dev/null
+++ b/crates/ltk_hashdb/src/cache.rs
@@ -0,0 +1,243 @@
+//! Shared, byte-capped store of decompressed arena frames.
+//!
+//! A published table is immutable, so a decompressed frame never goes stale - this is
+//! pure memoisation with nothing to invalidate. Frames are keyed by a dense, contiguous
+//! index, which lets the store be an N-way set-associative table sized once at open:
+//! no map, no eviction list, no per-insert allocation, and one lock per set, so
+//! concurrent readers stay off each other's backs without a sharding decision.
+//!
+//! Buffers from evicted frames are recycled, so a steady-state miss decompresses into a
+//! buffer that already exists instead of allocating a new one.
+
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, Mutex, PoisonError};
+
+/// One decompressed arena frame, shared by every path resolved out of it.
+pub(crate) struct Frame(Vec);
+
+impl Frame {
+ pub(crate) fn bytes(&self) -> &[u8] {
+ &self.0
+ }
+}
+
+impl From> for Frame {
+ fn from(bytes: Vec) -> Self {
+ Self(bytes)
+ }
+}
+
+/// Frames per set. Four is enough associativity that the arena's forward-walking
+/// access pattern never thrashes, and small enough that a set scan stays trivial.
+const WAYS: usize = 4;
+
+/// Recycled buffers kept for reuse. Scales with concurrent decompressions, not with
+/// the cache size - a buffer only sits here between one frame's eviction and the next
+/// frame's decompression.
+const SPARE_BUFFERS: usize = 8;
+
+/// A fixed-size set-associative cache of decompressed frames.
+pub(crate) struct FrameCache {
+ /// Empty when caching is disabled; every operation then no-ops.
+ sets: Box<[Mutex<[Slot; WAYS]>]>,
+
+ /// Monotonic access counter; a slot's last value orders eviction within its set.
+ clock: AtomicU64,
+
+ spare: Mutex>>,
+}
+
+#[derive(Default)]
+struct Slot {
+ /// Meaningful only while `frame` is `Some`.
+ index: u32,
+
+ used: u64,
+
+ frame: Option>,
+}
+
+impl FrameCache {
+ /// A cache holding at most `budget` bytes worth of `frame_size` frames.
+ ///
+ /// Never sized beyond the `frames` the file actually has, so a small table costs a
+ /// small table's worth of slots. A zero budget (or a table with no frames at all)
+ /// disables caching outright.
+ pub(crate) fn new(budget: usize, frame_size: usize, frames: usize) -> Self {
+ let wanted = match budget.checked_div(frame_size) {
+ Some(n) => n.min(frames),
+ None => 0,
+ };
+
+ Self {
+ sets: (0..wanted.div_ceil(WAYS))
+ .map(|_| Mutex::new(std::array::from_fn(|_| Slot::default())))
+ .collect(),
+ clock: AtomicU64::new(0),
+ spare: Mutex::new(Vec::new()),
+ }
+ }
+
+ /// How many frames this cache can hold at once; `0` when disabled.
+ pub(crate) fn capacity(&self) -> usize {
+ self.sets.len() * WAYS
+ }
+
+ /// The cached frame `index`, if it is still resident.
+ pub(crate) fn get(&self, index: u32) -> Option> {
+ let mut slots = self
+ .set(index)?
+ .lock()
+ .unwrap_or_else(PoisonError::into_inner);
+ let slot = slots
+ .iter_mut()
+ .find(|slot| slot.frame.is_some() && slot.index == index)?;
+
+ slot.used = self.clock.fetch_add(1, Ordering::Relaxed);
+ slot.frame.clone()
+ }
+
+ /// Publish `frame` under `index`, evicting the coldest frame in its set.
+ pub(crate) fn insert(&self, index: u32, frame: &Arc) {
+ let Some(set) = self.set(index) else { return };
+ let used = self.clock.fetch_add(1, Ordering::Relaxed);
+
+ let evicted = {
+ let mut slots = set.lock().unwrap_or_else(PoisonError::into_inner);
+ // Prefer the slot this frame already occupies (a racing reader decompressed
+ // it too), then an empty one, then the least recently used. `+ 1` keeps an
+ // occupied slot from tying with an empty one at clock zero.
+ let slot = match slots
+ .iter()
+ .position(|slot| slot.frame.is_some() && slot.index == index)
+ {
+ Some(position) => &mut slots[position],
+ None => slots
+ .iter_mut()
+ .min_by_key(|slot| match slot.frame {
+ None => 0,
+ Some(_) => slot.used.saturating_add(1),
+ })
+ .expect("WAYS is nonzero"),
+ };
+
+ slot.index = index;
+ slot.used = used;
+ slot.frame.replace(Arc::clone(frame))
+ };
+
+ if let Some(evicted) = evicted {
+ self.recycle(evicted);
+ }
+ }
+
+ /// A buffer to decompress into, reusing an evicted frame's allocation when there
+ /// is one to reuse.
+ pub(crate) fn take_buffer(&self, capacity: usize) -> Vec {
+ let mut spare = self.spare.lock().unwrap_or_else(PoisonError::into_inner);
+ match spare.pop() {
+ Some(mut buffer) => {
+ buffer.clear();
+ buffer.reserve(capacity);
+ buffer
+ }
+ None => Vec::with_capacity(capacity),
+ }
+ }
+
+ /// Reclaim an evicted frame's buffer, unless a caller still holds a path into it.
+ fn recycle(&self, frame: Arc) {
+ let Some(frame) = Arc::into_inner(frame) else {
+ return;
+ };
+
+ let mut spare = self.spare.lock().unwrap_or_else(PoisonError::into_inner);
+ if spare.len() < SPARE_BUFFERS {
+ let mut buffer = frame.0;
+ buffer.clear();
+ spare.push(buffer);
+ }
+ }
+
+ fn set(&self, index: u32) -> Option<&Mutex<[Slot; WAYS]>> {
+ if self.sets.is_empty() {
+ return None;
+ }
+
+ Some(&self.sets[index as usize % self.sets.len()])
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn frame(byte: u8) -> Arc {
+ Arc::new(Frame::from(vec![byte; 16]))
+ }
+
+ #[test]
+ fn a_zero_budget_disables_caching() {
+ let cache = FrameCache::new(0, 16, 100);
+ assert_eq!(cache.capacity(), 0);
+
+ cache.insert(0, &frame(1));
+ assert!(cache.get(0).is_none());
+ }
+
+ #[test]
+ fn slots_never_exceed_the_frames_in_the_file() {
+ // Budget for 1000 frames, but the file only has 3.
+ let cache = FrameCache::new(16_000, 16, 3);
+ assert_eq!(cache.capacity(), WAYS);
+ }
+
+ #[test]
+ fn frames_round_trip_until_their_set_fills() {
+ // One set: four ways, so the fifth frame mapping to it evicts the coldest.
+ let cache = FrameCache::new(16 * WAYS, 16, WAYS);
+ assert_eq!(cache.sets.len(), 1);
+
+ for i in 0..WAYS as u32 {
+ cache.insert(i, &frame(i as u8));
+ }
+ // Touch 0 so it is no longer the coldest, then force one eviction.
+ assert!(cache.get(0).is_some());
+ cache.insert(99, &frame(99));
+
+ assert!(cache.get(0).is_some(), "recently used frame survived");
+ assert!(cache.get(99).is_some(), "newly inserted frame is resident");
+ assert!(cache.get(1).is_none(), "coldest frame was evicted");
+ }
+
+ #[test]
+ fn evicted_buffers_are_recycled() {
+ let cache = FrameCache::new(16 * WAYS, 16, WAYS);
+ for i in 0..WAYS as u32 + 1 {
+ cache.insert(i, &frame(i as u8));
+ }
+
+ // The evicted frame's allocation came back, so this buffer is not a fresh one.
+ let buffer = cache.take_buffer(16);
+ assert!(buffer.is_empty());
+ assert!(buffer.capacity() >= 16);
+ assert!(
+ cache.spare.lock().unwrap().is_empty(),
+ "buffer was handed out"
+ );
+ }
+
+ /// A frame still held by a caller must not be recycled out from under them.
+ #[test]
+ fn a_held_frame_is_not_recycled() {
+ let cache = FrameCache::new(16 * WAYS, 16, WAYS);
+ let held = frame(0);
+ cache.insert(0, &held);
+ for i in 1..WAYS as u32 + 1 {
+ cache.insert(i, &frame(i as u8));
+ }
+
+ assert_eq!(held.bytes(), &[0u8; 16]);
+ assert!(cache.spare.lock().unwrap().is_empty());
+ }
+}
diff --git a/crates/ltk_hashdb/src/extended.rs b/crates/ltk_hashdb/src/extended.rs
deleted file mode 100644
index 5968d5c..0000000
--- a/crates/ltk_hashdb/src/extended.rs
+++ /dev/null
@@ -1,59 +0,0 @@
-//! A [`HashDb`] plus an in-memory overlay of extra entries.
-
-use std::borrow::Cow;
-use std::collections::HashMap;
-
-use crate::HashDb;
-
-/// A `HashDb` with a mutable in-memory overlay (e.g. runtime mod hashes). The base
-/// file is never mutated; lookups consult the overlay first, then the table.
-pub struct ExtendedHashDb {
- base: HashDb,
- overlay: HashMap>,
-}
-
-impl ExtendedHashDb {
- pub fn new(base: HashDb) -> Self {
- Self {
- base,
- overlay: HashMap::new(),
- }
- }
-
- pub fn insert(&mut self, hash: u64, path: impl Into>) {
- self.overlay.insert(hash, path.into());
- }
-
- /// Hash `path` with the base table's algorithm, insert it, and return the hash -
- /// "register this path" without knowing the algorithm.
- pub fn insert_path(&mut self, path: &str) -> u64 {
- let hash = self.base.hash_path(path);
- self.insert(hash, path);
- hash
- }
-
- pub fn extend<'a>(&mut self, it: impl IntoIterator) {
- self.overlay
- .extend(it.into_iter().map(|(k, p)| (k, Box::from(p))));
- }
-
- /// Overlay first, then the base table.
- pub fn get(&self, hash: u64) -> Option> {
- match self.overlay.get(&hash) {
- Some(path) => Some(Cow::Borrowed(&**path)),
- None => self.base.get(hash),
- }
- }
-
- pub fn contains(&self, hash: u64) -> bool {
- self.overlay.contains_key(&hash) || self.base.contains(hash)
- }
-
- pub fn base(&self) -> &HashDb {
- &self.base
- }
-
- pub fn overlay_len(&self) -> usize {
- self.overlay.len()
- }
-}
diff --git a/crates/ltk_hashdb/src/layered.rs b/crates/ltk_hashdb/src/layered.rs
index 816953c..fd13d84 100644
--- a/crates/ltk_hashdb/src/layered.rs
+++ b/crates/ltk_hashdb/src/layered.rs
@@ -1,9 +1,9 @@
//! An in-memory overlay layered over an ordered list of read-only base tables.
-use std::borrow::Cow;
use std::collections::HashMap;
+use std::fmt;
-use crate::{Casing, HashDb, HashKind, KeyWidth};
+use crate::{Casing, HashDb, HashKind, KeyWidth, PathRef};
/// The key configuration a base hashes under. Every base in a [`LayeredHashDb`]
/// must agree on this triple; a base that diverges can never be hit by a caller's
@@ -15,9 +15,9 @@ fn base_config(db: &HashDb) -> (KeyWidth, HashKind, Casing) {
/// A writable in-memory overlay on top of ordered read-only [`HashDb`] bases.
///
/// Lookups consult the overlay first, then each base in push order; the first hit
-/// wins. Base files are never mutated. This generalises [`ExtendedHashDb`] (one
-/// base) to the N-base case consumers need when several tables (e.g. League's
-/// `game` and `lcu`) sit under one overlay.
+/// wins. Base files are never mutated. One base covers the "table plus my own
+/// runtime hashes" case; several cover the one consumers reach for when a workload
+/// spans more than one table (e.g. League's `game` and `lcu` under one overlay).
///
/// # Base configuration invariant
///
@@ -31,8 +31,6 @@ fn base_config(db: &HashDb) -> (KeyWidth, HashKind, Casing) {
/// [`from_bases`](Self::from_bases) `debug_assert!` this; release builds skip the
/// check. League's `game`/`lcu` tables are uniform (XXH64 / U64 / case-insensitive),
/// so the common path always satisfies it.
-///
-/// [`ExtendedHashDb`]: crate::ExtendedHashDb
#[derive(Default)]
pub struct LayeredHashDb {
overlay: HashMap>,
@@ -107,9 +105,9 @@ impl LayeredHashDb {
}
/// Overlay first, then each base in push order; the first hit wins.
- pub fn get(&self, hash: u64) -> Option> {
+ pub fn get(&self, hash: u64) -> Option> {
if let Some(path) = self.overlay.get(&hash) {
- return Some(Cow::Borrowed(&**path));
+ return Some(PathRef::borrowed(path));
}
self.bases.iter().find_map(|base| base.get(hash))
}
@@ -125,9 +123,9 @@ impl LayeredHashDb {
/// order. This is the payoff over calling [`get`](Self::get) N times.
pub fn get_batch<'a>(
&'a self,
- hashes: &'a [u64],
- ) -> impl Iterator>)> + 'a {
- let mut results: Vec>> = Vec::new();
+ hashes: &[u64],
+ ) -> impl Iterator>)> + 'a {
+ let mut results: Vec>> = Vec::new();
results.resize_with(hashes.len(), || None);
// Layer 0: overlay, O(1) per hash. Positions still missing stay in
@@ -135,7 +133,7 @@ impl LayeredHashDb {
let mut residual: Vec = Vec::new();
for (i, &h) in hashes.iter().enumerate() {
match self.overlay.get(&h) {
- Some(path) => results[i] = Some(Cow::Borrowed(&**path)),
+ Some(path) => results[i] = Some(PathRef::borrowed(path)),
None => residual.push(i),
}
}
@@ -159,7 +157,101 @@ impl LayeredHashDb {
residual = next;
}
- hashes.iter().copied().zip(results)
+ let out: Vec<(u64, Option>)> = hashes.iter().copied().zip(results).collect();
+ out.into_iter()
+ }
+
+ /// Resolve a batch without collecting it, calling `f` per hash as it resolves.
+ ///
+ /// The streaming counterpart to [`get_batch`](Self::get_batch), with the same
+ /// staging: the overlay answers first, then each base takes the residual. Hits
+ /// arrive layer by layer and, within a base, in arena order rather than input
+ /// order, so the first argument is the hash's position in `hashes`; hashes no
+ /// layer answers are reported last.
+ pub fn for_each_batch(&self, hashes: &[u64], mut f: impl FnMut(usize, u64, Option<&str>)) {
+ let mut residual: Vec = Vec::new();
+ for (i, &hash) in hashes.iter().enumerate() {
+ match self.overlay.get(&hash) {
+ Some(path) => f(i, hash, Some(path)),
+ None => residual.push(i),
+ }
+ }
+
+ for base in &self.bases {
+ if residual.is_empty() {
+ break;
+ }
+
+ let sub: Vec = residual.iter().map(|&i| hashes[i]).collect();
+ let mut next: Vec = Vec::new();
+ base.for_each_batch(&sub, |p, hash, path| match path {
+ Some(path) => f(residual[p], hash, Some(path)),
+ None => next.push(residual[p]),
+ });
+ residual = next;
+ }
+
+ for i in residual {
+ f(i, hashes[i], None);
+ }
+ }
+
+ /// Copy a path into `buf`, replacing what was there. `false` for a miss.
+ ///
+ /// See [`HashDb::get_into`] - the layered form consults the overlay first, then
+ /// each base in push order.
+ pub fn get_into(&self, hash: u64, buf: &mut String) -> bool {
+ if let Some(path) = self.overlay.get(&hash) {
+ buf.clear();
+ buf.push_str(path);
+ return true;
+ }
+
+ self.bases.iter().any(|base| base.get_into(hash, buf))
+ }
+
+ /// Every entry, overlay first and then each base in priority order.
+ ///
+ /// Shadowed entries are yielded once, by the layer that answers them - so this
+ /// enumerates exactly what [`get`](Self::get) can resolve. Each base is walked in
+ /// its own arena order, so its frames decompress once.
+ pub fn iter(&self) -> impl Iterator)> {
+ let overlay = self
+ .overlay
+ .iter()
+ .map(|(&hash, path)| (hash, PathRef::borrowed(path)));
+
+ let bases = self
+ .bases
+ .iter()
+ .enumerate()
+ .flat_map(move |(layer, base)| {
+ base.iter()
+ .filter(move |(hash, _)| !self.shadows(*hash, layer))
+ });
+
+ overlay.chain(bases)
+ }
+
+ /// Whether a layer above `layer` already answers `hash`.
+ fn shadows(&self, hash: u64, layer: usize) -> bool {
+ self.overlay.contains_key(&hash)
+ || self.bases[..layer].iter().any(|base| base.contains(hash))
+ }
+
+ /// Entries across every base, counting a key that appears in several of them once
+ /// per base rather than once outright.
+ ///
+ /// An upper bound on what [`iter`](Self::iter) yields, and O(bases) to compute -
+ /// the exact figure would cost a lookup per entry, so it is `iter().count()` when
+ /// a caller genuinely needs it.
+ pub fn base_len(&self) -> usize {
+ self.bases.iter().map(HashDb::len).sum()
+ }
+
+ /// Whether every base has read cleanly so far - see [`HashDb::is_healthy`].
+ pub fn is_healthy(&self) -> bool {
+ self.bases.iter().all(HashDb::is_healthy)
}
/// The base tables, in priority order.
@@ -173,6 +265,16 @@ impl LayeredHashDb {
}
}
+impl fmt::Debug for LayeredHashDb {
+ /// Shape only - the overlay's size and each base's shape. Never entries.
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.debug_struct("LayeredHashDb")
+ .field("overlay", &self.overlay.len())
+ .field("bases", &self.bases)
+ .finish()
+ }
+}
+
#[cfg(test)]
mod tests {
use std::io::Cursor;
@@ -319,6 +421,71 @@ mod tests {
let _ = LayeredHashDb::from_bases(vec![u64_base, u32_base]);
}
+ /// `iter` enumerates exactly what `get` can resolve: every shadowed key is
+ /// yielded once, by the layer that answers it.
+ #[test]
+ fn iter_yields_each_key_once_from_its_answering_layer() {
+ let base0 = raw_db(&[(1, "base0/one"), (2, "base0/two")]);
+ let base1 = raw_db(&[(2, "base1/two"), (3, "base1/three")]);
+ let mut db = LayeredHashDb::from_bases(vec![base0, base1]);
+ db.insert(1, "overlay/one");
+
+ let mut seen: Vec<(u64, String)> = db
+ .iter()
+ .map(|(hash, path)| (hash, path.into_owned()))
+ .collect();
+ seen.sort();
+
+ assert_eq!(
+ seen,
+ vec![
+ (1, "overlay/one".to_owned()),
+ (2, "base0/two".to_owned()),
+ (3, "base1/three".to_owned()),
+ ]
+ );
+
+ // Every yielded pair is what `get` answers with.
+ for (hash, path) in &seen {
+ assert_eq!(db.get(*hash).as_deref(), Some(path.as_str()));
+ }
+
+ // `base_len` counts the shadowed key in both bases, as documented.
+ assert_eq!(db.base_len(), 4);
+ }
+
+ #[test]
+ fn for_each_batch_matches_get_batch() {
+ let base0 = raw_db(&[(10, "base0/ten"), (20, "shadowed")]);
+ let base1 = raw_db(&[(20, "base1/twenty"), (30, "base1/thirty")]);
+ let mut db = LayeredHashDb::from_bases(vec![base0, base1]);
+ db.insert(5, "overlay/five");
+
+ let probes = [5u64, 10, 20, 30, 999, 10];
+ let mut streamed: Vec<(u64, Option)> = vec![(0, None); probes.len()];
+ db.for_each_batch(&probes, |i, hash, path| {
+ streamed[i] = (hash, path.map(str::to_owned));
+ });
+
+ let collected: Vec<(u64, Option)> = db
+ .get_batch(&probes)
+ .map(|(hash, path)| (hash, path.map(|p| p.into_owned())))
+ .collect();
+ assert_eq!(streamed, collected);
+ }
+
+ /// Debug prints shape, never entries.
+ #[test]
+ fn debug_prints_shape_only() {
+ let mut db = LayeredHashDb::from_bases(vec![raw_db(&[(1, "secret/path.bin")])]);
+ db.insert(2, "overlay/secret.bin");
+
+ let shown = format!("{db:?}");
+ assert!(shown.contains("overlay: 1"), "{shown}");
+ assert!(shown.contains("entries: 1"), "{shown}");
+ assert!(!shown.contains("secret"), "{shown}");
+ }
+
#[test]
fn insert_path_uses_first_base() {
let base = raw_db(&[(1, "seed")]);
diff --git a/crates/ltk_hashdb/src/lib.rs b/crates/ltk_hashdb/src/lib.rs
index edb17b8..8defb6f 100644
--- a/crates/ltk_hashdb/src/lib.rs
+++ b/crates/ltk_hashdb/src/lib.rs
@@ -8,20 +8,21 @@
//!
//! See `docs/FORMAT.md` for the byte-level spec.
+mod cache;
mod error;
-mod extended;
mod hash;
mod header;
mod layered;
+mod path;
mod reader;
mod writer;
pub use error::{BuildError, OpenError, VerifyError};
-pub use extended::ExtendedHashDb;
pub use hash::{Casing, HashKind};
pub use header::{FORMAT_VERSION, HEADER_SIZE, MAGIC};
pub use layered::LayeredHashDb;
-pub use reader::HashDb;
+pub use path::PathRef;
+pub use reader::{HashDb, HashDbOptions, WeakHashDb, DEFAULT_FRAME_CACHE_BYTES};
pub use writer::{BuildStats, HashDbWriter};
/// Width of the integer keys in a table.
diff --git a/crates/ltk_hashdb/src/path.rs b/crates/ltk_hashdb/src/path.rs
new file mode 100644
index 0000000..c57c329
--- /dev/null
+++ b/crates/ltk_hashdb/src/path.rs
@@ -0,0 +1,263 @@
+//! [`PathRef`]: a resolved path, borrowed from wherever its bytes already live.
+
+use std::borrow::Borrow;
+use std::fmt;
+use std::sync::Arc;
+
+use crate::cache::Frame;
+
+/// A path resolved from a table, without copying its bytes.
+///
+/// Derefs to [`str`], so it reads like one at the call site - `path.ends_with(".dds")`,
+/// `&*path`, `path.to_owned()`, and `Option::as_deref` all work unchanged. What it adds
+/// is where the bytes come from: a raw arena lends them straight out of the mmap, and a
+/// compressed arena lends them out of a decompressed frame the table is already holding.
+/// Neither allocates. Only two cases do: an entry that straddles a frame boundary, and
+/// one whose bytes are not valid UTF-8 (replaced lossily, as [`HashDb::get`] has always
+/// done).
+///
+/// A `PathRef` keeps its frame alive, so holding many of them across lookups pins that
+/// many frames in memory. Call [`into_owned`](Self::into_owned) to detach one.
+///
+/// [`HashDb::get`]: crate::HashDb::get
+#[derive(Clone)]
+pub struct PathRef<'a> {
+ repr: Repr<'a>,
+}
+
+#[derive(Clone)]
+enum Repr<'a> {
+ /// Straight out of a raw arena's mmap, or out of a `LayeredHashDb` overlay.
+ Borrowed(&'a str),
+
+ /// A range of a shared decompressed frame, kept alive by the `Arc`.
+ ///
+ /// The range is always valid UTF-8: `PathRef::from_frame` checks it and falls
+ /// back to `Owned` when it is not.
+ Frame {
+ frame: Arc,
+ start: u32,
+ len: u32,
+ },
+
+ /// Spliced across a frame boundary, or lossily replaced invalid UTF-8.
+ Owned(Box),
+}
+
+impl<'a> PathRef<'a> {
+ /// Borrow a path that is already a `&str` (raw arena, overlay entry).
+ pub(crate) fn borrowed(path: &'a str) -> Self {
+ Self {
+ repr: Repr::Borrowed(path),
+ }
+ }
+
+ /// Take ownership of a path that had to be built (spliced or lossy).
+ pub(crate) fn owned(path: impl Into>) -> Self {
+ Self {
+ repr: Repr::Owned(path.into()),
+ }
+ }
+
+ /// Borrow `frame[start..start + len]`, keeping the frame alive.
+ ///
+ /// Falls back to an owned, lossily-replaced copy when those bytes are not valid
+ /// UTF-8 - which also establishes the invariant [`Deref`](std::ops::Deref) relies on.
+ pub(crate) fn from_frame(frame: Arc, start: usize, len: usize) -> Self {
+ let bytes = &frame.bytes()[start..start + len];
+ match std::str::from_utf8(bytes) {
+ Ok(_) => Self {
+ repr: Repr::Frame {
+ start: start as u32,
+ len: len as u32,
+ frame,
+ },
+ },
+ Err(_) => Self::owned(String::from_utf8_lossy(bytes).into_owned()),
+ }
+ }
+
+ /// The path as a plain `&str`.
+ pub fn as_str(&self) -> &str {
+ match &self.repr {
+ Repr::Borrowed(path) => path,
+ Repr::Frame { frame, start, len } => {
+ let bytes = &frame.bytes()[*start as usize..(*start + *len) as usize];
+ // SAFETY: `Repr::Frame` is built only by `from_frame`, which validates
+ // exactly this range as UTF-8 and takes the `Owned` branch when it fails.
+ // The frame is immutable behind its `Arc`, so those bytes cannot change
+ // between that check and this read.
+ unsafe { std::str::from_utf8_unchecked(bytes) }
+ }
+ Repr::Owned(path) => path,
+ }
+ }
+
+ /// Whether these bytes were copied rather than borrowed.
+ ///
+ /// False for the paths a table lends out directly - from a raw arena's mmap, from
+ /// a cached frame, or from an overlay - which is every path in a well-formed table
+ /// but the two exceptions: one spliced across a frame boundary, and one whose bytes
+ /// were not valid UTF-8 and got lossily replaced. Assert on it to pin down that a
+ /// hot path allocates nothing; [`into_owned`](Self::into_owned) is free when it is
+ /// true.
+ pub fn is_owned(&self) -> bool {
+ matches!(self.repr, Repr::Owned(_))
+ }
+
+ /// Detach the path from its frame, copying it if it was borrowed.
+ ///
+ /// Use this to keep a handful of paths out of a large scan without pinning the
+ /// frames they came from.
+ pub fn into_owned(self) -> String {
+ match self.repr {
+ Repr::Owned(path) => path.into_string(),
+ _ => self.as_str().to_owned(),
+ }
+ }
+}
+
+impl std::ops::Deref for PathRef<'_> {
+ type Target = str;
+
+ fn deref(&self) -> &str {
+ self.as_str()
+ }
+}
+
+impl AsRef for PathRef<'_> {
+ fn as_ref(&self) -> &str {
+ self.as_str()
+ }
+}
+
+impl Borrow for PathRef<'_> {
+ fn borrow(&self) -> &str {
+ self.as_str()
+ }
+}
+
+/// Prints the path itself, quoted - a `PathRef` is a string as far as a reader cares.
+impl fmt::Debug for PathRef<'_> {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ fmt::Debug::fmt(self.as_str(), f)
+ }
+}
+
+impl fmt::Display for PathRef<'_> {
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ f.write_str(self.as_str())
+ }
+}
+
+impl PartialEq for PathRef<'_> {
+ fn eq(&self, other: &Self) -> bool {
+ self.as_str() == other.as_str()
+ }
+}
+
+impl Eq for PathRef<'_> {}
+
+impl PartialOrd for PathRef<'_> {
+ fn partial_cmp(&self, other: &Self) -> Option {
+ Some(self.cmp(other))
+ }
+}
+
+impl Ord for PathRef<'_> {
+ fn cmp(&self, other: &Self) -> std::cmp::Ordering {
+ self.as_str().cmp(other.as_str())
+ }
+}
+
+impl std::hash::Hash for PathRef<'_> {
+ fn hash(&self, state: &mut H) {
+ self.as_str().hash(state);
+ }
+}
+
+impl PartialEq for PathRef<'_> {
+ fn eq(&self, other: &str) -> bool {
+ self.as_str() == other
+ }
+}
+
+impl PartialEq<&str> for PathRef<'_> {
+ fn eq(&self, other: &&str) -> bool {
+ self.as_str() == *other
+ }
+}
+
+impl PartialEq for PathRef<'_> {
+ fn eq(&self, other: &String) -> bool {
+ self.as_str() == other.as_str()
+ }
+}
+
+impl PartialEq> for str {
+ fn eq(&self, other: &PathRef<'_>) -> bool {
+ self == other.as_str()
+ }
+}
+
+impl PartialEq> for &str {
+ fn eq(&self, other: &PathRef<'_>) -> bool {
+ *self == other.as_str()
+ }
+}
+
+impl PartialEq> for String {
+ fn eq(&self, other: &PathRef<'_>) -> bool {
+ self.as_str() == other.as_str()
+ }
+}
+
+impl From> for String {
+ fn from(path: PathRef<'_>) -> Self {
+ path.into_owned()
+ }
+}
+
+impl From> for Box {
+ fn from(path: PathRef<'_>) -> Self {
+ match path.repr {
+ Repr::Owned(path) => path,
+ _ => path.as_str().into(),
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn borrowed_and_owned_compare_and_print_alike() {
+ let borrowed = PathRef::borrowed("assets/foo.dds");
+ let owned = PathRef::owned("assets/foo.dds");
+
+ assert_eq!(borrowed, owned);
+ assert_eq!(borrowed, "assets/foo.dds");
+ assert_eq!("assets/foo.dds", borrowed);
+ assert_eq!(borrowed.to_string(), "assets/foo.dds");
+ assert_eq!(format!("{owned:?}"), "\"assets/foo.dds\"");
+
+ // Deref means str's whole API is available without an accessor.
+ assert!(borrowed.ends_with(".dds"));
+ assert_eq!(Some(owned).as_deref(), Some("assets/foo.dds"));
+ }
+
+ /// A frame-backed path borrows its bytes; invalid UTF-8 falls back to a lossy copy.
+ #[test]
+ fn frame_backed_paths_borrow_or_replace() {
+ let frame = Arc::new(Frame::from(b"aa/one.binbb/two.bin".to_vec()));
+ let path = PathRef::from_frame(Arc::clone(&frame), 10, 10);
+ assert_eq!(path, "bb/two.bin");
+ assert!(matches!(path.repr, Repr::Frame { .. }));
+
+ let invalid = Arc::new(Frame::from(vec![b'a', 0xff, b'b']));
+ let path = PathRef::from_frame(invalid, 0, 3);
+ assert_eq!(path, "a\u{fffd}b");
+ assert!(matches!(path.repr, Repr::Owned(_)));
+ }
+}
diff --git a/crates/ltk_hashdb/src/reader.rs b/crates/ltk_hashdb/src/reader.rs
index 4f412e3..66041c8 100644
--- a/crates/ltk_hashdb/src/reader.rs
+++ b/crates/ltk_hashdb/src/reader.rs
@@ -1,24 +1,132 @@
//! Read-only, mmap-backed `.hashdb` hash table.
use std::borrow::Cow;
+use std::cell::RefCell;
use std::collections::HashMap;
+use std::fmt;
use std::fs::File;
use std::ops::Range;
use std::path::Path;
-use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
+use std::sync::{Arc, Weak};
use xxhash_rust::xxh3::Xxh3;
use zeekstd::SeekTable;
+use crate::cache::{Frame, FrameCache};
use crate::header::Header;
-use crate::{Casing, HashKind, KeyWidth, OpenError, VerifyError};
+use crate::{Casing, HashKind, KeyWidth, OpenError, PathRef, VerifyError};
+
+/// Decompressed frame bytes a table caches by default: 4 MiB, i.e. 256 frames at the
+/// published 16 KiB frame size.
+///
+/// Enough that a consumer walking a WAD's chunks in path order keeps hitting the frames
+/// it just decompressed, small enough to be an unremarkable cost per open table.
+pub const DEFAULT_FRAME_CACHE_BYTES: usize = 4 << 20;
+
+/// Open-time knobs for a [`HashDb`].
+///
+/// ```no_run
+/// # use ltk_hashdb::HashDb;
+/// // A table used for one bulk pass needs no cache; one behind a UI wants a bigger one.
+/// let scratch = HashDb::options().frame_cache_bytes(0).open("game.lhdb")?;
+/// let resident = HashDb::options().frame_cache_bytes(32 << 20).open("game.lhdb")?;
+/// # Ok::<(), ltk_hashdb::OpenError>(())
+/// ```
+#[derive(Debug, Clone, Copy)]
+pub struct HashDbOptions {
+ frame_cache_bytes: usize,
+}
+
+impl Default for HashDbOptions {
+ fn default() -> Self {
+ Self {
+ frame_cache_bytes: DEFAULT_FRAME_CACHE_BYTES,
+ }
+ }
+}
+
+impl HashDbOptions {
+ pub fn new() -> Self {
+ Self::default()
+ }
+
+ /// Cap the decompressed frames this table keeps cached; `0` disables caching.
+ ///
+ /// Only compressed arenas cache anything - a raw arena is read straight out of the
+ /// mmap, so the budget is ignored.
+ pub fn frame_cache_bytes(mut self, bytes: usize) -> Self {
+ self.frame_cache_bytes = bytes;
+ self
+ }
+
+ /// mmap `path` read-only and validate it.
+ ///
+ /// # Errors
+ ///
+ /// Fails if the file cannot be opened or mapped, or if the header or section
+ /// bounds do not validate - see [`OpenError`].
+ ///
+ /// # Safety of the mapping
+ ///
+ /// See [`HashDb::open`] for the obligation a caller takes on by mapping a file.
+ pub fn open(self, path: impl AsRef) -> Result {
+ let file = File::open(path)?;
+ // SAFETY: `Mmap::map` is unsound if the mapped bytes change underneath us, so
+ // every writer of a `.hashdb` this crate ships must leave published files
+ // immutable. `ltk_mimir_cache` upholds that: `commit` writes a new versioned
+ // filename and renames it into place rather than over an existing one, and `gc`
+ // only unlinks (an unlinked file keeps its pages alive for anyone still mapping
+ // it). A caller mapping a file some other process may truncate or rewrite in
+ // place gets undefined behaviour, not an error - `HashDb::open` documents this.
+ let mmap = unsafe { memmap2::Mmap::map(&file)? };
+ HashDb::from_backing(Backing::Mmap(mmap), self)
+ }
+
+ /// Open an in-memory image (embedded tables, tests).
+ ///
+ /// # Errors
+ ///
+ /// Fails if the header or section bounds do not validate - see [`OpenError`].
+ pub fn open_bytes(self, bytes: impl Into>) -> Result {
+ HashDb::from_backing(Backing::Bytes(bytes.into()), self)
+ }
+}
/// A read-only `.hashdb` hash table.
///
/// `open` validates the (untrusted) header and section bounds. Lookups binary-search
/// the mmap'd key array, so a miss never touches the arena; a hit on a compressed
-/// arena decompresses only the containing frame(s).
+/// arena decompresses only the containing frame, and keeps it cached for the lookups
+/// after it.
+///
+/// Cloning is cheap - every clone shares one mapping, one seek table, and one frame
+/// cache - so a `HashDb` is passed around rather than reopened.
+#[derive(Clone)]
pub struct HashDb {
+ inner: Arc,
+}
+
+/// A handle that does not keep its table mapped.
+///
+/// For a registry of open tables: keep these rather than [`HashDb`]s so the registry
+/// does not pin every table anyone ever opened. [`upgrade`](WeakHashDb::upgrade) hands
+/// back a live handle while one is still held elsewhere.
+#[derive(Clone, Debug)]
+pub struct WeakHashDb {
+ inner: Weak,
+}
+
+impl WeakHashDb {
+ /// A live handle to the table, or `None` once the last one was dropped.
+ pub fn upgrade(&self) -> Option {
+ self.inner.upgrade().map(|inner| HashDb { inner })
+ }
+}
+
+/// Everything a table's clones share: the bytes, where the sections are, and the
+/// frames decompressed out of them so far.
+struct Inner {
backing: Backing,
header: Header,
keys: Range,
@@ -29,8 +137,14 @@ pub struct HashDb {
/// Present iff the arena is a zeekstd seekable stream.
seek_table: Option,
+ cache: FrameCache,
+
/// Frames decompressed so far; misses must never bump it (see unit tests).
decompressions: AtomicU64,
+
+ /// Cleared for good the first time a lookup swallows a decompression failure,
+ /// so "this build knows nothing" can be told apart from "this file is broken".
+ healthy: AtomicBool,
}
enum Backing {
@@ -47,24 +161,109 @@ impl Backing {
}
}
-/// A decompressed run of frames (raw-arena range it covers + the bytes), so
-/// in-order consumers decompress each frame once rather than once per entry.
-type FrameCache = Option<(Range, Vec)>;
+/// Entry bytes as they were found: lent by the mmap, lent by a cached frame, or
+/// spliced together because the entry crossed a frame boundary.
+enum Bytes<'a> {
+ Borrowed(&'a [u8]),
+ Frame {
+ frame: Arc,
+ start: usize,
+ len: usize,
+ },
+ Spliced(Vec),
+}
+
+impl Bytes<'_> {
+ fn as_slice(&self) -> &[u8] {
+ match self {
+ Self::Borrowed(bytes) => bytes,
+ Self::Frame { frame, start, len } => &frame.bytes()[*start..*start + *len],
+ Self::Spliced(bytes) => bytes,
+ }
+ }
+}
+
+impl<'a> From> for PathRef<'a> {
+ fn from(bytes: Bytes<'a>) -> Self {
+ match bytes {
+ Bytes::Borrowed(bytes) => match String::from_utf8_lossy(bytes) {
+ Cow::Borrowed(path) => PathRef::borrowed(path),
+ Cow::Owned(path) => PathRef::owned(path),
+ },
+ Bytes::Frame { frame, start, len } => PathRef::from_frame(frame, start, len),
+ Bytes::Spliced(bytes) => match String::from_utf8(bytes) {
+ Ok(path) => PathRef::owned(path),
+ Err(e) => PathRef::owned(String::from_utf8_lossy(e.as_bytes()).into_owned()),
+ },
+ }
+ }
+}
+
+thread_local! {
+ /// One decompression context per thread: creating one per frame would cost more
+ /// than the decompression, and sharing one across threads would serialize them.
+ static DCTX: RefCell>> = const { RefCell::new(None) };
+}
+
+/// Run `f` against this thread's decompression context, creating it on first use.
+fn with_dctx(
+ f: impl FnOnce(&mut zstd::bulk::Decompressor<'static>) -> Result,
+) -> Result {
+ DCTX.with(|cell| {
+ let mut slot = cell.borrow_mut();
+ let dctx = match &mut *slot {
+ Some(dctx) => dctx,
+ slot => slot.insert(zstd::bulk::Decompressor::new()?),
+ };
+
+ f(dctx)
+ })
+}
impl HashDb {
/// mmap `path` read-only and validate it.
+ ///
+ /// # Errors
+ ///
+ /// Fails if the file cannot be opened or mapped, or if the header or section
+ /// bounds do not validate - see [`OpenError`].
+ ///
+ /// # Safety of the mapping
+ ///
+ /// The mapping is only sound while the file's bytes do not change. Published
+ /// `.hashdb` files are immutable by contract - a new version ships as a new
+ /// filename, and old versions are only ever unlinked - which is what discharges
+ /// the `unsafe` here. If you manage your own tables, uphold the same rule: never
+ /// truncate or rewrite a file in place while a `HashDb` maps it. Doing so is
+ /// undefined behaviour rather than an error you can catch. Build to a temporary
+ /// name and rename, or use [`open_bytes`](HashDb::open_bytes) for images you
+ /// mutate.
pub fn open(path: impl AsRef) -> Result {
- let file = File::open(path)?;
- let mmap = unsafe { memmap2::Mmap::map(&file)? };
- Self::from_backing(Backing::Mmap(mmap))
+ HashDbOptions::default().open(path)
}
/// Open an in-memory image (embedded tables, tests).
+ ///
+ /// # Errors
+ ///
+ /// Fails if the header or section bounds do not validate - see [`OpenError`].
pub fn open_bytes(bytes: impl Into>) -> Result {
- Self::from_backing(Backing::Bytes(bytes.into()))
+ HashDbOptions::default().open_bytes(bytes)
+ }
+
+ /// Open-time knobs - the frame cache budget, today.
+ pub fn options() -> HashDbOptions {
+ HashDbOptions::default()
+ }
+
+ /// A handle that does not keep this table mapped - see [`WeakHashDb`].
+ pub fn downgrade(&self) -> WeakHashDb {
+ WeakHashDb {
+ inner: Arc::downgrade(&self.inner),
+ }
}
- fn from_backing(backing: Backing) -> Result {
+ fn from_backing(backing: Backing, options: HashDbOptions) -> Result {
let data = backing.bytes();
let header = Header::decode(data)?;
@@ -123,30 +322,97 @@ impl HashDb {
None
};
+ // Size the cache to this table: never more slots than it has frames, and
+ // nothing at all for a raw arena, which is read straight out of the mmap.
+ let cache = match &seek_table {
+ Some(st) => FrameCache::new(
+ options.frame_cache_bytes,
+ st.max_frame_size_decomp() as usize,
+ st.num_frames() as usize,
+ ),
+ None => FrameCache::new(0, 0, 0),
+ };
+
// Per-entry extents aren't validated here (keeps `open` O(1)); each read
// bounds-checks its own, reading out-of-bounds as a miss. `verify()` reports them.
Ok(Self {
- backing,
- header,
- keys,
- offsets,
- lengths,
- arena,
- seek_table,
- decompressions: AtomicU64::new(0),
+ inner: Arc::new(Inner {
+ backing,
+ header,
+ keys,
+ offsets,
+ lengths,
+ arena,
+ seek_table,
+ cache,
+ decompressions: AtomicU64::new(0),
+ healthy: AtomicBool::new(true),
+ }),
})
}
- /// Look up a hash. Raw arenas borrow the path from the mmap; compressed arenas
- /// decompress its frame(s). Returns `None` for a miss or an entry that won't
- /// decompress (corrupt file - see [`HashDb::verify`]).
- pub fn get(&self, hash: u64) -> Option> {
- self.index_of(hash).and_then(|i| self.str_at(i))
+ /// Look up a hash. The path is lent by the mmap or by a cached frame, so a hit
+ /// allocates nothing. Returns `None` for a miss or an entry that won't decompress
+ /// (corrupt file - see [`HashDb::verify`]).
+ pub fn get(&self, hash: u64) -> Option> {
+ let i = self.inner.index_of(hash)?;
+ self.inner.lookup(i).map(PathRef::from)
+ }
+
+ /// Look up a hash, surfacing a corrupt arena instead of reporting it as a miss.
+ ///
+ /// [`get`](HashDb::get) follows the format's rule that an entry which will not
+ /// decompress reads as a miss. That is right for resolving names and wrong for
+ /// telling a user why every name in an install came back unknown, so this returns
+ /// the failure at the call site instead. `Ok(None)` is a genuine miss.
+ ///
+ /// # Errors
+ ///
+ /// Fails with [`VerifyError`] when the entry's frame will not decompress or its
+ /// extent runs outside the arena - always a corrupt file.
+ pub fn try_get(&self, hash: u64) -> Result>, VerifyError> {
+ let Some(i) = self.inner.index_of(hash) else {
+ return Ok(None);
+ };
+
+ Ok(self.inner.bytes_at(i)?.map(PathRef::from))
+ }
+
+ /// Copy a path into `buf`, replacing what was there. `false` for a miss.
+ ///
+ /// The counterpart to [`get`](HashDb::get) for a caller looping over a reusable
+ /// buffer: it copies the bytes out and holds no frame afterwards, where a retained
+ /// [`PathRef`] keeps its frame resident.
+ pub fn get_into(&self, hash: u64, buf: &mut String) -> bool {
+ let Some(i) = self.inner.index_of(hash) else {
+ return false;
+ };
+ let Some(bytes) = self.inner.lookup(i) else {
+ return false;
+ };
+
+ buf.clear();
+ match std::str::from_utf8(bytes.as_slice()) {
+ Ok(path) => buf.push_str(path),
+ Err(_) => buf.push_str(&String::from_utf8_lossy(bytes.as_slice())),
+ }
+
+ true
+ }
+
+ /// Whether every lookup so far has read cleanly.
+ ///
+ /// Turns false for good the first time a lookup hits an entry that will not
+ /// decompress - bit rot, a truncated write - and stays false. Nothing re-verifies
+ /// an installed table after its download checksum, so this is the cheap signal that
+ /// a table needs [`verify`](HashDb::verify) rather than a redownload of the world.
+ pub fn is_healthy(&self) -> bool {
+ self.inner.healthy.load(Ordering::Relaxed)
}
/// Membership test; never touches the arena.
pub fn contains(&self, hash: u64) -> bool {
- self.index_of(hash).is_some()
+ self.inner.index_of(hash).is_some()
}
/// Bulk lookup. Hits resolve in arena order so each frame decompresses at most
@@ -154,147 +420,189 @@ impl HashDb {
pub fn get_batch<'a>(
&'a self,
hashes: &[u64],
- ) -> impl Iterator>)> + 'a {
- let indices: Vec> = hashes.iter().map(|&h| self.index_of(h)).collect();
+ ) -> impl Iterator>)> + 'a {
+ let indices: Vec> = hashes.iter().map(|&h| self.inner.index_of(h)).collect();
let mut order: Vec = (0..hashes.len()).collect();
// Resolve hits in arena order (misses sort last) so each frame decompresses once.
- order.sort_unstable_by_key(|&p| indices[p].map_or(u64::MAX, |i| self.offset_at(i)));
+ order.sort_unstable_by_key(|&p| indices[p].map_or(u64::MAX, |i| self.inner.offset_at(i)));
- let mut results: Vec>> = Vec::new();
+ let mut results: Vec>> = Vec::new();
results.resize_with(hashes.len(), || None);
- let mut cache: FrameCache = None;
for p in order {
if let Some(i) = indices[p] {
- results[p] = self.str_at_cached(i, &mut cache);
+ results[p] = self.inner.lookup(i).map(PathRef::from);
}
}
- let out: Vec<(u64, Option>)> = hashes.iter().copied().zip(results).collect();
+ let out: Vec<(u64, Option>)> = hashes.iter().copied().zip(results).collect();
out.into_iter()
}
+ /// Resolve a batch without collecting it, calling `f` per hash as it resolves.
+ ///
+ /// The allocation-free batch: where [`get_batch`](HashDb::get_batch) materialises
+ /// every result before yielding the first, this hands each one straight to `f`.
+ /// Calls arrive in **arena order**, not input order - that is what lets each frame
+ /// decompress once - so the first argument is the hash's position in `hashes`.
+ pub fn for_each_batch(&self, hashes: &[u64], mut f: impl FnMut(usize, u64, Option<&str>)) {
+ let indices: Vec> = hashes.iter().map(|&h| self.inner.index_of(h)).collect();
+ let mut order: Vec = (0..hashes.len()).collect();
+ order.sort_unstable_by_key(|&p| indices[p].map_or(u64::MAX, |i| self.inner.offset_at(i)));
+
+ for p in order {
+ let path = indices[p].and_then(|i| self.inner.lookup(i));
+ match path {
+ Some(bytes) => {
+ let path = PathRef::from(bytes);
+ f(p, hashes[p], Some(&path));
+ }
+ None => f(p, hashes[p], None),
+ }
+ }
+ }
+
pub fn len(&self) -> usize {
- self.header.entry_count as usize
+ self.inner.len()
}
pub fn is_empty(&self) -> bool {
- self.header.entry_count == 0
+ self.inner.header.entry_count == 0
}
pub fn key_width(&self) -> KeyWidth {
- self.header.key_width
+ self.inner.header.key_width
}
pub fn hash_kind(&self) -> HashKind {
- self.header.hash_kind
+ self.inner.header.hash_kind
}
/// Whether the keys hash the lowercased path (from the `case_insensitive`
/// header flag).
pub fn casing(&self) -> Casing {
- self.header.casing()
+ self.inner.header.casing()
}
/// Whether the arena is zeekstd-compressed on disk.
pub fn is_compressed(&self) -> bool {
- self.header.arena_compressed()
+ self.inner.header.arena_compressed()
}
/// Total length of all path strings (the raw arena), in bytes.
pub fn arena_decompressed_size(&self) -> u64 {
- self.header.arena_decompressed_size
+ self.inner.header.arena_decompressed_size
}
/// Bytes the arena occupies on disk (== decompressed size for raw arenas).
pub fn arena_compressed_size(&self) -> u64 {
- self.header.arena_compressed_size
+ self.inner.header.arena_compressed_size
}
/// Number of zstd frames this table has decompressed over its lifetime.
#[cfg(test)]
pub(crate) fn decompressions(&self) -> u64 {
- self.decompressions.load(Ordering::Relaxed)
+ self.inner.decompressions.load(Ordering::Relaxed)
}
/// Hash a path string with **this table's** algorithm and casing rule (from
/// the `hash_kind` header field - falling back on key width when
/// unspecified - and the `case_insensitive` flag).
pub fn hash_path(&self, path: &str) -> u64 {
- self.header
- .hash_kind
- .hash(path, self.header.casing(), self.header.key_width)
+ self.inner.header.hash_kind.hash(
+ path,
+ self.inner.header.casing(),
+ self.inner.header.key_width,
+ )
}
/// Iterate entries in arena order (path order, **not** key order) so each frame
/// decompresses once. Entries that fail to decompress are skipped; `verify()` reports them.
- pub fn iter(&self) -> impl Iterator)> {
- let mut cache: FrameCache = None;
- self.arena_order().into_iter().filter_map(move |i| {
- let path = self.str_at_cached(i, &mut cache)?;
- Some((self.key_at(i), path))
+ pub fn iter(&self) -> impl Iterator)> {
+ self.inner.arena_order().into_iter().filter_map(move |i| {
+ let bytes = self.inner.lookup(i)?;
+ Some((self.inner.key_at(i), PathRef::from(bytes)))
})
}
/// Opt-in fully-resident mode: decode everything into an owned map.
pub fn load_all(&self) -> HashMap> {
- self.iter()
- .map(|(k, s)| (k, s.into_owned().into_boxed_str()))
- .collect()
+ self.iter().map(|(k, path)| (k, path.into())).collect()
}
/// Full integrity check, skipped by `open`:
/// - xxh3 checksum over the stored sections
/// - keys strictly ascending
/// - every entry in bounds and valid UTF-8 in the arena
+ ///
+ /// # Errors
+ ///
+ /// Fails with [`VerifyError`] on the first problem it finds: a checksum mismatch,
+ /// out-of-order keys, an entry that runs past the arena, one that is not valid
+ /// UTF-8, or a frame that will not decompress.
pub fn verify(&self) -> Result<(), VerifyError> {
- let data = self.backing.bytes();
+ let inner = &*self.inner;
+ let data = inner.backing.bytes();
let mut hasher = Xxh3::new();
- hasher.update(&data[self.keys.clone()]);
- hasher.update(&data[self.offsets.clone()]);
- hasher.update(&data[self.lengths.clone()]);
- hasher.update(&data[self.arena.clone()]);
- if hasher.digest() != self.header.checksum {
+ hasher.update(&data[inner.keys.clone()]);
+ hasher.update(&data[inner.offsets.clone()]);
+ hasher.update(&data[inner.lengths.clone()]);
+ hasher.update(&data[inner.arena.clone()]);
+ if hasher.digest() != inner.header.checksum {
return Err(VerifyError::ChecksumMismatch);
}
- let n = self.len();
+ let n = inner.len();
for i in 1..n {
- if self.key_at(i - 1) >= self.key_at(i) {
+ if inner.key_at(i - 1) >= inner.key_at(i) {
return Err(VerifyError::Malformed("keys not strictly ascending"));
}
}
- if self.seek_table.is_none() {
- let arena = &data[self.arena.clone()];
+ // Compressed arenas are walked in arena order so each frame decompresses once
+ // and only the current run is resident - never the whole arena at once. A raw
+ // arena is already resident, so key order is as good and needs no permutation.
+ if inner.seek_table.is_none() {
for i in 0..n {
- let slice = self
- .extent_of(i)
- .and_then(|(start, end)| arena.get(start as usize..end as usize))
- .ok_or(VerifyError::Malformed("entry extends past the arena"))?;
- if std::str::from_utf8(slice).is_err() {
- return Err(VerifyError::Malformed("entry is not valid UTF-8"));
- }
+ inner.verify_entry(i)?;
}
- return Ok(());
- }
-
- // Compressed: walk entries in arena order so each frame decompresses once
- // and only the current run is resident - never the whole arena at once.
- let mut cache: FrameCache = None;
- for i in self.arena_order() {
- let (start, end) = self
- .extent_of(i)
- .ok_or(VerifyError::Malformed("entry extends past the arena"))?;
- if start == end {
- continue;
- }
- let slice = self.frame_bytes(start, end, &mut cache)?;
- if std::str::from_utf8(slice).is_err() {
- return Err(VerifyError::Malformed("entry is not valid UTF-8"));
+ } else {
+ for i in inner.arena_order() {
+ inner.verify_entry(i)?;
}
}
+
Ok(())
}
+}
+
+impl fmt::Debug for HashDb {
+ /// Shape only - counts, widths, and flags. Never entries.
+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+ let inner = &*self.inner;
+ f.debug_struct("HashDb")
+ .field("entries", &inner.header.entry_count)
+ .field("key_width", &inner.header.key_width)
+ .field("hash_kind", &inner.header.hash_kind)
+ .field("casing", &inner.header.casing())
+ .field("compressed", &inner.header.arena_compressed())
+ .field(
+ "arena_decompressed_size",
+ &inner.header.arena_decompressed_size,
+ )
+ .field("arena_compressed_size", &inner.header.arena_compressed_size)
+ .field(
+ "frames",
+ &inner.seek_table.as_ref().map_or(0, SeekTable::num_frames),
+ )
+ .field("cached_frames", &inner.cache.capacity())
+ .finish()
+ }
+}
+
+impl Inner {
+ fn len(&self) -> usize {
+ self.header.entry_count as usize
+ }
/// Entry indices sorted by arena offset (path order); walking them this way
/// decompresses each frame once, keeping only the current run resident.
@@ -339,84 +647,129 @@ impl HashDb {
(end <= self.header.arena_decompressed_size).then_some((start, end))
}
- /// The path for entry `i`, or `None` if out of bounds. Invalid UTF-8 is replaced
- /// lossily rather than panicking; `verify()` reports both.
- fn str_at(&self, i: usize) -> Option> {
- self.str_at_cached(i, &mut None)
- }
+ /// Entry `i`'s bytes, or `None` if its extent runs past the arena.
+ ///
+ /// Errors mean a corrupt file ([`VerifyError`]); the lookup path swallows them
+ /// into a miss, only `verify` surfaces them.
+ fn bytes_at(&self, i: usize) -> Result>, VerifyError> {
+ let Some((start, end)) = self.extent_of(i) else {
+ return Ok(None);
+ };
- fn str_at_cached(&self, i: usize, cache: &mut FrameCache) -> Option> {
- let (start, end) = self.extent_of(i)?;
- if self.seek_table.is_none() {
+ let Some(seek_table) = self.seek_table.as_ref() else {
let range = self.arena.start + start as usize..self.arena.start + end as usize;
- return Some(String::from_utf8_lossy(&self.backing.bytes()[range]));
- }
+ return Ok(Some(Bytes::Borrowed(&self.backing.bytes()[range])));
+ };
if start == end {
- return Some(Cow::Borrowed(""));
+ return Ok(Some(Bytes::Borrowed(&[])));
}
- let bytes = self.frame_bytes(start, end, cache).ok()?;
- Some(Cow::Owned(String::from_utf8_lossy(bytes).into_owned()))
+
+ let first = seek_table.frame_index_decomp(start);
+ let last = seek_table.frame_index_decomp(end - 1);
+ let frame = self.frame(first)?;
+ let frame_start = seek_table.frame_start_decomp(first)?;
+ let offset = (start - frame_start) as usize;
+
+ if first == last {
+ let len = (end - start) as usize;
+ if offset + len > frame.bytes().len() {
+ return Err(VerifyError::Malformed("entry extends past its frame"));
+ }
+
+ return Ok(Some(Bytes::Frame {
+ frame,
+ start: offset,
+ len,
+ }));
+ }
+
+ // The entry straddles a frame boundary - roughly one entry per frame. Splice
+ // the pieces into a buffer of its own rather than lending out either frame.
+ let mut spliced = Vec::with_capacity((end - start) as usize);
+ let head = frame
+ .bytes()
+ .get(offset..)
+ .ok_or(VerifyError::Malformed("entry starts past its frame"))?;
+ spliced.extend_from_slice(head);
+
+ for index in first + 1..=last {
+ let frame = self.frame(index)?;
+ let frame_start = seek_table.frame_start_decomp(index)?;
+ let take = (end - frame_start).min(frame.bytes().len() as u64) as usize;
+ let tail = frame.bytes().get(..take).ok_or(VerifyError::Malformed(
+ "frame shorter than the seek table says",
+ ))?;
+ spliced.extend_from_slice(tail);
+ }
+
+ Ok(Some(Bytes::Spliced(spliced)))
}
- /// Decompressed bytes for extent `start..end`, filling `cache` with the
- /// containing frame(s) on a miss. Caller must handle the empty (`start == end`) case.
- ///
- /// Errors mean a corrupt file ([`VerifyError`]); the lookup path swallows
- /// them into a miss, only `verify` surfaces them.
- fn frame_bytes<'c>(
- &self,
- start: u64,
- end: u64,
- cache: &'c mut FrameCache,
- ) -> Result<&'c [u8], VerifyError> {
- let covered = cache
- .as_ref()
- .is_some_and(|(range, _)| range.start <= start && end <= range.end);
- if !covered {
- let st = self.seek_table.as_ref().unwrap();
- let lo = st.frame_index_decomp(start);
- let hi = st.frame_index_decomp(end - 1);
- let (cov_start, bytes) = self.read_frames(lo, hi)?;
- *cache = Some((cov_start..cov_start + bytes.len() as u64, bytes));
+ /// Entry `i`'s bytes for a lookup: a failure to decompress reads as a miss, and
+ /// trips the health flag on its way past.
+ fn lookup(&self, i: usize) -> Option> {
+ match self.bytes_at(i) {
+ Ok(bytes) => bytes,
+ Err(_) => {
+ self.healthy.store(false, Ordering::Relaxed);
+ None
+ }
+ }
+ }
+
+ /// Check one entry the way `verify` needs it checked: in bounds and valid UTF-8.
+ fn verify_entry(&self, i: usize) -> Result<(), VerifyError> {
+ let bytes = self
+ .bytes_at(i)?
+ .ok_or(VerifyError::Malformed("entry extends past the arena"))?;
+ if std::str::from_utf8(bytes.as_slice()).is_err() {
+ return Err(VerifyError::Malformed("entry is not valid UTF-8"));
+ }
+
+ Ok(())
+ }
+
+ /// Frame `index`, decompressing it only if it is not already cached.
+ fn frame(&self, index: u32) -> Result, VerifyError> {
+ if let Some(frame) = self.cache.get(index) {
+ return Ok(frame);
}
- let (range, bytes) = cache.as_ref().unwrap();
- Ok(&bytes[(start - range.start) as usize..(end - range.start) as usize])
+
+ // Decompressed outside any cache lock, so concurrent readers never wait on each
+ // other. Two threads racing on one frame both decompress it; the loser's copy is
+ // simply dropped, which is cheaper than serializing every miss.
+ let frame = Arc::new(Frame::from(self.decompress(index)?));
+ self.cache.insert(index, &frame);
+
+ Ok(frame)
}
- /// Decompress frames `lo..=hi`, returning the run's decompressed-space start
- /// offset plus the bytes. Frame content is untrusted, so every extent and
- /// output size is checked.
- fn read_frames(&self, lo: u32, hi: u32) -> Result<(u64, Vec), VerifyError> {
- let st = self.seek_table.as_ref().expect("compressed arena");
+ /// Decompress one frame. Frame content is untrusted, so the extent and the
+ /// resulting size are both checked.
+ fn decompress(&self, index: u32) -> Result, VerifyError> {
+ let seek_table = self.seek_table.as_ref().expect("compressed arena");
let arena = &self.backing.bytes()[self.arena.clone()];
- let d_start = st.frame_start_decomp(lo)?;
- let total = st.frame_end_decomp(hi)? - d_start;
+ let start = seek_table.frame_start_comp(index)? as usize;
+ let end = seek_table.frame_end_comp(index)? as usize;
+ let size = seek_table.frame_size_decomp(index)? as usize;
+ let compressed = arena
+ .get(start..end)
+ .ok_or(VerifyError::Malformed("frame extent out of arena bounds"))?;
+
// Cap the capacity hint at the (header-pinned) arena size so a corrupt seek
- // table can't force a huge allocation; a full read still allocates once.
- let cap = total.min(self.header.arena_decompressed_size) as usize;
- // Decompress each frame straight into `out` via a cursor - one reusable
- // context, no per-frame buffer, no second copy.
- let mut out = std::io::Cursor::new(Vec::with_capacity(cap));
- let mut dctx = zstd::bulk::Decompressor::new()?;
- for f in lo..=hi {
- let c_start = st.frame_start_comp(f)? as usize;
- let c_end = st.frame_end_comp(f)? as usize;
- let d_size = st.frame_size_decomp(f)? as usize;
- let frame = arena
- .get(c_start..c_end)
- .ok_or(VerifyError::Malformed("frame extent out of arena bounds"))?;
- self.decompressions.fetch_add(1, Ordering::Relaxed);
- out.get_mut().reserve(d_size);
- let pos = out.position();
- let n = dctx.decompress_to_buffer(frame, &mut out)?;
- if n != d_size {
- return Err(VerifyError::Malformed(
- "frame decompressed to unexpected size",
- ));
- }
- out.set_position(pos + n as u64);
+ // table can't force a huge allocation.
+ let capacity = size.min(self.header.arena_decompressed_size as usize);
+ let mut buffer = self.cache.take_buffer(capacity);
+
+ self.decompressions.fetch_add(1, Ordering::Relaxed);
+ let written = with_dctx(|dctx| Ok(dctx.decompress_to_buffer(compressed, &mut buffer)?))?;
+ if written != size {
+ return Err(VerifyError::Malformed(
+ "frame decompressed to unexpected size",
+ ));
}
- Ok((d_start, out.into_inner()))
+
+ Ok(buffer)
}
}
@@ -449,6 +802,18 @@ mod tests {
use crate::{Compression, HashDbWriter, KeyWidth};
fn compressed_db(frame_size: u32) -> HashDb {
+ compressed_db_with(frame_size, super::DEFAULT_FRAME_CACHE_BYTES)
+ }
+
+ fn compressed_db_with(frame_size: u32, cache_bytes: usize) -> HashDb {
+ HashDb::options()
+ .frame_cache_bytes(cache_bytes)
+ .open_bytes(compressed_bytes(frame_size))
+ .expect("open")
+ }
+
+ /// 100 clustered paths, keyed `i * 3`, spanning several frames.
+ fn compressed_bytes(frame_size: u32) -> Vec {
let mut w = HashDbWriter::new(
KeyWidth::U64,
Compression::Zeekstd {
@@ -464,7 +829,7 @@ mod tests {
}
let mut out = Cursor::new(Vec::new());
w.build(&mut out).expect("build");
- HashDb::open_bytes(out.into_inner()).expect("open")
+ out.into_inner()
}
/// A miss is decided by the key array alone - never a frame decompression.
@@ -475,10 +840,10 @@ mod tests {
assert_eq!(db.get(probe), None);
}
assert!(!db.contains(999));
- assert_eq!(db.decompressions.load(Ordering::Relaxed), 0);
+ assert_eq!(db.inner.decompressions.load(Ordering::Relaxed), 0);
assert!(db.get(0).is_some());
- assert!(db.decompressions.load(Ordering::Relaxed) > 0);
+ assert!(db.inner.decompressions.load(Ordering::Relaxed) > 0);
}
/// In-order iteration decompresses each frame once, not once per entry.
@@ -486,9 +851,98 @@ mod tests {
fn iter_decompresses_each_frame_once() {
let db = compressed_db(128);
assert_eq!(db.iter().count(), 100);
- let frames = db.seek_table.as_ref().unwrap().num_frames() as u64;
+ let frames = db.inner.seek_table.as_ref().unwrap().num_frames() as u64;
assert!(frames > 1, "fixture should span multiple frames");
// Boundary-straddling entries decompress both frames, so allow one re-read each.
- assert!(db.decompressions.load(Ordering::Relaxed) <= 2 * frames);
+ assert!(db.inner.decompressions.load(Ordering::Relaxed) <= 2 * frames);
+ }
+
+ /// The point of the cache: repeating a lookup must not decompress again, and
+ /// neighbouring keys share the frame their neighbour just paid for.
+ #[test]
+ fn cached_frames_are_not_decompressed_twice() {
+ let db = compressed_db(128);
+ assert!(db.get(0).is_some());
+ let after_first = db.decompressions();
+
+ for _ in 0..10 {
+ assert!(db.get(0).is_some());
+ }
+ assert_eq!(db.decompressions(), after_first, "repeat lookups were free");
+
+ // A clone shares the cache rather than starting its own.
+ let clone = db.clone();
+ assert!(clone.get(0).is_some());
+ assert_eq!(clone.decompressions(), after_first);
+ }
+
+ /// A table whose arena no longer decompresses must degrade to misses, say so via
+ /// `is_healthy`, and surface the reason through `try_get` - never panic.
+ #[test]
+ fn a_corrupt_frame_reports_unhealthy() {
+ let mut bytes = compressed_bytes(128);
+ // The seek table lives at the end of the arena, so scribbling over the start
+ // leaves the file openable and breaks only the frames it lands on.
+ let arena_offset = u64::from_le_bytes(bytes[40..48].try_into().unwrap()) as usize;
+ for byte in &mut bytes[arena_offset..arena_offset + 64] {
+ *byte = 0xff;
+ }
+
+ let db = HashDb::open_bytes(bytes).expect("open");
+ assert!(db.is_healthy(), "nothing has been read yet");
+
+ let keys: Vec = (0..100u64).map(|i| i * 3).collect();
+ let swallowed = keys.iter().filter(|&&k| db.get(k).is_none()).count();
+ assert!(swallowed > 0, "the corrupt frame should swallow lookups");
+ assert!(!db.is_healthy(), "and say so afterwards");
+
+ // The same lookups report the corruption when asked to.
+ let surfaced = keys.iter().filter(|&&k| db.try_get(k).is_err()).count();
+ assert_eq!(surfaced, swallowed);
+
+ // Entries in the untouched frames still resolve.
+ assert!(keys.iter().any(|&k| db.get(k).is_some()));
+ }
+
+ #[test]
+ fn for_each_batch_matches_get_batch() {
+ let db = compressed_db(128);
+ let probes = [0u64, 3, 297, 1, 99, 0];
+
+ let mut streamed: Vec<(u64, Option)> = vec![(0, None); probes.len()];
+ db.for_each_batch(&probes, |i, hash, path| {
+ streamed[i] = (hash, path.map(str::to_owned));
+ });
+
+ let collected: Vec<(u64, Option)> = db
+ .get_batch(&probes)
+ .map(|(hash, path)| (hash, path.map(|p| p.into_owned())))
+ .collect();
+ assert_eq!(streamed, collected);
+ }
+
+ #[test]
+ fn get_into_reuses_the_callers_buffer() {
+ let db = compressed_db(128);
+ let mut buf = String::from("stale contents");
+
+ assert!(db.get_into(0, &mut buf));
+ assert_eq!(buf, "assets/characters/champ0/skins/skin0.bin");
+
+ // A miss leaves the buffer alone rather than clearing it.
+ assert!(!db.get_into(1, &mut buf));
+ assert_eq!(buf, "assets/characters/champ0/skins/skin0.bin");
+ }
+
+ /// With caching off, every lookup pays for its own frame - the old behaviour,
+ /// still available for one-shot passes.
+ #[test]
+ fn a_disabled_cache_decompresses_every_time() {
+ let db = compressed_db_with(128, 0);
+ assert!(db.get(0).is_some());
+ let after_first = db.decompressions();
+
+ assert!(db.get(0).is_some());
+ assert!(db.decompressions() > after_first);
}
}
diff --git a/crates/ltk_hashdb/tests/roundtrip.rs b/crates/ltk_hashdb/tests/roundtrip.rs
index b96a0db..e8c9cd2 100644
--- a/crates/ltk_hashdb/tests/roundtrip.rs
+++ b/crates/ltk_hashdb/tests/roundtrip.rs
@@ -1,10 +1,9 @@
//! Round-trip and behavioral tests: txt-shaped data → `HashDbWriter` → `HashDb`.
-use std::borrow::Cow;
use std::io::Cursor;
use ltk_hashdb::{
- BuildError, Casing, Compression, ExtendedHashDb, HashDb, HashDbWriter, HashKind, KeyWidth,
+ BuildError, Casing, Compression, HashDb, HashDbWriter, HashKind, KeyWidth, LayeredHashDb,
OpenError, VerifyError,
};
@@ -42,13 +41,14 @@ const GAME_ENTRIES: &[(u64, &str)] = &[
),
];
-/// `docs/CONSUMERS.md` promises a `HashDb` can be shared across threads
-/// (all lookups take `&self`); keep this a compile-time guarantee.
+/// `docs/CONSUMERS.md` promises a reader can be shared across threads (all
+/// lookups take `&self`); keep this a compile-time guarantee for every public
+/// one, `LayeredHashDb` included - it is the type most consumers hold.
#[test]
-fn hashdb_is_send_sync() {
+fn readers_are_send_sync() {
fn assert_send_sync() {}
assert_send_sync::();
- assert_send_sync::();
+ assert_send_sync::();
}
#[test]
@@ -103,11 +103,12 @@ fn empty_table() {
db.verify().expect("verify");
}
+/// A raw arena lends its bytes straight out of the mapping - no copy per lookup.
#[test]
-fn get_returns_borrowed_for_raw_arena() {
+fn get_borrows_from_a_raw_arena() {
let bytes = build(KeyWidth::U64, HashKind::Xxh64, GAME_ENTRIES);
let db = HashDb::open_bytes(bytes).expect("open");
- assert!(matches!(db.get(1), Some(Cow::Borrowed(_))));
+ assert!(!db.get(1).expect("hit").is_owned());
}
#[test]
@@ -192,7 +193,12 @@ fn compressed_roundtrip() {
assert!(db.is_compressed());
for &(k, p) in GAME_ENTRIES {
assert_eq!(db.get(k).as_deref(), Some(p), "frame_size {frame_size}");
- assert!(matches!(db.get(k), Some(Cow::Owned(_))));
+ // A frame larger than the whole arena holds every entry whole, so the
+ // hit lends its bytes out of the cached frame rather than copying them.
+ // The tiny frame sizes above are the straddling case, which must splice.
+ if frame_size == 1 << 20 {
+ assert!(!db.get(k).expect("hit").is_owned());
+ }
}
assert_eq!(db.get(2), None);
db.verify().expect("verify");
@@ -335,28 +341,28 @@ fn bad_magic_rejected() {
}
#[test]
-fn extended_overlay_first_then_base() {
+fn overlay_shadows_a_real_base_table() {
let bytes = build(KeyWidth::U64, HashKind::Xxh64, GAME_ENTRIES);
let db = HashDb::open_bytes(bytes).expect("open");
- let mut ext = ExtendedHashDb::new(db);
+ let mut layered = LayeredHashDb::from_bases(vec![db]);
// Base entries still resolve.
assert_eq!(
- ext.get(1).as_deref(),
+ layered.get(1).as_deref(),
Some("assets/characters/aatrox/aatrox.bin")
);
// Overlay shadows the base.
- ext.insert(1, "overridden/path.bin");
- assert_eq!(ext.get(1).as_deref(), Some("overridden/path.bin"));
+ layered.insert(1, "overridden/path.bin");
+ assert_eq!(layered.get(1).as_deref(), Some("overridden/path.bin"));
- // insert_path hashes with the base table's algorithm.
+ // insert_path hashes with the first base's algorithm.
let path = "assets/custom/mod/thing.dds";
- let h = ext.insert_path(path);
- assert_eq!(h, ext.base().hash_path(path));
- assert_eq!(ext.get(h).as_deref(), Some(path));
- assert!(ext.contains(h));
- assert_eq!(ext.overlay_len(), 2);
+ let h = layered.insert_path(path).expect("has a base");
+ assert_eq!(h, layered.bases()[0].hash_path(path));
+ assert_eq!(layered.get(h).as_deref(), Some(path));
+ assert!(layered.contains(h));
+ assert_eq!(layered.overlay_len(), 2);
}
#[test]
diff --git a/crates/ltk_mimir_cache/README.md b/crates/ltk_mimir_cache/README.md
index dfb02b7..a63e643 100644
--- a/crates/ltk_mimir_cache/README.md
+++ b/crates/ltk_mimir_cache/README.md
@@ -100,4 +100,4 @@ See [`docs/CONSUMERS.md`](../../docs/CONSUMERS.md) for the consumer-facing integ
## License
-MIT OR Apache-2.0.
+Apache-2.0. Copyright 2026 Crauzer <0xcrauzer@proton.me>.
diff --git a/crates/ltk_mimir_cache/src/store.rs b/crates/ltk_mimir_cache/src/store.rs
index cf8d385..a3cc0e8 100644
--- a/crates/ltk_mimir_cache/src/store.rs
+++ b/crates/ltk_mimir_cache/src/store.rs
@@ -1,9 +1,11 @@
//! [`HashStore`]: the shared cache directory and everything that reads or mutates it -
//! opening the active table, committing new immutable versions, and GC'ing old ones.
+use std::collections::HashMap;
use std::path::{Path, PathBuf};
+use std::sync::{Arc, Mutex, PoisonError};
-use ltk_hashdb::{HashDb, LayeredHashDb};
+use ltk_hashdb::{HashDb, LayeredHashDb, WeakHashDb};
use crate::manifest::{Manifest, Source, TableEntry};
use crate::{
@@ -21,9 +23,18 @@ const TABLE_EXT: &str = "lhdb";
///
/// Construction is cheap and does not touch the filesystem; the directory is created
/// lazily on the first [`commit`](HashStore::commit).
+///
+/// A store also remembers the tables it has opened through
+/// [`open_shared`](HashStore::open_shared), weakly - clones of a store share that
+/// register, two stores built separately do not, and a table drops out of it as soon as
+/// the last handle to it does.
#[derive(Debug, Clone)]
pub struct HashStore {
dir: PathBuf,
+
+ /// Tables opened through `open_shared`, keyed by their active file. Weak, so the
+ /// register never keeps a superseded version mapped.
+ opened: Arc>>,
}
/// One table to install in a [`commit`](HashStore::commit) call.
@@ -63,14 +74,15 @@ impl HashStore {
/// Resolve the cache directory from the environment / platform. Does not
/// create it.
pub fn discover() -> Result {
- Ok(Self {
- dir: dir::resolve()?,
- })
+ Ok(Self::at(dir::resolve()?))
}
/// Use an explicit cache directory (tests, `--dir` overrides).
pub fn at(dir: impl Into) -> Self {
- Self { dir: dir.into() }
+ Self {
+ dir: dir.into(),
+ opened: Arc::new(Mutex::new(HashMap::new())),
+ }
}
/// The cache directory.
@@ -103,10 +115,53 @@ impl HashStore {
/// Structure is validated on open; the download-time sha256 in the manifest is
/// trusted, so this stays cheap and lazy. Use [`HashDb::verify`] for a full
/// checksum pass.
+ ///
+ /// Every call maps the file afresh. Use [`open_shared`](HashStore::open_shared)
+ /// when the same table may already be open in this process.
pub fn open(&self, table: Table) -> Result {
Ok(HashDb::open(self.path_for(table)?)?)
}
+ /// Open the active version of `table`, reusing a handle this store already has.
+ ///
+ /// [`open`](HashStore::open) maps the file and parses its seek table every time -
+ /// on `game` that is over ten thousand frame records for a table the process may
+ /// already have open. This hands back the existing handle instead, and because the
+ /// register is keyed on the manifest's active filename, an update published in the
+ /// meantime opens the new version by itself rather than serving the old one.
+ ///
+ /// Prefer it wherever a table is opened more than once. The returned handle is a
+ /// [`HashDb`] like any other - cheap to clone, shared frame cache.
+ ///
+ /// # Errors
+ ///
+ /// Fails like [`open`](HashStore::open): a missing manifest, a table the manifest
+ /// does not carry, or a file that does not validate.
+ pub fn open_shared(&self, table: Table) -> Result {
+ let path = self.path_for(table)?;
+ if let Some(db) = self.registered(&path) {
+ return Ok(db);
+ }
+
+ // Opened outside the lock: two threads racing on one table each get a working
+ // handle and the loser's simply isn't the one registered, which is cheaper than
+ // holding a lock across an mmap and a seek-table parse.
+ let db = HashDb::open(&path)?;
+
+ let mut opened = self.opened.lock().unwrap_or_else(PoisonError::into_inner);
+ opened.insert(path, db.downgrade());
+ // Superseded versions leave dead entries behind; sweep them while we hold the
+ // lock, so a long-lived store doesn't accumulate one per update.
+ opened.retain(|_, weak| weak.upgrade().is_some());
+
+ Ok(db)
+ }
+
+ fn registered(&self, path: &Path) -> Option {
+ let opened = self.opened.lock().unwrap_or_else(PoisonError::into_inner);
+ opened.get(path).and_then(WeakHashDb::upgrade)
+ }
+
/// Open several tables, pairing each with its result so callers can warn-and-skip
/// missing ones instead of aborting on the first error. Results are returned in
/// `tables` order.
@@ -121,11 +176,14 @@ impl HashStore {
/// A tool stays usable when a table is missing: its hashes just miss. This is
/// the shape most WAD consumers want - e.g.
/// `open_layered(&[Table::Game, Table::Lcu])`.
+ ///
+ /// Tables are opened through [`open_shared`](HashStore::open_shared), so calling
+ /// this twice does not map anything twice.
pub fn open_layered(&self, tables: &[Table]) -> (LayeredHashDb, Vec<(Table, OpenError)>) {
let mut layered = LayeredHashDb::new();
let mut errors = Vec::new();
- for (table, res) in self.open_many(tables) {
- match res {
+ for &table in tables {
+ match self.open_shared(table) {
Ok(db) => layered.push_base(db),
Err(e) => errors.push((table, e)),
}
diff --git a/crates/ltk_mimir_cache/tests/store.rs b/crates/ltk_mimir_cache/tests/store.rs
index 10cb3c3..8216908 100644
--- a/crates/ltk_mimir_cache/tests/store.rs
+++ b/crates/ltk_mimir_cache/tests/store.rs
@@ -427,3 +427,35 @@ fn gc_handles_mapped_superseded_file() {
assert!(mapped.contains(0x2222));
assert!(store.open(Table::Game).unwrap().contains(0x3333));
}
+
+/// `open_shared` hands back the handle it already has, and follows the manifest to a
+/// new version once one is published.
+#[test]
+fn open_shared_reuses_handles_until_the_version_changes() {
+ let tmp = tempdir().unwrap();
+ let store = HashStore::at(tmp.path());
+
+ let one = build_table(&tmp.path().join("game-1.build"), &[(1, "assets/one.bin")]);
+ store
+ .commit(&[CommitItem::new(Table::Game, "1", &one)], None)
+ .unwrap();
+
+ let first = store.open_shared(Table::Game).unwrap();
+ let second = store.open_shared(Table::Game).unwrap();
+ assert_eq!(first.get(1).as_deref(), Some("assets/one.bin"));
+ assert_eq!(second.get(1).as_deref(), Some("assets/one.bin"));
+
+ // Publishing a new version repoints the manifest, so the next call opens that one
+ // rather than serving the handle it already holds.
+ let two = build_table(
+ &tmp.path().join("game-2.build"),
+ &[(1, "assets/one.bin"), (2, "assets/two.bin")],
+ );
+ store
+ .commit(&[CommitItem::new(Table::Game, "2", &two)], None)
+ .unwrap();
+
+ let after = store.open_shared(Table::Game).unwrap();
+ assert_eq!(after.len(), 2, "followed the manifest to the new version");
+ assert_eq!(first.len(), 1, "the old handle keeps reading the old file");
+}
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
index 85edd10..3d2c0bc 100644
--- a/docs/BENCHMARKS.md
+++ b/docs/BENCHMARKS.md
@@ -17,7 +17,7 @@ arena**, the **16 KiB default frame size**, and the **level 19** publishing defa
```
`bench_real` also writes its measurements to `/bench_real.json`; the
- README performance charts (`docs/assets/bench-*.svg`) regenerate from that report
+ performance charts in `DESIGN.md` (`docs/assets/bench-*.svg`) regenerate from that report
via `cargo run -p ltk_hashdb --example gen_charts`. The arena-layout chart is the
exception - its numbers come from `compression_lab` and live in `gen_charts.rs`.
diff --git a/docs/CONSUMERS.md b/docs/CONSUMERS.md
index 69e4ba3..82035f3 100644
--- a/docs/CONSUMERS.md
+++ b/docs/CONSUMERS.md
@@ -141,33 +141,32 @@ size in private memory, so reach for it last.
## Extending a table with custom hashes
Mod tooling often introduces paths the community tables don't know. The sanctioned way
-is `ExtendedHashDb` - an in-memory overlay consulted before the immutable base, so you
-don't hand-roll a second map plus fallback:
+is `LayeredHashDb` - an in-memory overlay consulted before one or more immutable base
+tables, so you don't hand-roll a second map plus fallback:
```rust
-use ltk_hashdb::ExtendedHashDb;
+use ltk_hashdb::LayeredHashDb;
-let mut ext = ExtendedHashDb::new(store.open(Table::Game)?);
+let mut db = LayeredHashDb::from_bases(vec![store.open(Table::Game)?]);
-// Hashes with the base table's algorithm and returns the hash:
-let h = ext.insert_path("assets/mymod/custom.dds");
-ext.insert(precomputed_hash, "assets/mymod/other.bin"); // or bring your own hash
-ext.extend(pairs); // or bulk-load
+// Hashes with the first base's algorithm and returns the hash:
+let h = db.insert_path("assets/mymod/custom.dds").expect("has a base");
+db.insert(precomputed_hash, "assets/mymod/other.bin"); // or bring your own hash
+db.extend(pairs); // or bulk-load
-assert!(ext.contains(h));
-let path = ext.get(h); // overlay first, then base
+assert!(db.contains(h));
+let path = db.get(h); // overlay first, then each base in order
```
-The base file is never mutated. `ext.base()` exposes the underlying `HashDb`;
-`ext.overlay_len()` counts overlay-only entries. Overlay entries are per-process and
-not persisted - if you want them shared or durable, contribute them upstream to the
+Base files are never mutated. `db.bases()` exposes the underlying tables in priority
+order; `db.overlay_len()` counts overlay-only entries. Overlay entries are per-process
+and not persisted - if you want them shared or durable, contribute them upstream to the
CommunityDragon txt lists (the canonical source).
### Layering several base tables under one overlay
-When a workload spans **more than one** table - WAD chunk resolution consults both
-`Game` and `Lcu` - reach for `LayeredHashDb` instead of hand-rolling a `Vec`
-plus fallback. It is `ExtendedHashDb` generalised to N ordered bases: lookups consult
+The same type takes N bases, which is what you want when a workload spans **more than
+one** table - WAD chunk resolution consults both `Game` and `Lcu`. Lookups consult
the overlay first, then each base in push order, first hit wins.
```rust
diff --git a/docs/DESIGN.md b/docs/DESIGN.md
new file mode 100644
index 0000000..461add8
--- /dev/null
+++ b/docs/DESIGN.md
@@ -0,0 +1,147 @@
+# Design
+
+> Why mimir exists, how the `.hashdb` format is put together, and what it measures.
+> For the byte-level specification see [`FORMAT.md`](FORMAT.md); for the integration
+> API see [`CONSUMERS.md`](CONSUMERS.md).
+
+## Why this exists
+
+Almost every League of Legends tool - WAD unpackers, `.bin` inspectors, mod loaders,
+asset browsers - hits the same wall: the game identifies files and fields by **hash**,
+not by name. To show a human-readable path you need a **hash table** that maps each
+hash back to its original string.
+
+Today that table is the CommunityDragon `hashes.*.txt` set: **~348 MB of plain text**,
+which every tool downloads and keeps around. That approach has two costs that compound
+the moment a machine runs more than one of these tools:
+
+- **Memory.** To resolve hashes efficiently a program has to load the whole table into
+ an in-memory map. Run three tools that each need the game hashes and you pay for
+ **three private copies** of the same hundreds-of-megabytes table, all resident at once.
+- **Startup & distribution.** Every tool ships (or re-downloads) the same giant text
+ files and spends time parsing them into a map before it can answer a single query.
+
+**mimir** replaces that text blob with a purpose-built, **read-only** binary format for
+hash storage. The design goals are, in order:
+
+1. **Usable as shipped** - no unpack or full-expansion step; a consumer `mmap`s the file
+ and immediately does lookups.
+2. **Small** - the game table drops from ~348 MB of text to roughly **~50 MB** on disk.
+3. **Memory-efficient across processes** - the file is memory-mapped, so the OS page
+ cache holds **one** copy that every tool on the machine shares. Resident RAM stays
+ low because pages are faulted in lazily and dropped under pressure, and a lookup
+ *miss* touches zero string data.
+
+## How it works
+
+A `.hashdb` file is a single logical table laid out for direct, zero-parse use over an
+`mmap`:
+
+- **Sorted key array** - the integer hashes, stored strictly ascending so a lookup is a
+ **binary search straight over the mapped bytes**. A miss is decided here and never
+ reads any string data.
+- **Parallel offset + length arrays** - for a found key, where its path lives in the arena
+ and how long it is.
+- **String arena** - all the path strings concatenated with no separators, compressed as
+ a **Zstandard Seekable Format** stream. The seek table means a hit decompresses just the
+ **one small frame** that holds its path (default 16 KiB frames), not the whole table -
+ so partial, on-demand reads stay cheap. Paths are packed in **lexicographic order** so
+ a directory's files land in the same frames, which both compresses far better (~4× vs.
+ key order on the real game table) and makes directory-local batch lookups touch fewer
+ frames.
+
+The file is immutable once published; updates ship as new versioned files, and a downloaded
+file is treated as untrusted - the header is validated on open and every read bounds-checks
+its own extent. See [`FORMAT.md`](FORMAT.md) for the byte-level specification.
+
+Because it's memory-mapped and read-only, a lazy consumer (say, a mod loader) can open the
+table only when it first needs to resolve a hash, share the page cache with every other
+mimir-backed tool running, and drop the handle to reclaim its (already small) footprint.
+
+### Reading a path without copying it
+
+A lookup returns a `PathRef`, which borrows rather than allocates. A raw arena lends the
+bytes straight out of the mapping; a compressed arena lends them out of the decompressed
+frame the table is holding, keeping that frame alive for as long as the path is. Only two
+cases copy: an entry that straddles a frame boundary (about one per frame), and one whose
+bytes are not valid UTF-8. `PathRef` derefs to `str`, so it reads like one at the call
+site.
+
+Decompressed frames are cached in a fixed, byte-capped, N-way set-associative table sized
+at open - frames are keyed by a dense index, so no map and no eviction list is needed, and
+one lock per set keeps concurrent readers off each other. Buffers from evicted frames are
+recycled, so a steady-state miss decompresses into an allocation that already exists.
+Because published files are immutable, this is pure memoisation with nothing to
+invalidate.
+
+This matters because point lookups are not a niche shape: `ltk_wad`'s `PathResolver` is
+point-shaped by construction, so every WAD extractor resolves one hash per chunk whether
+it wants to or not.
+
+## Performance
+
+Real-data measurements against the CommunityDragon `hashes.*.txt` snapshot of
+2026-07-07 (~2.97 M entries across 8 tables). Full tables, methodology, and
+reproduction steps are in [`BENCHMARKS.md`](BENCHMARKS.md).
+
+
+
+
+
+
+The whole corpus drops from **~253 MiB of txt to ~52 MiB** of `.hashdb` - and the
+binary is usable as-shipped: `open` is a header validation plus an `mmap`, with no
+parse or expansion step before the first lookup.
+
+
+
+
+
+
+A hit decompresses exactly one small frame; batched lookups amortize that by resolving in
+arena order. A **miss is decided by binary search over the raw key section and never
+touches string data** - ~143 ns whether the file is raw or compressed, which matters
+because hash hunting hammers misses.
+
+Those point-lookup figures predate the reader's frame cache. With it, one archive's worth
+of paths (20 000 entries of the 2 291 324-entry `game-2026-08-14` table, probed in hash
+order) measures **1.07 µs cached against 5.92 µs uncached, and 0.66 µs batched** - point
+lookups land within 1.6× of the batch path instead of an order of magnitude off it. Keys
+scattered across the whole table, where no cache holds the working set, are unchanged at
+~6.7 µs. Reproduce with:
+
+```sh
+cargo run --release -p ltk_hashdb --example frame_cache --
+```
+
+
+
+
+
+
+Sorting the arena by path packs each directory into the same frames, so the seekable
+arena compresses **~4× better than key order** - beating even a solid, non-seekable
+zstd stream - while making hits faster and directory-local batches frame-coherent.
+
+## Distribution
+
+The txt lists stay canonical: they are what the community PRs against and what git
+merges. The binary is a generated release artifact, rebuilt from them, never the source
+of truth.
+
+Tables ship as versioned GitHub release assets (`game-2026-08-14.lhdb`) alongside a
+`manifest.json` naming the active version, sha256, and entry count per table. A machine
+keeps one shared cache directory - `%LOCALAPPDATA%\LeagueToolkit\hashes` on Windows,
+`$XDG_DATA_HOME/LeagueToolkit/hashes` on Linux, `~/Library/Application Support/…` on
+macOS - so every mimir-backed tool on it reads the same files through the same page
+cache. Installs are atomic: table files land under immutable versioned names first, and
+the manifest is swapped last, so a reader sees either the whole old version or the whole
+new one. A single-updater lock keeps two tools from downloading at once; readers never
+take it.
+
+## Planned work
+
+[`ROADMAP.md`](ROADMAP.md) tracks what is next, in dependency order, along with the three
+constraints that govern it - the version and schema equality gates that make every format
+change additive, the reserved header fields that are the room left to be additive in, and
+the pre-publish window in which breaking Rust API changes are free.
diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
new file mode 100644
index 0000000..acff2f8
--- /dev/null
+++ b/docs/ROADMAP.md
@@ -0,0 +1,424 @@
+# Roadmap
+
+> Planned work on the format, the reader, and the shared cache, in dependency order.
+> Findings are lettered (A/B/C) from a consumer-integration review; tasks are numbered by wave.
+
+Everything here is `ltk_hashdb`, `ltk_mimir_cache`, or `ltk_mimir_cli`. The waves are
+ordered by what each unblocks and by three constraints that are not negotiable - read
+those first, because two of them decide what may ship at all.
+
+## Ordering constraints
+
+### 1. `version` and `schema` are equality gates, so everything is additive from here
+
+`Header::decode` rejects any `version != FORMAT_VERSION` (`header.rs:114`), and
+`Manifest::from_slice` rejects any `schema != SCHEMA_VERSION` (`manifest.rs:76`). Both
+compare for equality, not for a floor.
+
+The shared cache exists so that several independently-versioned tools point at one
+directory. That premise breaks the moment a version is bumped: the first tool to sync
+publishes files and a manifest that every not-yet-upgraded tool on the machine refuses.
+The manifest gate is the sharper of the two - it is not confined to reading a table, so a
+consumer that merely reports cache status fails outright rather than degrading.
+
+**`schema` stays at `1` and `FORMAT_VERSION` stays at `1`.** Task 3.1 relaxes the gates
+for future readers; it cannot help the ones already shipped, which is the argument for
+never bumping rather than for bumping once more first.
+
+### 2. The additive room is in the reserved fields, not in the flag bits
+
+| Room | Where | Status |
+|------|-------|--------|
+| Header reserved `[u8;8]` | bytes 72..80 (`header.rs:20`) | **Safe.** Written as zero, ignored on read - wide enough for a section offset. |
+| Header reserved `[u8;2]` | bytes 14..16 (`header.rs:12`) | **Safe**, but only 2 bytes - a discriminant or a small count, not an offset. |
+| New manifest fields | `manifest.rs` | **Safe.** serde ignores unknown fields; existing optional fields already use `#[serde(default)]`. |
+| New `flags` bits | `header.rs:38`, checked at `header.rs:120` | **Not safe.** `KNOWN_FLAGS` rejects any unknown bit, and `FORMAT.md` specifies "other bits must be 0". |
+
+So a new *feature* can be announced through a reserved header field without a version
+bump, but a new *flag* cannot. Task 3.1 opens the flag byte; task 4.1 is written to need
+only the reserved `u64`, so it does not depend on that half landing first.
+
+### 3. Breaking Rust API changes are free until the first crates.io release
+
+`ltk_hashdb` and `ltk_mimir_cache` sit at `0.1.0` with `release = true, publish = true` in
+`release-plz.toml`, but no crate tag exists yet (`git tag -l` shows only the
+`hashes-*` table-data namespace) and neither crate has a changelog. There are no external
+consumers pinned to a published version.
+
+Every breaking signature change is therefore one rev bump in one downstream repo today,
+and a semver event forever after. **Wave 2 exists to spend that window before the first
+`cargo publish`.**
+
+## Wave 1 - Additive reader work
+
+Purely additive: no signature changes, so this wave can ship to existing git-pinned
+consumers without touching a call site.
+
+### 1.1 Move `HashDb`'s state behind an `Arc` and add a frame cache
+
+> Findings **A1** and **B4**. One refactor, not two - the shared read-only state and the
+> shared frame cache want the same `Arc`.
+
+`FrameCache` (`reader.rs:52`) is a local `&mut` threaded through one call. It lives for
+the duration of a `get_batch` (`reader.rs:154`), an `iter` (`reader.rs:230`), or a
+`verify` (`reader.rs:249`), and dies when that call returns. `get` (`reader.rs:143`)
+passes `&mut None`. Nothing caches a decompressed frame across calls.
+
+The cost is the gap the benchmarks already record: on `game` a point hit is 8.5 µs against
+3.6 µs batched and 143 ns for a miss; on `binentries` it is 10.5 µs against 1.2 µs. That
+whole difference is frame decompression, and it is re-paid on every lookup even when the
+previous one decompressed the very frame the answer is in. Because the arena is
+path-ordered, consecutive lookups in a real workload are usually in the same frame - so a
+consumer resolving an archive's chunks one at a time decompresses a handful of frames tens
+of thousands of times.
+
+Point lookups are not a niche shape. `ltk_wad`'s `PathResolver` is point-shaped by
+construction, so every WAD extractor resolves one hash per chunk whether it wants to or
+not.
+
+- `HashDb { inner: Arc }`, deriving `Clone`; the mmap, header, section ranges, and
+ seek table move into `Inner`.
+- A byte-capped LRU of decompressed frames beside them - a few MiB by default,
+ `with_frame_cache(bytes)` to tune, `0` to disable. Published files are immutable, so this
+ is pure memoisation with no invalidation.
+- `HashStore::open_shared(table)` keeping a weak cache keyed on the manifest's active
+ filename, so reopening after an update happens by itself. `open` (`store.rs:106`) today
+ re-mmaps and re-parses the seek table on every call - ~10 400 frame records for `game`.
+
+Design constraints: a frame held behind a `Mutex` cannot be borrowed out of the guard, so
+`get` keeps returning `Cow::Owned` and the allocation only goes away with task 1.2.
+Measure lock contention before choosing between one mutex, sharding by frame index, and a
+thread-local cache.
+
+**Done when** a bench resolving one archive's chunks point-wise lands within a small factor
+of the same set batched; the `misses_never_decompress` invariant still holds; and
+`Clone + Send + Sync` is asserted for `HashDb` *and* `LayeredHashDb`.
+
+### 1.2 Closure and buffer accessors for allocation-free reads
+
+> Finding **A3**. Depends on 1.1.
+
+For a compressed arena - every published `.lhdb` - `str_at_cached` (`reader.rs:348`) always
+takes the owned branch, because there is no borrow to be had from a decompression buffer
+that dies with the call. `BENCHMARKS.md` already names the frame buffer as the only
+per-lookup allocation. A consumer building a whole-install index therefore allocates a
+`String` per chunk and keeps a fraction of them.
+
+- `with_str(&self, hash, f: impl FnOnce(&str) -> R) -> Option`, reading straight out
+ of the cached frame.
+- `get_into(&self, hash, &mut String) -> bool` for callers that own a reusable buffer.
+- Document that `with_str`'s closure runs under the cache lock and must stay short.
+
+**Done when** a filter-only pass over a WAD's chunks - resolve, test the extension, discard
+- allocates nothing per hit.
+
+### 1.3 `Debug` on every public reader
+
+> Finding **B3**.
+
+None of `HashDb`, `LayeredHashDb` (`layered.rs:36` derives `Default` only), or
+`ExtendedHashDb` (`extended.rs:10`) implements `Debug`. Consumers wrapping them end up
+hand-writing `Debug` for their own types to compensate. Print counts, widths, and flags -
+never entries.
+
+`tests/roundtrip.rs:48` asserts `Send + Sync` for `HashDb` and `ExtendedHashDb` but not for
+`LayeredHashDb`, which is the type most consumers actually hold. Add it.
+
+**Done when** all three print their shape and the assertion covers all three.
+
+### 1.4 Tell a corrupt table apart from a table of unknown hashes
+
+> Finding **A4**.
+
+`str_at_cached` (`reader.rs:348`) swallows every `frame_bytes` (`reader.rs:366`) error into
+`None`. That is right for a lookup path - `FORMAT.md` specifies a failed frame as a miss -
+but it means a table whose arena no longer decompresses degrades silently into "this build
+knows nothing". A consumer renders an entire install under hex names with no signal to
+distinguish that from an incomplete table.
+
+Nothing re-verifies an installed table after its download checksum (`store.rs:106`: the
+manifest sha256 is trusted), so bit rot and truncating writes both land here.
+
+- A sticky `AtomicBool` set on the first swallowed error, plus `HashDb::is_healthy()`.
+- Optionally `try_get -> Result, VerifyError>` for callers that want the error
+ at the call site.
+
+**Done when** a table with a deliberately corrupted frame reports unhealthy after the first
+affected lookup, and `get` still never panics.
+
+### 1.5 Write down why the mmap is sound
+
+> Finding **A5**.
+
+`unsafe { memmap2::Mmap::map(&file)? }` (`reader.rs:58`) carries no safety comment. What
+discharges it - published `.lhdb` files are immutable, `commit` never renames over an
+existing name, and `gc` only unlinks - is documented on `HashStore` in a different crate.
+`CONSUMERS.md` explicitly invites using `ltk_hashdb` alone for embedded or self-built
+tables, and such a consumer gets no warning that truncating a mapped file is undefined
+behaviour rather than an error.
+
+Add a `// SAFETY:` at the block and a paragraph in `HashDb::open`'s docs stating the
+caller's obligation.
+
+## Wave 2 - Breaking API cleanup
+
+Every breaking signature change worth making, batched into one release. See constraint 3.
+
+### 2.1 Give `Table` its own metadata
+
+> Finding **B2**. Blocks 2.2.
+
+`Table` (`lib.rs:38`) exposes `id()` (`lib.rs:63`) and `from_id()` (`lib.rs:77`) and nothing
+else. It cannot report its key width, hash algorithm, or casing, though those are fixed per
+table and the workspace knows them - so they are written out three times: the CLI's private
+`Table` enum (`ltk_mimir_cli/src/main.rs:32`, methods at `:43`-`:64`), `bundle.rs`'s
+`SPECS` (`ltk_mimir_cli/src/bundle.rs:76`), and prose in consumers explaining which tables
+share a hash universe.
+
+- `key_width()`, `hash_kind()`, `casing()`, and a `universe()` discriminator on the library
+ enum.
+- Delete the CLI's duplicate methods and fold `TableSpec` (`bundle.rs:49`) down to input
+ filename plus split flag.
+
+**Done when** each table's width, algorithm, and casing is stated once in the workspace.
+
+### 2.2 Make the layered key-config invariant a hard error
+
+> Finding **B1**. Depends on 2.1.
+
+`LayeredHashDb` requires every base to agree on `(key_width, hash_kind, casing)`, because
+lookups take a pre-hashed `u64` and no base re-hashes. It documents this at length and then
+enforces it with `debug_assert_eq!` (`layered.rs:57` in `from_bases`, `layered.rs:75` in
+`push_base`) - which compiles out of release builds.
+
+In release, layering `binfields` under `binentries` does not fail. It answers a property
+hash with an object's path, across four unrelated 32-bit FNV-1a universes totalling
+~500 000 rows, often enough to be a certainty rather than a risk. A wrong name is worse
+than a number.
+
+- `push_base` and `from_bases` return `Result<_, KeyConfigMismatch>` naming both configs.
+- `open_layered` (`store.rs:124`) refuses a mismatched set using `Table::universe()` before
+ it opens anything.
+
+**Done when** layering `binfields` under `binentries` fails in a release build.
+
+### 2.3 `#[non_exhaustive]`, `Display`, and `FromStr` on `Table`
+
+> Finding **B7**.
+
+The table set will grow, and without `#[non_exhaustive]` each addition is a breaking
+change. `Display`/`FromStr` over `id()` remove `table.id().to_owned()` from every consumer
+boundary. `OpenError::TableNotFound` formats with `{0:?}`, so it prints `BinEntries` where
+every other surface says `binentries`. An optional serde feature belongs here too.
+
+### 2.4 Split `Casing` so the League rule has its own name
+
+> Finding **A6**. Narrows a documented decision; does not reverse it.
+
+`FORMAT.md` already specifies Unicode-aware lowercasing deliberately, and already carries
+the stability note that only the ASCII part of the mapping is bit-stable across toolchains,
+recommending that non-ASCII publishers pre-lowercase and hash case-sensitively. The gap is
+not that the decision is wrong - it is that `Casing::Insensitive` (`hash.rs:18`, variant at
+`hash.rs:27`) is one name covering two rules, and the League tables, which are the
+overwhelmingly common case, get the Unicode one by default.
+
+Introduce `Casing::AsciiInsensitive` as what League tables mean and what
+`FLAG_CASE_INSENSITIVE` maps to for them; keep the Unicode variant as a deliberate,
+separately-named choice or drop it (see *Open questions*). No data migration - every
+published table is ASCII, so the bytes on disk do not change, only which variant names
+them. `FORMAT.md`'s `case_insensitive` section needs updating either way.
+
+### 2.5 Signature parity and a streaming batch
+
+> Finding **B5**.
+
+`HashDb::get_batch` (`reader.rs:154`) collects eagerly so the returned iterator does not
+borrow the input. `LayeredHashDb::get_batch` (`layered.rs:126`) keeps the borrow, because
+its tail zips over the input slice - so the same method on the two types has two contracts,
+and the layered one refuses a temporary.
+
+- Drop the `'a` from the layered signature's `hashes`.
+- Add `for_each_batch(&self, hashes, f: impl FnMut(usize, u64, Option<&str>))` to both,
+ which inherits task 1.2's allocation-free path. Both current forms materialise a `Vec` of
+ every result before yielding the first.
+
+### 2.6 Retire `ExtendedHashDb`, grow `LayeredHashDb`
+
+> Finding **B6**.
+
+`layered.rs` already describes `LayeredHashDb` as `ExtendedHashDb` generalised to N bases,
+and it has everything the narrow type has plus `get_batch`. Deprecate `ExtendedHashDb`,
+lead `CONSUMERS.md` with the layered type rather than teaching the narrower one first, and
+add `iter()` and `len()` to `LayeredHashDb` - today it cannot be enumerated at all.
+
+## Wave 3 - Distribution
+
+Task 3.1 gates the rest of this wave *and* the flag-byte half of wave 4. Nothing here may
+add a manifest field or a header flag until the version gates stop being fatal.
+
+### 3.1 Make the version gates survivable
+
+> Finding **A7**. Blocks 3.2, 3.3, and the flag-byte option in 4.1.
+
+- **Manifest.** Accept `schema >= SCHEMA_VERSION` (`manifest.rs:76`) and rely on serde
+ ignoring unknown fields. Add an optional `min_reader_schema` for the day something
+ genuinely incompatible is needed. Policy: `schema` stays at `1`.
+- **Header.** Split `flags` into a required byte - unknown bit rejects, today's behaviour
+ at `header.rs:120` - and an optional byte whose unknown bits are ignored. Update
+ `FORMAT.md`, which currently specifies "other bits must be 0".
+- **Manifest.** `format_version` per `TableEntry` (`manifest.rs:50`), so a reader skips a
+ table it cannot open the way `UpdateReport::unknown_tables` already skips an unknown id.
+- **Release.** Publish a per-format channel filename alongside `latest`, so an old build
+ keeps updating within the format it can read. `ReleaseSource::github` hardcodes
+ `/releases/latest/download`.
+
+**Done when** a manifest carrying an unknown table, an unknown field, and a higher `schema`
+still installs the tables this build understands, and a `.hashdb` with an unknown
+*optional* flag bit opens normally.
+
+### 3.2 Version and time on `TableEntry`
+
+> Finding **C6**. Depends on 3.1.
+
+`TableEntry` (`manifest.rs:50`) carries `file`, `sha256`, `entries`, and `key_width` - no
+version and no timestamp. The version exists only inside the filename, extracted by
+`version_of`, which is private (`update.rs:345`). `generated_at` (`manifest.rs:23`) is a
+`String`, so "how stale is this cache" is every consumer's parsing problem despite the
+crate already depending on `time`.
+
+Add `version`; expose `generated_at` parsed beside the raw string.
+
+**Done when** a consumer can show `game · 2026-07-10` instead of `game-2026-07-10.lhdb`.
+
+### 3.3 Move provenance onto the table
+
+> Finding **C5**. Depends on 3.1.
+
+`commit` (`store.rs:155`) assigns `manifest.source = source` wholesale (`store.rs:169`), so
+a run installing only `game` restamps `repo`, `commit`, and `inputs_sha256` for the seven
+tables it did not touch - the manifest then claims a CommunityDragon commit for tables
+built from a different one.
+
+Move `commit` and `inputs_sha256` onto `TableEntry`; keep a manifest-level record named for
+what it actually is, the last run.
+
+### 3.4 Add `HashStore::check`
+
+> Finding **C1**.
+
+`update` (`update.rs:161`) is the only path to the remote manifest, and it takes the
+exclusive lock before fetching anything. There is no lock-free "what would change?", so a
+UI cannot show an updates-available state and a startup check cannot stay out of the CLI's
+way.
+
+Expose the first half of the private `plan` (`update.rs:226`) as
+`check(&remote) -> Result, _>`: fetch, compare sha256s, install nothing,
+take no lock.
+
+**Done when** a consumer can render "3 tables behind" without touching the update lock.
+
+### 3.5 Stream the download and stop copying the result
+
+> Findings **C3** and **C2**. One change - the copy only exists because the fetcher hands
+> back a whole buffer.
+
+`Fetch` (`update.rs:29`) and `AsyncFetch` (`update.rs:66`) return `Vec`. For each table
+the bytes then move four times: `verify_and_stage` (`update.rs:269`) runs `sha256_bytes`
+(`fsutil.rs:46`) over the buffer and writes a `.download.tmp`; `commit` opens that file,
+`atomic_copy` (`fsutil.rs:36`) reads and writes it again into a second temp, renames, and
+`sha256_file` (`fsutil.rs:51`) reads the destination a third time to recompute a digest it
+was just handed.
+
+A ~52 MiB corpus therefore costs several hundred MiB of I/O and peaks at the size of the
+largest table in memory (38.3 MiB for `game`). The buffer return also makes byte-level
+progress and mid-download cancellation impossible, which is why the bundled fetchers are
+documented as silent.
+
+- `Fetch::fetch_to(&self, filename, sink: &mut dyn Write)` as the primitive, with the
+ `Vec` form kept as a default method over it. Same for `AsyncFetch`.
+- Stage straight into the cache directory while hashing the stream.
+- A `CommitItem` constructor that *takes* an already-staged sibling - rename, reuse the
+ digest - alongside the existing copy path for files built elsewhere.
+
+**Done when** installing the full corpus reads and writes each table's bytes once, peak
+memory does not track the largest table, and a caller's sink can report bytes and cancel.
+
+### 3.6 Say who holds the update lock
+
+> Finding **C4**.
+
+`UpdateLock::try_acquire` (`lock.rs:20`) is non-blocking and reports only presence - no
+holder, no start time, no bounded wait. Consumers can only surface "another process is
+already syncing", which reads the same as a crashed updater.
+
+Write pid and an RFC-3339 start time into the lock file's body, and expose `lock_holder()`
+plus `lock_update_timeout(Duration)`. The OS lock stays the source of truth; the body
+exists only so a message can name who and since when.
+
+### 3.7 A cheap integrity tier between `open` and `verify`
+
+> Finding **C7**. Pairs with 1.4.
+
+`open` validates structure only; `verify` (`reader.rs:249`) reads the whole file. Add
+`verify_index()` - checksum the keys, offsets, and lengths sections and skip the arena.
+Milliseconds rather than a full pass, and it catches most real post-install damage.
+Together with 1.4's health flag a consumer gets both a proactive check and a reactive
+signal.
+
+## Wave 4 - Format capability
+
+### 4.1 Record arena order in the file
+
+> Finding **A2**. Depends on 3.1 only if the flag-byte variant is chosen; the reserved-`u64`
+> variant below needs nothing.
+
+The writer sorts the arena lexicographically by path (`writer.rs:84`-`93`) - that ordering
+is what earns the ~4× compression win. But offsets and lengths are stored in *key* order,
+so nothing can walk the arena forward without first reconstructing the permutation.
+`arena_order` (`reader.rs:301`) does that with an `O(n log n)` sort over an `n`-word
+allocation, on every `iter` and every `verify`: on `game` that is a 2 086 643-element sort
+and ~16 MiB of scratch, recomputed per call. `BENCHMARKS.md` already lists the vector under
+*Memory profile*.
+
+The larger cost is what it forecloses. A table that is already a sorted list of strings
+cannot answer "which paths start with `assets/characters/ahri/`" - the query an asset
+browser and a name-autocomplete both want. Consumers work around it by downloading the
+CommunityDragon txt list separately just to enumerate names.
+
+Two candidate layouts:
+
+| Layout | Content | Size on `game` |
+|--------|---------|---------------:|
+| **Sparse per-frame index** | Per zstd frame: entry index and arena offset of the first entry starting in it | ~10 400 frames × 12 B ≈ **122 KiB** |
+| Dense rank array | `arena_rank → entry_index`, `u32 × entry_count` | ~8 MiB (+21 % on a 38.3 MiB file) |
+
+Point the reader at the section from the reserved header `u64` at bytes 72..80
+(`header.rs:20`), which current readers already ignore - so no version bump, no flag bit,
+and a reader built before this change still opens the new file.
+
+Then: `values()` streaming with no sort and no allocation, `prefix(&str, limit)` as a
+binary search over the arena, and `iter`/`verify` stop rebuilding the permutation.
+
+`FORMAT.md` needs a new section; its "readers must treat the arena as opaque bytes
+addressed by (offset, length)" invariant is unaffected, since this adds an index rather
+than changing how an entry is addressed.
+
+**Done when** `iter` on the game table allocates no index vector,
+`prefix("assets/characters/ahri/")` returns in single-digit milliseconds, and a reader
+built before the change still opens the file. Decide between the two layouts against a
+measured prefix benchmark, not on paper.
+
+## Open questions
+
+- **Does `ExtendedHashDb` get deleted or deprecated forever?** Deprecation costs nothing
+ but keeps two overlapping types in the docs; deletion is free right now (constraint 3)
+ and a semver event later.
+- **Does the Unicode `Casing` variant survive task 2.4?** Nothing in the League corpus
+ needs it, `FORMAT.md` already tells non-ASCII publishers to pre-lowercase instead, and
+ keeping it means keeping a way to hash a path to a key no published table holds. Dropping
+ it would simplify `hash.rs` and remove the Unicode-version stability caveat from the spec.
+- **Should mimir ship a `PathResolver` impl for `ltk_wad`?** Every WAD consumer writes the
+ same ~40-line adapter. The dependency direction is the awkward part: `ltk_wad`
+ implementing the trait for `LayeredHashDb` is cleaner than mimir depending on `ltk_wad`,
+ but it puts the impl in a crate that would then need an `ltk_hashdb` dependency. A
+ feature-gated impl here is the third option.
diff --git a/docs/assets/logo.png b/docs/assets/logo.png
new file mode 100644
index 0000000..1fd1ae7
Binary files /dev/null and b/docs/assets/logo.png differ
-
-
-
+
+ 
+
+
+