diff --git a/README.md b/README.md index 32cd3b1..e697b05 100644 --- a/README.md +++ b/README.md @@ -136,9 +136,10 @@ use ltk_mimir_cache::{HashStore, Table}; let store = HashStore::discover()?; // Missing tables are reported, not fatal - the tool stays usable and their hashes miss. -let (mut db, errors) = store.open_layered(&[Table::Game, Table::Lcu]); +// The call itself only fails if you ask for tables from different hash universes. +let (mut db, errors) = store.open_layered(&[Table::Game, Table::Lcu])?; for (table, e) in &errors { - eprintln!("skipping {table:?}: {e}"); + eprintln!("skipping {table}: {e}"); } // Register a path your mod introduced; it is hashed with the first base's algorithm. @@ -148,8 +149,10 @@ assert_eq!(db.get(hash).as_deref(), Some("assets/mymod/custom.dds")); > [!NOTE] > Every base must agree on key width, hash algorithm, and casing, because lookups take a -> hash the caller already computed and no base re-hashes it. `game` and `lcu` do; the four -> 32-bit `bin*` tables are separate hash universes and must not be layered together. +> hash the caller already computed and no base re-hashes it - `push_base` returns a +> `KeyConfigMismatch` rather than layering one that doesn't. `game` and `lcu` agree; the +> four 32-bit `bin*` tables agree too, and still must not be layered, because they are +> separate hash *universes* - so `open_layered` refuses that set outright. ### Enumerating a table @@ -193,7 +196,7 @@ use ltk_hashdb::{Casing, Compression, HashDbWriter, HashKind, KeyWidth}; let mut writer = HashDbWriter::new(KeyWidth::U64, Compression::default()) .hash_kind(HashKind::Xxh64) // recorded, so readers can hash new paths - .casing(Casing::Insensitive); // League tables hash the lowercased path + .casing(Casing::AsciiInsensitive); // League tables hash the ASCII-lowercased path writer.insert(hash, "assets/characters/ahri/ahri.bin"); writer.extend(pairs); @@ -244,15 +247,24 @@ frame cache. `Send + Sync`. | `discover` · `at` | resolve the platform cache dir, or point at your own | | `open_shared` | open the active version, reusing a handle this store already has | | `open` · `open_many` | open a fresh mapping, one table or several | -| `open_layered` | open several into one `LayeredHashDb`, reporting per-table errors | +| `open_layered` | open several of one universe into a `LayeredHashDb`, reporting per-table errors | | `manifest` · `path_for` | what is installed, and where | | `update` · `update_async` | compare → download → verify → install → GC | | `commit` · `gc` · `try_lock_update` | publish versions, sweep old ones, take the lock | +**`Table`** - which logical table, and how it hashes. + +| Method | | +|---|---| +| `ALL` · `id` · `Display` · `FromStr` · serde | the stable spellings (`game`, `binentries`, `rst-xxh3`) | +| `key_config` · `key_width` · `hash_kind` · `casing` | how this table's keys were produced | +| `universe` | which hashes it can answer - only same-universe tables may be layered | + **`PathRef`** - a resolved path. `Deref`, plus `as_str`, `is_owned` (whether the bytes were copied rather than borrowed), and `into_owned`. -**`HashDbWriter`** - `new` → `hash_kind` / `casing` → `insert` / `extend` → `build`. +**`HashDbWriter`** - `new` → `hash_kind` / `casing` → `insert` / `extend` → `build`, or +`with_key_config` when a `Table` already states all three. ## CLI diff --git a/crates/ltk_hashdb/examples/bench_real.rs b/crates/ltk_hashdb/examples/bench_real.rs index 73a8564..9233a9a 100644 --- a/crates/ltk_hashdb/examples/bench_real.rs +++ b/crates/ltk_hashdb/examples/bench_real.rs @@ -270,7 +270,7 @@ fn build( ) -> (u64, ltk_hashdb::BuildStats) { let mut w = HashDbWriter::new(key_width, compression) .hash_kind(hash_kind) - .casing(Casing::Insensitive); + .casing(Casing::AsciiInsensitive); w.extend(entries.iter().map(|(k, p)| (*k, p.as_str()))); let file = BufWriter::new(File::create(out).expect("create output")); let stats = w.build(file).expect("build"); diff --git a/crates/ltk_hashdb/src/error.rs b/crates/ltk_hashdb/src/error.rs index e370bd2..b8436ae 100644 --- a/crates/ltk_hashdb/src/error.rs +++ b/crates/ltk_hashdb/src/error.rs @@ -3,6 +3,8 @@ use thiserror::Error; +use crate::KeyConfig; + /// Errors from opening a `.hashdb` file ([`HashDb::open`] / [`HashDb::open_bytes`]): /// I/O, or the untrusted header/section-bounds validation rejecting the file. /// @@ -71,3 +73,22 @@ pub enum BuildError { #[error("zstd seekable format error")] Zeekstd(#[from] zeekstd::Error), } + +/// A base rejected by [`LayeredHashDb`] because it does not hash its keys the way +/// the rest of the layer does. +/// +/// [`LayeredHashDb`]: crate::LayeredHashDb +#[derive(Debug, Clone, Copy, PartialEq, Eq, Error)] +#[error( + "base {index} is keyed {found}, but the layer is keyed {expected}; a base that hashes differently can never be hit by a caller's precomputed probe" +)] +pub struct KeyConfigMismatch { + /// Position of the rejected base, counting the ones already layered. + pub index: usize, + + /// What the layer hashes under - its first base's configuration. + pub expected: KeyConfig, + + /// What the rejected base hashes under. + pub found: KeyConfig, +} diff --git a/crates/ltk_hashdb/src/hash.rs b/crates/ltk_hashdb/src/hash.rs index b47a448..e8390c9 100644 --- a/crates/ltk_hashdb/src/hash.rs +++ b/crates/ltk_hashdb/src/hash.rs @@ -5,31 +5,46 @@ //! [`crate::HashDb::hash_path`]. Unit tests pin the case-insensitive results to //! `ltk_hash`'s `WadHash`/`BinHash` (League paths are ASCII, where they coincide). +use std::fmt; + use xxhash_rust::xxh3::xxh3_64; use xxhash_rust::xxh64::xxh64; use crate::KeyWidth; -/// Whether a table's keys hash the path as given or its lowercased form. +/// Whether a table's keys hash the path as given or its ASCII-lowercased form. /// /// Stored as the `case_insensitive` header flag, orthogonal to [`HashKind`]: /// the algorithm says *how* the bytes are hashed, the casing says *which* bytes. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)] pub enum Casing { /// Hash the path bytes exactly as given. #[default] Sensitive, - /// Lowercase the path before hashing (all League tables). The mapping is - /// Unicode-aware ([`str::to_lowercase`]); on the ASCII paths League ships - /// it reduces to plain `A-Z` → `a-z`, and non-ASCII paths get sensible - /// case-insensitivity for free. - Insensitive, + /// Map `A-Z` to `a-z` before hashing, leaving every other byte alone (all + /// League tables). + /// + /// The mapping is deliberately ASCII-only: a byte substitution with no locale, + /// no Unicode tables, and no toolchain drift, so a key computed today still + /// resolves years from now. Bytes outside `A-Z` pass through untouched, so a + /// publisher whose paths are not ASCII should lowercase them however it likes + /// and hash [`Sensitive`](Casing::Sensitive). + AsciiInsensitive, +} + +impl fmt::Display for Casing { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(match self { + Self::Sensitive => "case-sensitive", + Self::AsciiInsensitive => "ascii-case-insensitive", + }) + } } /// The hash algorithm a table's keys were produced with. The casing rule is /// recorded separately (see [`Casing`]). -#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Default)] #[repr(u8)] pub enum HashKind { /// Not recorded. [`HashKind::hash`] falls back on key width: @@ -61,11 +76,10 @@ impl HashKind { /// Hash `path` with this algorithm under `casing`. `key_width` resolves the /// [`HashKind::Unspecified`] fallback. /// - /// Insensitive hashing is allocation-free for ASCII paths (all League data): - /// they lowercase into a stack buffer. Only non-ASCII or longer-than-buffer - /// paths pay the full Unicode [`str::to_lowercase`] allocation. This sits on - /// the hunt engine's hot path - millions of candidates per round - where the - /// stack path measures ~2-3× faster than the allocating one. + /// Insensitive hashing is allocation-free for paths up to 512 bytes - every + /// path League ships - which lowercase into a stack buffer; longer ones pay + /// one allocation. This sits on the hunt engine's hot path, millions of + /// candidates per round, where the stack path measures ~2-3× faster. pub fn hash(self, path: &str, casing: Casing, key_width: KeyWidth) -> u64 { let kind = match self { Self::Unspecified => match key_width { @@ -75,16 +89,23 @@ impl HashKind { other => other, }; + // ASCII lowercasing is a per-byte map that leaves every byte of a + // multi-byte sequence alone, so it needs no `is_ascii` guard - only a + // buffer big enough to hold the path. match casing { Casing::Sensitive => kind.hash_bytes(path.as_bytes()), - Casing::Insensitive if path.is_ascii() && path.len() <= LOWER_STACK => { + Casing::AsciiInsensitive if path.len() <= LOWER_STACK => { let mut buf = [0u8; LOWER_STACK]; let lowered = &mut buf[..path.len()]; lowered.copy_from_slice(path.as_bytes()); lowered.make_ascii_lowercase(); kind.hash_bytes(lowered) } - Casing::Insensitive => kind.hash_bytes(path.to_lowercase().as_bytes()), + Casing::AsciiInsensitive => { + let mut lowered = path.as_bytes().to_vec(); + lowered.make_ascii_lowercase(); + kind.hash_bytes(&lowered) + } } } @@ -100,8 +121,67 @@ impl HashKind { } } -/// Mixed-case ASCII paths up to this length lowercase on the stack; longer ones -/// fall back to a heap allocation. Real paths max out around 200 bytes. +impl fmt::Display for HashKind { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(match self { + Self::Unspecified => "unspecified", + Self::Xxh64 => "xxh64", + Self::Fnv1a32 => "fnv1a32", + Self::Xxh3 => "xxh3", + }) + } +} + +/// How a table's keys were produced: key width, hash algorithm, casing rule. +/// +/// The three only ever mean something together - a `u64` probe is answerable by a +/// table only when all three of them agree - so they travel as one value. +/// [`HashDb::key_config`](crate::HashDb::key_config) reports a table's, and +/// [`LayeredHashDb`](crate::LayeredHashDb) requires every base to share one. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct KeyConfig { + key_width: KeyWidth, + hash_kind: HashKind, + casing: Casing, +} + +impl KeyConfig { + /// A configuration from its three parts. + pub const fn new(key_width: KeyWidth, hash_kind: HashKind, casing: Casing) -> Self { + Self { + key_width, + hash_kind, + casing, + } + } + + pub const fn key_width(self) -> KeyWidth { + self.key_width + } + + pub const fn hash_kind(self) -> HashKind { + self.hash_kind + } + + pub const fn casing(self) -> Casing { + self.casing + } + + /// Hash `path` the way this table's keys were produced. + pub fn hash(self, path: &str) -> u64 { + self.hash_kind.hash(path, self.casing, self.key_width) + } +} + +impl fmt::Display for KeyConfig { + /// `u64/xxh64/ascii-case-insensitive`. + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}/{}/{}", self.key_width, self.hash_kind, self.casing) + } +} + +/// Mixed-case paths up to this length lowercase on the stack; longer ones fall +/// back to a heap allocation. Real paths max out around 200 bytes. const LOWER_STACK: usize = 512; /// FNV-1a 32 over raw bytes (`ltk_hash::BinHash` only exposes a lowercasing form). @@ -123,16 +203,16 @@ mod tests { fn insensitive_matches_ltk_hash() { let p = "DATA/Characters/Aatrox/Aatrox.bin"; assert_eq!( - HashKind::Xxh64.hash(p, Casing::Insensitive, KeyWidth::U64), + HashKind::Xxh64.hash(p, Casing::AsciiInsensitive, KeyWidth::U64), *WadHash::hash_str(p) ); assert_eq!( - HashKind::Fnv1a32.hash(p, Casing::Insensitive, KeyWidth::U32), + HashKind::Fnv1a32.hash(p, Casing::AsciiInsensitive, KeyWidth::U32), *BinHash::hash_str(p) as u64 ); // Known FNV-1a-lower vector (from ltk_hash's own tests). assert_eq!( - HashKind::Fnv1a32.hash("TEST", Casing::Insensitive, KeyWidth::U32), + HashKind::Fnv1a32.hash("TEST", Casing::AsciiInsensitive, KeyWidth::U32), 0xafd071e5 ); } @@ -140,14 +220,25 @@ mod tests { #[test] fn insensitive_lowercases_before_hashing() { for kind in [HashKind::Xxh64, HashKind::Fnv1a32, HashKind::Xxh3] { - let a = kind.hash("ASSETS/Foo.DDS", Casing::Insensitive, KeyWidth::U64); - let b = kind.hash("assets/foo.dds", Casing::Insensitive, KeyWidth::U64); + let a = kind.hash("ASSETS/Foo.DDS", Casing::AsciiInsensitive, KeyWidth::U64); + let b = kind.hash("assets/foo.dds", Casing::AsciiInsensitive, KeyWidth::U64); assert_eq!(a, b, "{kind:?}"); + } + } + + /// The rule is ASCII-only on purpose: non-ASCII case pairs stay distinct + /// rather than tracking a Unicode table that can move under a published file. + #[test] + fn insensitive_leaves_non_ascii_alone() { + for kind in [HashKind::Xxh64, HashKind::Fnv1a32, HashKind::Xxh3] { + let upper = kind.hash("assets/É.dds", Casing::AsciiInsensitive, KeyWidth::U64); + let lower = kind.hash("assets/é.dds", Casing::AsciiInsensitive, KeyWidth::U64); + assert_ne!(upper, lower, "{kind:?}"); - // The lowercasing is Unicode-aware, not ASCII-only. - let a = kind.hash("assets/É.dds", Casing::Insensitive, KeyWidth::U64); - let b = kind.hash("assets/é.dds", Casing::Insensitive, KeyWidth::U64); - assert_eq!(a, b, "{kind:?} (unicode)"); + // A non-ASCII byte must not disturb the ASCII part of the same path. + let a = kind.hash("É/Foo.DDS", Casing::AsciiInsensitive, KeyWidth::U64); + let b = kind.hash("É/foo.dds", Casing::AsciiInsensitive, KeyWidth::U64); + assert_eq!(a, b, "{kind:?}"); } } @@ -168,7 +259,7 @@ mod tests { for kind in [HashKind::Xxh64, HashKind::Fnv1a32, HashKind::Xxh3] { assert_eq!( kind.hash(p, Casing::Sensitive, KeyWidth::U64), - kind.hash(p, Casing::Insensitive, KeyWidth::U64), + kind.hash(p, Casing::AsciiInsensitive, KeyWidth::U64), "{kind:?}" ); } @@ -179,19 +270,19 @@ mod tests { ); } - /// By definition `Insensitive` must equal lowercase-then-`Sensitive`; pin the - /// stack-buffer / heap fast paths (and the buffer boundary) to it. + /// By definition `AsciiInsensitive` must equal ascii-lowercase-then-`Sensitive`; + /// pin the stack-buffer and heap paths (and the buffer boundary) to it. #[test] - fn insensitive_fast_paths_match_reference() { + fn insensitive_paths_match_reference() { let long_mixed = "A".repeat(600) + "/File.DDS"; let mut cases = vec![ String::new(), "a".into(), - "assets/foo.dds".into(), // ASCII: stack buffer + "assets/foo.dds".into(), // stack buffer "ASSETS/Foo.DDS".into(), - "ässets/FÖÖ.dds".into(), // non-ASCII: heap + "ässets/FÖÖ.dds".into(), // non-ASCII: only the ASCII bytes move "É".into(), - long_mixed, // ASCII past the stack buffer: heap + long_mixed, // past the stack buffer: heap ]; for len in [511, 512, 513] { cases.push("A".repeat(len)); // exactly around the stack-buffer boundary @@ -200,8 +291,8 @@ mod tests { for kind in [HashKind::Xxh64, HashKind::Fnv1a32, HashKind::Xxh3] { for path in &cases { assert_eq!( - kind.hash(path, Casing::Insensitive, KeyWidth::U64), - kind.hash(&path.to_lowercase(), Casing::Sensitive, KeyWidth::U64), + kind.hash(path, Casing::AsciiInsensitive, KeyWidth::U64), + kind.hash(&path.to_ascii_lowercase(), Casing::Sensitive, KeyWidth::U64), "{kind:?} {path:?}" ); } @@ -211,7 +302,7 @@ mod tests { #[test] fn unspecified_falls_back_on_key_width() { let p = "data/characters/aatrox/aatrox.bin"; - for casing in [Casing::Sensitive, Casing::Insensitive] { + for casing in [Casing::Sensitive, Casing::AsciiInsensitive] { assert_eq!( HashKind::Unspecified.hash(p, casing, KeyWidth::U64), HashKind::Xxh64.hash(p, casing, KeyWidth::U64) diff --git a/crates/ltk_hashdb/src/header.rs b/crates/ltk_hashdb/src/header.rs index b9bb477..4f74ae3 100644 --- a/crates/ltk_hashdb/src/header.rs +++ b/crates/ltk_hashdb/src/header.rs @@ -32,7 +32,12 @@ pub const HEADER_SIZE: usize = 80; /// Header flag: the arena is a zeekstd seekable stream rather than raw bytes. pub(crate) const FLAG_ARENA_COMPRESSED: u8 = 1 << 0; -/// Header flag: the keys hash the lowercased path ([`Casing::Insensitive`]). +/// Header flag: the keys hash the ASCII-lowercased path +/// ([`Casing::AsciiInsensitive`]). +/// +/// Should a Unicode-aware rule ever be wanted, it gets its own value in the +/// reserved byte at offset 14 rather than a second flag bit - unknown flag bits +/// are rejected, unknown reserved bytes are not. pub(crate) const FLAG_CASE_INSENSITIVE: u8 = 1 << 1; const KNOWN_FLAGS: u8 = FLAG_ARENA_COMPRESSED | FLAG_CASE_INSENSITIVE; @@ -75,7 +80,7 @@ impl Header { pub fn casing(&self) -> Casing { if self.flags & FLAG_CASE_INSENSITIVE != 0 { - Casing::Insensitive + Casing::AsciiInsensitive } else { Casing::Sensitive } diff --git a/crates/ltk_hashdb/src/layered.rs b/crates/ltk_hashdb/src/layered.rs index fd13d84..b59b6c4 100644 --- a/crates/ltk_hashdb/src/layered.rs +++ b/crates/ltk_hashdb/src/layered.rs @@ -3,14 +3,7 @@ use std::collections::HashMap; use std::fmt; -use crate::{Casing, HashDb, HashKind, KeyWidth, PathRef}; - -/// The key configuration a base hashes under. Every base in a [`LayeredHashDb`] -/// must agree on this triple; a base that diverges can never be hit by a caller's -/// precomputed probe (see the type-level invariant). -fn base_config(db: &HashDb) -> (KeyWidth, HashKind, Casing) { - (db.key_width(), db.hash_kind(), db.casing()) -} +use crate::{HashDb, KeyConfigMismatch, PathRef}; /// A writable in-memory overlay on top of ordered read-only [`HashDb`] bases. /// @@ -24,13 +17,16 @@ fn base_config(db: &HashDb) -> (KeyWidth, HashKind, Casing) { /// Lookups take a `u64` the caller already computed, and each base binary-searches /// its own key set with that raw value - there is no per-base re-hashing. So every /// base (and any path registered via [`insert_path`](Self::insert_path)) must share -/// the same key configuration: [`key_width`](HashDb::key_width), -/// [`hash_kind`](HashDb::hash_kind), and [`casing`](HashDb::casing). A base that -/// diverges is silently unreachable - the caller's probes were hashed for a -/// different scheme, so they can never match it. [`push_base`](Self::push_base) and -/// [`from_bases`](Self::from_bases) `debug_assert!` this; release builds skip the -/// check. League's `game`/`lcu` tables are uniform (XXH64 / U64 / case-insensitive), -/// so the common path always satisfies it. +/// one [`KeyConfig`](crate::KeyConfig). A base that diverges is unreachable - the caller's probes were +/// hashed for a different scheme, so they can never match it - which is why +/// [`push_base`](Self::push_base) and [`from_bases`](Self::from_bases) refuse one +/// instead of layering it. League's `game`/`lcu` tables are uniform (u64 / xxh64 / +/// ascii-case-insensitive), so the common path always satisfies it. +/// +/// A shared key config is necessary, not sufficient: it says two bases *can* answer +/// each other's probes, not that they *should*. `binentries` and `binfields` share +/// one and mean entirely different things, so `HashStore::open_layered` refuses that +/// pairing on top of this check. #[derive(Default)] pub struct LayeredHashDb { overlay: HashMap>, @@ -46,39 +42,53 @@ impl LayeredHashDb { } /// Layer an empty overlay over `bases`, in the given priority order (`bases[0]` - /// shadows `bases[1]`, and so on). In debug builds, asserts every base shares - /// the first one's key configuration (see the type-level invariant). - pub fn from_bases(bases: Vec) -> Self { + /// shadows `bases[1]`, and so on). + /// + /// # Errors + /// + /// [`KeyConfigMismatch`] if any base hashes its keys differently from `bases[0]` + /// - see the type-level invariant. + pub fn from_bases(bases: Vec) -> Result { if let Some((first, rest)) = bases.split_first() { - let cfg = base_config(first); - for db in rest { - debug_assert_eq!( - base_config(db), - cfg, - "LayeredHashDb bases must share key config (key_width/hash_kind/casing); \ - a divergent base is unreachable by a caller's precomputed probe" - ); + let expected = first.key_config(); + for (i, db) in rest.iter().enumerate() { + let found = db.key_config(); + if found != expected { + return Err(KeyConfigMismatch { + index: i + 1, + expected, + found, + }); + } } } - Self { + Ok(Self { overlay: HashMap::new(), bases, - } + }) } /// Append a lower-priority base below all existing ones. - pub fn push_base(&mut self, db: HashDb) { + /// + /// # Errors + /// + /// [`KeyConfigMismatch`] if `db` hashes its keys differently from the first base + /// - see the type-level invariant. The layer is left untouched. + pub fn push_base(&mut self, db: HashDb) -> Result<(), KeyConfigMismatch> { if let Some(first) = self.bases.first() { - debug_assert_eq!( - base_config(&db), - base_config(first), - "LayeredHashDb bases must share key config (key_width/hash_kind/casing); \ - a divergent base is unreachable by a caller's precomputed probe" - ); + let (expected, found) = (first.key_config(), db.key_config()); + if found != expected { + return Err(KeyConfigMismatch { + index: self.bases.len(), + expected, + found, + }); + } } self.bases.push(db); + Ok(()) } /// Insert an overlay entry (e.g. a runtime mod hash). Shadows every base. @@ -328,7 +338,7 @@ mod tests { fn layering_order_shadows_lower_layers() { let base0 = raw_db(&[(1, "base0/one"), (2, "base0/two")]); let base1 = raw_db(&[(2, "base1/two"), (3, "base1/three")]); - let mut db = LayeredHashDb::from_bases(vec![base0, base1]); + let mut db = LayeredHashDb::from_bases(vec![base0, base1]).expect("uniform bases"); db.insert(1, "overlay/one"); // Overlay shadows base 0. @@ -347,7 +357,7 @@ mod tests { fn get_and_get_batch_agree_in_input_order() { let base0 = raw_db(&[(10, "base0/ten"), (20, "shadowed")]); let base1 = raw_db(&[(20, "base1/twenty"), (30, "base1/thirty")]); - let mut db = LayeredHashDb::from_bases(vec![base0, base1]); + let mut db = LayeredHashDb::from_bases(vec![base0, base1]).expect("uniform bases"); db.insert(5, "overlay/five"); // Mixed set: overlay hit, base-0 hit, base-1 hit, miss, duplicate. @@ -369,7 +379,7 @@ mod tests { let base = compressed_db(256); let frames = base.decompressions(); // 0 before any read assert_eq!(frames, 0); - let db = LayeredHashDb::from_bases(vec![base]); + let db = LayeredHashDb::from_bases(vec![base]).expect("uniform bases"); // Batch every real key (i*3 for i in 0..100) plus some misses. let mut probes: Vec = (0..100u64).map(|i| i * 3).collect(); @@ -401,24 +411,33 @@ mod tests { assert!(db.bases().is_empty()); } + /// A U32 base under a U64 one is unreachable, so it is refused - in release + /// builds too, which is the point of this not being a `debug_assert!`. #[test] - #[should_panic(expected = "must share key config")] fn push_base_rejects_divergent_key_config() { let u64_base = raw_db_width(KeyWidth::U64, &[(1, "u64/one")]); let u32_base = raw_db_width(KeyWidth::U32, &[(2, "u32/two")]); - let mut db = LayeredHashDb::from_bases(vec![u64_base]); + let mut db = LayeredHashDb::from_bases(vec![u64_base]).expect("one base"); - // Debug-only guard: layering a U32 base under a U64 base is unreachable. - db.push_base(u32_base); + let err = db.push_base(u32_base).expect_err("divergent base refused"); + assert_eq!(err.index, 1); + assert_eq!(err.expected.key_width(), KeyWidth::U64); + assert_eq!(err.found.key_width(), KeyWidth::U32); + assert_eq!(db.bases().len(), 1, "the layer is left untouched"); } #[test] - #[should_panic(expected = "must share key config")] fn from_bases_rejects_divergent_key_config() { let u64_base = raw_db_width(KeyWidth::U64, &[(1, "u64/one")]); let u32_base = raw_db_width(KeyWidth::U32, &[(2, "u32/two")]); - let _ = LayeredHashDb::from_bases(vec![u64_base, u32_base]); + let err = LayeredHashDb::from_bases(vec![u64_base, u32_base]).expect_err("divergent base"); + assert_eq!(err.index, 1); + let msg = err.to_string(); + assert!( + msg.contains("u64/") && msg.contains("u32/"), + "names both configs: {msg}" + ); } /// `iter` enumerates exactly what `get` can resolve: every shadowed key is @@ -427,7 +446,7 @@ mod tests { fn iter_yields_each_key_once_from_its_answering_layer() { let base0 = raw_db(&[(1, "base0/one"), (2, "base0/two")]); let base1 = raw_db(&[(2, "base1/two"), (3, "base1/three")]); - let mut db = LayeredHashDb::from_bases(vec![base0, base1]); + let mut db = LayeredHashDb::from_bases(vec![base0, base1]).expect("uniform bases"); db.insert(1, "overlay/one"); let mut seen: Vec<(u64, String)> = db @@ -458,7 +477,7 @@ mod tests { fn for_each_batch_matches_get_batch() { let base0 = raw_db(&[(10, "base0/ten"), (20, "shadowed")]); let base1 = raw_db(&[(20, "base1/twenty"), (30, "base1/thirty")]); - let mut db = LayeredHashDb::from_bases(vec![base0, base1]); + let mut db = LayeredHashDb::from_bases(vec![base0, base1]).expect("uniform bases"); db.insert(5, "overlay/five"); let probes = [5u64, 10, 20, 30, 999, 10]; @@ -477,7 +496,8 @@ mod tests { /// Debug prints shape, never entries. #[test] fn debug_prints_shape_only() { - let mut db = LayeredHashDb::from_bases(vec![raw_db(&[(1, "secret/path.bin")])]); + let mut db = + LayeredHashDb::from_bases(vec![raw_db(&[(1, "secret/path.bin")])]).expect("one base"); db.insert(2, "overlay/secret.bin"); let shown = format!("{db:?}"); @@ -489,7 +509,7 @@ mod tests { #[test] fn insert_path_uses_first_base() { let base = raw_db(&[(1, "seed")]); - let mut db = LayeredHashDb::from_bases(vec![base]); + let mut db = LayeredHashDb::from_bases(vec![base]).expect("uniform bases"); let path = "assets/characters/aatrox/aatrox.bin"; let hash = db.insert_path(path).expect("has a base"); assert_eq!(db.get(hash).as_deref(), Some(path)); diff --git a/crates/ltk_hashdb/src/lib.rs b/crates/ltk_hashdb/src/lib.rs index 8defb6f..675ea3f 100644 --- a/crates/ltk_hashdb/src/lib.rs +++ b/crates/ltk_hashdb/src/lib.rs @@ -17,8 +17,8 @@ mod path; mod reader; mod writer; -pub use error::{BuildError, OpenError, VerifyError}; -pub use hash::{Casing, HashKind}; +pub use error::{BuildError, KeyConfigMismatch, OpenError, VerifyError}; +pub use hash::{Casing, HashKind, KeyConfig}; pub use header::{FORMAT_VERSION, HEADER_SIZE, MAGIC}; pub use layered::LayeredHashDb; pub use path::PathRef; @@ -26,7 +26,7 @@ pub use reader::{HashDb, HashDbOptions, WeakHashDb, DEFAULT_FRAME_CACHE_BYTES}; pub use writer::{BuildStats, HashDbWriter}; /// Width of the integer keys in a table. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum KeyWidth { /// 32-bit keys (bin tables: FNV-1a). U32, @@ -45,6 +45,15 @@ impl KeyWidth { } } +impl std::fmt::Display for KeyWidth { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str(match self { + Self::U32 => "u32", + Self::U64 => "u64", + }) + } +} + /// Arena compression strategy. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Compression { diff --git a/crates/ltk_hashdb/src/reader.rs b/crates/ltk_hashdb/src/reader.rs index 66041c8..72ee6fb 100644 --- a/crates/ltk_hashdb/src/reader.rs +++ b/crates/ltk_hashdb/src/reader.rs @@ -15,7 +15,7 @@ use zeekstd::SeekTable; use crate::cache::{Frame, FrameCache}; use crate::header::Header; -use crate::{Casing, HashKind, KeyWidth, OpenError, PathRef, VerifyError}; +use crate::{Casing, HashKind, KeyConfig, KeyWidth, OpenError, PathRef, VerifyError}; /// Decompressed frame bytes a table caches by default: 4 MiB, i.e. 256 frames at the /// published 16 KiB frame size. @@ -477,12 +477,19 @@ impl HashDb { self.inner.header.hash_kind } - /// Whether the keys hash the lowercased path (from the `case_insensitive` - /// header flag). + /// Whether the keys hash the ASCII-lowercased path (from the + /// `case_insensitive` header flag). pub fn casing(&self) -> Casing { self.inner.header.casing() } + /// Width, algorithm, and casing as one value - what a probe must be hashed + /// under to be answerable here, and what every base of a + /// [`LayeredHashDb`](crate::LayeredHashDb) must agree on. + pub fn key_config(&self) -> KeyConfig { + KeyConfig::new(self.key_width(), self.hash_kind(), self.casing()) + } + /// Whether the arena is zeekstd-compressed on disk. pub fn is_compressed(&self) -> bool { self.inner.header.arena_compressed() @@ -508,11 +515,7 @@ impl HashDb { /// the `hash_kind` header field - falling back on key width when /// unspecified - and the `case_insensitive` flag). pub fn hash_path(&self, path: &str) -> u64 { - self.inner.header.hash_kind.hash( - path, - self.inner.header.casing(), - self.inner.header.key_width, - ) + self.key_config().hash(path) } /// Iterate entries in arena order (path order, **not** key order) so each frame diff --git a/crates/ltk_hashdb/src/writer.rs b/crates/ltk_hashdb/src/writer.rs index e090784..8e33d33 100644 --- a/crates/ltk_hashdb/src/writer.rs +++ b/crates/ltk_hashdb/src/writer.rs @@ -7,7 +7,7 @@ use xxhash_rust::xxh3::Xxh3; use crate::header::{ Header, OffsetWidth, FLAG_ARENA_COMPRESSED, FLAG_CASE_INSENSITIVE, HEADER_SIZE, }; -use crate::{BuildError, Casing, Compression, HashKind, KeyWidth}; +use crate::{BuildError, Casing, Compression, HashKind, KeyConfig, KeyWidth}; /// Collects `(key, path)` pairs, then [`HashDbWriter::build`] sorts by key, dedups, /// assigns arena offsets, and writes the file. @@ -29,6 +29,9 @@ pub struct BuildStats { } impl HashDbWriter { + /// A writer for `key_width` keys, with the algorithm unrecorded and the paths + /// hashed as given; [`hash_kind`](Self::hash_kind) and [`casing`](Self::casing) + /// fill those in. pub fn new(key_width: KeyWidth, compression: Compression) -> Self { Self { key_width, @@ -39,6 +42,16 @@ impl HashDbWriter { } } + /// A writer for a table whose key configuration is already known - a League + /// table, say, whose [`Table::key_config`] states all three at once. + /// + /// [`Table::key_config`]: https://docs.rs/ltk_mimir_cache/latest/ltk_mimir_cache/enum.Table.html#method.key_config + pub fn with_key_config(config: KeyConfig, compression: Compression) -> Self { + Self::new(config.key_width(), compression) + .hash_kind(config.hash_kind()) + .casing(config.casing()) + } + /// Record the algorithm the keys were hashed with, so readers can hash new /// paths via `HashDb::hash_path`. pub fn hash_kind(mut self, kind: HashKind) -> Self { @@ -46,7 +59,7 @@ impl HashDbWriter { self } - /// Record whether the keys hash the lowercased path ([`Casing::Insensitive`], + /// Record whether the keys hash the ASCII-lowercased path ([`Casing::AsciiInsensitive`], /// all League tables) or the path as given. Defaults to [`Casing::Sensitive`]. pub fn casing(mut self, casing: Casing) -> Self { self.casing = casing; @@ -150,7 +163,7 @@ impl HashDbWriter { (compressed, FLAG_ARENA_COMPRESSED) } }; - if self.casing == Casing::Insensitive { + if self.casing == Casing::AsciiInsensitive { flags |= FLAG_CASE_INSENSITIVE; } diff --git a/crates/ltk_hashdb/tests/golden.rs b/crates/ltk_hashdb/tests/golden.rs index f70077f..d60c1f3 100644 --- a/crates/ltk_hashdb/tests/golden.rs +++ b/crates/ltk_hashdb/tests/golden.rs @@ -71,7 +71,7 @@ fn build_bytes( ) -> Vec { let mut w = HashDbWriter::new(key_width, compression) .hash_kind(hash_kind) - .casing(Casing::Insensitive); + .casing(Casing::AsciiInsensitive); w.extend(entries.iter().map(|(&k, p)| (k, p.as_str()))); let mut out = Cursor::new(Vec::new()); w.build(&mut out).expect("build"); @@ -138,7 +138,7 @@ fn golden_parity() { // independent xxh64) - so tolerate a handful, not zero. let mut mismatches = 0usize; for (&k, p) in &entries { - if hash_kind.hash(p, Casing::Insensitive, key_width) != k { + if hash_kind.hash(p, Casing::AsciiInsensitive, key_width) != k { mismatches += 1; if mismatches <= 5 { eprintln!("{file}: upstream hash mismatch: {k:#x} {p:?}"); diff --git a/crates/ltk_hashdb/tests/roundtrip.rs b/crates/ltk_hashdb/tests/roundtrip.rs index e8c9cd2..4ce3ee7 100644 --- a/crates/ltk_hashdb/tests/roundtrip.rs +++ b/crates/ltk_hashdb/tests/roundtrip.rs @@ -16,7 +16,7 @@ fn build_with( // The fixtures are League-shaped, so record the League casing rule. let mut w = HashDbWriter::new(key_width, compression) .hash_kind(hash_kind) - .casing(Casing::Insensitive); + .casing(Casing::AsciiInsensitive); w.extend(entries.iter().copied()); let mut out = Cursor::new(Vec::new()); let stats = w.build(&mut out).expect("build"); @@ -296,7 +296,7 @@ fn zero_frame_size_rejected() { fn hash_path_uses_table_algorithm() { let bytes = build(KeyWidth::U32, HashKind::Fnv1a32, &[]); let db = HashDb::open_bytes(bytes).expect("open"); - assert_eq!(db.casing(), Casing::Insensitive); + assert_eq!(db.casing(), Casing::AsciiInsensitive); assert_eq!(db.hash_path("TEST"), 0xafd071e5); } @@ -344,7 +344,7 @@ fn bad_magic_rejected() { fn overlay_shadows_a_real_base_table() { let bytes = build(KeyWidth::U64, HashKind::Xxh64, GAME_ENTRIES); let db = HashDb::open_bytes(bytes).expect("open"); - let mut layered = LayeredHashDb::from_bases(vec![db]); + let mut layered = LayeredHashDb::from_bases(vec![db]).expect("one base"); // Base entries still resolve. assert_eq!( diff --git a/crates/ltk_hashdb/tests/snapshot.rs b/crates/ltk_hashdb/tests/snapshot.rs index 82fb002..ba088ac 100644 --- a/crates/ltk_hashdb/tests/snapshot.rs +++ b/crates/ltk_hashdb/tests/snapshot.rs @@ -21,7 +21,7 @@ fn hex_dump(bytes: &[u8]) -> String { fn fixture_file_bytes() { let mut w = HashDbWriter::new(KeyWidth::U64, Compression::None) .hash_kind(HashKind::Xxh64) - .casing(Casing::Insensitive); + .casing(Casing::AsciiInsensitive); w.insert(0x0123_4567_89ab_cdef, "assets/a.dds"); w.insert(0x0000_0000_0000_0042, "data/b.bin"); let mut out = Cursor::new(Vec::new()); @@ -34,7 +34,7 @@ fn fixture_file_bytes() { fn fixture_u32_header() { let mut w = HashDbWriter::new(KeyWidth::U32, Compression::None) .hash_kind(HashKind::Fnv1a32) - .casing(Casing::Insensitive); + .casing(Casing::AsciiInsensitive); w.insert(0xafd0_71e5, "test"); let mut out = Cursor::new(Vec::new()); w.build(&mut out).expect("build"); diff --git a/crates/ltk_mimir_cache/src/error.rs b/crates/ltk_mimir_cache/src/error.rs index e45fbc2..03168ee 100644 --- a/crates/ltk_mimir_cache/src/error.rs +++ b/crates/ltk_mimir_cache/src/error.rs @@ -5,7 +5,7 @@ use std::path::PathBuf; use thiserror::Error; -use crate::Table; +use crate::{HashUniverse, Table}; /// Errors from resolving the platform cache directory /// ([`HashStore::discover`](crate::HashStore::discover)). @@ -13,6 +13,41 @@ use crate::Table; #[error("could not determine a platform cache directory")] pub struct NoCacheDirError; +/// A string that names no [`Table`] ([`Table::from_str`](std::str::FromStr::from_str)). +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ParseTableError { + input: String, +} + +impl ParseTableError { + pub(crate) fn new(input: &str) -> Self { + Self { + input: input.to_owned(), + } + } + + /// The string that failed to parse. + pub fn input(&self) -> &str { + &self.input + } +} + +impl std::fmt::Display for ParseTableError { + /// Lists every accepted id, so a typo is one message away from being fixed. + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "unknown table {:?}; expected one of ", self.input)?; + for (i, table) in Table::ALL.iter().enumerate() { + if i > 0 { + f.write_str(", ")?; + } + f.write_str(table.id())?; + } + Ok(()) + } +} + +impl std::error::Error for ParseTableError {} + /// Errors from reading, parsing, or writing `manifest.json`. #[derive(Debug, Error)] pub enum ManifestError { @@ -36,11 +71,35 @@ pub enum OpenError { #[error(transparent)] Manifest(#[from] ManifestError), - #[error("table {0:?} is not in the manifest")] + #[error("table {0} is not in the manifest")] TableNotFound(Table), #[error("opening the table file")] HashDb(#[from] ltk_hashdb::OpenError), + + #[error("the table file does not hash its keys the way this table is defined to")] + KeyConfig(#[from] ltk_hashdb::KeyConfigMismatch), +} + +/// Refusal to layer tables drawn from different hash universes +/// ([`HashStore::open_layered`](crate::HashStore::open_layered)). +#[derive(Debug, Clone, Copy, Error)] +#[error( + "cannot layer {table} ({found}) with {first} ({expected}): a hash means nothing outside \ + its own universe, so one table would answer the other with a confident, wrong path" +)] +pub struct UniverseMismatch { + /// The first table in the requested set - the universe the rest must match. + pub first: Table, + + /// That table's universe. + pub expected: HashUniverse, + + /// The table that does not belong to it. + pub table: Table, + + /// The universe it belongs to instead. + pub found: HashUniverse, } /// Errors from installing tables ([`HashStore::commit`](crate::HashStore::commit)). diff --git a/crates/ltk_mimir_cache/src/lib.rs b/crates/ltk_mimir_cache/src/lib.rs index f82c78c..125fafa 100644 --- a/crates/ltk_mimir_cache/src/lib.rs +++ b/crates/ltk_mimir_cache/src/lib.rs @@ -16,9 +16,13 @@ mod fsutil; mod lock; mod manifest; mod store; +mod table; mod update; -pub use error::{CommitError, GcError, ManifestError, NoCacheDirError, OpenError, UpdateError}; +pub use error::{ + CommitError, GcError, ManifestError, NoCacheDirError, OpenError, ParseTableError, + UniverseMismatch, UpdateError, +}; #[cfg(feature = "reqwest")] pub use fetch::ReqwestFetch; #[cfg(feature = "ureq")] @@ -28,53 +32,5 @@ pub use fetch::{HttpFetchError, ReleaseSource}; pub use lock::UpdateLock; pub use manifest::{Manifest, Source, TableEntry, SCHEMA_VERSION}; pub use store::{CommitItem, GcReport, HashStore}; +pub use table::{HashUniverse, Table}; pub use update::{AsyncFetch, Fetch, UpdateOptions, UpdateOutcome, UpdateReport}; - -/// The logical hash tables, each stored as its own `.lhdb` file. -/// -/// The two RST variants hash the same strings with different algorithms (XXH64 -/// vs XXH3 for RST v5+), so they are separate tables (see `docs/FORMAT.md`). -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] -pub enum Table { - Game, - Lcu, - BinEntries, - BinTypes, - BinFields, - BinHashes, - Rst, - RstXxh3, -} - -impl Table { - /// Every logical table, in a stable order. - pub const ALL: [Table; 8] = [ - Table::Game, - Table::Lcu, - Table::BinEntries, - Table::BinTypes, - Table::BinFields, - Table::BinHashes, - Table::Rst, - Table::RstXxh3, - ]; - - /// The stable string id used in filenames and manifest keys. - pub fn id(self) -> &'static str { - match self { - Table::Game => "game", - Table::Lcu => "lcu", - Table::BinEntries => "binentries", - Table::BinTypes => "bintypes", - Table::BinFields => "binfields", - Table::BinHashes => "binhashes", - Table::Rst => "rst", - Table::RstXxh3 => "rst-xxh3", - } - } - - /// Parse a table from its [`id`](Table::id). - pub fn from_id(id: &str) -> Option { - Table::ALL.into_iter().find(|t| t.id() == id) - } -} diff --git a/crates/ltk_mimir_cache/src/store.rs b/crates/ltk_mimir_cache/src/store.rs index a3cc0e8..cfea53b 100644 --- a/crates/ltk_mimir_cache/src/store.rs +++ b/crates/ltk_mimir_cache/src/store.rs @@ -9,7 +9,8 @@ use ltk_hashdb::{HashDb, LayeredHashDb, WeakHashDb}; use crate::manifest::{Manifest, Source, TableEntry}; use crate::{ - dir, fsutil, CommitError, GcError, ManifestError, NoCacheDirError, OpenError, Table, UpdateLock, + dir, fsutil, CommitError, GcError, ManifestError, NoCacheDirError, OpenError, Table, + UniverseMismatch, UpdateLock, }; /// The manifest filename inside the cache directory. @@ -179,17 +180,50 @@ impl HashStore { /// /// Tables are opened through [`open_shared`](HashStore::open_shared), so calling /// this twice does not map anything twice. - pub fn open_layered(&self, tables: &[Table]) -> (LayeredHashDb, Vec<(Table, OpenError)>) { + /// + /// # Errors + /// + /// [`UniverseMismatch`] if `tables` spans more than one + /// [`HashUniverse`](crate::HashUniverse) - `binentries` under `binfields`, say, + /// where one table would answer the other's hashes with an unrelated path. That + /// is decided before anything is opened. + /// + /// A table that is missing, unreadable, or keyed differently than it claims is + /// *not* an error here: it lands in the returned per-table list and is left out + /// of the layer. + pub fn open_layered( + &self, + tables: &[Table], + ) -> Result<(LayeredHashDb, Vec<(Table, OpenError)>), UniverseMismatch> { + if let Some((&first, rest)) = tables.split_first() { + let expected = first.universe(); + if let Some(&table) = rest.iter().find(|t| t.universe() != expected) { + return Err(UniverseMismatch { + first, + expected, + table, + found: table.universe(), + }); + } + } + let mut layered = LayeredHashDb::new(); let mut errors = Vec::new(); for &table in tables { match self.open_shared(table) { - Ok(db) => layered.push_base(db), + // One universe implies one key config, so this only fires on a file + // that is not the table it is filed under - a mislabelled download, + // not a caller mistake. Skip it like any other unusable table. + Ok(db) => { + if let Err(e) = layered.push_base(db) { + errors.push((table, e.into())); + } + } Err(e) => errors.push((table, e)), } } - (layered, errors) + Ok((layered, errors)) } /// Try to become the single updater without blocking. `Ok(None)` means another diff --git a/crates/ltk_mimir_cache/src/table.rs b/crates/ltk_mimir_cache/src/table.rs new file mode 100644 index 0000000..fd98c61 --- /dev/null +++ b/crates/ltk_mimir_cache/src/table.rs @@ -0,0 +1,297 @@ +//! The logical hash tables, and the hash universe each one publishes into. + +use std::fmt; +use std::str::FromStr; + +use ltk_hashdb::{Casing, HashKind, KeyConfig, KeyWidth}; +use serde::de::Error as _; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; + +use crate::ParseTableError; + +/// The logical hash tables, each stored as its own `.lhdb` file. +/// +/// The two RST variants hash the same strings with different algorithms (XXH64 +/// vs XXH3 for RST v5+), so they are separate tables (see `docs/FORMAT.md`). +/// +/// The set grows as CommunityDragon publishes new lists, so this is +/// `#[non_exhaustive]`: match with a `_` arm, and iterate [`ALL`](Table::ALL) +/// rather than writing the variants out. It serializes as its [`id`](Table::id), +/// which is also what [`FromStr`] accepts. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum Table { + Game, + Lcu, + BinEntries, + BinTypes, + BinFields, + BinHashes, + Rst, + RstXxh3, +} + +impl Table { + /// Every logical table, in a stable order. + pub const ALL: &'static [Table] = &[ + Table::Game, + Table::Lcu, + Table::BinEntries, + Table::BinTypes, + Table::BinFields, + Table::BinHashes, + Table::Rst, + Table::RstXxh3, + ]; + + /// The stable string id used in filenames and manifest keys. + pub const fn id(self) -> &'static str { + match self { + Table::Game => "game", + Table::Lcu => "lcu", + Table::BinEntries => "binentries", + Table::BinTypes => "bintypes", + Table::BinFields => "binfields", + Table::BinHashes => "binhashes", + Table::Rst => "rst", + Table::RstXxh3 => "rst-xxh3", + } + } + + /// Parse a table from its [`id`](Table::id). + pub fn from_id(id: &str) -> Option
{ + Table::ALL.iter().copied().find(|t| t.id() == id) + } + + /// Which universe of hashed strings this table's keys are drawn from. + pub fn universe(self) -> HashUniverse { + match self { + Table::Game | Table::Lcu => HashUniverse::WadPath, + Table::BinEntries => HashUniverse::BinEntry, + Table::BinTypes => HashUniverse::BinType, + Table::BinFields => HashUniverse::BinField, + Table::BinHashes => HashUniverse::BinHash, + Table::Rst => HashUniverse::RstXxh64, + Table::RstXxh3 => HashUniverse::RstXxh3, + } + } + + /// How this table's keys were produced, from its + /// [`universe`](Table::universe). + pub fn key_config(self) -> KeyConfig { + self.universe().key_config() + } + + /// Key width: 8 bytes for the WAD path and RST tables, 4 for the bin tables. + pub fn key_width(self) -> KeyWidth { + self.key_config().key_width() + } + + /// The algorithm this table's keys were hashed with. + pub fn hash_kind(self) -> HashKind { + self.key_config().hash_kind() + } + + /// The casing rule: every League table hashes the ASCII-lowercased string. + pub fn casing(self) -> Casing { + self.key_config().casing() + } +} + +impl fmt::Display for Table { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.id()) + } +} + +/// The universe of strings a set of hashes was drawn from. +/// +/// Grows with [`Table`], so it is `#[non_exhaustive]` for the same reason. +/// +/// A hash only means something inside its universe: `binentries` and `binfields` +/// are both 32-bit FNV-1a over an ASCII-lowercased string, so a property hash +/// looked up in the entry table can collide with an unrelated object path and +/// come back with a confident, wrong answer. Tables may therefore only be layered +/// together (see [`HashStore::open_layered`](crate::HashStore::open_layered)) when +/// they share a universe - `game` and `lcu`, which are two halves of one WAD path +/// space, and nothing else. +/// +/// A shared [`KeyConfig`] is the weaker, mechanical half of that: it is what +/// `ltk_hashdb` can check without knowing what the strings mean. Universes are +/// where the meaning lives, so each one states its key configuration and the +/// tables read theirs off it. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +#[non_exhaustive] +pub enum HashUniverse { + /// WAD chunk paths (`game`, `lcu`). + WadPath, + + /// `.bin` entry (object) paths. + BinEntry, + + /// `.bin` class and struct names. + BinType, + + /// `.bin` property (field) names. + BinField, + + /// Other strings that appear hashed inside `.bin` values. + BinHash, + + /// RST stringtable keys, XXH64. + RstXxh64, + + /// RST stringtable keys, XXH3 (RST v5+). + RstXxh3, +} + +impl HashUniverse { + /// How every table in this universe hashes its strings. + /// + /// This is the workspace's single statement of each table's width, algorithm, + /// and casing; [`Table::key_config`] and the CLI both read it from here. + pub fn key_config(self) -> KeyConfig { + // Every League list is hashed from the lowercased string. + let casing = Casing::AsciiInsensitive; + match self { + Self::WadPath | Self::RstXxh64 => { + KeyConfig::new(KeyWidth::U64, HashKind::Xxh64, casing) + } + Self::RstXxh3 => KeyConfig::new(KeyWidth::U64, HashKind::Xxh3, casing), + Self::BinEntry | Self::BinType | Self::BinField | Self::BinHash => { + KeyConfig::new(KeyWidth::U32, HashKind::Fnv1a32, casing) + } + } + } + + /// The stable string id, as it appears in diagnostics. + pub fn id(self) -> &'static str { + match self { + Self::WadPath => "wad-path", + Self::BinEntry => "bin-entry", + Self::BinType => "bin-type", + Self::BinField => "bin-field", + Self::BinHash => "bin-hash", + Self::RstXxh64 => "rst-xxh64", + Self::RstXxh3 => "rst-xxh3", + } + } +} + +impl fmt::Display for HashUniverse { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.id()) + } +} + +impl FromStr for Table { + type Err = ParseTableError; + + /// Accepts exactly the [`id`](Table::id) spellings - `binentries`, not + /// `BinEntries` or `bin-entries`. + fn from_str(s: &str) -> Result { + Table::from_id(s).ok_or_else(|| ParseTableError::new(s)) + } +} + +impl Serialize for Table { + fn serialize(&self, serializer: S) -> Result { + serializer.serialize_str(self.id()) + } +} + +impl<'de> Deserialize<'de> for Table { + fn deserialize>(deserializer: D) -> Result { + let id = String::deserialize(deserializer)?; + Table::from_id(&id).ok_or_else(|| D::Error::custom(ParseTableError::new(&id))) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn ids_round_trip() { + for &table in Table::ALL { + assert_eq!(Table::from_id(table.id()), Some(table)); + assert_eq!(table.to_string(), table.id()); + assert_eq!(table.id().parse(), Ok(table)); + } + assert_eq!(Table::from_id("nope"), None); + } + + /// Serde round-trips through the id, not the variant name. + #[test] + fn serde_uses_the_id() { + for &table in Table::ALL { + let json = serde_json::to_string(&table).unwrap(); + assert_eq!(json, format!("{:?}", table.id())); + assert_eq!(serde_json::from_str::
(&json).unwrap(), table); + } + + let err = serde_json::from_str::
("\"BinEntries\"").unwrap_err(); + assert!(err.to_string().contains("binentries"), "{err}"); + } + + #[test] + fn parse_error_lists_the_accepted_ids() { + let err = "bin-entries".parse::
().unwrap_err(); + assert_eq!(err.input(), "bin-entries"); + let msg = err.to_string(); + assert!( + msg.contains("binentries") && msg.contains("rst-xxh3"), + "{msg}" + ); + } + + /// The facts the CLI and the bundler used to each carry a copy of. + #[test] + fn key_configs_match_the_published_tables() { + use Table::*; + for table in [Game, Lcu, Rst] { + assert_eq!(table.key_width(), KeyWidth::U64, "{table}"); + assert_eq!(table.hash_kind(), HashKind::Xxh64, "{table}"); + } + assert_eq!(RstXxh3.key_width(), KeyWidth::U64); + assert_eq!(RstXxh3.hash_kind(), HashKind::Xxh3); + for table in [BinEntries, BinTypes, BinFields, BinHashes] { + assert_eq!(table.key_width(), KeyWidth::U32, "{table}"); + assert_eq!(table.hash_kind(), HashKind::Fnv1a32, "{table}"); + } + for &table in Table::ALL { + assert_eq!(table.casing(), Casing::AsciiInsensitive, "{table}"); + } + } + + /// The point of having a universe at all: it is strictly finer than the key + /// config, so a shared config is not licence to layer two tables. + #[test] + fn universes_are_finer_than_key_configs() { + use Table::*; + assert_eq!(Game.universe(), Lcu.universe(), "one WAD path space"); + + for a in [BinEntries, BinTypes, BinFields, BinHashes] { + for b in [BinEntries, BinTypes, BinFields, BinHashes] { + assert_eq!(a.key_config(), b.key_config(), "{a} vs {b}"); + assert_eq!(a == b, a.universe() == b.universe(), "{a} vs {b}"); + } + } + } + + #[test] + fn every_table_hashes_under_its_own_config() { + // The one path all eight tables can agree on the spelling of. + let path = "data/characters/aatrox/aatrox.bin"; + for &table in Table::ALL { + let config = table.key_config(); + assert_eq!( + config.hash(path), + config + .hash_kind() + .hash(path, config.casing(), config.key_width()), + "{table}" + ); + } + } +} diff --git a/crates/ltk_mimir_cache/tests/store.rs b/crates/ltk_mimir_cache/tests/store.rs index 8216908..57e3da9 100644 --- a/crates/ltk_mimir_cache/tests/store.rs +++ b/crates/ltk_mimir_cache/tests/store.rs @@ -8,7 +8,7 @@ use std::sync::Arc; use ltk_hashdb::{Compression, HashDbWriter, KeyWidth}; use ltk_mimir_cache::{ - CommitError, CommitItem, HashStore, ManifestError, OpenError, Source, Table, + CommitError, CommitItem, HashStore, HashUniverse, ManifestError, OpenError, Source, Table, }; use tempfile::tempdir; @@ -148,7 +148,9 @@ fn open_layered_skips_missing_and_respects_order() { // Both present: no errors, each table's unique key resolves, and the earlier // table shadows the later one on the shared key. - let (db, errors) = store.open_layered(&[Table::Game, Table::Lcu]); + let (db, errors) = store + .open_layered(&[Table::Game, Table::Lcu]) + .expect("both are WAD path tables"); assert!(errors.is_empty()); assert_eq!(db.get(0x1111).as_deref(), Some("game/only")); assert_eq!(db.get(0x2222).as_deref(), Some("lcu/only")); @@ -156,7 +158,9 @@ fn open_layered_skips_missing_and_respects_order() { assert_eq!(db.bases().len(), 2); // Reversing the request order flips which table wins the shared key. - let (db, _) = store.open_layered(&[Table::Lcu, Table::Game]); + let (db, _) = store + .open_layered(&[Table::Lcu, Table::Game]) + .expect("both are WAD path tables"); assert_eq!(db.get(0xAAAA).as_deref(), Some("lcu/shared")); } @@ -177,7 +181,9 @@ fn open_layered_reports_missing_table_but_stays_usable() { ) .unwrap(); - let (db, errors) = store.open_layered(&[Table::Game, Table::Lcu]); + let (db, errors) = store + .open_layered(&[Table::Game, Table::Lcu]) + .expect("both are WAD path tables"); // The present table still resolves; the missing one is reported, not fatal. assert!(db.contains(0x1111)); @@ -189,6 +195,35 @@ fn open_layered_reports_missing_table_but_stays_usable() { )); } +/// `binentries` and `binfields` are both u32 FNV-1a but mean different things, so +/// layering them would answer a property hash with an object path. Refused up front, +/// in release builds too - nothing is even opened. +#[test] +fn open_layered_refuses_tables_from_different_universes() { + let tmp = tempdir().unwrap(); + let store = HashStore::at(tmp.path()); + + let err = store + .open_layered(&[Table::BinEntries, Table::BinFields]) + .expect_err("different universes"); + + assert_eq!(err.first, Table::BinEntries); + assert_eq!(err.expected, HashUniverse::BinEntry); + assert_eq!(err.table, Table::BinFields); + assert_eq!(err.found, HashUniverse::BinField); + assert_eq!( + Table::BinEntries.key_config(), + Table::BinFields.key_config(), + "refused despite sharing a key config - that is the point" + ); + + // No manifest was ever written here, so a set that did open would have failed + // with TableNotFound instead; the universe check runs before any of that. + assert!(store + .open_layered(&[Table::BinEntries, Table::BinEntries]) + .is_ok()); +} + #[test] fn open_many_pairs_each_table_with_its_result() { let tmp = tempdir().unwrap(); diff --git a/crates/ltk_mimir_cli/src/bundle.rs b/crates/ltk_mimir_cli/src/bundle.rs index 61ff79d..238dd23 100644 --- a/crates/ltk_mimir_cli/src/bundle.rs +++ b/crates/ltk_mimir_cli/src/bundle.rs @@ -13,7 +13,7 @@ use std::io::{BufWriter, Read}; use std::path::{Path, PathBuf}; use anyhow::{bail, Context, Result}; -use ltk_hashdb::{Casing, Compression, HashDbWriter, HashKind, KeyWidth}; +use ltk_hashdb::{Compression, HashDbWriter}; use ltk_mimir_cache::{CommitItem, HashStore, Source, Table}; use sha2::{Digest, Sha256}; @@ -46,25 +46,18 @@ pub struct Options { /// Which CDragon txt file feeds each table. `split` also gathers numbered /// `.` parts - the data repo splits `hashes.game.txt` purely to dodge /// GitHub's file-size limit, so parts and the unsplit file are the same list. +/// +/// Only the input layout lives here; a table's key width, hash algorithm, and +/// casing come from [`Table::key_config`]. struct TableSpec { table: Table, - key_width: KeyWidth, - hash_kind: HashKind, input: &'static str, split: bool, } -const fn spec( - table: Table, - key_width: KeyWidth, - hash_kind: HashKind, - input: &'static str, - split: bool, -) -> TableSpec { +const fn spec(table: Table, input: &'static str, split: bool) -> TableSpec { TableSpec { table, - key_width, - hash_kind, input, split, } @@ -74,62 +67,14 @@ const fn spec( /// not consumed: tables store full-width hashes only; the /// full-width `.xxh64` / `.xxh3` lists cover RST. const SPECS: [TableSpec; 8] = [ - spec( - Table::Game, - KeyWidth::U64, - HashKind::Xxh64, - "hashes.game.txt", - true, - ), - spec( - Table::Lcu, - KeyWidth::U64, - HashKind::Xxh64, - "hashes.lcu.txt", - false, - ), - spec( - Table::BinEntries, - KeyWidth::U32, - HashKind::Fnv1a32, - "hashes.binentries.txt", - false, - ), - spec( - Table::BinTypes, - KeyWidth::U32, - HashKind::Fnv1a32, - "hashes.bintypes.txt", - false, - ), - spec( - Table::BinFields, - KeyWidth::U32, - HashKind::Fnv1a32, - "hashes.binfields.txt", - false, - ), - spec( - Table::BinHashes, - KeyWidth::U32, - HashKind::Fnv1a32, - "hashes.binhashes.txt", - false, - ), - spec( - Table::Rst, - KeyWidth::U64, - HashKind::Xxh64, - "hashes.rst.xxh64.txt", - false, - ), - spec( - Table::RstXxh3, - KeyWidth::U64, - HashKind::Xxh3, - "hashes.rst.xxh3.txt", - false, - ), + spec(Table::Game, "hashes.game.txt", true), + spec(Table::Lcu, "hashes.lcu.txt", false), + spec(Table::BinEntries, "hashes.binentries.txt", false), + spec(Table::BinTypes, "hashes.bintypes.txt", false), + spec(Table::BinFields, "hashes.binfields.txt", false), + spec(Table::BinHashes, "hashes.binhashes.txt", false), + spec(Table::Rst, "hashes.rst.xxh64.txt", false), + spec(Table::RstXxh3, "hashes.rst.xxh3.txt", false), ]; pub fn run(opts: &Options) -> Result<()> { @@ -262,10 +207,7 @@ fn build_table( out: &Path, compression: Compression, ) -> Result { - // Every League table hashes the lowercased path. - let mut writer = HashDbWriter::new(spec.key_width, compression) - .hash_kind(spec.hash_kind) - .casing(Casing::Insensitive); + let mut writer = HashDbWriter::with_key_config(spec.table.key_config(), compression); for file in files { read_hash_lines(file, |hash, _, path| { writer.insert(hash, path); diff --git a/crates/ltk_mimir_cli/src/main.rs b/crates/ltk_mimir_cli/src/main.rs index 44b1c38..d5b541d 100644 --- a/crates/ltk_mimir_cli/src/main.rs +++ b/crates/ltk_mimir_cli/src/main.rs @@ -11,9 +11,10 @@ use std::io::{BufRead, BufReader, BufWriter, Write}; use std::path::{Path, PathBuf}; use anyhow::{bail, Context, Result}; -use clap::{Parser, Subcommand, ValueEnum}; -use ltk_hashdb::{Casing, Compression, HashDb, HashDbWriter, HashKind, KeyWidth}; -use ltk_mimir_cache::{HashStore, Table as CacheTable}; +use clap::builder::TypedValueParser as _; +use clap::{Parser, Subcommand}; +use ltk_hashdb::{Compression, HashDb, HashDbWriter}; +use ltk_mimir_cache::{HashStore, Table}; use ltk_mimir_gen::guessers::{ CharacterSkin, CrossReference, ExtensionSwap, NumericRange, PrefixVariants, RegionLocale, SeedStrings, WordAdd, WordSubstitution, @@ -27,53 +28,14 @@ struct Cli { command: Command, } -/// The logical CDragon tables; picks key width and hash algorithm. -#[derive(Debug, Clone, Copy, ValueEnum)] -enum Table { - Game, - Lcu, - BinEntries, - BinTypes, - BinFields, - BinHashes, - Rst, - RstXxh3, -} - -impl Table { - fn key_width(self) -> KeyWidth { - match self { - Self::Game | Self::Lcu | Self::Rst | Self::RstXxh3 => KeyWidth::U64, - _ => KeyWidth::U32, - } - } - - fn hash_kind(self) -> HashKind { - match self { - Self::Game | Self::Lcu | Self::Rst => HashKind::Xxh64, - Self::RstXxh3 => HashKind::Xxh3, - _ => HashKind::Fnv1a32, - } - } - - /// Every League table hashes the lowercased path. - fn casing(self) -> Casing { - Casing::Insensitive - } - - /// The corresponding shared-cache table. - fn cache(self) -> CacheTable { - match self { - Self::Game => CacheTable::Game, - Self::Lcu => CacheTable::Lcu, - Self::BinEntries => CacheTable::BinEntries, - Self::BinTypes => CacheTable::BinTypes, - Self::BinFields => CacheTable::BinFields, - Self::BinHashes => CacheTable::BinHashes, - Self::Rst => CacheTable::Rst, - Self::RstXxh3 => CacheTable::RstXxh3, - } - } +/// Accept the library table ids (`game`, `binentries`, `rst-xxh3`, ...) as +/// `--table` values, with completions and an error listing them. +/// +/// `Table` is a foreign type, so it cannot derive clap's `ValueEnum`; restricting +/// the raw value to `Table::ALL` and parsing it back is the same thing by hand. +fn table_parser() -> impl clap::builder::TypedValueParser { + clap::builder::PossibleValuesParser::new(Table::ALL.iter().map(|t| t.id())) + .map(|id| Table::from_id(&id).expect("clap accepted only known ids")) } #[derive(Subcommand)] @@ -85,7 +47,7 @@ enum Command { input: PathBuf, /// Which logical table this is (sets key width + hash algorithm). - #[arg(long)] + #[arg(long, value_parser = table_parser())] table: Table, /// Output .hashdb file. @@ -118,7 +80,12 @@ enum Command { /// Resolve from the shared cache's active version of this table instead /// (cache dir: MIMIR_DIR override, else the platform data dir). - #[arg(long, conflicts_with = "file", required_unless_present = "file")] + #[arg( + long, + conflicts_with = "file", + required_unless_present = "file", + value_parser = table_parser() + )] table: Option
, }, @@ -162,7 +129,7 @@ enum Command { wad: Vec, /// Which logical table (sets key width, hash algorithm, guesser preset). - #[arg(long)] + #[arg(long, value_parser = table_parser())] table: Table, /// Extra candidate strings (one per line) checked verbatim, e.g. @@ -347,9 +314,7 @@ fn read_hash_lines(input: &Path, mut on_entry: impl FnMut(u64, &str, &str)) -> R } fn build(input: PathBuf, table: Table, out: PathBuf, compression: Compression) -> Result<()> { - let mut writer = HashDbWriter::new(table.key_width(), compression) - .hash_kind(table.hash_kind()) - .casing(table.casing()); + let mut writer = HashDbWriter::with_key_config(table.key_config(), compression); read_hash_lines(&input, |hash, _, path| { writer.insert(hash, path); })?; @@ -380,7 +345,7 @@ fn gen_hashes( max_skin: u32, out: PathBuf, ) -> Result<()> { - let mut ctx = GuessContext::new(table.hash_kind(), table.casing(), table.key_width()); + let mut ctx = GuessContext::new(table.key_config()); let mut known_hashes = HashSet::new(); for input in &known { let mut paths = Vec::new(); @@ -499,7 +464,7 @@ fn gen_hashes( let mut out_file = BufWriter::new(File::create(&out).with_context(|| format!("creating {}", out.display()))?); - let hex_width = 2 * table.key_width().bytes(); + let hex_width = 2 * table.key_config().key_width().bytes(); // The game-class CDragon lists store paths lowercased (the bin lists keep // original casing); match, so merging finds into a list never produces // case-only duplicates of the same hash. @@ -536,9 +501,9 @@ fn get(hash: &str, file: Option, table: Option
) -> Result<()> { (None, Some(table)) => { let store = HashStore::discover()?; let db = store - .open(table.cache()) - .with_context(|| format!("opening {table:?} from the shared cache"))?; - (db, format!("the shared cache ({table:?})")) + .open(table) + .with_context(|| format!("opening {table} from the shared cache"))?; + (db, format!("the shared cache ({table})")) } (None, None) => unreachable!("clap requires --file or --table"), }; diff --git a/crates/ltk_mimir_gen/src/context.rs b/crates/ltk_mimir_gen/src/context.rs index 074e20c..e2a34b7 100644 --- a/crates/ltk_mimir_gen/src/context.rs +++ b/crates/ltk_mimir_gen/src/context.rs @@ -4,7 +4,7 @@ use std::collections::HashSet; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::{Mutex, OnceLock}; -use ltk_hashdb::{Casing, HashKind, KeyWidth}; +use ltk_hashdb::KeyConfig; use crate::guessers::util::champ_of; use crate::UnknownSet; @@ -45,14 +45,12 @@ const BUILTIN_EXTENSIONS: &[&str] = &[ ]; /// Everything a [`crate::Guesser`] draws candidates from: the known-path corpus, -/// the set of still-unknown hashes, and the table's hash algorithm. +/// the set of still-unknown hashes, and the table's key configuration. /// /// [`crate::Hunt::run`] grows the corpus and shrinks the unknown set as rounds /// resolve hashes. pub struct GuessContext { - hash_kind: HashKind, - casing: Casing, - key_width: KeyWidth, + key_config: KeyConfig, known: Vec>, unknown: UnknownSet, // Derived from `known`; rebuilt lazily after each promotion. @@ -62,11 +60,11 @@ pub struct GuessContext { } impl GuessContext { - pub fn new(hash_kind: HashKind, casing: Casing, key_width: KeyWidth) -> Self { + /// A context for a table hashed under `key_config` - for a League table, + /// `Table::key_config()`. + pub fn new(key_config: KeyConfig) -> Self { Self { - hash_kind, - casing, - key_width, + key_config, known: Vec::new(), unknown: UnknownSet::new(Vec::new()), wordlist: OnceLock::new(), @@ -97,16 +95,9 @@ impl GuessContext { self.unknown = UnknownSet::new(keys); } - pub fn hash_kind(&self) -> HashKind { - self.hash_kind - } - - pub fn casing(&self) -> Casing { - self.casing - } - - pub fn key_width(&self) -> KeyWidth { - self.key_width + /// How this table's keys were produced - what candidates are hashed under. + pub fn key_config(&self) -> KeyConfig { + self.key_config } pub fn known_paths(&self) -> &[Box] { @@ -119,7 +110,7 @@ impl GuessContext { /// Hash a candidate with this table's algorithm and casing rule. pub fn hash_candidate(&self, candidate: &str) -> u64 { - self.hash_kind.hash(candidate, self.casing, self.key_width) + self.key_config.hash(candidate) } /// Vocabulary mined from the known corpus: path segments split on @@ -160,7 +151,7 @@ impl GuessContext { } /// File extensions (no leading dot): everything seen in the known corpus - /// merged with [`BUILTIN_EXTENSIONS`]. Sorted and deduped. + /// merged with `BUILTIN_EXTENSIONS`. Sorted and deduped. pub fn extensions(&self) -> &[Box] { self.extensions.get_or_init(|| { let mut exts: HashSet<&str> = BUILTIN_EXTENSIONS.iter().copied().collect(); @@ -204,9 +195,7 @@ impl GuessContext { /// Where guessers report candidates. Hashes the candidate, tests it against /// the unknown set, and collects hits. Shared across rayon threads. pub struct CandidateSink<'a> { - hash_kind: HashKind, - casing: Casing, - key_width: KeyWidth, + key_config: KeyConfig, unknown: &'a UnknownSet, tried: AtomicU64, found: Mutex>, @@ -215,9 +204,7 @@ pub struct CandidateSink<'a> { impl<'a> CandidateSink<'a> { pub fn new(ctx: &'a GuessContext) -> Self { Self { - hash_kind: ctx.hash_kind, - casing: ctx.casing, - key_width: ctx.key_width, + key_config: ctx.key_config, unknown: &ctx.unknown, tried: AtomicU64::new(0), found: Mutex::new(Vec::new()), @@ -228,7 +215,7 @@ impl<'a> CandidateSink<'a> { pub fn check(&self, candidate: &str) { self.tried.fetch_add(1, Ordering::Relaxed); - let hash = self.hash_kind.hash(candidate, self.casing, self.key_width); + let hash = self.key_config.hash(candidate); if !self.unknown.contains(hash) { return; } diff --git a/crates/ltk_mimir_gen/src/guessers/cross.rs b/crates/ltk_mimir_gen/src/guessers/cross.rs index c6ec4c9..e2fc47e 100644 --- a/crates/ltk_mimir_gen/src/guessers/cross.rs +++ b/crates/ltk_mimir_gen/src/guessers/cross.rs @@ -58,10 +58,14 @@ fn strip_plugin_mount(path: &str) -> Option<&str> { #[cfg(test)] mod tests { use super::*; - use ltk_hashdb::{Casing, HashKind, KeyWidth}; + use ltk_hashdb::{Casing, HashKind, KeyConfig, KeyWidth}; fn hunt(known: &[&str], unknown_paths: &[&str]) -> Vec { - let mut ctx = GuessContext::new(HashKind::Xxh64, Casing::Insensitive, KeyWidth::U64); + let mut ctx = GuessContext::new(KeyConfig::new( + KeyWidth::U64, + HashKind::Xxh64, + Casing::AsciiInsensitive, + )); ctx.add_known(known.iter().map(|s| s.to_string())); let hashes: Vec = unknown_paths .iter() diff --git a/crates/ltk_mimir_gen/src/lib.rs b/crates/ltk_mimir_gen/src/lib.rs index 812aa80..d8cc59a 100644 --- a/crates/ltk_mimir_gen/src/lib.rs +++ b/crates/ltk_mimir_gen/src/lib.rs @@ -10,10 +10,11 @@ //! build on it. //! //! ```no_run -//! use ltk_hashdb::{Casing, HashKind, KeyWidth}; +//! use ltk_hashdb::{Casing, HashKind, KeyConfig, KeyWidth}; //! use ltk_mimir_gen::{GuessContext, Hunt}; //! -//! let mut ctx = GuessContext::new(HashKind::Xxh64, Casing::Insensitive, KeyWidth::U64); +//! let config = KeyConfig::new(KeyWidth::U64, HashKind::Xxh64, Casing::AsciiInsensitive); +//! let mut ctx = GuessContext::new(config); //! ctx.add_known(["assets/characters/ahri/skins/skin01/ahri_tx.dds".to_owned()]); //! ctx.add_unknown([0x123456789abcdef0]); // e.g. mined from a WAD's chunk table //! let report = Hunt::default_game().run(&mut ctx); diff --git a/crates/ltk_mimir_gen/tests/hunt.rs b/crates/ltk_mimir_gen/tests/hunt.rs index c47ce99..65d23d0 100644 --- a/crates/ltk_mimir_gen/tests/hunt.rs +++ b/crates/ltk_mimir_gen/tests/hunt.rs @@ -1,7 +1,7 @@ //! End-to-end hunts over synthetic corpora: seed a few known paths, hash the //! target paths into the unknown set, and assert the guessers rediscover them. -use ltk_hashdb::{Casing, HashKind, KeyWidth}; +use ltk_hashdb::{Casing, HashKind, KeyConfig, KeyWidth}; use ltk_mimir_gen::guessers::{ CharacterSkin, ExtensionSwap, NumericRange, PrefixVariants, RegionLocale, SeedStrings, WordAdd, WordSubstitution, @@ -9,11 +9,15 @@ use ltk_mimir_gen::guessers::{ use ltk_mimir_gen::{GuessContext, Hunt}; fn xxh64(s: &str) -> u64 { - HashKind::Xxh64.hash(s, Casing::Insensitive, KeyWidth::U64) + HashKind::Xxh64.hash(s, Casing::AsciiInsensitive, KeyWidth::U64) } fn ctx_with(known: &[&str], targets: &[&str]) -> GuessContext { - let mut ctx = GuessContext::new(HashKind::Xxh64, Casing::Insensitive, KeyWidth::U64); + let mut ctx = GuessContext::new(KeyConfig::new( + KeyWidth::U64, + HashKind::Xxh64, + Casing::AsciiInsensitive, + )); ctx.add_known(known.iter().copied().map(Box::from)); ctx.add_unknown(targets.iter().map(|t| xxh64(t))); ctx @@ -185,7 +189,11 @@ fn dry_round_terminates() { #[test] fn empty_inputs_are_safe() { - let mut ctx = GuessContext::new(HashKind::Xxh64, Casing::Insensitive, KeyWidth::U64); + let mut ctx = GuessContext::new(KeyConfig::new( + KeyWidth::U64, + HashKind::Xxh64, + Casing::AsciiInsensitive, + )); let report = Hunt::default_game().run(&mut ctx); assert!(report.resolved.is_empty()); @@ -199,9 +207,13 @@ fn empty_inputs_are_safe() { fn fnv1a32_tables_hash_case_insensitively() { // Bin tables keep original-case strings but hash lowercased. let target = "Data/Spells/AhriOrbMissile.lua"; - let mut ctx = GuessContext::new(HashKind::Fnv1a32, Casing::Insensitive, KeyWidth::U32); + let mut ctx = GuessContext::new(KeyConfig::new( + KeyWidth::U32, + HashKind::Fnv1a32, + Casing::AsciiInsensitive, + )); ctx.add_known(["Data/Spells/AhriOrbMissile.luabin".to_owned()]); - ctx.add_unknown([HashKind::Fnv1a32.hash(target, Casing::Insensitive, KeyWidth::U32)]); + ctx.add_unknown([HashKind::Fnv1a32.hash(target, Casing::AsciiInsensitive, KeyWidth::U32)]); let report = Hunt::new().with(ExtensionSwap).run(&mut ctx); assert_eq!(report.resolved.len(), 1); diff --git a/docs/CONSUMERS.md b/docs/CONSUMERS.md index 82035f3..2c3630f 100644 --- a/docs/CONSUMERS.md +++ b/docs/CONSUMERS.md @@ -147,7 +147,7 @@ tables, so you don't hand-roll a second map plus fallback: ```rust use ltk_hashdb::LayeredHashDb; -let mut db = LayeredHashDb::from_bases(vec![store.open(Table::Game)?]); +let mut db = LayeredHashDb::from_bases(vec![store.open(Table::Game)?])?; // Hashes with the first base's algorithm and returns the hash: let h = db.insert_path("assets/mymod/custom.dds").expect("has a base"); @@ -176,9 +176,9 @@ let store = HashStore::discover()?; // Open the WAD path tables into one layered reader; missing tables are reported, // not fatal - the tool stays usable and their hashes just miss. -let (mut db, errors) = store.open_layered(&[Table::Game, Table::Lcu]); +let (mut db, errors) = store.open_layered(&[Table::Game, Table::Lcu])?; for (table, e) in &errors { - eprintln!("skipping {table:?}: {e}"); + eprintln!("skipping {table}: {e}"); } db.insert(precomputed_hash, "assets/mymod/custom.bin"); // overlay writes as before @@ -193,6 +193,13 @@ for (hash, path) in db.get_batch(&chunk_hashes) { } ``` +`open_layered` only fails outright on a set spanning more than one hash universe - +`&[Table::BinEntries, Table::BinFields]`, say, where one table would answer the other's +hashes with an unrelated path. That is checked before anything is opened. Everything +else (a missing table, an unreadable file, a file that isn't the table it's filed under) +comes back in the per-table error list, so a partial cache still gives you a usable +reader. + `open_layered` is the convenience most WAD consumers want; `open_many` is the lower-level primitive it's built from - it pairs each requested table with its `Result` so you can warn-and-skip instead of aborting on the first @@ -324,7 +331,7 @@ use ltk_hashdb::{Casing, Compression, HashDbWriter, HashKind, KeyWidth}; let mut w = HashDbWriter::new(KeyWidth::U64, Compression::default()) // 16 KiB frames, level 19 .hash_kind(HashKind::Xxh64) // recorded so readers can `hash_path` - .casing(Casing::Insensitive); // keys hash the lowercased path (League rule); + .casing(Casing::AsciiInsensitive); // keys hash the ASCII-lowercased path (League rule); // defaults to Sensitive (hash bytes as given) w.insert(hash, "assets/characters/aatrox/aatrox.bin"); @@ -353,11 +360,11 @@ for path-shaped tokens), and the chunk table *is* the unknown set: ```rust use ltk_mimir_gen::guessers::SeedStrings; use ltk_mimir_gen::{mine_wad, GuessContext, Hunt}; -use ltk_hashdb::{Casing, HashKind, KeyWidth}; +use ltk_mimir_cache::Table; let mined = mine_wad("Ahri.wad.client".as_ref())?; // seed strings + chunk hashes -let mut ctx = GuessContext::new(HashKind::Xxh64, Casing::Insensitive, KeyWidth::U64); +let mut ctx = GuessContext::new(Table::Game.key_config()); ctx.add_known(db.iter().map(|(_, p)| p.into_owned())); // corpus to mutate from ctx.add_unknown(mined.chunk_hashes.into_iter().filter(|&h| !db.contains(h))); diff --git a/docs/FORMAT.md b/docs/FORMAT.md index d5526d2..64e81cb 100644 --- a/docs/FORMAT.md +++ b/docs/FORMAT.md @@ -75,24 +75,25 @@ RST hashes are stored **full-width** (no truncation/masking). ### `case_insensitive` (flags bit1) -Whether the keys hash the **lowercased** path (bit set) or the path bytes as -given (bit clear). All League tables set it - the game hashes lowercased +Whether the keys hash the **ASCII-lowercased** path (bit set) or the path bytes +as given (bit clear). All League tables set it - the game hashes lowercased paths - so a consumer hashing a new path must lowercase it first to match. -Paths are UTF-8 and the lowercasing is Unicode-aware, so non-League tables get -full UTF-8 case-insensitivity: the path is mapped through the Unicode **full** -lowercase mapping - Rust's `str::to_lowercase`, which is per-character *plus* -the context-sensitive final-sigma rule (`Σ` → `ς` at the end of a word, `σ` -elsewhere) - then UTF-8 encoded and hashed. Implementations in other languages -must match that mapping exactly, not a per-character-only one. League paths are -ASCII, where this reduces to plain `A-Z` → `a-z` and coincides with `ltk_hash`'s -`WadHash`/`BinHash` on all real data. - -Stability note: Unicode case mappings can gain entries in new Unicode versions, -so only the ASCII part of the mapping (`A-Z` → `a-z`) is guaranteed bit-stable -across toolchains. Hashes of non-ASCII paths could in principle drift when the -producer's Unicode tables update; publishers of non-ASCII tables who need -long-term stability should pre-lowercase their paths and hash case-sensitively. +The mapping is exactly `A-Z` → `a-z`, byte by byte. Every other byte is hashed +untouched, including every byte of a multi-byte UTF-8 sequence, so `É` and `é` +are different keys. This is a deliberate narrowing: a byte substitution has no +locale, no Unicode tables and no toolchain drift behind it, so a key computed +from a given path is the same key forever, and an implementation in any language +is a three-line loop. On League data - all ASCII - it coincides with `ltk_hash`'s +`WadHash`/`BinHash`. + +A publisher whose paths are not ASCII should case-fold them however its domain +requires and then hash **case-sensitively**, which keeps the folding rule (and +its version) on the publisher's side of the file rather than the reader's. Should +a Unicode-aware rule ever be specified, it takes a value in the reserved byte at +offset 14 (0 = defer to bit1) rather than a second flag bit: unknown flag bits +are rejected on open, unknown reserved bytes are ignored, so the addition would +be readable by builds that predate it. ## Sections