feat(diskann): optional lattice-embed backend for f32 distance kernels, plus a native cross-dimension parity test

[ohdearquant]

What this is, stated plainly

A fourth backend for ruvector-diskann's two f32 distance kernels, behind an off-by-default
lattice-simd feature.

This one is a substitution, not a gap-fill, and the difference is worth being clear about.
The wasm argument that motivates the same feature in ruvector-core does not transfer here:
ruvector-diskann already has hand-written simd128 kernels for both metrics, so wasm is covered
and lattice is not filling a hole. On native, SimSIMD already vectorizes. This is one SIMD
implementation swapped for another, and it should be judged on what the swap buys rather than on a
speed claim.

What the swap buys

A pure-Rust dependency in place of a C one. simsimd builds its kernels through build.rs ->
cc::Build -> c/lib.c, so enabling simd here needs a working C toolchain at build time.
lattice-embed's kernels are Rust std::arch intrinsics with runtime dispatch and no build
script of their own. One honesty note on the toolchain claim: this does not make the enabled
build C-free, because lattice-embed's mandatory blake3 dependency compiles a C NEON source
through its build script on aarch64 targets, so cross-compiling with lattice-simd enabled
still needs a target C compiler there. The swap removes the C dependency from the distance
kernels themselves, not from the feature's whole dependency tree.

One dependency entry rather than a target-gated one. simsimd is declared under
[target.'cfg(not(target_arch = "wasm32"))'.dependencies] because it does not build for wasm32.
lattice-embed does, so it is a plain dependency.

One kernel set across the workspace. If ruvector-core routes through lattice, leaving
diskann on a different SIMD library means two implementations of the same arithmetic, with
independent rounding behaviour, inside one search path.

On performance claims. When this was first opened, no measurement was attached: at the time
the host could not certify A/B deltas in the plausible range (an A/A control on byte-identical
source produced differences up to 9.97%). The measurement section added later, below, supersedes
this position and states its own conditions and limits.

The change

l2_squared and inner_product each gain a lattice-simd arm ahead of the existing three. The
arms stay mutually exclusive and exhaustive, so exactly one route is selected for any
feature/target combination (the other backends' helper functions still compile; only the
dispatch arm actually called changes):

condition	backend
lattice-simd	lattice_embed::simd
else, wasm32 + simd128	existing wasm_simd128_*
else, non-wasm32 + simd	existing SimSIMD
else	existing scalar

Default builds (default = []) are untouched and use the scalar path; simd (SimSIMD)
is opt-in.

SpatialSimilarity::inner is a plain alias for dot — it forwards directly, with no 1 - x
transform — so the negation both paths apply is over the same raw dot product. I checked that at
the SimSIMD source rather than inferring it from the method name, because had the two differed by
a constant, the scalar fallback sitting beside it would already disagree with the SimSIMD path
today.

scalar_inner_product becomes unreachable in a lattice-simd build. It is annotated
#[cfg_attr(feature = "lattice-simd", allow(dead_code))] rather than deleted or made pub: it is
still the fallback the simd backend calls when SimSIMD returns None. (The parity test's
oracle is a separate local naive_inner_product, not this function.)

A test gap this turned up, independent of lattice

The cross-dimension parity tests in this file are gated on
#[cfg(all(test, target_arch = "wasm32", target_feature = "simd128"))]. They are good tests: a
14-dimension grid straddling lane widths and remainders, with a documented tolerance. They only
ever run for wasm32.

The native backends' fixed-value checks all sat at dim 3 — below every lane width in play, so
they exercise a remainder loop and nothing else. (An identical-vectors check did run at dim 128,
but identical inputs cannot catch a wrong-value bug; no prior non-identical, reference-checked
native test exercised dimensions above 3.)

I did not want to assert the consequence of that without measuring it, so I injected a
dropped-remainder bug — truncating the distance computation to the first 3 lanes, which is exactly
correct at dim <= 3 and wrong above it — and ran the pre-existing suite against it:

arm	pre-existing tests (new one excluded)	wall time
unmutated	34 passed, 0 failed	253.07s
dropped-remainder bug injected	34 passed, 0 failed	2.66s

The load-bearing result there is the pass/fail column, not the clock: the suite does not catch the
bug. On the wall times, one scope note — those two runs were sequential on a machine I did not
hold exclusively, so treat them as an order-of-magnitude observation rather than a controlled
timing measurement. The ratio is far too large to be contention, but I would rather bound the
claim than have it read as a benchmark.

That collapse is still the interesting part: those tests build indexes and drive the
wide-dimension distance path constantly, they simply never compare a value against a reference,
so an incorrect result above dim 3 has nothing asserting against it.

backend_matches_scalar_reference closes that, and fails on that same injected bug. It is
backend-independent by construction, so it covers the SimSIMD path shipping today, the scalar
path, and the lattice path alike, and it keeps its value in this file whether or not you take the
rest of this PR. Its references are naive single-pass loops rather than the crate's own scalar_*
functions, which are themselves 4-accumulator implementations — using those as the reference would
let a shared reduction-order bug pass.

Verification

arm	result
test --features lattice-simd --lib	36 passed, 0 failed, 1 ignored
test --features simd --lib (SimSIMD, opt-in)	35 passed, 0 failed, 1 ignored
test --lib (scalar, the default)	35 passed, 0 failed, 1 ignored
clippy --features lattice-simd --all-targets -- -D warnings	clean
clippy --all-targets -- -D warnings	clean

The scalar-oracle parity test runs in every configuration (named in each arm's output), which is
what shows it is backend-independent rather than compiled into one configuration; the
lattice-simd arm's one extra test is the cfg-gated routing witness.

Mutation-checked per adapter, not per patch, since mutating a patch as a unit only certifies
its best line:

mutation	lattice-simd arm	default arm
l2 adapter: squared_euclidean_distance -> euclidean_distance	fails	stays green
inner adapter: drop the negation	fails	stays green
dropped-remainder bug (above)	fails	n/a, lattice arm only

Each mutation leaving the default arm green is what proves it stayed inside its own cfg block
instead of breaking the crate outright.

Two consequences worth stating

This resolves a second copy of lattice-embed into the workspace. ruvector-core pins
0.6.1; this pins 0.7.1, which is the version the other in-flight lattice PRs move to. ^0.6
and ^0.7 cannot unify, so until those land, a build enabling both ruvector-core/lattice-embeddings
and ruvector-diskann/lattice-simd compiles lattice-embed twice. Verified with cargo tree
rather than read off the lockfile. Whichever of these PRs lands second will want a lockfile
rebase.

Debug-profile test time differed noticeably between arms (the lattice arm ran longer than the
SimSIMD arm). I am deliberately not putting numbers on that: the arms ran sequentially on a shared
machine under varying load, it is a debug profile, and I have not controlled it. Flagging it only
so it is not a surprise, and it is worth a release-profile check before adoption.

Cost

lattice-embed requires Rust >= 1.93. Enabling this feature raises the effective MSRV for whoever
turns it on. The default build is unaffected and stays on the workspace MSRV. That is the real
price and it is why the feature is opt-in.

Measurement

When this section was first written the crate had no benchmark to run, and the
position taken was that the harness should be its own reviewable change. That
harness is now #783; the numbers below come from cherry-picking #783's commit
onto this PR's head locally (clean pick). If #783 lands first, this branch
measures as-is.

Apple silicon Mac mini, macOS, aarch64, rustc 1.93.0, cargo bench -p ruvector-diskann --bench distance, Criterion --measurement-time 10. Same
commit both phases, only the feature flag differs. This crate's default is
empty, so off is the scalar path. CPU idle sampled every 20s inside each
measured phase: off minimum 85%, on minimum 78% — both phases quiet.

benchmark	off	on (lattice-embed)	change
l2_squared/384	115.47 ns	18.85 ns	-83.6%
l2_squared/1536	459.15 ns	79.57 ns	-82.7%
inner_product/384	67.62 ns	19.89 ns	-70.6%
inner_product/1536	268.71 ns	73.83 ns	-72.5%
scalar_l2_squared/1536	460.23 ns	446.40 ns	-3.1%
pq_asymmetric_distance/1536	822.77 ns	829.73 ns	+0.7%

(128/768 dims behave the same as their neighbours; all p = 0.00; seeded-random
inputs.)

The last two rows are the table's own controls, and they are why the first
four are believable. scalar_l2_squared is deliberately not routed by this
feature and moves only ~3%; pq_asymmetric_distance is untouched and moves
under 1%. If the harness or the machine had shifted between phases, those two
rows are where it would show.

Kernel-level numbers only: nothing here measures DiskANN search end to end,
and the sibling PRs show the kernel-to-end-to-end ratio varies widely by
crate. The feature stays off by default.

---

Note on this PR's CI. Tests (core-and-rest) is red on every branch in this repository,
including main: the job is cancelled at its 240-minute cap while still compiling and never
reaches the test phase. #786 restores the exclusion list that the shard's packages: value loses
to a shell comment, #784 unblocks the ruvector-filter test target that the compiler cannot
finish, and #787 fixes a deadlock waiting behind both. That failure is not caused by this branch.