feat(tiny-dancer-core): make the cosine backend selectable, with a lattice-simd option

[ohdearquant]

What this is

ruvector-tiny-dancer-core scores candidates through one cosine kernel, wired directly to
SimSIMD through a hard dependency. The crate has no [features] section at all, so there is no
way to build it against a different kernel, and no way to build it without a SIMD library. This
adds a feature axis with three mutually exclusive arms and leaves the default on what ships
today.

features	backend
default (simd-simsimd)	SimSIMD, unchanged
lattice-simd	lattice-embed kernels
--no-default-features	scalar

lattice-simd wins if both are enabled, so the arms stay mutually exclusive and exhaustive.

Let me be straight about what this does not buy you

I went into this expecting the wasm argument that motivates the same feature in ruvector-core
to apply here, and it does not. I measured before writing the PR rather than after, so this
section is a correction to my own premise rather than a caveat bolted onto a claim.

The pure-Rust-dependency argument does not transfer to this crate. It is real for
ruvector-diskann, where SimSIMD is the reason a C toolchain is needed. Here
ruvector-tiny-dancer-core also hard-depends on rusqlite with bundled, which compiles
SQLite from C. Removing SimSIMD's C-ness does not make this crate buildable without a C
compiler, so I am not going to pretend it does.

This does not fix the wasm build, for reasons in the next section.

What is left is smaller and still worth having: a backend that can be selected at all, a scalar
path where there was none, and — for callers that opt in — the option of converging on one
kernel set instead of two implementations of the same arithmetic with independent rounding
behaviour inside one search path. To be precise about state versus option: the default build
still selects SimSIMD and the lockfile currently carries two lattice-embed versions, so this
PR offers the convergence, it does not put the workspace in that state.

Performance. An earlier revision of this description made no performance claim because the
local host could not certify A/B deltas in the plausible range (an A/A control on byte-identical
source produced differences up to 9.97%). Numbers were subsequently taken on a host that can
support the claim; see the Measurement section below.

Something I found while scoping this, unrelated to the change

ruvector-tiny-dancer-wasm does not build for wasm32-unknown-unknown. It is
crate-type = ["cdylib", "rlib"] with wasm-bindgen, and on unmodified main:

cargo check -p ruvector-tiny-dancer-wasm --target wasm32-unknown-unknown
error: the wasm*-unknown-unknown targets are not supported by default,
       you may need to enable the "js" feature
   --> getrandom-0.2.17/src/lib.rs:346:9

That is the first blocker, not the only one. Shimming it locally the way
ruvector-diskann/Cargo.toml already does for the same crate:

[target.'cfg(target_arch = "wasm32")'.dependencies]
getrandom02 = { package = "getrandom", version = "0.2", features = ["js"] }

moves the failure to the next one:

cargo:warning=sqlite3/sqlite3.c:14678:10: fatal error: 'stdio.h' file not found
error occurred in cc-rs: ... --target=wasm32-unknown-unknown ... -c sqlite3/sqlite3.c

libsqlite3-sys building bundled SQLite for wasm32, reached through the unconditional
rusqlite dependency in ruvector-tiny-dancer-core. I stopped there and reverted the probe, so
I do not know whether SimSIMD is a third blocker behind it — the build never gets that far.

Why CI does not catch it: ruvector-tiny-dancer-wasm is only ever compiled for the native
host. It appears in the nextest package list in ci.yml and in an --exclude list in another
native job, and none of the PR-triggered CI workflows build it for wasm32-unknown-unknown
(the release workflow does run wasm-pack build for this crate, so the categorical statement
stops at PR CI). The wasm32 target jobs that
do exist (build-attention.yml, build-graph-transformer.yml, metabiohacker-ci.yml) cover
other crates. A gate whose trigger set does not intersect what its instrument actually reaches
produces greens that read as evidence.

I am reporting this rather than fixing it because the fix is a scoping decision about your
storage layer, not a drive-by. If you do want the wasm path, the order looks like: gate
rusqlite behind a feature or a target, shim getrandom, and then the SIMD layer is the next
thing in the way — which this PR makes removable rather than removes: lattice-embed compiles
for wasm32 and SimSIMD does not, but ruvector-tiny-dancer-wasm still consumes this crate's
default features, so its manifest would also need default-features = false plus lattice-simd
for the swap to reach it. That is an ordering observation, not a claim that this
PR unlocks anything on its own.

Conventions, which are the interesting part

The two libraries disagree on what "cosine" returns. SpatialSimilarity::cosine returns a
distance, which is why the existing code reads 1.0 - f32::cosine(a, b).
lattice_embed::simd::cosine_similarity returns a similarity directly, so its adapter has no
1 - x. The arms are not interchangeable text and a copy-paste between them would silently
invert the score.

They also disagree on zero norms. lattice-embed returns 0.0 for a zero norm, which collides
with the genuine "orthogonal" answer. The behaviour this path has always had is preserved on
every arm:

• two all-zero vectors score 1.0
• a zero vector against a non-zero one scores 0.0
• a length mismatch stays an error

I did not assume those values, I measured them: backend_boundary_conventions_agree asserts them
exactly and passes on the unmodified default arm, which is what establishes them as the
incumbent behaviour rather than my guess at it. Restoring the all-zero case costs a cold-path
check that only runs when the kernel already returned 0.0.

For the record, 1.0 for two all-zero vectors is arguably the wrong answer, since cosine is
undefined there. I have deliberately not changed it. A backend swap that also changes semantics
is two changes wearing one hat, and which value that should be is your call, not mine.

Verification

arm	result
test --lib (default, SimSIMD)	33 passed, 0 failed
test --features lattice-simd --lib	33 passed, 0 failed
test --no-default-features --lib (scalar)	33 passed, 0 failed
test --no-default-features --features lattice-simd --lib	33 passed, 0 failed
clippy --all-targets -- -D warnings (default)	clean
clippy --features lattice-simd --all-targets -- -D warnings	clean
clippy --no-default-features --all-targets -- -D warnings	clean
clippy --no-default-features --features lattice-simd --all-targets -- -D warnings	clean

Same test count in every arm, and both new tests are named in each arm's output, which is what
shows they are backend-independent rather than compiled into one configuration.

Mutation-checked per adapter, not per patch, since mutating a patch as a unit only certifies
its best line. Each mutation must fail its own arm and leave the default arm green — the second
half is what proves it stayed inside its own cfg block instead of breaking the crate outright.

mutation	own arm	default arm
lattice adapter: cosine_similarity(a, b) -> (a, a)	fails (2 of 5)	green
lattice: drop the all-zero -> 1.0 restoration	fails (1 of 5)	green
scalar: drop the sqrt on both norms	fails (1 of 5)	green
scalar: all-zero returns 0.0 instead of 1.0	fails (1 of 5)	green

The probe echoes each mutation's diff before running, so a regex that silently failed to apply
cannot be scored as a passing arm, and it reports an instrument failure rather than a result if
no test suite actually ran.

Two consequences worth stating

This resolves a second copy of lattice-embed into the workspace. ruvector-core pins
0.6.1; this pins 0.7.1, which is the version the other in-flight lattice PRs move to. ^0.6
and ^0.7 cannot unify, so until those land, a build enabling both
ruvector-core/lattice-embeddings and ruvector-tiny-dancer-core/lattice-simd compiles
lattice-embed twice. Both versions are visible in Cargo.lock. Whichever of these PRs lands
second will want a lockfile rebase.

The lockfile diff is lattice-embed only. The resolver also wanted to move tempfile's
getrandom edge from 0.3.4 to 0.4.3; that is unrelated to this change and not required by
it, so it is reverted here and cargo check -p ruvector-tiny-dancer-core --features lattice-simd --locked resolves cleanly without it.

Cost

lattice-embed requires Rust >= 1.93. Enabling this feature raises the effective MSRV for
whoever turns it on. The default build is unaffected and stays on the workspace MSRV. That is the
real price and it is why the feature is opt-in and off by default.

Measurement

Apple silicon Mac mini, macOS, aarch64, rustc 1.93.0, the crate's own
feature_engineering bench target, Criterion --measurement-time 10. Both
phases are the same commit and differ only by the feature flag.

The baseline here is not scalar. This crate's default is ["simd-simsimd"],
so the off side is SimSIMD and this table is a backend-to-backend comparison,
which is the harder of the two comparisons a backend PR can be asked to win.

benchmark	off (SimSIMD)	on (lattice-embed)	change (95% CI)	p
cosine_similarity_384d	107.71 ns	73.377 ns	-34.52% .. -32.04%	0.00
feature_weighting (3 cases)	53.26-53.98 ns	47.76-48.28 ns	-11.75% .. -7.15%	0.00

CPU idle was sampled every 20s inside each measured phase, three samples each.
Off phase minimum 76%, on phase minimum 68%. The on phase dipped slightly below
the bar I hold myself to, and the direction of that dip runs against the on side
rather than for it.

One number here is worth flagging as an open question rather than a result. In
ruvector-core, the same SimSIMD-to-lattice swap on 384-dimension cosine
measures roughly -52%; here the same swap on a 384-dimension cosine measures
-33%, and the SimSIMD side itself costs 107.7 ns here against roughly 57 ns
there for what should be the same kernel and the same dimension. The difference
is wrapper overhead in this crate's cosine path, not the kernel, and it is
untouched by this PR. It is the reason the end-to-end feature_weighting numbers
land at -8 to -10% rather than tracking the kernel figure: the kernel is not the
whole cost on this path.

---

Note on this PR's CI. Tests (core-and-rest) is red on every branch in this repository,
including main: the job is cancelled at its 240-minute cap while still compiling and never
reaches the test phase. #786 restores the exclusion list that the shard's packages: value loses
to a shell comment, #784 unblocks the ruvector-filter test target that the compiler cannot
finish, and #787 fixes a deadlock waiting behind both. That failure is not caused by this branch.