Bench harness + obs_mobility unit fix by jspaezp · Pull Request #65 · TalusBio/timsbuktoolkit

jspaezp · 2026-05-18T20:10:17Z

Summary

Fixture-driven bench harness under bench/: stage_fixture, push_fixture, wandb_bench, entrapment analyser (FDRBench Method 1/1B), peptide-list pipeline, plus tests.
Fix obs_mobility unit bug in timsseek scoring: percent error was being added to absolute 1/k0, biasing observed mobility ~50% high. Now converts pct -> abs via ref_mobility/100 before combining. Also splits NaN gating in weighted_ms1, names the magic .take(3) constant, renames a confusing variable shadow.
Refit supersimpleprediction on hela_iccoff_gt20peps after the unit fix: holdout MAPE 1.36% (was 1.83% in the prior docstring). New tool at scripts/refit_mobility.py to redo this on any results.parquet.
Bench staging path bugs: stage_fixture emitted relative paths into the staged TOML which then failed schema validation; target_peptides was unconditionally staged though the schema made it optional. wandb_bench s3_download_file calls now go through a _materialize_uri helper so absolute local paths pass through.

Test plan

cargo check --workspace clean
cargo test -p timsseek --doc supersimpleprediction passes (new expected value 1.144405)
bench stage hela_iccoff_gt20peps -> wandb_bench --fixtures-dir bench_out/staged runs end to end
bench stage hela_smoke_shuffled (entrapment fixture, shuffled mode w/ pairing) -> wandb_bench runs end to end and logs entrap/* scalars
obs_mobility distribution post-fix: mean ~0.92, range [0.71, 1.24], residual obs-ref bias dropped from ~+0.5 to ~+0.03 (real instrument offset, not a unit bug)
(followup, separate) chase the residual +0.03 obs-ref bias - likely asymmetric ridge picking or MS1<->MS2 mobility offset
(followup, separate) q05 entrapment combined FDR 0.109 vs nominal 0.05 on smoke fixture - investigate

…ypes, test tightening)

…x tuple

…unt)

Adds resolve_dbs() that turns a list of --db spec strings into one concatenated FASTA on disk (local, S3, UniProt proteome/accession). Individual accession specs are coalesced into a single batched HTTP call. Adds bench/_s3.py wrapping aws s3 cp for download/upload/upload-dir.

… layout)

ruff reformatted 6 bench/ files (magic trailing comma, line-length normalisation).

…ixtures)

…s, --force to override)

…_db set

…ent metadata Replace fasta/entrapment_fasta with target_peptides/entrapment_peptides plus entrapment_ratio, entrapment_mode (foreign|shuffled), and pairing fields. Adds cross-field consistency validation and has_pairing() helper on Fixture.

…ED entrapment

… histogram

…fixture

… update gt20peps fixture

obs_mobility was computed by adding the percent-of-ref mobility error back to ref_mobility, giving values biased ~50% high. avg_delta_mobs now converts pct -> absolute via ref_mobility/100 before feeding the collector, so obs_mob = ref + abs_delta is dimensionally correct. delta_ms1_ms2_mobility likewise switched to absolute 1/k0 units. Side cleanups in offsets.rs: - weighted_ms1 NaN gate was checking mz_error_ppm only, poisoning the mobility accumulator (or dropping valid mobility) on mismatched NaNs. Now gates each dimension independently. - magic .take(3) on FRAGMENT_TOP_N=7 named as FRAGMENT_OBS_MOB_TOP_N. - renamed shadow mz/mob in with_sorted_offsets to ms1_mob/ms2_mob. supersimpleprediction refit on hela_iccoff_gt20peps (34k IDs, holdout MAPE 1.36% vs prior 1.83% claim) using scripts/refit_mobility.py.

stage_fixture emitted relative paths (bench_out/cache/...) into the staged TOML, which then failed the schema validator that requires s3:// or absolute local paths. Now resolves to absolute via dst.resolve() on both _stage_one_file and _stage_one_dir. Also guards target_peptides for the gt20peps fixture (target_peptides is optional since e6339f4 but stage still unconditionally tried to download it). wandb_bench grew a _materialize_uri helper used in the entrapment block so target_peptides / entrapment_peptides / pairing pass through when already local instead of erroring out of s3_download_file. scripts/stage_gt20.sh: one-shot wrapper for the gt20peps fixture.

jspaezp added 30 commits May 8, 2026 20:14

chore(bench): add bench dependency group and gitignore bench_out/

d6d020f

feat(bench): pydantic schema and loader for fixture TOMLs

4c93b7f

fix(bench): clean up fixture schema (dead tomli fallback, validator t…

090bd0d

…ypes, test tightening)

feat(bench): polymorphic --db spec classifier

5c52907

fix(bench): drop unused fixture, add wrong-extension test, sort suffi…

d046d4c

…x tuple

feat(bench): uniprot REST fetch helpers (proteome + accession batch)

22902fe

chore: configure ty to use project .venv

bb6634a

fix(bench): tighten uniprot test (ty-clean url access, exact chunk co…

ca77ef4

…unt)

feat(bench): aho-corasick peptide classification for entrapment

399b4bc

feat(bench): empirical FDR curve and PNG plot for entrapment

ccce272

feat(bench): analyse() entry point returning wandb-ready scalars

ce7517f

feat(bench): push_fixture.py CLI scaffolding + TOML emitter

cc3a063

feat(bench): push_fixture.py upload + speclib build pipeline

5118703

feat(bench): rewrite bench runner skeleton with fixture selection

108e720

feat(bench): timsseek invocation + wandb logging + entrapment hook

b853108

fix(bench): pass res_dir as timsseek --output-dir (matches its native…

1451eeb

… layout)

docs(bench): add default config and README for fixture harness

89a3c0e

style(bench): apply task fmt

602860e

ruff reformatted 6 bench/ files (magic trailing comma, line-length normalisation).

fix(bench): ty-clean field_name access; use real S3 prefix in README

7ba9e25

feat(bench): fixture inputs accept absolute local paths (for staged f…

6cf5438

…ixtures)

feat(bench): --fixtures-dir flag on wandb_bench (for staged fixtures)

d3b4539

chore: disable patching

85cf7de

feat(bench): idempotent push_fixture uploads (skip existing s3 object…

3eac4ff

…s, --force to override)

feat(bench): stage_fixture CLI for local cache + offline reruns

af12aad

docs(bench): add staging + idempotent uploads sections

3f9c154

feat(bench): k-mer-based search-space normalization for entrapment FDR

5070f11

feat(bench): --request-delay-ms flag for speclib_build_cli throttle

c82d3e3

fix(bench): build speclib from merged target+entrap fasta when entrap…

a0c0392

…_db set

feat(bench): default fetch_proteome to reviewed-only (Swiss-Prot)

bc12861

jspaezp added 11 commits May 9, 2026 10:01

feat(bench): trypsin digest + algorithm-1 shuffled-entrapment helpers

6589605

feat(bench)!: rewrite push_fixture for peptide-list pipeline + SHUFFL…

84e4b7e

…ED entrapment

feat(bench)!: rewrite entrapment.py with FDRBench Method 1/1B + score…

41a01d3

… histogram

feat(bench)!: thread peptide-list schema through wandb_bench + stage_…

e667afd

…fixture

docs(bench): peptide-list pipeline + Algorithm 1/2 + FDRBench estimators

30bb2cb

feat(bench): target_peptides optional (only required for entrapment);…

e6339f4

… update gt20peps fixture

fix(bench): speclib_build_cli flag is --peptide-list, not --peptides

a4c5ef6

ci: stuff

9564f56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bench harness + obs_mobility unit fix#65

Bench harness + obs_mobility unit fix#65
jspaezp wants to merge 41 commits into
mainfrom
infra/benchmark_fixture

jspaezp commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jspaezp commented May 18, 2026

Summary

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant