Reproducibility Review (main)

[Robert-Roth]

Repository: Arcadia-Science/FPguide
Branch: main
Commit: 29935e8cb7
Reviewed: 2026-09-03 21:27 UTC

Reproducibility review — Arcadia-Science/FPguide

Summary

This repository is unusually well documented for reproducibility: every folder carries its own README with ordered commands, the root README maps each entry point onto a four-tier "what can you actually regenerate" table, the conda environment is fully pinned with lock files plus OS/driver notes, and the tracked artifacts were deliberately chosen so the cheap entry points (msa_conservation/, experiment/, Section 1 of GFP_DMS/visualization.ipynb) run from a bare clone. Someone could clone this today and, after one ~325 MB release download, re-run the GFP_DMS analysis and the EGFP campaign analysis as documented. The remaining friction is at the heavy end: the in-silico-test/ deployed checkpoints are not published and their refit is not deterministic, and the committed design CSVs make the documented campaign "run" commands no-ops on a fresh clone.

Critical issues

None found.

Warnings

• The in-silico-test/ deployed checkpoints are not published, and refitting them is not bit-reproducible. design_common.SURR_CKPT (trained_models/surrogate_final/cnn-max-d1_trainval.pt, design_common.py:147) and ORAC_CKPT (trained_models/oracle_sweep/cnn-max-d1_s0.pt, line 148) fall under the repo-wide *.pt ignore, as does trained_models/surrogate_cv3.csv, from which train_final_surrogate.py:40 derives its epoch budget. The rebuild path is documented (root README Tier 3, in-silico-test/README.md pipeline), but it is the full chain — ~2 GB of ESM-2/ProstT5 caches, sweep_peak_oracle.py --role both, cv_all_surrogate.py, train_final_surrogate.py — and train_final_surrogate.py:52 sets only torch.manual_seed(0) with no cuDNN determinism pinned, so a reproducer's surrogate will differ slightly from the one behind the numbers in in-silico-test/README.md ("Design results"). Either ship these two checkpoints alongside the reference clouds, or state in the README that exact numeric agreement is not expected.
• The documented campaign run commands silently do nothing on a fresh clone. The design CSVs under design-campaign-EGFP/*/designs*/ are committed, and fpdesign/campaign.py:670–690 skips any pair/cell whose CSV already covers --trials. build_argparser (campaign.py:182–214) exposes no --force — only --rescore, --backfill-ppl, --probe — so bash run_sweep.sh / bash run_campaign.sh from the campaign README's "Run" section print "cached" and exit. Add a line telling reproducers to pass a different --outdir (or delete the CSVs) to actually re-execute. in-silico-test's equivalent _complete() skip is documented under "Notes and gotchas"; the campaign's is not.

Polish/nice to have

• in-silico-test/sweep_results.ipynb cell 6 does a bare pd.read_csv("trained_models/surrogate_cv3.csv") and raises FileNotFoundError on a clone, whereas the root README's Tier 3 table says those entry points "fail with the command to run, not a traceback". load_sweep in the same notebook already degrades gracefully; the CV cell could too.
• Paths with spaces and parentheses — experiment/plasmid constructs/ and (bright)his-morange_msa_07-in-pet-28a.fasta — force quoting and URL-escaping (experiment/README.md already has to write plasmid%20constructs).
• ~175 MB of RCSB PDBx cache is committed under structures/experimental/ even though pockets.py self-populates it on a miss (structures/README.md says as much). 1G7K.pdbx and 1GFL.pdbx additionally exist at both structures/ and structures/experimental/, though that duplication is functional (pockets.STRUCT reads the top level, experimental_window the subdirectory).
• The root README's Environment section documents only conda env create -f environment.yml; the exact-restore path (environment.linux-64.lock plus requirements.linux-64.txt plus pip install -e ./fpbase-extractor) is only in comments inside environment.yml. Worth surfacing.
• Stale internal references: GFP_DMS/brightness_threshold.py:16 and GFP_DMS/embed_dms.py:16 name transform_dms.py, now transform_avgfp_dms.py; the stage bootstraps in in-silico-test/1_surrogate_oracle_training/*.py and 2_design_task_specification/*.py still prepend a removed msa/ directory to sys.path (harmless, but misleading).
• GFP_DMS/README.md lane A ends in jupyter nbconvert --to notebook --execute --inplace visualization.ipynb, which rewrites a tracked 268 KB notebook and dirties git status on every reproduction run; --output to a scratch path would avoid that.

Suggestions

• Publish the in-silico-test/ surrogate and oracle checkpoints (and surrogate_cv3.csv) in the same release as reference-cloud-v1. They are small, and this is the single change that moves the pub's core design experiment from "re-derive over many GPU-hours, non-deterministically" to "download and re-run".
• Give the design engines an explicit --force, or have the wrapper scripts default to a timestamped --outdir, so "re-run the campaign" is one documented command rather than a manual delete of committed files.

---

Automated reproducibility review • model: claude-opus-5 • tokens: 569,237 in / 6,953 out • cache: 569,235 written, 0 read