Target discovery benchmark

A benchmark of coding agents on a single open-ended research task: discovering candidate disease–gene associations de novo from single-cell RNA-seq (scRNA-seq) data.

Objective

Each agent is asked to design and evaluate Python workflows that surface candidate disease-associated genes purely from expression data — not to look up associations that are already known. A workflow exposes a frozen entrypoint:

def predict(data_path: str, params: dict) -> pd.DataFrame:  # -> columns: gene, score

The real grading criterion is generalization: the selected predict() is run unmodified on held-out indications (different diseases, tissues, and datasets). The two development indications (A and B) are only a proxy for comparing workflows during development. The full task specification lives in prompts/.

Prompt variants

The agents were run under three progressively more constrained prompts:

Prompt	What it asks for
prompts/base_prompt.md	Baseline task; any sound methodology allowed (including standard differential expression).
prompts/explore_prompt.md	Base + must go beyond standard DE testing — embedding/multivariate/network/foundation-model approaches instead.
prompts/explore_and_iterate_prompt.md	Explore + an added iterative refinement phase (evaluate → revise → re-evaluate), with the iteration history documented.

Concrete input paths in the prompts are templated as {PATH_TO_INPUT_DATA}.

Agents

Prefix	Agent	Model
codex_*	OpenAI Codex	GPT-5.5
cs_*	Claude Science	Claude Opus 4.8
kimi_*	Kimi	Moonshot Kimi

All three agents (codex_*, cs_*, kimi_*) cover all three prompt variants.

Repository layout

prompts/                     # the three task prompts (see above)
runs/
  <agent>_<variant>/         # one folder per (agent × prompt variant)
    *.py                     # the agent's solution: workflow / predict / evaluate code
    config/<INDICATION>.json # per-indication params for predict() (codex_*/cs_* only)

For codex_* and cs_*, each config/<INDICATION>.json holds the params dict for one disease, e.g.:

{"label_key":"Condition","disease_values":["ALS"],"control_values":["PN"],
 "celltype_key":"CellType","donor_key":"Donor"}

The kimi_* folders have no config/ directory — that agent expressed its per-indication params inside its run scripts rather than as standalone config files, so there is no equivalent to carry over.

The runs/*/ folders contain the solution code the agents produced; input datasets and generated outputs (e.g. *.h5ad, ranked-target CSVs) are not tracked here.