Verify DeepTempo Direct-Edit + GPT-5-Codex Location Oracle result

[samarmstrong]

Please verify the DeepTempo Direct-Edit + GPT-5-Codex Location Oracle submission in PR #2.

Submitted result:

• Strict PoC+Unit: 77/230 = 33.48%
• PoC-only: 82/230 = 35.65%
• Setting: PatchEval Location Oracle
• Model: gpt-5-codex via OpenRouter
• Entry type: LLM / non-agent direct edit harness
• Returned API cost: approximately $0.181/CVE

Generation method:

• Direct search/replace edits -> git diff
• Official PatchEval problem statement
• Oracle vulnerable file content
• One compiler-feedback repair round for compiler/build failures
• No PoC or unit-test feedback during generation

Step-by-step reproduction:

1. Clone the DeepTempo harness:

   git clone https://github.com/armstrongflg/tempo-ingest tempo-ingest
   cd tempo-ingest

2. Clone and set up PatchEval at the path expected by the harness:

   git clone https://github.com/bytedance/PatchEval research/spikes/patcheval-upstream
   python3 -m venv research/spikes/patcheval/.venv
   research/spikes/patcheval/.venv/bin/python -m pip install -r research/spikes/patcheval-upstream/requirements.txt
   cd research/spikes/patcheval-upstream/scripts
   ../../patcheval/.venv/bin/python download_images.py
   cd ../../../../

3. Set the OpenRouter API key without printing it:

   export OPENROUTER_API_KEY=...

4. Use the pinned model id:

   openai/gpt-5-codex
   

5. Generate the initial 230 patches:

   PYTHONUNBUFFERED=1 research/spikes/patcheval/.venv/bin/python \
     research/spikes/patcheval/scripts/generate_edit_patches_v2.py \
     --model openai/gpt-5-codex \
     --cve-file research/spikes/patcheval/full230_cves.txt \
     --output research/spikes/patcheval/outputs/full230_gpt5codex_v2_patches.json \
     --usage-output research/spikes/patcheval/outputs/full230_gpt5codex_v2_usage.json \
     --raw-output research/spikes/patcheval/outputs/full230_gpt5codex_v2_raw.json \
     --request-timeout-seconds 1200 \
     --max-generation-retries 2

6. Run initial PatchEval validation:

   PYTHONUNBUFFERED=1 research/spikes/patcheval/.venv/bin/python \
     research/spikes/patcheval/scripts/run_disk_safe_validation.py \
     --patch-file research/spikes/patcheval/outputs/full230_gpt5codex_v2_patches.json \
     --output full230_gpt5codex_v2_initial \
     --cve-file research/spikes/patcheval/full230_cves.txt

7. Generate one compiler-feedback repair round for compiler/build failures only:

   PYTHONUNBUFFERED=1 research/spikes/patcheval/.venv/bin/python \
     research/spikes/patcheval/scripts/generate_compile_feedback_repairs.py \
     --model openai/gpt-5-codex \
     --validation-output research/spikes/patcheval-upstream/patcheval/evaluation/evaluation_output/full230_gpt5codex_v2_initial \
     --base-patch-file research/spikes/patcheval/outputs/full230_gpt5codex_v2_patches.json \
     --output research/spikes/patcheval/outputs/full230_gpt5codex_v2_compile_repairs.json \
     --usage-output research/spikes/patcheval/outputs/full230_gpt5codex_v2_compile_repairs_usage.json \
     --raw-output research/spikes/patcheval/outputs/full230_gpt5codex_v2_compile_repairs_raw.json \
     --request-timeout-seconds 1200

8. Merge repaired patches into full230_gpt5codex_v2_final_patches.json, validate the final patch set, and compare against the submitted summary.json. The canonical local orchestration script that performs these steps is:

   research/spikes/patcheval/scripts/supervise_full230_v2_pipeline.sh

Reproduction notes:

• The submitted patches are plain diffs.
• Local macOS/Colima validation required Docker/bind-mount fixes so /workspace/fix.patch remained available after container startup; Linux hosts matching the PatchEval README may not need those local harness fixes.
• Positive controls using embedded reference patches passed 3/3 strict locally.
• LLM generation is not guaranteed bit-for-bit deterministic; please expect possible variance of roughly 1-2 CVEs on a fresh generation run.