Skip to content

Add DeepTempo Direct-Edit + GPT-5-Codex location oracle result#2

Open
samarmstrong wants to merge 2 commits into
PatchEval:mainfrom
samarmstrong:submit-direct-edit-gpt5-codex
Open

Add DeepTempo Direct-Edit + GPT-5-Codex location oracle result#2
samarmstrong wants to merge 2 commits into
PatchEval:mainfrom
samarmstrong:submit-direct-edit-gpt5-codex

Conversation

@samarmstrong

Copy link
Copy Markdown

This PR submits a DeepTempo Location Oracle result for Direct-Edit + GPT-5-Codex.

Result summary:

  • Setting: PatchEval Location Oracle, comparable to the Location Oracle leaderboard tab.
  • Entry type: LLM / non-agent direct edit harness.
  • Model: gpt-5-codex via OpenRouter.
  • Strict PoC+Unit: 77/230 = 33.48%.
  • PoC-only: 82/230 = 35.65%.
  • Returned API cost: approximately $0.181/CVE.

Method disclosure:

  • Direct search/replace edits -> git diff.
  • Used the official problem statement and oracle vulnerable file content.
  • One compiler-feedback repair round for compiler/build failures.
  • No PoC or unit-test feedback was used during generation.

Operational notes:

  • Running the upstream harness locally on macOS/Colima required Docker/bind-mount fixes so /workspace/fix.patch stayed available after container startup.
  • The submitted patches are plain diffs and do not depend on those local harness changes.
  • Positive controls with embedded reference patches passed 3/3 strict locally.
  • Reproduction instructions are included in 2026-07-09__direct-edit-gpt-5-codex__location-oracle/REPRODUCTION.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant