If you would like to submit your model results to PatchEval leaderboard, please follow the steps below.
-
Setup: Fork and clone the PatchEval repository. Set up the evaluation environment as described in the repositoryβs README.md.
-
Run evaluation: You can evaluate your own results using the provided script (e.g.,
patcheval/evaluation/run_evaluation.py).a. Prepare your patch file:
[ { "cve": "CVE-ID", "fix_patch": "YOUR_PATCH (diff)" }, ]b. Run the evaluation command:
cd patcheval/evaluation python run_evaluation.py \ --output example \ --patch_file YOUR_PATCH_PATH \ -
Submission: Your submission directory should contain the following files and structure:
-
summary.json: Evaluation results -
run_evaluation.log: Evaluation logs -
your_patch.json: Must include 'cve' and 'fix_patch' keys -
logs/: Evaluation artifacts for each CVE. The folder structure should be as follows:logs βββ CVE-ID β βββ fix.patch β βββ success_output.log βββ CVE-ID βββ error_output.log βββ fix.patch -
metadata.yaml: Metadata of your submission, including how it will be displayed on the website:- name: The name of your leaderboard entry
- site: URL/link to more information about your system
- verified: false (See below for results verification)
-
trajs/: (Optional) Traces that show how your agent repair each CVE
-
-
Create a pull request to the repository with a new folder.
A Verified β badge on the leaderboard indicates that the PatchEval team has independently reproduced your patch generation results. To request verification for your submission, please follow these steps:
- Open an Issue: Create a new issue in GitHub repository to request verification.
- Provide Instructions: In the issue, include detailed, step-by-step instructions for running your model on the PatchEval benchmark.
- Await Confirmation: Once the team successfully reproduces your results, your submission will receive the βVerifiedβ checkmark on the leaderboard.