Skip to content

PatchEval/experiments

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

3 Commits
Β 
Β 
Β 
Β 

Repository files navigation

PatchEval: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilitie

Python License

Submit to PatchEval Leaderboard

If you would like to submit your model results to PatchEval leaderboard, please follow the steps below.

  1. Setup: Fork and clone the PatchEval repository. Set up the evaluation environment as described in the repository’s README.md.

  2. Run evaluation: You can evaluate your own results using the provided script (e.g., patcheval/evaluation/run_evaluation.py).

    a. Prepare your patch file:

    [
        {
            "cve": "CVE-ID",
            "fix_patch": "YOUR_PATCH (diff)"
        },
    ]

    b. Run the evaluation command:

    cd patcheval/evaluation
    python run_evaluation.py \
        --output example \
        --patch_file YOUR_PATCH_PATH \
  3. Submission: Your submission directory should contain the following files and structure:

    • summary.json: Evaluation results

    • run_evaluation.log: Evaluation logs

    • your_patch.json: Must include 'cve' and 'fix_patch' keys

    • logs/: Evaluation artifacts for each CVE. The folder structure should be as follows:

      logs
      β”œβ”€β”€ CVE-ID
      β”‚   β”œβ”€β”€ fix.patch
      β”‚   └── success_output.log
      └── CVE-ID
          β”œβ”€β”€ error_output.log
          └── fix.patch
    • metadata.yaml: Metadata of your submission, including how it will be displayed on the website:

      • name: The name of your leaderboard entry
      • site: URL/link to more information about your system
      • verified: false (See below for results verification)
    • trajs/: (Optional) Traces that show how your agent repair each CVE

  4. Create a pull request to the repository with a new folder.

Verify Your Results

A Verified βœ… badge on the leaderboard indicates that the PatchEval team has independently reproduced your patch generation results. To request verification for your submission, please follow these steps:

  1. Open an Issue: Create a new issue in GitHub repository to request verification.
  2. Provide Instructions: In the issue, include detailed, step-by-step instructions for running your model on the PatchEval benchmark.
  3. Await Confirmation: Once the team successfully reproduces your results, your submission will receive the β€œVerified” checkmark on the leaderboard.

About

No description, website, or topics provided.

Resources

License

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors