Extend the real-repo benchmark (umbra_eval/detection/real_repos.py) with another
public, deliberately-vulnerable app so umbra-eval realrepo covers more ground.
Good candidates: WebGoat (Java), Damn Vulnerable GraphQL App, RailsGoat (Ruby),
Vulnerable Node apps — pick one you can pin to a commit.
Where: add a RealRepoCase(...) with a pinned commit (reproducibility),
languages, and a cited provenance. Conservative expect_at_least is optional
(report-only), not a strict assertion.
Acceptance criteria
See CONTRIBUTING.md.
Extend the real-repo benchmark (
umbra_eval/detection/real_repos.py) with anotherpublic, deliberately-vulnerable app so
umbra-eval realrepocovers more ground.Good candidates: WebGoat (Java), Damn Vulnerable GraphQL App, RailsGoat (Ruby),
Vulnerable Node apps — pick one you can pin to a commit.
Where: add a
RealRepoCase(...)with a pinned commit (reproducibility),languages, and a citedprovenance. Conservativeexpect_at_leastis optional(report-only), not a strict assertion.
Acceptance criteria
umbra-eval realrepoclones + scans it and reports findings by category.pytest -q+ruffgreen (the real-repo run itself is network-gated / notasserted in unit tests).
See
CONTRIBUTING.md.