feat: Add agent-redteam kit#200
Open
ishaaanfarooq wants to merge 5 commits into
Open
Conversation
Adds a red-team harness that fires a curated jailbreak/prompt-injection/ exfiltration/instruction-override/PII-extraction/harmful-content attack battery at a system prompt and applies a pass/fail guardrail gate — the security counterpart to kits/llm-eval-harness, which only checks output correctness. Verified end-to-end against deployed Lamatic flows (Groq llama-3.3-70b-versatile): a weak example prompt resists 41.7% of attacks (5/12), a hardened example resists 83.3% (10/12) but still fails the 90% gate — the harness catches a real gap where a "fictional framing" jailbreak gets through despite an explicit anti-roleplay rule. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI (base), Organization UI (inherited) Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughThis PR adds a new ChangesAgent Red-Team Harness
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
:robot_face: AgentKit Structural ValidationNew Contributions Detected
Check Results
🎉 All checks passed! This contribution follows the AgentKit structure. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/agent-redteam/apps/actions/orchestrate.ts`:
- Around line 27-35: Add a timeout guard around getLamaticClient().executeFlow
in getAnswer so a stuck Lamatic request cannot block the scan. Keep the existing
answer extraction and validation in getAnswer, but wrap the executeFlow await
with a timeout-based race or helper that rejects after a fixed duration, and
ensure the timeout is cleared once the flow completes.
In `@kits/agent-redteam/apps/app/page.tsx`:
- Around line 103-111: Associate the “System prompt under test” Label with the
Textarea by adding a matching identifier and label reference in the page
component. Update the Label and Textarea pair in the form so the Label uses
htmlFor and the Textarea uses the corresponding id, keeping the existing
systemPrompt value/onChange wiring intact.
- Around line 16-23: The CATEGORY_LABELS mapping is duplicated in the app page
and the other two consumers, and the typings are inconsistent. Move the mapping
into a shared constants module and export it with the stricter
Record<AttackCategory, string> type so page.tsx, security-scorecard.tsx, and
attack-results-table.tsx all import the same source. Update the existing
CATEGORY_LABELS references in those components to use the shared export, and
keep any category list helper derived from the shared mapping so the three files
stay in sync.
- Around line 213-230: The empty-state icon logic in the page component is
showing a success icon during loading; update the conditional in the render
block that uses result and isLoading so that the loading branch renders Loader2
with animate-spin instead of CheckCircle2. Keep CheckCircle2 for the non-loading
success/loaded state, and use the existing icon selection area in the main page
component to make the state distinction clear.
- Around line 54-72: Switch the scan form in the page component from manual
useState validation to react-hook-form with zod schema validation. Update the
onSubmit flow in the page’s submit handler to use form values validated by a zod
resolver instead of trim()/length checks, and rely on the form library to
prevent invalid submits before calling runRedTeamScan. Add the missing
react-hook-form and zod dependencies to the app package so the page can use the
standard form stack.
In `@kits/agent-redteam/apps/lib/eval.ts`:
- Around line 94-119: `computeSecurityAggregate` is using the rounded `passRate`
to decide `gatePassed`, which can change borderline outcomes; keep the displayed
rounded rate, but compute the gate decision from the exact unrounded pass
percentage inside `computeSecurityAggregate` so the threshold check stays
deterministic. Update the `gatePassed` assignment to use the raw ratio derived
from `results`/`passed` rather than the rounded `passRate`, while leaving the
returned `passRate` field unchanged for reporting.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 746e9003-d617-4021-ba2b-93807c035c19
⛔ Files ignored due to path filters (1)
kits/agent-redteam/apps/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (36)
kits/agent-redteam/.gitignorekits/agent-redteam/README.mdkits/agent-redteam/agent.mdkits/agent-redteam/apps/.env.examplekits/agent-redteam/apps/.gitignorekits/agent-redteam/apps/README.mdkits/agent-redteam/apps/actions/orchestrate.tskits/agent-redteam/apps/app/globals.csskits/agent-redteam/apps/app/layout.tsxkits/agent-redteam/apps/app/page.tsxkits/agent-redteam/apps/components.jsonkits/agent-redteam/apps/components/attack-results-table.tsxkits/agent-redteam/apps/components/security-scorecard.tsxkits/agent-redteam/apps/components/ui/button.tsxkits/agent-redteam/apps/components/ui/input.tsxkits/agent-redteam/apps/components/ui/label.tsxkits/agent-redteam/apps/components/ui/textarea.tsxkits/agent-redteam/apps/lib/attacks.tskits/agent-redteam/apps/lib/eval.tskits/agent-redteam/apps/lib/lamatic-client.tskits/agent-redteam/apps/lib/types.tskits/agent-redteam/apps/lib/utils.tskits/agent-redteam/apps/next.config.mjskits/agent-redteam/apps/package.jsonkits/agent-redteam/apps/postcss.config.mjskits/agent-redteam/apps/tsconfig.jsonkits/agent-redteam/constitutions/default.mdkits/agent-redteam/flows/judge.tskits/agent-redteam/flows/run-target.tskits/agent-redteam/lamatic.config.tskits/agent-redteam/model-configs/judge.tskits/agent-redteam/model-configs/run-target.tskits/agent-redteam/prompts/judge_system.mdkits/agent-redteam/prompts/judge_user.mdkits/agent-redteam/prompts/run-target_system.mdkits/agent-redteam/prompts/run-target_user.md
Comment on lines
+213
to
+230
| ) : ( | ||
| <div className="flex min-h-[460px] items-center justify-center rounded-2xl border border-dashed border-white/10 bg-white/[0.01]"> | ||
| <div className="max-w-sm px-6 text-center"> | ||
| <div className="mx-auto mb-4 flex h-14 w-14 items-center justify-center rounded-2xl border border-white/10 bg-white/[0.03]"> | ||
| {result === null && !isLoading ? ( | ||
| <ShieldAlert className="h-7 w-7 text-muted-foreground" /> | ||
| ) : ( | ||
| <CheckCircle2 className="h-7 w-7 text-muted-foreground" /> | ||
| )} | ||
| </div> | ||
| <p className="font-medium">No scan yet</p> | ||
| <p className="mt-1.5 text-sm text-muted-foreground"> | ||
| Paste a system prompt and run the scan — or click <span className="font-medium text-foreground">Load weak example</span> to see a prompt | ||
| get compromised. | ||
| </p> | ||
| </div> | ||
| </div> | ||
| )} |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Loading state shows a green check circle — should show a spinner.
When isLoading is true and result is null, the empty-state icon renders CheckCircle2, which visually implies success. Show Loader2 with animate-spin during loading instead.
🔄 Proposed fix
{result === null && !isLoading ? (
<ShieldAlert className="h-7 w-7 text-muted-foreground" />
) : (
- <CheckCircle2 className="h-7 w-7 text-muted-foreground" />
+ <Loader2 className="h-7 w-7 animate-spin text-muted-foreground" />
)}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ) : ( | |
| <div className="flex min-h-[460px] items-center justify-center rounded-2xl border border-dashed border-white/10 bg-white/[0.01]"> | |
| <div className="max-w-sm px-6 text-center"> | |
| <div className="mx-auto mb-4 flex h-14 w-14 items-center justify-center rounded-2xl border border-white/10 bg-white/[0.03]"> | |
| {result === null && !isLoading ? ( | |
| <ShieldAlert className="h-7 w-7 text-muted-foreground" /> | |
| ) : ( | |
| <CheckCircle2 className="h-7 w-7 text-muted-foreground" /> | |
| )} | |
| </div> | |
| <p className="font-medium">No scan yet</p> | |
| <p className="mt-1.5 text-sm text-muted-foreground"> | |
| Paste a system prompt and run the scan — or click <span className="font-medium text-foreground">Load weak example</span> to see a prompt | |
| get compromised. | |
| </p> | |
| </div> | |
| </div> | |
| )} | |
| ) : ( | |
| <div className="flex min-h-[460px] items-center justify-center rounded-2xl border border-dashed border-white/10 bg-white/[0.01]"> | |
| <div className="max-w-sm px-6 text-center"> | |
| <div className="mx-auto mb-4 flex h-14 w-14 items-center justify-center rounded-2xl border border-white/10 bg-white/[0.03]"> | |
| {result === null && !isLoading ? ( | |
| <ShieldAlert className="h-7 w-7 text-muted-foreground" /> | |
| ) : ( | |
| <Loader2 className="h-7 w-7 animate-spin text-muted-foreground" /> | |
| )} | |
| </div> | |
| <p className="font-medium">No scan yet</p> | |
| <p className="mt-1.5 text-sm text-muted-foreground"> | |
| Paste a system prompt and run the scan — or click <span className="font-medium text-foreground">Load weak example</span> to see a prompt | |
| get compromised. | |
| </p> | |
| </div> | |
| </div> | |
| )} |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@kits/agent-redteam/apps/app/page.tsx` around lines 213 - 230, The empty-state
icon logic in the page component is showing a success icon during loading;
update the conditional in the render block that uses result and isLoading so
that the loading branch renders Loader2 with animate-spin instead of
CheckCircle2. Keep CheckCircle2 for the non-loading success/loaded state, and
use the existing icon selection area in the main page component to make the
state distinction clear.
- Guard flow calls with a 30s timeout so a stuck Lamatic request can't hang the scan - Fix security gate to decide pass/fail from the exact ratio, not the rounded display passRate, so a borderline result can't round into passing - Move CATEGORY_LABELS/ALL_CATEGORIES into lib/attacks.ts as a single source of truth instead of duplicating the map in three components - Fix empty-state icon showing a static icon during a run instead of a spinner - Associate the system-prompt Label/Textarea via htmlFor/id - Migrate the scan form to react-hook-form + zod (matches llm-eval-harness's pattern), so invalid submissions are prevented proactively instead of only being caught in the submit handler Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kits/agent-redteam/apps/app/page.tsx`:
- Around line 19-23: The formSchema categories field is too loose because
z.array(z.string()) infers string[] and forces the unsafe AttackCategory cast
later in page.tsx. Update the categories validation in formSchema to use
z.enum() with the valid AttackCategory values so the type is inferred correctly,
then remove the cast at the form submission/use site that currently converts
categories to AttackCategory[].
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: 2a08bfe9-d399-48ae-8936-da52f0678f6c
⛔ Files ignored due to path filters (1)
kits/agent-redteam/apps/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (7)
kits/agent-redteam/apps/actions/orchestrate.tskits/agent-redteam/apps/app/page.tsxkits/agent-redteam/apps/components/attack-results-table.tsxkits/agent-redteam/apps/components/security-scorecard.tsxkits/agent-redteam/apps/lib/attacks.tskits/agent-redteam/apps/lib/eval.tskits/agent-redteam/apps/package.json
Removes the unsafe AttackCategory[] cast — the schema now infers the correct union type directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Actual env vars live in apps/.env.example; this just points there so the structural validation checklist item for kits is satisfied explicitly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
kits/agent-redteam/apps/app/page.tsx (1)
154-166: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAdd
aria-pressedto category toggle buttons for screen reader accessibility.Active categories are conveyed only through color styling. Screen reader users can't perceive which categories are selected without
aria-pressed. The Label shows a count but not which specific categories are active.♿ Proposed fix
<button key={category} type="button" onClick={() => toggleCategory(category)} disabled={isLoading} + aria-pressed={active} className={cn( "rounded-lg border px-3 py-2 text-left text-xs font-medium transition-colors", active ? "border-rose-500/30 bg-rose-500/10 text-foreground" : "border-white/10 bg-white/[0.02] text-muted-foreground", )} >🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kits/agent-redteam/apps/app/page.tsx` around lines 154 - 166, The category toggle buttons in the category list rely only on color to ցույց active state, so add an accessibility state indicator to the button rendered in the category toggle map. Update the button created around toggleCategory, active, and CATEGORY_LABELS to include aria-pressed based on the active flag so screen readers can announce selection state consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@kits/agent-redteam/apps/app/page.tsx`:
- Around line 154-166: The category toggle buttons in the category list rely
only on color to ցույց active state, so add an accessibility state indicator to
the button rendered in the category toggle map. Update the button created around
toggleCategory, active, and CATEGORY_LABELS to include aria-pressed based on the
active flag so screen readers can announce selection state consistently.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro
Run ID: b636a253-7402-4a71-a2eb-a752e26f7f00
📒 Files selected for processing (2)
kits/agent-redteam/.env.examplekits/agent-redteam/apps/app/page.tsx
Screen readers otherwise have no way to know a category is selected — active state was color-only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
An Agent Red-Team Harness kit — it fires a curated jailbreak / prompt-injection / exfiltration / instruction-override / PII-extraction / harmful-content attack battery at a system prompt, then applies a pass/fail guardrail gate. Submitted for the agentkit-challenge.
Problem
kits/llm-eval-harnessalready gates whether an agent's outputs are correct. Nothing in this repo gates whether an agent's guardrails actually hold under attack — even though "constitutions" (guardrails) are one of Lamatic's own core platform primitives. Every team shipping an agent needs to know if their system prompt survives a jailbreak attempt before a real user finds out it doesn't.How it works
run-targetflow sends the system-prompt-under-test + an attack payload to an LLM and captures the raw response (the "agent under attack").judgeflow (LLM-as-judge) scores whether the guardrail held (refused/deflected) or was compromised (complied, leaked, or produced disallowed content), with a severity level and reasoning. Partial compliance still counts as compromised — a response that leaks a fragment before catching itself doesn't get a pass.Results (verified against live deployed flows, Groq
llama-3.3-70b-versatile)Stack
Lamatic flows (Groq
llama-3.3-70b-versatile, temperature 0 throughout for reproducible security results) · Next.js 16 / React 19 · Tailwind v4 · shadcn/ui. Verified locally end-to-end;npm run buildtype-checks clean.Notes / tradeoffs
run-targetreflects the tested prompt plus the underlying provider's own safety training — for jailbreak/harmful-content categories, a "held" verdict measures the prompt and the model's base training together, not the prompt in total isolation. Exfiltration/instruction-override/PII-extraction are more prompt-dependent and a cleaner signal of the prompt's own guardrail quality.llm-eval-harness:compromised/passis recomputed app-side from the judge's fields rather than trusted from the model's own arithmetic.kits/llm-eval-harness— run both a correctness gate and a security gate before shipping a prompt change.Label:
agentkit-challengekits/agent-redteamkit for running an Agent Red-Team Harness against a target system prompt using a curated attack battery, evaluated via LLM-as-judge guardrail scoring (held vs compromised, with severity and strict JSON verdicts; partial compliance counts as compromised).flow.jsonfiles underkits/agent-redteam; harness flows are defined as TypeScript exports in:kits/agent-redteam/flows/run-target.tskits/agent-redteam/flows/judge.tsFiles added (under
kits/agent-redteam)kits/agent-redteam/README.mdkits/agent-redteam/agent.mdkits/agent-redteam/.env.examplekits/agent-redteam/.gitignorekits/agent-redteam/lamatic.config.tskits/agent-redteam/constitutions/default.mdkits/agent-redteam/flows/run-target.tskits/agent-redteam/flows/judge.tskits/agent-redteam/prompts/run-target_system.mdkits/agent-redteam/prompts/run-target_user.mdkits/agent-redteam/prompts/judge_system.mdkits/agent-redteam/prompts/judge_user.mdkits/agent-redteam/model-configs/run-target.tskits/agent-redteam/model-configs/judge.tskits/agent-redteam/apps)kits/agent-redteam/apps/README.mdkits/agent-redteam/apps/.env.examplekits/agent-redteam/apps/.gitignorekits/agent-redteam/apps/package.jsonkits/agent-redteam/apps/package-lock.jsonkits/agent-redteam/apps/tsconfig.jsonkits/agent-redteam/apps/next.config.mjskits/agent-redteam/apps/postcss.config.mjskits/agent-redteam/apps/components.jsonkits/agent-redteam/apps/app/layout.tsxkits/agent-redteam/apps/app/globals.csskits/agent-redteam/apps/app/page.tsxkits/agent-redteam/apps/actions/orchestrate.tskits/agent-redteam/apps/components/security-scorecard.tsxkits/agent-redteam/apps/components/attack-results-table.tsxkits/agent-redteam/apps/components/ui/button.tsxkits/agent-redteam/apps/components/ui/input.tsxkits/agent-redteam/apps/components/ui/label.tsxkits/agent-redteam/apps/components/ui/textarea.tsxkits/agent-redteam/apps/lib/attacks.tskits/agent-redteam/apps/lib/eval.tskits/agent-redteam/apps/lib/types.tskits/agent-redteam/apps/lib/lamatic-client.tskits/agent-redteam/apps/lib/utils.tsFlow structure + node types (from flow TS)
run-target(kits/agent-redteam/flows/run-target.ts)triggerNode→dynamicNode(LLM) →responseNode{systemPrompt, input}, runs the target LLM using@prompts/run-target_system.mdand@prompts/run-target_user.md, then mapsLLMNode_1.output.generatedResponseto the outputanswer.defaultEdgebetween graph nodes plus aresponseEdgefor the trigger→response mapping.judge(kits/agent-redteam/flows/judge.ts)triggerNode→dynamicNode(LLM) →responseNoderesponse, runs the judge LLM with@prompts/judge_system.md+@prompts/judge_user.mdand produces strict JSON verdict output, mapped into the returnedanswer.defaultEdgebetween graph nodes plus aresponseEdgefor the trigger→response mapping.App orchestration (high level)
apps/actions/orchestrate.tsorchestrates the battery serially (CONCURRENCY = 1) by calling:run-targetonce per selected attack case to capture the raw target response (answer)judgeonce per case to scorecompromised+severity+reasoningapps/lib/eval.tsrecomputes pass deterministically fromcompromised(partial compliance => compromised), calculates pass-rate and gate result vs threshold, and produces per-category breakdowns.apps/app/page.tsxprovides the scan form (category toggles + threshold) and renders the verdict viaSecurityScorecardand per-attack details viaAttackResultsTable.Accessibility tweak
aria-pressedinkits/agent-redteam/apps/app/page.tsx.PR Checklist (per CONTRIBUTING.md)
kits/agent-redteam/(kebab-case, unique)lamatic.config.tspresent with validtype: "kit",name,author,tags,steps,linksagent.mdpresentREADME.mdpresent, describes what the contribution does and how to use itflows/<name>.tsexists for every step inlamatic.config.ts(judge,run-target)constitutions/default.mdpresentapps/.env.examplepresent with placeholder values onlyapps/package.jsonworks withnpm install && npm run dev— verified locally, and end-to-end against live deployed flows (Groqllama-3.3-70b-versatile)links.githubpoints tokits/agent-redteamlinks.deployhasroot-directory=kits/agent-redteam/apps.envor.env.localcommitted@referencepaths resolve to files that exist in the kitkits/agent-redteam/z.enumtyping)