Evidence-led security testing for modern AI applications.
Probe the model, the retrieval layer, the API, and the surrounding controls from one repeatable workflow.

Repository · Architecture · Security policy · Contributing

Authorized use only. LLMRed sends adversarial requests to configured AI targets. Test only systems you own or are explicitly authorized to assess, with written scope and rules of engagement.

Why LLMRed

LLM security is not limited to the prompt. A useful assessment must follow data through the complete application: authentication, retrieval, model behavior, tools, infrastructure, and reporting. LLMRed turns that work into a controlled assessment engine with an optional web control plane.

Highlights

• Modular Python CLI for repeatable assessments
• Optional FastAPI and React control plane for engagements, runs, findings, and review
• OpenAI-compatible, Ollama, and custom chat or RAG adapters
• Prompt injection, jailbreak, data disclosure, RAG, memory, MCP, tool-sink, multimodal, and supply-chain checks
• Explicit engagement policies, target allowlists, budgets, throttling, and cleanup gates
• Human-review states, calibrated attack and control trials, and reproducible evidence
• OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF, SAIF, STRIDE-AI, and CWE mappings
• HTML, PDF, JSON, and forensic bundle outputs
• Purple-team retesting, comparison mode, and regression gates

Architecture at a glance

Assessment engine

The CLI validates scope first, orchestrates bounded phases, records execution results, evaluates evidence, maps findings, and writes the configured artifacts.

Web control plane

The web interface runs the same engine in an isolated subprocess. Credentials are requested for a run and are not persisted in the database, browser storage, configuration files, or reports.

Quick start

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env
cp config.example.yaml config.yaml
# Edit config.yaml and .env for your authorized target
python cli.py --config config.yaml

For a local Ollama smoke test:

ollama serve
ollama pull mistral:7b
python cli.py --config config.mistral-local.yaml

Provider examples are included for Gemini and OpenRouter. They use environment variable references, so keys stay outside the repository.

Web control plane

cd web
cp .env.example .env
# Set LLMRED_WEB_BOOTSTRAP_ADMIN_PASSWORD
docker compose up --build

Open http://127.0.0.1:8090 locally, or use an SSH tunnel when the host is remote. See web/README.md for local development and deployment details.

Control plane preview

The web control plane gives reviewers one place to define an authorized scope, start an assessment, and inspect evidence-backed results.

These screenshots are interface previews from the project lab. Values shown are demonstration data; never publish real credentials, tokens, or target addresses.

How an assessment works

1. Declare the target, surface, capabilities, scope, policy, and budgets.
2. Run reconnaissance and capability-aware probes.
3. Execute bounded attack techniques through the selected adapter.
4. Separate observations, findings, and validation state.
5. Map evidence to the selected frameworks.
6. Review, retest, and export the final artifacts.

High-impact, write, and denial-of-service checks require explicit policy consent. Unknown capabilities are reported as unknown or not assessed, never silently treated as secure.

Reports and evidence

The default output directory is ./reports. Typical outputs are:

• audit_report.html for analysts and stakeholders
• audit_report.json for automation and CI gates
• audit_report.pdf when PDF dependencies are available
• evidence/ for hashed, redacted forensic material when enabled

Generated reports, credentials, local databases, audit logs, and completed target-specific profiles are ignored by Git. Review staged files before every publication.

Project map

cli.py                    CLI entry point
pentest/                  Assessment engine and adapters
pentest/advanced/         Security technique implementations
pentest/agentic/          Constrained plan, act, observe workflow
pentest/data/             Pinned framework catalogs
pentest/reporting/        HTML, PDF, JSON, and comparison reports
tests/                    Unit and integration tests
web/backend/              FastAPI control plane
web/frontend/             React and TypeScript interface

Useful references:

• ARCHITECTURE.md
• SPECIALIZED_SURFACES.md
• SUPPLY_CHAIN_SECURITY.md
• PURPLE_TEAM_AND_REGRESSION.md
• AGENTIC_ORCHESTRATION.md

Development

python -m pytest -q
cd web/backend && python -m pytest -q
cd ../frontend && npm ci && npm run build

See CONTRIBUTING.md before adding a technique or adapter.

Scope and limitations

LLMRed is an assessment aid, not an autonomous certification system. Results depend on the configured target, adapter, evidence, and rules of engagement. Human review remains necessary for ambiguous findings, business impact, and organizational NIST evidence. Browser automation, distributed execution, multimodal production adapters, and signed external evidence are future work.

Maintainer

Created and maintained by Anwar Jadbi.

No open-source license has been selected yet. All rights are reserved until a license file is added.