Evaluation Infrastructure for AI Agents
-
Updated
Feb 25, 2026 - TypeScript
#
ai-evals
Here are 39 public repositories matching this topic...
Benchmarking Open-Ended Inference Optimization by AI Agents
-
Updated
May 16, 2026 - Python
Learn to evaluate AI products for production — 21 hands-on lessons on evals, metrics, fairness, agents, red teaming, and release decisions for working PMs.
bootcamp red-teaming rag prompt-engineering llmops ai-product-management llm-evaluation claude-code ai-pm ai-evals
-
Updated
May 6, 2026
Eval-first AI agent that triages property maintenance emails. The real work is the eval system around it: trace-driven error analysis, code graders and validated LLM-as-judge (TPR/TNR), component and end-to-end evals, a failure taxonomy, and a CI regression gate. LangGraph, FastAPI, Langfuse.
python evaluation openai ai-agents pydantic fastapi ai-engineering prompt-engineering llmops langfuse llm-evaluation langgraph llm-as-a-judge llm-observability agentic-ai context-engineering ai-evals
-
Updated
May 26, 2026 - Python
🔍 Benchmark jailbreak resilience in LLMs with JailBench for clear insights and improved model defenses against jailbreak attempts.
python flask analytics openai alignment model-evaluation ai-safety security-testing red-teaming model-robustness anthropic litellm content-safety llm-jailbreaks tool-calling llm-benchmark ai-evals textual-tui
-
Updated
May 30, 2026 - Python
Playbook for PMs shipping AI products with PRDs, evals, HITL, launch gates, cost, and observability.
-
Updated
May 26, 2026 - TypeScript
Benchmark LLM jailbreak resilience across providers with standardized tests, adversarial mode, rich analytics, and a clean Web UI.
python flask analytics openai alignment model-evaluation ai-safety security-testing red-teaming model-robustness prompt-injection anthropic litellm content-safety llm-jailbreaks tool-calling llm-benchmark adversarial-testing ai-evals textual-tui
-
Updated
Aug 12, 2025 - Python
Collection of frameworks and tools for AI evalations, including tool-use, agentic AI, MCP, and multimodal
-
Updated
May 25, 2026 - Python
Free, local Langfuse OSS setup with Ollama for LLM evaluation, scoring, and datasets.
docker open-source self-hosted free no-cost local-llm ollama langfuse llm-evaluation prompt-evaluation offline-ai llm-as-judge llm-observability ai-evals
-
Updated
Apr 13, 2026 - TypeScript
End-to-end AI evals orchestration platform for comparing LLM outputs across providers with transcription, structured logging, human review, and Supabase-backed decision tracking.
gemini openai multi-model transcription human-in-the-loop model-comparison supabase anthropic llm-evaluation ai-evals evaluation-pipeline
-
Updated
Mar 10, 2026 - TypeScript
Unofficial TypeScript starter for deterministic local contract testing around Foundry-oriented workflows with Themis.
react typescript schema-validation themis contract-testing osdk developer-tooling agentic-workflows ai-evals foundry-workflows
-
Updated
Mar 28, 2026 - TypeScript
Dali is an open infrastructure project focused on citation integrity, evidentiary lineage, and reproducibility for legal AI systems. It evaluates whether AI generated legal citations and workflows remain attributable, reconstructable, and verifiable across modern AI environments.
open-source benchmark oss mcp provenance reproducibility legaltech evidence legal-ai ai-evaluation open-infrastructure ai-evals deterministic-ai legal-citations defensible-ai legal-infrastructure citation-integrity evidentiary-infrastructure
-
Updated
May 28, 2026 - Python
Offline, auditable benchmark for one-shot LLM market decisions.
-
Updated
May 30, 2026 - Python
Experimentation framework for LLM systems using simulated users, conversational behavioral metrics, and causal inference to evaluate prompt strategies, temperature, and model scaling.
-
Updated
Mar 8, 2026 - Python
AI-powered remote job aggregator with true-remote filtering, fuzzy skill match, and a working AI Evals Engineer portfolio (Hamel/Shreya methodology, calibrated judges, failure taxonomy).
typescript nextjs job-search posthog supabase ai-evaluation llm inngest prompt-engineering anthropic ai-evals
-
Updated
May 14, 2026 - HTML
A lightweight workbench for dataset-driven agent and LLM evaluation.
-
Updated
May 1, 2026 - Python
Mercor — AI-powered talent marketplace and labeling workforce
-
Updated
May 24, 2026
Harbor-format AI evaluation tasks for synthetic adtech revenue operations workflows
-
Updated
May 25, 2026 - Python
Lightweight eval framework for LLMs & AI apps combining deterministic scoring, LLM-as-judge, and regression testing.
-
Updated
Apr 8, 2026 - Python
Improve this page
Add a description, image, and links to the ai-evals topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the ai-evals topic, visit your repo's landing page and select "manage topics."