Summary
Autoevals fills a real gap for teams shipping LLM features: knowing whether a model's answer is actually good. Instead of hand-grading outputs, engineers in TypeScript or Python score completions with LLM-as-a-judge templates like Factuality and Battle, plus heuristic and statistical metrics, a shared Score shape across both languages with prompts you can tweak per example.
Different judge models are already part of the design: Autoevals honors OPENAI_BASE_URL, accepts custom OpenAI-compatible clients via init(), and ships a LiteLLM adapter for native keys. Because judges are LLM calls themselves, the model behind them drives both eval quality and cost. Adding OrcaRouter as an optional provider would give Autoevals users one OpenAI-compatible endpoint spanning many chat and reasoning models, turning judge-model comparison into a config change.
Proposal
I'd like to propose adding OrcaRouter as an optional provider for Autoevals. It would sit alongside the existing OpenAI-compatible and LiteLLM paths, and nothing about the current defaults would change. OrcaRouter exposes an OpenAI-compatible API and uses standard API-key authentication, so the natural integration point is the abstraction Autoevals already ships: a grader could target OrcaRouter's endpoint with an ORCAROUTER_API_KEY through the existing OPENAI_BASE_URL / init() client configuration, with no changes to scorer templates or code.
Why it could matter for Autoevals users
A few OrcaRouter capabilities look especially relevant to evaluation workloads:
- Model breadth: chat, reasoning, image, and video models behind one endpoint, so a given evaluation can be run against different judge models without new SDK wiring.
- Automatic model routing and provider failover: helpful for long batch eval runs that should keep going when one upstream provider is down.
- Usage tracking and budgets: visibility into what a large eval run actually costs.
- Prompt caching: cuts repeated cost when the same grading prompt is evaluated across many examples.
OrcaRouter is already present in the open-source ecosystem: it has a provider page in promptfoo's docs, a plugin on the Dify Marketplace, an LLM-provider integration in NocoBase, and a declarative provider in goose, with more listed at https://www.orcarouter.ai/built-with.
Transparency and next steps
Transparency note: OrcaRouter runs an optional open-source partner program in which approved OSS projects can receive a 5% revenue share from OrcaRouter usage attributed to their integration. Participation is not a condition for the integration, and I'm happy to follow whatever disclosure or governance norms the Autoevals maintainers prefer.
To be clear, I'm an engineer on the OrcaRouter team, and this is a proposal; no code has been written or tested against Autoevals. I'd welcome maintainers' thoughts on whether an optional OrcaRouter provider fits Autoevals and on the best integration point; with your approval, I'd be glad to submit an implementation PR for review. Thanks for your time!
Summary
Autoevals fills a real gap for teams shipping LLM features: knowing whether a model's answer is actually good. Instead of hand-grading outputs, engineers in TypeScript or Python score completions with LLM-as-a-judge templates like Factuality and Battle, plus heuristic and statistical metrics, a shared Score shape across both languages with prompts you can tweak per example.
Different judge models are already part of the design: Autoevals honors
OPENAI_BASE_URL, accepts custom OpenAI-compatible clients viainit(), and ships a LiteLLM adapter for native keys. Because judges are LLM calls themselves, the model behind them drives both eval quality and cost. Adding OrcaRouter as an optional provider would give Autoevals users one OpenAI-compatible endpoint spanning many chat and reasoning models, turning judge-model comparison into a config change.Proposal
I'd like to propose adding OrcaRouter as an optional provider for Autoevals. It would sit alongside the existing OpenAI-compatible and LiteLLM paths, and nothing about the current defaults would change. OrcaRouter exposes an OpenAI-compatible API and uses standard API-key authentication, so the natural integration point is the abstraction Autoevals already ships: a grader could target OrcaRouter's endpoint with an
ORCAROUTER_API_KEYthrough the existingOPENAI_BASE_URL/init()client configuration, with no changes to scorer templates or code.Why it could matter for Autoevals users
A few OrcaRouter capabilities look especially relevant to evaluation workloads:
OrcaRouter is already present in the open-source ecosystem: it has a provider page in promptfoo's docs, a plugin on the Dify Marketplace, an LLM-provider integration in NocoBase, and a declarative provider in goose, with more listed at https://www.orcarouter.ai/built-with.
Transparency and next steps
Transparency note: OrcaRouter runs an optional open-source partner program in which approved OSS projects can receive a 5% revenue share from OrcaRouter usage attributed to their integration. Participation is not a condition for the integration, and I'm happy to follow whatever disclosure or governance norms the Autoevals maintainers prefer.
To be clear, I'm an engineer on the OrcaRouter team, and this is a proposal; no code has been written or tested against Autoevals. I'd welcome maintainers' thoughts on whether an optional OrcaRouter provider fits Autoevals and on the best integration point; with your approval, I'd be glad to submit an implementation PR for review. Thanks for your time!