feat: deterministic assertions (is_json, json_schema, contains, regex) that skip the LLM judge on failure

README.md

@@ -18,6 +18,7 @@ PromptLens runs golden test sets against multiple models, scores outputs using L
 - **Multi-Provider Support** - Test Anthropic (Claude), OpenAI (GPT), Google (Gemini), You.com, and local models (Ollama, LM Studio)
 - **Tool/Function Calling Evaluation** - Test tool usage with automatic + LLM judge scoring across 5 criteria
 - **LLM-as-Judge Scoring** - Automated evaluation using another LLM with configurable criteria
+- **Deterministic Assertions** - Zero-token checks (`is_json`, `json_schema`, `contains`, `not_contains`, `regex`, `starts_with`) that run before the judge; a failed assertion skips the judge call entirely
 - **Cost & Latency Tracking** - Monitor per-query costs and response times across models
 - **Beautiful Reports** - Interactive HTML reports with charts, comparisons, and detailed results
 - **Multiple Export Formats** - HTML, JSON, CSV, Markdown, and JUnit XML outputs
@@ -115,6 +116,47 @@ test_cases:
 
 Save as `my_tests.yaml`.
 
+### Deterministic Assertions (zero judge tokens)
+
+Test cases can declare an `assert` block of deterministic checks that run locally before the LLM judge. When any assertion fails, the case is marked failed and the judge call is skipped, so judge tokens are only spent on responses that pass the cheap checks first. This matters for structured-output evaluation: with schema-constrained generation now GA across providers, the question is no longer "is it valid JSON" but "are the values right", and you should not pay an LLM judge to check either.
+
+```yaml
+test_cases:
+  - id: "extract-001"
+    query: "Extract name and age as JSON from: 'Jane Doe, 34'"
+    expected_behavior: "Return valid JSON with correct values"
+    assert:
+      - type: is_json
+      - type: json_schema
+        value:
+          type: object
+          required: ["name", "age"]
+          properties:
+            name: { type: string }
+            age: { type: integer }
+      - type: contains
+        value: "Jane"
+```
+
+Supported assertion types:
+
+| Type | Value | Passes when |
+|------|-------|-------------|
+| `is_json` | none | The response parses as JSON |
+| `json_schema` | JSON Schema (mapping) | The response parses as JSON and validates against the schema |
+| `contains` | string | The response contains the substring (case-sensitive) |
+| `not_contains` | string | The response does not contain the substring |
+| `regex` | string | `re.search` finds a match in the response |
+| `starts_with` | string | The response (ignoring leading whitespace) starts with the prefix |
+
+Behavior details:
+
+- All assertions in a case are always evaluated, so a single run reports every failure at once.
+- Assertion outcomes appear in all export formats. In JUnit XML, a failed assertion is a `<failure type="AssertionFailed">`, so CI fails the build.
+- Assertion failures also fail the `--fail-under` quality gate, regardless of judge scores.
+- A golden set that uses only assertions (no judge needed) passes the gate when every assertion passes.
+- Cases without an `assert` block behave exactly as before. See `examples/golden_sets/structured_output.yaml` for a complete example.
+
 ### Creating a Configuration File
 
 ```yaml

examples/golden_sets/structured_output.yaml

@@ -0,0 +1,45 @@
+name: "Structured Output Evaluation"
+description: "Deterministic assertions for JSON and structured responses. Failed assertions skip the LLM judge, so these checks cost zero judge tokens."
+version: "1.0"
+
+test_cases:
+  - id: "so-001"
+    query: "Extract the name, email, and age from this text as JSON with keys name, email, age: 'Jane Doe, 34, reachable at jane@example.com'"
+    expected_behavior: "Return valid JSON with the correct name, email, and age values"
+    category: "extraction"
+    tags: ["json", "extraction"]
+    assert:
+      - type: is_json
+      - type: json_schema
+        value:
+          type: object
+          required: ["name", "email", "age"]
+          properties:
+            name:
+              type: string
+            email:
+              type: string
+            age:
+              type: integer
+
+  - id: "so-002"
+    query: "Classify the sentiment of 'I absolutely love this product!' Reply with exactly one word: positive, negative, or neutral."
+    expected_behavior: "Reply with the single word 'positive'"
+    category: "classification"
+    tags: ["classification", "sentiment"]
+    assert:
+      - type: regex
+        value: "^\\s*(positive|negative|neutral)\\s*$"
+      - type: contains
+        value: "positive"
+
+  - id: "so-003"
+    query: "Summarize the benefits of unit testing in two sentences. Do not use marketing language like 'game-changer' or 'revolutionary'."
+    expected_behavior: "A concise two-sentence summary in plain technical language"
+    category: "summarization"
+    tags: ["summarization", "style"]
+    assert:
+      - type: not_contains
+        value: "game-changer"
+      - type: not_contains
+        value: "revolutionary"

promptlens/cli.py

@@ -54,10 +54,17 @@ def _remove_path_if_exists(path: Path) -> None:
 
 
 def _check_fail_under(result: "RunResult", fail_under: float) -> list:
-    """Return models whose average judge score falls below the gate.
+    """Return models that fail the quality gate.
+
+    A model fails the gate when its average judge score falls below the
+    threshold, or when any of its test cases failed a deterministic
+    assertion (assertion failures are definitive, regardless of scores).
 
     A model with no judge scores at all also fails the gate, since the gate
-    cannot be evaluated without scores and a silent pass would be misleading.
+    cannot be evaluated without scores and a silent pass would be
+    misleading. The one exception: a model whose every test case carried
+    assertions that all passed is a deterministic pass and does not need
+    judge scores.
 
     Args:
         result: The completed run result
@@ -68,8 +75,23 @@ def _check_fail_under(result: "RunResult", fail_under: float) -> list:
     """
     failing = []
     for model in result.models_tested:
+        model_results = [r for r in result.results if r.model_response.model == model]
         avg = result.get_average_score(model)
-        if avg is None or avg < fail_under:
+
+        has_assertion_failure = any(
+            r.assertions_passed() is False for r in model_results
+        )
+        if has_assertion_failure:
+            failing.append((model, avg))
+            continue
+
+        if avg is None:
+            all_deterministic_pass = bool(model_results) and all(
+                r.assertions_passed() for r in model_results
+            )
+            if not all_deterministic_pass:
+                failing.append((model, avg))
+        elif avg < fail_under:
             failing.append((model, avg))
     return failing
 
@@ -228,7 +250,18 @@ def run(
                 )
                 for model, avg in failing_models:
                     avg_display = f"{avg:.2f}" if avg is not None else "no scores"
-                    console.print(f"  {model}: average judge score {avg_display}")
+                    assertion_failures = sum(
+                        1
+                        for r in result.results
+                        if r.model_response.model == model
+                        and r.assertions_passed() is False
+                    )
+                    line = f"  {model}: average judge score {avg_display}"
+                    if assertion_failures:
+                        line += (
+                            f", {assertion_failures} case(s) with failed assertions"
+                        )
+                    console.print(line)
                 sys.exit(2)
             console.print(
                 f"\n[bold green]✓ Quality gate passed (--fail-under {fail_under:g})[/bold green]"

promptlens/exporters/junit_exporter.py

@@ -7,10 +7,15 @@
 
 Mapping rules:
     - A test case whose model response errored is reported as an <error>.
+    - A test case with one or more failed deterministic assertions is
+      reported as a <failure> (assertion failures skip LLM judging, so this
+      is checked before the judge score).
     - A test case whose judge score is below the failure threshold is
       reported as a <failure>.
     - A test case that was never judged (judging disabled or judge failed)
-      is reported as <skipped>, so CI does not report a false pass.
+      and has no passing assertions is reported as <skipped>, so CI does
+      not report a false pass. A case whose assertions all passed counts as
+      a pass even without a judge score.
     - Everything else is a pass.
 """
 
@@ -126,13 +131,35 @@ def _build_suite(
 
             response_error = eval_result.model_response.error
             judge_score = eval_result.judge_score
+            failed_assertions = [
+                a for a in eval_result.assertion_results if not a.passed
+            ]
 
             if response_error:
                 errors += 1
                 error_el = ET.SubElement(testcase, "error")
                 error_el.set("message", _truncate(response_error, 300))
                 error_el.set("type", "ModelResponseError")
                 error_el.text = response_error
+            elif failed_assertions:
+                failures += 1
+                failure_el = ET.SubElement(testcase, "failure")
+                failure_el.set(
+                    "message",
+                    f"{len(failed_assertions)} assertion(s) failed: "
+                    + ", ".join(a.type for a in failed_assertions),
+                )
+                failure_el.set("type", "AssertionFailed")
+                failure_el.text = (
+                    f"Query: {_truncate(eval_result.query, 500)}\n"
+                    + "\n".join(
+                        f"[{a.type}] {_truncate(a.message, 500)}"
+                        for a in failed_assertions
+                    )
+                )
+            elif judge_score is None and eval_result.assertion_results:
+                # All assertions passed and no judge score: deterministic pass.
+                pass
             elif judge_score is None:
                 skipped += 1
                 skipped_el = ET.SubElement(testcase, "skipped")
@@ -163,6 +190,13 @@ def _build_suite(
                 f"cost_usd: {eval_result.model_response.cost_usd or 0.0}",
                 f"tokens_used: {eval_result.model_response.tokens_used or 0}",
             ]
+            if eval_result.assertion_results:
+                passed_count = sum(
+                    1 for a in eval_result.assertion_results if a.passed
+                )
+                out_lines.append(
+                    f"assertions: {passed_count}/{len(eval_result.assertion_results)} passed"
+                )
             if judge_score is not None:
                 out_lines.append(f"judge_score: {judge_score.score}")
                 out_lines.append(

promptlens/judges/__init__.py

@@ -1,6 +1,7 @@
 """LLM-as-judge evaluation."""
 
+from promptlens.judges.assertions import evaluate_assertions
 from promptlens.judges.base import BaseJudge
 from promptlens.judges.llm_judge import LLMJudge
 
-__all__ = ["BaseJudge", "LLMJudge"]
+__all__ = ["BaseJudge", "LLMJudge", "evaluate_assertions"]