[Feature]: honor target.trace on the HTTP endpoint target, and add a per-case corpus channel for RAG targets (+ offer: a TypeScript target companion)

[charris-msft]

Problem you're trying to solve

We integrated ASSERT against a TypeScript chat product that uses retrieval-augmented generation, driving it through the HTTP endpoint target, and ran the full pipeline (systematize -> taxonomy -> test-set generation -> inference -> judge) against an Azure-hosted reasoning model using keyless Entra auth.

Systematization was genuinely strong - it derived a sensible behavior taxonomy for our product from a short natural-language spec. Two gaps stopped us from getting real value out of the run, and both look structural for non-Python and/or retrieval-grounded targets. (I checked the tracker first - the auth and temperature items I would otherwise have raised are already handled, see the note at the end.)

1. target.trace is honored only on callable, not on endpoint

docs/targets/callable.md documents that OTel traces take the judge from 1/8 to 8/8 observability. But the endpoint path constructs a plain HTTP session that does not consume target.trace. A non-Python agent must use endpoint, so it is structurally capped at 1/8 (final response text only).

We implemented an OpenInference span emitter on our side and confirmed the spans are well-formed, but there is no way for ASSERT to read them back over HTTP. The judge therefore cannot cite tool calls, routing decisions, or intermediate model calls for any non-Python target - which is exactly the "right answer for the wrong reason" class the trace path exists to catch.

2. No per-case corpus channel, so RAG / grounded targets cannot be evaluated

The target contract is {message, history}. A retrieval-grounded product needs each generated test case's own source document delivered to the target, and there is nowhere to put it.

Concretely, in our run: ASSERT generated groundedness prompts referencing documents that existed only inside the generated scenario. Our target correctly answered "I do not have access to those documents" - and the judge scored that correct refusal as policy_violation + overrefusal + wrong_tool, because from its point of view the assistant declined instead of using retrieval. Every groundedness flag in that run was an artifact of the missing channel rather than a real defect, so the run produced no usable signal on the dimension we most wanted to measure.

Proposed solution or behavior

For (1): honor target.trace on the endpoint path - either read back from the configured collector (same as the callable path), or let the target return spans / a trace id on the response envelope for ASSERT to correlate.

For (2): an optional per-case context / documents field on the generated case, passed through to the target and surfaced to the judge as the grounding source. Even a passthrough blob the target is responsible for interpreting would be enough - the key is having a channel, so generated groundedness cases can reach a retrieval target with their own source material.

Offer: a TypeScript target companion. To do the above we built a small dependency-light TypeScript package: the {message, history} -> {response} HTTP server, an OpenInference span-tree builder, a dependency-free OTLP/JSON exporter, and the target contract types, with tests. It has no dependencies on our product - node builtins and fetch only.

The OpenInference auto-instrumentation ecosystem is Python-heavy while a lot of agent products are Node/TS, so if a TS companion would be useful to you we would be glad to contribute it and shape it however you prefer.

Alternatives considered

• Python callable wrapper around our TS agent. Rejected: it would mean reimplementing or subprocess-shelling our agent runtime, and the point was to evaluate the real product as deployed.
• Injecting the corpus via the behavior spec / system prompt. Does not work for generated cases - each case needs its own source document, and the spec is global to the suite.
• Building the transcripts ourselves and calling the judge in-process. This is what we ended up doing to get a controlled comparison, and it works, but it bypasses ASSERT's generative pipeline entirely - which is the part we actually wanted.

Use case context

TypeScript multi-agent chat product with RAG over user-uploaded documents, evaluated against an Azure-hosted reasoning model with local auth disabled (Entra ID only, no API keys).

Two notes that may be useful to you, both offered as validation rather than requests:

• The keyless path from feat(auth): Azure Managed Identity / Entra ID auth for azure/* models #237 works well. With ASSERT_AZURE_USE_AAD=1 and az login against an account with disableLocalAuth: true, we ran a long evaluation with zero auth failures. Worth calling out explicitly: because azure_ad_token_provider is a callback rather than a static token, the credential is re-invoked per request and refreshes automatically, so the usual "long run dies when the AAD token expires" failure mode does not occur. That is a nice property and I did not see it stated anywhere - it might be worth a line in the docs, since it is the first thing an enterprise user will worry about.

• We may have corroborating data for [Bug]: Built-in policy_violation ORs over permissible nodes, coupling it with overrefusal and forcing a custom-dimension workaround for ACS A/B evals. #272. We ran a detection test using synthesized known-bad responses, scored by ASSERT's real judge under two different judge models. Groundedness and direct-attack safety defects were caught at 100% under both judges, but the overrefusal mutants were caught by one judge and missed entirely by the other. The judge that missed them explained itself directly: "the provided taxonomy has no populated behavior_categories to score, so there is no returned node judgment with violated = true to trigger this dimension." That is consistent with the node-coupling described in [Bug]: Built-in policy_violation ORs over permissible nodes, coupling it with overrefusal and forcing a custom-dimension workaround for ACS A/B evals. #272, and suggests a practical consequence worth noting there: overrefusal recall can become judge-model-dependent when the dimension has no populated taxonomy to fire from. Happy to add detail on that issue if it is useful.

Thanks for building this - the spec-driven approach is the right idea, and systematization in particular did better than we expected.

Acknowledgements

• I searched existing issues and discussions and did not find a duplicate.
• I'd be willing to contribute a PR for this (optional).