feat(tracing): add base example of inference.net tracing for grc assi…

[MJSPollard]

Hey guys 👋

I know you came to the Inference office and met with Sam and Amar earlier last week and expressed interest in setting up our new tracing product in your platform. We should probably open up a Slack Connect channel to chat more and figure out what the best setup would be for you if you're still interested

This PR is just the bare-minimum example so you can see traces start populating in your Inference dashboard. From there you can use HALO (our open source RLM based agent optimizer) product to ask questions about your traces, derive insights, get recommendations and then iterate on your agent harnesses. We also offer broader observability for the one-shot LLM calls you have all over the platform (questionnaires, embeddings, vendor-risk scraping, evidence forms, etc.) if that's something you'd like to explore. From there you can train your own cheaper/faster models that are fine tuned specifically for the task they are doing.

It's probably best if you test this out yourselves and play around in the Inference dashboard. I noticed your main task automation agent looks like it's closed-source / in a separate enterprise service. This product could be used to improve that too.

I didn't set up a full local version of this repo as I wasn't sure where/if you want to incorporate our tracing and what you want to include in the spans and how you want to group them. I believe it should work but hard to know without full setup and testing. It seemed quicker to just set up an example and hand off to you guys if you're interested in the full setup. If there are any issues just let us know and we can take a look or you can play around with it on your dev server to get it how you want so you can experiment with the results on our dashboard.

Check out our docs if you want to learn more. We also have a cli tool that can assist in adding tracing to any repo.

---

What does this PR do?

Adds Inference.net Catalyst tracing to the two Vercel AI SDK agents in this repo so the team can see model calls, tool calls, token usage, and errors in the Catalyst dashboard.

Scope (intentionally narrow):

• GRC Assistant (apps/api/src/assistant-chat) — service name compai-grc-assistant
• Policy Editor AI (apps/app/src/app/api/policies/[policyId]/chat) — service name compai-policy-editor

Only these two streamText calls are instrumented. The other ~50 single-shot generateText / generateObject / embed calls (questionnaire, embeddings, vendor risk scraping, evidence forms, etc.) are deliberately left untraced — they're not multi-step agents and can be added later if cost/latency monitoring on those flows becomes useful.

What was added:

• apps/api/src/inference-tracing.ts and apps/app/src/lib/inference-tracing.ts — small singleton modules that call setup() from @inference/tracing once at app boot and expose a getAITelemetry(functionId) helper
• initTracing() call in apps/api/src/main.ts (before NestFactory.create) and in apps/app/src/instrumentation.ts (Next.js boot hook)
• experimental_telemetry: getAITelemetry(...) passed on the two streamText calls
• shutdownTracing() wired into the existing SIGTERM/SIGINT handler in apps/api/src/main.ts to flush the last batch of spans on graceful shutdown (Next.js has no equivalent hook, so apps/app may lose its trailing batch on deploy — acceptable per the canonical Catalyst AI SDK docs)
• New env vars documented in both .env.example files: CATALYST_OTLP_TOKEN, CATALYST_OTLP_ENDPOINT

Not in scope:

• The "Automated Evidence" task automation agent — it lives in a separate enterprise service behind NEXT_PUBLIC_ENTERPRISE_API_URL and can't be instrumented from this repo.

• Manual AGENT parent spans with agent.id / session.id attributes. Catalyst's docs suggest this for nicer Agents-dashboard grouping (agent-identity). Skipped here because I couldn't validate async-context propagation through the AI SDK's streaming spans without running the apps. Easy follow-up if the dashboard grouping isn't good enough by default.

• Fixes #XXXX — N/A

• Fixes COMP-XXXX — N/A

Visual Demo

N/A — this is observability wiring with no end-user-visible behavior. Verification happens in the Catalyst dashboard after setting the env vars; see "How should this be tested?" below.

Mandatory Tasks (DO NOT REMOVE)

• I have self-reviewed the code.
• I have updated the developer docs in /docs if this PR makes changes that would require a documentation change. N/A — no public docs impact; internal env vars are documented in .env.example.
• I confirm automated tests are in place that prove my fix is effective or that my feature works. N/A — this is tracing wiring; verification requires running each app with real Catalyst credentials and observing traces in the Catalyst dashboard. There's no meaningful unit-test for OTel SDK initialization that doesn't just re-test the upstream library.

How should this be tested?

Environment variables (set in appropriate places):

CATALYST_OTLP_TOKEN=<your Inference API key from https://inference.net>
CATALYST_OTLP_ENDPOINT=https://telemetry.inference.net

Both are documented in apps/api/.env.example and apps/app/.env.example.

Steps:

1. Set the env vars above in the .env files. Get the token from the inference dashboard under API Keys.
2. Test both assistants with a couple questions.
3. Open the Inference Catalyst dashboard and go to the traces or agent tab:
   • compai-grc-assistant should show ai.streamText, ai.streamText.doStream, and ai.toolCall spans
   • compai-policy-editor should show the same for the Anthropic-backed policy chat

What to look for in a healthy trace:

• Model name (gpt-5 for GRC, claude-sonnet-4-6 for policy editor)
• Token usage (input/output/total)
• Tool call names, arguments, and results when the agent uses tools
• operation.name containing grc-assistant or policy-editor
• Finish reason on completion, error status on failures

If traces don't appear, just ping or there are any issues just ping us and we can take a look. You can also check https://docs.inference.net/integrations/traces/troubleshooting.

Checklist

(All affirmative bullets removed per the template's instructions.)

---

Summary by cubic

Add Inference.net Catalyst tracing to the GRC Assistant and Policy Editor agents so you can see model calls, tool usage, token counts, and errors in the Catalyst dashboard. This is a minimal integration to validate tracing before wider coverage.

• New Features

  • Instrumented two streamText calls with experimental_telemetry via getAITelemetry(...).
  • Bootstrapped tracing with initTracing() in both apps; added shutdownTracing() on API shutdown.
  • Service names: compai-grc-assistant and compai-policy-editor.
  • Added dependency @inference/tracing.

• Migration

  • Set CATALYST_OTLP_TOKEN and CATALYST_OTLP_ENDPOINT in both apps’ envs.
  • Restart API and web; send a few messages to each agent.
  • Verify spans in the Catalyst dashboard under the two service names.

^{Written for commit 5ae2d89. Summary will update on new commits. Review in cubic}

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[vercel]

@MJSPollard is attempting to deploy a commit to the Comp AI Team on Vercel.

A member of the Team first needs to authorize it.

[cubic-dev-ai]

No issues found across 11 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}