From 34614c34a79b6d1f663e4f021d8c772935a2b303 Mon Sep 17 00:00:00 2001 From: Jonathan Haas Date: Tue, 9 Jun 2026 06:06:46 +0000 Subject: [PATCH] docs: fill in discovered platform env values and add .env.example Replace example URLs/IDs with actual values discovered from evalops/deploy and evalops/platform: - Gateway URL: https://llm-gateway.evalops.dev/v1 - Platform URL: https://api.evalops.dev - Organization ID: org_evalops - Workspace ID: evalops Also add .env.example for quick setup and note that registration/span ingest are gated on EVA-157 (agent-mcp not publicly exposed). --- .env.example | 29 +++++++++++++++++++++++++++++ .gitignore | 1 + docs/install-config.md | 31 +++++++++++++++++++------------ 3 files changed, 49 insertions(+), 12 deletions(-) create mode 100644 .env.example diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..61cf482 --- /dev/null +++ b/.env.example @@ -0,0 +1,29 @@ +# Hermes EvalOps Plugin — environment configuration +# Copy to .env and fill in the two tokens. + +# --- Gateway (live — routes LLM calls through EvalOps) --- +HERMES_EVALOPS_GATEWAY_URL=https://llm-gateway.evalops.dev/v1 +HERMES_EVALOPS_GATEWAY_TOKEN= +HERMES_EVALOPS_GATEWAY_MODEL=evalops-default + +# --- Platform (gated on EVA-157 — agent-mcp not publicly exposed yet) --- +HERMES_EVALOPS_PLATFORM_URL=https://api.evalops.dev +HERMES_EVALOPS_PLATFORM_TOKEN= +HERMES_EVALOPS_ORGANIZATION_ID=org_evalops +HERMES_EVALOPS_WORKSPACE_ID=evalops + +# --- Agent identity --- +HERMES_EVALOPS_AGENT_ID=hermes-evalops +HERMES_EVALOPS_AGENT_TYPE=hermes +HERMES_EVALOPS_AGENT_NAME="Hermes EvalOps" +HERMES_EVALOPS_AGENT_VERSION=0.1.0 +HERMES_EVALOPS_AUTH_PROVIDER=google + +# --- Registration & span ingest (defaults: MCP stubs) --- +# HERMES_EVALOPS_REGISTRATION_ENDPOINT=mcp +# HERMES_EVALOPS_SPAN_INGEST_ENDPOINT=mcp:self_diagnostic + +# --- Safety knobs --- +# HERMES_EVALOPS_CAPTURE_PAYLOADS=0 +# HERMES_EVALOPS_TIMEOUT_SECONDS=10 +# HERMES_EVALOPS_ENABLED=1 diff --git a/.gitignore b/.gitignore index 86d424c..2509e72 100644 --- a/.gitignore +++ b/.gitignore @@ -7,4 +7,5 @@ __pycache__/ build/ dist/ *.egg-info/ +.env diff --git a/docs/install-config.md b/docs/install-config.md index 88e56c4..86f5f91 100644 --- a/docs/install-config.md +++ b/docs/install-config.md @@ -46,17 +46,18 @@ trace. Required for live gateway calls: ```sh -export HERMES_EVALOPS_GATEWAY_URL="https://gateway.evalops.example/v1/chat/completions" -export HERMES_EVALOPS_GATEWAY_TOKEN="..." +export HERMES_EVALOPS_GATEWAY_URL="https://llm-gateway.evalops.dev/v1" +export HERMES_EVALOPS_GATEWAY_TOKEN="..." # Identity bootstrap key or issued agent token ``` -Required for live registration/span ingest: +Required for live registration/span ingest (gated on EVA-157 — agent-mcp not +yet publicly exposed): ```sh -export HERMES_EVALOPS_PLATFORM_URL="https://platform.evalops.example" -export HERMES_EVALOPS_PLATFORM_TOKEN="..." -export HERMES_EVALOPS_ORGANIZATION_ID="org_..." -export HERMES_EVALOPS_WORKSPACE_ID="ws_..." +export HERMES_EVALOPS_PLATFORM_URL="https://api.evalops.dev" +export HERMES_EVALOPS_PLATFORM_TOKEN="..." # Same identity token +export HERMES_EVALOPS_ORGANIZATION_ID="org_evalops" +export HERMES_EVALOPS_WORKSPACE_ID="evalops" ``` Useful knobs: @@ -69,16 +70,22 @@ export HERMES_EVALOPS_CAPTURE_PAYLOADS="0" export HERMES_EVALOPS_TIMEOUT_SECONDS="10" ``` -EVA-157 swap points: +EVA-157 swap points (current defaults use in-process MCP stubs): ```sh -export HERMES_EVALOPS_REGISTRATION_ENDPOINT="/v1/agent-mcp/external-agents/register" +# Current defaults — run against local stubs with no network or credentials: +# HERMES_EVALOPS_REGISTRATION_ENDPOINT=mcp +# HERMES_EVALOPS_SPAN_INGEST_ENDPOINT=mcp:self_diagnostic +# +# Once EVA-157 lands and agent-mcp is publicly exposed, swap to: +export HERMES_EVALOPS_REGISTRATION_ENDPOINT="mcp" export HERMES_EVALOPS_SPAN_INGEST_ENDPOINT="/v1/agent-runtime/external-spans" ``` -Those defaults are intentionally contract-shaped placeholders. Once EVA-157 -lands, replace them with the final scoped `agent-mcp` registration and span -ingest endpoints, or replace `EvalOpsPlatformClient` with generated clients. +The `mcp` registration endpoint connects to the platform's agent-mcp +StreamableHTTP endpoint. The `mcp:self_diagnostic` span ingest posts tool spans +as self-diagnostic events through the same MCP session. Once EVA-157 ships, +replace `mcp:self_diagnostic` with the scoped span ingest path. ## Runtime Behavior