Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Hermes EvalOps Plugin — environment configuration
# Copy to .env and fill in the two tokens.

# --- Gateway (live — routes LLM calls through EvalOps) ---
HERMES_EVALOPS_GATEWAY_URL=https://llm-gateway.evalops.dev/v1
HERMES_EVALOPS_GATEWAY_TOKEN=
HERMES_EVALOPS_GATEWAY_MODEL=evalops-default

# --- Platform (gated on EVA-157 — agent-mcp not publicly exposed yet) ---
HERMES_EVALOPS_PLATFORM_URL=https://api.evalops.dev
HERMES_EVALOPS_PLATFORM_TOKEN=
HERMES_EVALOPS_ORGANIZATION_ID=org_evalops
HERMES_EVALOPS_WORKSPACE_ID=evalops

# --- Agent identity ---
HERMES_EVALOPS_AGENT_ID=hermes-evalops
HERMES_EVALOPS_AGENT_TYPE=hermes
HERMES_EVALOPS_AGENT_NAME="Hermes EvalOps"
HERMES_EVALOPS_AGENT_VERSION=0.1.0
HERMES_EVALOPS_AUTH_PROVIDER=google

# --- Registration & span ingest (defaults: MCP stubs) ---
# HERMES_EVALOPS_REGISTRATION_ENDPOINT=mcp
# HERMES_EVALOPS_SPAN_INGEST_ENDPOINT=mcp:self_diagnostic

# --- Safety knobs ---
# HERMES_EVALOPS_CAPTURE_PAYLOADS=0
# HERMES_EVALOPS_TIMEOUT_SECONDS=10
# HERMES_EVALOPS_ENABLED=1
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ __pycache__/
build/
dist/
*.egg-info/
.env

31 changes: 19 additions & 12 deletions docs/install-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,17 +46,18 @@ trace.
Required for live gateway calls:

```sh
export HERMES_EVALOPS_GATEWAY_URL="https://gateway.evalops.example/v1/chat/completions"
export HERMES_EVALOPS_GATEWAY_TOKEN="..."
export HERMES_EVALOPS_GATEWAY_URL="https://llm-gateway.evalops.dev/v1"
export HERMES_EVALOPS_GATEWAY_TOKEN="..." # Identity bootstrap key or issued agent token
```

Required for live registration/span ingest:
Required for live registration/span ingest (gated on EVA-157 — agent-mcp not
yet publicly exposed):

```sh
export HERMES_EVALOPS_PLATFORM_URL="https://platform.evalops.example"
export HERMES_EVALOPS_PLATFORM_TOKEN="..."
export HERMES_EVALOPS_ORGANIZATION_ID="org_..."
export HERMES_EVALOPS_WORKSPACE_ID="ws_..."
export HERMES_EVALOPS_PLATFORM_URL="https://api.evalops.dev"
export HERMES_EVALOPS_PLATFORM_TOKEN="..." # Same identity token
export HERMES_EVALOPS_ORGANIZATION_ID="org_evalops"
export HERMES_EVALOPS_WORKSPACE_ID="evalops"
```

Useful knobs:
Expand All @@ -69,16 +70,22 @@ export HERMES_EVALOPS_CAPTURE_PAYLOADS="0"
export HERMES_EVALOPS_TIMEOUT_SECONDS="10"
```

EVA-157 swap points:
EVA-157 swap points (current defaults use in-process MCP stubs):

```sh
export HERMES_EVALOPS_REGISTRATION_ENDPOINT="/v1/agent-mcp/external-agents/register"
# Current defaults — run against local stubs with no network or credentials:
# HERMES_EVALOPS_REGISTRATION_ENDPOINT=mcp
# HERMES_EVALOPS_SPAN_INGEST_ENDPOINT=mcp:self_diagnostic
#
# Once EVA-157 lands and agent-mcp is publicly exposed, swap to:
export HERMES_EVALOPS_REGISTRATION_ENDPOINT="mcp"
export HERMES_EVALOPS_SPAN_INGEST_ENDPOINT="/v1/agent-runtime/external-spans"
```

Those defaults are intentionally contract-shaped placeholders. Once EVA-157
lands, replace them with the final scoped `agent-mcp` registration and span
ingest endpoints, or replace `EvalOpsPlatformClient` with generated clients.
The `mcp` registration endpoint connects to the platform's agent-mcp
StreamableHTTP endpoint. The `mcp:self_diagnostic` span ingest posts tool spans
as self-diagnostic events through the same MCP session. Once EVA-157 ships,
replace `mcp:self_diagnostic` with the scoped span ingest path.

## Runtime Behavior

Expand Down
Loading