diff --git a/.env.example b/.env.example index a65f7824a..bb0892d1a 100644 --- a/.env.example +++ b/.env.example @@ -36,7 +36,8 @@ POWERCONTEXT_SERVER_LOGGING_FORMAT=console POWERCONTEXT_SERVER_LOGGING_ACCESS=true POWERCONTEXT_SERVER_METRICS_ENABLED=true POWERCONTEXT_SERVER_TRACING_ENABLED=false -# OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:6006/v1/traces +# The OpenTelemetry SDK appends /v1/traces to this base URL. +# OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:6006 # OTEL_SERVICE_NAME=powercontext # Database -------------------------------------------------------------------- diff --git a/docs/en/docs/how-to/trace-with-langfuse.md b/docs/en/docs/how-to/trace-with-langfuse.md new file mode 100644 index 000000000..1ac9c0bfd --- /dev/null +++ b/docs/en/docs/how-to/trace-with-langfuse.md @@ -0,0 +1,141 @@ +--- +title: Trace with Langfuse +description: Export PowerContext transport, application, and inference spans to Langfuse through standard OTLP configuration. +--- + +# Trace with Langfuse + +PowerContext exports OpenTelemetry spans for transport and application operations. When tracing is enabled, the +generation and embedding calls that PowerContext itself constructs are traced too, so one trace shows the request, the +Memory operation, and the model calls underneath it. + +This guide sends those spans to [Langfuse](https://langfuse.com) through its OTLP endpoint. It needs no PowerContext +code change and no Langfuse SDK: the standard OpenTelemetry variables from [Trace with Phoenix](trace-with-phoenix.md) +point the exporter at Langfuse instead. + +## Start Langfuse + +Langfuse self-hosting runs several services (web, worker, PostgreSQL, ClickHouse, Redis, and MinIO) with Docker +Compose: + +```bash +git clone https://github.com/langfuse/langfuse.git +cd langfuse +docker compose up -d +``` + +Open , create a user, an organization, and a project, then create an API key pair in the project +settings. Keep the public key (`pk-lf-...`) and the secret key (`sk-lf-...`) at hand; they authenticate the exporter +below. The OTLP endpoint requires Langfuse v3.22.0 or later. This guide was verified with Langfuse 4.10.0. + +For a reproducible local setup, [headless initialization](https://langfuse.com/self-hosting/headless-initialization) +creates the organization, project, user, and keys from environment variables instead of the UI. Langfuse Cloud works +the same way as a self-hosted instance: skip the compose step and replace `http://localhost:3000` below with the base +URL of your region, such as `https://cloud.langfuse.com` or `https://us.cloud.langfuse.com`. + +## Install the export dependency + +Recording and export require the `tracing-otlp` extra: + +```bash +uv tool install --force "powercontext[cli,server,tracing-otlp] @ git+https://github.com/oceanbase/powercontext.git@master" +``` + +Without this extra, enabling tracing fails at startup with an explicit error instead of silently dropping spans. + +## Configure and start the Server + +Langfuse authenticates OTLP requests with HTTP Basic authentication built from the project keys. Enable tracing, point +the exporter at Langfuse, and configure a generation model so inference spans have something to record: + +```bash +export LANGFUSE_PUBLIC_KEY=pk-lf-replace-me +export LANGFUSE_SECRET_KEY=sk-lf-replace-me +LANGFUSE_AUTH=$(printf '%s:%s' "$LANGFUSE_PUBLIC_KEY" "$LANGFUSE_SECRET_KEY" | base64 | tr -d '\n') + +export POWERCONTEXT_SERVER_TRACING_ENABLED=true +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:3000/api/public/otel +export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic ${LANGFUSE_AUTH},x-langfuse-ingestion-version=4" +export OTEL_SERVICE_NAME=powercontext-server +export POWERCONTEXT_SERVER_INFERENCE_GENERATION_MODEL=provider:model-name +powercontext server run +``` + +The OpenTelemetry SDK appends `/v1/traces` to `OTEL_EXPORTER_OTLP_ENDPOINT`, so the spans arrive at +`http://localhost:3000/api/public/otel/v1/traces`, the traces endpoint Langfuse expects. Langfuse accepts OTLP over +HTTP only, which is the protocol of the exporter installed by the `tracing-otlp` extra. The +`x-langfuse-ingestion-version=4` header makes Langfuse process the spans immediately; without it, Langfuse documents +that ingestion can lag by up to ten minutes. Set the provider credentials your generation model needs; PowerContext +records neither them nor the exporter headers. + +## Trigger one inference request + +Set `POWERCONTEXT_SCOPE_ID` to an existing ID returned by `create_scope`, capture a Source, then convert it into +Memory: + +```bash +curl -X POST http://localhost:8000/v1/sources/content \ + -H 'content-type: application/json' \ + -d "{\"scope_id\":\"${POWERCONTEXT_SCOPE_ID}\",\"source_id\":\"task-1\",\"content\":\"I always book aisle seats.\"}" +``` + +```bash +curl -X POST http://localhost:8000/v1/memory/flush \ + -H 'content-type: application/json' \ + -d "{\"scope_id\":\"${POWERCONTEXT_SCOPE_ID}\"}" +``` + +Memory extraction runs during the flush, not during capture. + +## Read the trace + +Open , select the project, and open the **Traces** view. Langfuse names a trace after its root +span, so the flush appears as `HTTP flush_memory`. Every PowerContext span becomes an observation, and Langfuse infers +the observation type from the GenAI attributes on the span: + +| Observation | Type | Meaning | +| --- | --- | --- | +| `HTTP flush_memory` | SPAN | The inbound HTTP request. Its `attributes.powercontext.request.id` metadata matches the `X-PowerContext-Request-ID` response header. | +| `powercontext flush_memory` | SPAN | The application operation, independent of the transport that invoked it. | +| `memory.flush` | SPAN | The Runtime stage that processes the Source window. The other stage spans, such as `scope.context`, `scope.lock`, `memory.search`, and `context.build`, are SPAN observations as well. | +| `memory_extraction run` | AGENT | One PowerContext generation task. Langfuse names it from the span's `logfire.msg` attribute, so Pydantic AI's `invoke_agent memory_extraction` span appears under this name. | +| `chat ` | GENERATION | One request to the model provider, with the model name, latency, and input, output, and total token usage. | + +The other generation tasks follow the same pattern with their own names, such as `experience_incubation run` and +`memory_rerank run`. + +An MCP request produces `MCP mcp.tools.call` as the root observation. FastMCP adds a `TOOL` observation named after the +tool, and the `powercontext ` span and its stages nest beneath it. Readiness probes are deliberately not +traced. + +Span attributes appear in each observation's metadata as `attributes.`, and resource attributes as +`resourceAttributes.`. To find the trace of one request, filter observations on the metadata key +`attributes.powercontext.request.id` with the value of the `X-PowerContext-Request-ID` response header. Failed +operations carry the `ERROR` level and `attributes.error.type`. + +Langfuse derives the cost of a generation from its model definitions, which match the model name; models it does not +recognize show usage but no cost until you add a definition under the project's model settings. Token usage and cost +can then be aggregated in the Langfuse dashboards and Metrics API. + +Spans are exported in batches, so allow a few seconds before refreshing. Scheduled background activations arrive as +their own traces, as described in the "Scheduled background spans" section of +[Trace with Phoenix](trace-with-phoenix.md). + +## What is not exported + +PowerContext configures inference instrumentation to exclude content. Observations carry model identifiers, token +usage, durations, and error categories. Prompts, model responses, Memory content, search queries, and vectors are +excluded, so the input and output panels of a generation show only the role and part types of each message, never its +text. PowerContext sets no Langfuse user, session, or tag attributes either, so the user and session views stay empty +and traces are located through metadata instead. + +## Stop Langfuse + +```bash +docker compose down +``` + +Add `-v` to delete the stored traces as well. + +Span names and attributes follow the Pydantic AI GenAI semantic conventions and can change when that dependency is +upgraded across a major version. Do not treat them as a stable contract. diff --git a/docs/en/docs/reference/configuration.md b/docs/en/docs/reference/configuration.md index 87583939e..f26803316 100644 --- a/docs/en/docs/reference/configuration.md +++ b/docs/en/docs/reference/configuration.md @@ -277,7 +277,9 @@ that do not use the `powercontext` command may omit the `cli` extra. Enabling tracing also produces spans for the generation and embedding calls that PowerContext constructs, without recording prompts, model responses, Memory content, or vectors. See -[Trace with Phoenix](../how-to/trace-with-phoenix.md) for a working configuration. +[Trace with Phoenix](../how-to/trace-with-phoenix.md) for a working configuration, and +[Trace with Langfuse](../how-to/trace-with-langfuse.md) for a backend that authenticates the exporter through +`OTEL_EXPORTER_OTLP_HEADERS`. To use OceanBase, provide its URL through your environment or secret manager: diff --git a/docs/zh/docs/how-to/trace-with-langfuse.md b/docs/zh/docs/how-to/trace-with-langfuse.md new file mode 100644 index 000000000..2338fb7d7 --- /dev/null +++ b/docs/zh/docs/how-to/trace-with-langfuse.md @@ -0,0 +1,129 @@ +--- +title: 用 Langfuse 查看 trace +description: 通过标准 OTLP 配置，把 PowerContext 的 transport、application 和推理 span 导出到 Langfuse。 +--- + +# 用 Langfuse 查看 trace + +PowerContext 会为 transport 和 application 操作导出 OpenTelemetry span。启用 tracing 后，PowerContext 自己构造的 +generation 与 embedding 调用也会被 trace，因此一条 trace 里可以同时看到请求、Memory 操作，以及其下的模型调用。 + +本文把这些 span 通过 OTLP 端点发送到 [Langfuse](https://langfuse.com)。整个过程不需要改动 PowerContext 代码，也不需要 +Langfuse SDK：只是把 [用 Phoenix 查看 trace](trace-with-phoenix.md) 中的标准 OpenTelemetry 变量改为指向 Langfuse。 + +## 启动 Langfuse + +Langfuse 自托管通过 Docker Compose 运行多个服务（web、worker、PostgreSQL、ClickHouse、Redis 和 MinIO）： + +```bash +git clone https://github.com/langfuse/langfuse.git +cd langfuse +docker compose up -d +``` + +打开 ，创建用户、organization 和 project，然后在 project 设置里创建一对 API key。记下 public key +（`pk-lf-...`）和 secret key（`sk-lf-...`），下文用它们为 exporter 鉴权。OTLP 端点要求 Langfuse v3.22.0 及以上；本文 +在 Langfuse 4.10.0 上验证。 + +如需可复现的本地环境，[headless initialization](https://langfuse.com/self-hosting/headless-initialization) 可以通过 +环境变量直接创建 organization、project、用户和 key，而不必经过 UI。Langfuse Cloud 的用法与自托管相同：跳过 compose +步骤，把下文的 `http://localhost:3000` 换成所在区域的基础 URL，例如 `https://cloud.langfuse.com` 或 +`https://us.cloud.langfuse.com`。 + +## 安装导出依赖 + +recording 和 export 需要 `tracing-otlp` extra： + +```bash +uv tool install --force "powercontext[cli,server,tracing-otlp] @ git+https://github.com/oceanbase/powercontext.git@master" +``` + +缺少该 extra 时，启用 tracing 会在启动阶段直接报错，而不是静默丢弃 span。 + +## 配置并启动 Server + +Langfuse 用 project key 组成的 HTTP Basic 认证来鉴权 OTLP 请求。启用 tracing、把 exporter 指向 Langfuse，并配置一个 +generation model，让推理 span 有内容可记录： + +```bash +export LANGFUSE_PUBLIC_KEY=pk-lf-replace-me +export LANGFUSE_SECRET_KEY=sk-lf-replace-me +LANGFUSE_AUTH=$(printf '%s:%s' "$LANGFUSE_PUBLIC_KEY" "$LANGFUSE_SECRET_KEY" | base64 | tr -d '\n') + +export POWERCONTEXT_SERVER_TRACING_ENABLED=true +export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:3000/api/public/otel +export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic ${LANGFUSE_AUTH},x-langfuse-ingestion-version=4" +export OTEL_SERVICE_NAME=powercontext-server +export POWERCONTEXT_SERVER_INFERENCE_GENERATION_MODEL=provider:model-name +powercontext server run +``` + +OpenTelemetry SDK 会在 `OTEL_EXPORTER_OTLP_ENDPOINT` 后追加 `/v1/traces`，因此 span 最终发往 +`http://localhost:3000/api/public/otel/v1/traces`，正是 Langfuse 期望的 traces 端点。Langfuse 只接受 OTLP over HTTP， +与 `tracing-otlp` extra 安装的 exporter 协议一致。`x-langfuse-ingestion-version=4` 头让 Langfuse 立即处理这些 +span；Langfuse 文档指出，缺少该头时摄入最多可能延迟十分钟。按所选 generation model 的要求设置 provider 凭据； +PowerContext 既不会记录凭据，也不会记录 exporter 的请求头。 + +## 触发一次推理请求 + +将 `POWERCONTEXT_SCOPE_ID` 设置为 `create_scope` 返回的已有 ID，先捕获一个 Source，再把它转成 Memory： + +```bash +curl -X POST http://localhost:8000/v1/sources/content \ + -H 'content-type: application/json' \ + -d "{\"scope_id\":\"${POWERCONTEXT_SCOPE_ID}\",\"source_id\":\"task-1\",\"content\":\"I always book aisle seats.\"}" +``` + +```bash +curl -X POST http://localhost:8000/v1/memory/flush \ + -H 'content-type: application/json' \ + -d "{\"scope_id\":\"${POWERCONTEXT_SCOPE_ID}\"}" +``` + +Memory extraction 发生在 flush 阶段，而不是捕获阶段。 + +## 查看 trace + +打开 ，选择 project，进入 **Traces** 视图。Langfuse 用根 span 命名 trace，因此这次 flush 显示为 +`HTTP flush_memory`。PowerContext 的每个 span 都会成为一个 observation，Langfuse 根据 span 上的 GenAI 属性推断 +observation 类型： + +| Observation | 类型 | 含义 | +| --- | --- | --- | +| `HTTP flush_memory` | SPAN | 入站 HTTP 请求。其 metadata 中的 `attributes.powercontext.request.id` 与响应头 `X-PowerContext-Request-ID` 一致。 | +| `powercontext flush_memory` | SPAN | application 操作，与调用它的 transport 无关。 | +| `memory.flush` | SPAN | 实际处理 Source window 的 Runtime stage。其他 stage span（如 `scope.context`、`scope.lock`、`memory.search`、`context.build`）同样是 SPAN observation。 | +| `memory_extraction run` | AGENT | 一次 PowerContext generation 任务。Langfuse 取 span 的 `logfire.msg` 属性作为名称，因此 Pydantic AI 的 `invoke_agent memory_extraction` span 以这个名字出现。 | +| `chat ` | GENERATION | 一次发往模型 provider 的请求，包含模型名、耗时，以及 input、output 和 total token 用量。 | + +其他 generation 任务遵循同样的模式，例如 `experience_incubation run` 和 `memory_rerank run`。 + +MCP 请求以 `MCP mcp.tools.call` 作为根 observation。FastMCP 会添加一个以工具名命名的 `TOOL` observation， +`powercontext ` span 及其 stage 嵌套在其下。readiness 探活被有意排除在 trace 之外。 + +span 属性以 `attributes.` 的形式出现在每个 observation 的 metadata 中，resource 属性则是 +`resourceAttributes.`。要定位某次请求的 trace，请用响应头 `X-PowerContext-Request-ID` 的值过滤 metadata key +`attributes.powercontext.request.id`。失败的操作带有 `ERROR` level 和 `attributes.error.type`。 + +Langfuse 根据模型定义匹配模型名来推算 generation 成本；未识别的模型只显示用量而没有成本，直到你在 project 的模型 +设置中添加定义。之后即可在 Langfuse 的 dashboard 与 Metrics API 中汇总 token 用量和成本。 + +span 是批量导出的，刷新前请稍等几秒。定时后台激活会以独立 trace 到达，见 +[用 Phoenix 查看 trace](trace-with-phoenix.md) 中的「定时后台 span」一节。 + +## 哪些内容不会被导出 + +PowerContext 在配置推理 instrumentation 时关闭了内容记录。observation 只携带模型标识、token 用量、耗时和错误类别； +prompt、模型响应、Memory 内容、搜索 query 和向量都不会被导出，因此 generation 的 input 与 output 面板只显示每条 +消息的 role 和 part 类型，不会显示正文。PowerContext 也不设置 Langfuse 的 user、session 或 tag 属性，因此用户与 +会话视图保持为空，trace 需要通过 metadata 定位。 + +## 停止 Langfuse + +```bash +docker compose down +``` + +追加 `-v` 可同时删除已存储的 trace。 + +span 名与属性遵循 Pydantic AI 的 GenAI 语义约定，跨大版本升级该依赖时可能变化，不应视为稳定契约。 diff --git a/docs/zh/docs/reference/configuration.md b/docs/zh/docs/reference/configuration.md index 0991b55ec..8999fb7d8 100644 --- a/docs/zh/docs/reference/configuration.md +++ b/docs/zh/docs/reference/configuration.md @@ -263,7 +263,8 @@ OpenTelemetry 环境变量进行配置。不使用 `powercontext` command 的 pr `cli` extra。 启用 tracing 后，PowerContext 自己构造的 generation 与 embedding 调用也会产生 span，且不记录 prompt、模型响应、 -Memory 内容或向量。可运行的配置见 [用 Phoenix 查看 trace](../how-to/trace-with-phoenix.md)。 +Memory 内容或向量。可运行的配置见 [用 Phoenix 查看 trace](../how-to/trace-with-phoenix.md)；需要通过 +`OTEL_EXPORTER_OTLP_HEADERS` 为 exporter 鉴权的后端示例见 [用 Langfuse 查看 trace](../how-to/trace-with-langfuse.md)。 使用 OceanBase 时，通过环境或 secret manager 提供 URL：