java-obs-agent

A framework-agnostic Java observability sidecar agent — drop a single -javaagent jar onto any Java (Servlet) service, then chat with it (via a built-in web UI, an MCP client like Claude Code, or curl) to understand the service's live state: request volume, latency, slow calls, errors, logs, thread dumps, method-level traces.

It's the thing between Arthas (in-JVM diagnostics, no AI/aggregation) and SkyWalking (APM with distributed traces, no conversational agent) — plus the Datadog Bits AI pattern (chat-with-your-service), but self-hosted and attachable to a single JVM.

Built and validated end-to-end on a real Spring Boot service (the interview-guide project): attach → chat → "which route is slowest?" → agent calls its tools → answers with real metrics, then trace_request to pin the slow method in the call tree.

What it does

Attach the agent to a running Java service (no restart needed — dynamic attach like Arthas). The agent runs inside the target JVM:

Captures per-route request metrics (count, p50/p95/p99, error rate), slow calls, errors — into in-JVM ring buffers (zero external storage).
Captures recent logs (Logback appender) + thread snapshots (JMX).
Exposes a MCP server (JSON-RPC over HTTP) — any MCP client (Claude Code, Cursor) can chat with the service.
Exposes a built-in chat web UI (/chat) — an OpenAI-compatible LLM (BYO API key) orchestrates the tools to answer your questions in natural language.
On-demand method-level diagnostics: watch_method (capture a method's invocations) and trace_request (full method call-tree, Arthas-trace style).
Leaves a Source SPI seam so a future SkyWalking/Tempo backend plugs in without changing tools (Phase 3).

Quick start

1. Build

cd <repo-root>
JAVA_HOME=<path-to-jdk-21> ./gradlew :obs-agent-core:shadowJar

Produces obs-agent-core/build/libs/obs-agent-core-0.1.0-all.jar (shaded + relocated ByteBuddy/Jackson; slf4j uses the host's). Use the -all jar (has the manifest + deps); not the plain one.

JDK 21 (GraalVM 21.0.11) builds Java 17 bytecode. The system default java is JDK 8 — always set JAVA_HOME to JDK 21 for ./gradlew.

2. Attach to a running service (dynamic, no restart)

# find the service PID (e.g. on port 8080)
PID=$(netstat -ano | grep LISTENING | grep ":8080" | head -1 | awk '{print $NF}')

# attach the agent into the running JVM (Arthas-style)
JAVA_HOME=<path-to-jdk-21>
JAR="<repo-root>/obs-agent-core/build/libs/obs-agent-core-0.1.0-all.jar"
ARGS='port=9911,token=tok,enableInstrumentation=true,chat.apiKey=<YOUR_LLM_KEY>,chat.base_url=<https://...>,chat.model=<glm-4|gpt-4o-mini|...>'
"$JAVA_HOME/bin/java" --add-modules jdk.attach -cp scripts Attach "$PID" "$JAR" "$ARGS"

The agent starts an MCP server + (optional) chat UI on port 9911 inside the target JVM. It stays until the JVM restarts.

The Attach helper is a one-liner using com.sun.tools.attach.VirtualMachine.loadAgent (the agentmain entry point). It ships at scripts/Attach.java. To use it, compile once: "$JAVA_HOME/bin/javac" --add-modules jdk.attach -d scripts scripts/Attach.java.

2b. Or attach at JVM startup (`-javaagent`, persistent across restarts)

java -javaagent:<repo-root>/obs-agent-core/build/libs/obs-agent-core-0.1.0-all.jar=port=9911,token=tok,enableInstrumentation=true -jar your-app.jar

Use this if you want the agent resident from boot (survives restarts). Dynamic attach (above) is for ad-hoc diagnosis without restart.

3. Use it

Chat web UI (browser): open http://localhost:9911/chat?token=tok — a chat box; type questions, the LLM calls the tools and answers.

Claude Code (MCP client):

claude mcp add --transport http obs-agent http://localhost:9911/mcp --header "Authorization: Bearer tok"
# then chat: "which route is slowest? any recent errors?"

curl (manual tool call):

curl -X POST http://localhost:9911/mcp \
  -H "Authorization: Bearer tok" -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"recent_metrics","arguments":{"route":"/api/x"}}}'

All endpoints require Authorization: Bearer <token>. The chat UI page also accepts ?token=<token> (browsers can't send headers on URL navigation).

Configuration

All config is in the agent args (comma-separated key=value), optionally overridden by env vars (OBSAGENT_PORT, OBSAGENT_HOST, OBSAGENT_TOKEN).

Key	Default	Description
`port`	9000	MCP + chat HTTP port
`host`	`127.0.0.1`	Network interface to bind. Defaults to loopback (not exposed). Set `0.0.0.0` only behind a firewall/reverse proxy and with `token` set.
`token`	(none → no auth)	MCP/chat Bearer token. Set it if the port is exposed beyond localhost.
`window`	300s	Metrics/log retention window
`bucket`	1s	Metrics time bucket
`slowThreshold`	500ms	Slow-call threshold
`logBuffer`	2000	Log ring buffer size
`enableInstrumentation`	false	Master switch for `watch_method`/`trace_request` (instrumentation tier)
`watchLimit`	10	Max captures per `watch_method`
`watchTtl`	60s	`watch_method` auto-uninstall TTL
`scrubPatterns`	(none)	Regexes (semicolon-separated) to scrub from tool results (tokens/PII)
`chat.apiKey`	(none → chat off)	LLM API key. Set to enable the built-in chat.
`chat.base_url`	—	OpenAI-compatible base URL (e.g. `https://api.openai.com/v1`, `https://open.bigmodel.cn/api/paas/v4`, `https://ark.cn-beijing.volces.com/api/coding/v3`)
`chat.model`	—	Model name (e.g. `gpt-4o-mini`, `glm-4`, `glm-5.2`)

When chat.apiKey is unset, only MCP is exposed (chat UI returns 404).

Tools

Read-only (always available):

Tool	What it does
`get_routes`	Known routes + health (count, error rate, p95)
`recent_metrics(route?, window_seconds)`	Per-route count / p50/p95/p99 / error rate / qps
`slow_calls(top)`	Slowest recent requests (with traceId if host sets MDC)
`recent_errors(top)`	Recent error requests
`search_logs(query, level?, traceId?)`	Search the in-JVM log ring (host Logback)
`thread_dump(top, by=cpu\|blocked)`	Top CPU threads + deadlocks

Instrumentation tier (gated by enableInstrumentation=true, default off):

Tool	What it does
`watch_method(class, method, limit?, ttl_seconds?)`	Start capturing a method's invocations (args/return/duration/exception). Returns immediately (two-phase).
`watch_results(class, method)`	Read the captures from a prior `watch_method`.
`trace_request(class_pattern, ttl_seconds?, limit?)`	Start a method call-tree trace (Arthas-`trace`) for classes matching the regex. Returns immediately.
`trace_results()`	Read the call trees from a prior `trace_request`.

Two-phase design: watch_method/trace_request install + return immediately; the trace/watch stays active for ttl_seconds. Generate traffic (hit the endpoint, or rely on frontend traffic), then call watch_results/trace_results to read. This is critical — a single blocking call would prevent the caller (LLM) from generating traffic during the window (→ empty results).

trace_request usage

class_pattern is a regex. Use the FQCN for specific classes.
To see cross-class call trees (controller → service), trace both classes with |: e.g. class_pattern="interview.guide.modules.x.KnowledgeBaseController|interview.guide.modules.x.service.KnowledgeBaseListService".
Repeated trace_request calls reset the prior trace before installing the new one (no advice stacking).
Example flow (via chat or MCP):
1. trace_request("...KnowledgeBaseController|...KnowledgeBaseListService", ttl_seconds=15)
2. hit the endpoint a few times
3. trace_results() → tree: controller.getStatistics (335ms) → service.getStatistics (333ms)

Architecture

┌──────────────────────────────────────────────────────────┐
│  LLM clients (BYO): chat UI (/chat) / Claude Code / curl  │
└──────────────────┬───────────────────────────────────────┘
                   │ MCP (JSON-RPC over HTTP) + /chat (POST), token auth
┌──────────────────▼───────────────────────────────────────┐
│  obs-agent-core  (inside the target JVM, shaded jar)      │
│  ┌───────────┐  ┌────────────┐  ┌─────────────────────┐   │
│  │ MCP server│  │ ChatEndpoint│  │  ToolRegistry       │   │
│  │ (JdkHttp) │  │ (LlmClient, │  │ (10 tools)          │   │
│  │           │  │  agent loop)│  │                     │   │
│  └───────────┘  └────────────┘  └──────────┬──────────┘   │
│  ┌──────────────────────────────────────────▼──────────┐ │
│  │ Source SPI (extension seam):                        │ │
│  │ MetricsSource / LogSource / TraceSource /            │ │
│  │ TopologySource  (+ capability flags)                 │ │
│  └──────────────────────────┬───────────────────────────┘ │
│  ┌──────────────────────────▼──────────────────────────┐│
│  │ Phase 1 in-JVM impls: ring buffers / Logback / JMX   ││
│  └──────────────────────────────────────────────────────┘│
│  ┌──────────────────────────────────────────────────────┐│
│  │ ByteBuddy instrumentation (shaded + isolated):        ││
│  │ - ServletMetricsInstrumentation (javax+jakarta)       ││
│  │ - MethodInstrumenter (watch)  - MethodTracer (trace)   ││
│  └──────────────────────────────────────────────────────┘│
└────────────────────────────────────────────┬──────────────┘
              (future) ┌──────────────────────▼──────────────┐
                       │ Phase 3: SkyWalkingTraceSource etc.  │
                       │ (query OAP, zero-change to tools)    │
                       └──────────────────────────────────────┘

Key design points:

Single shaded jar — ByteBuddy/Jackson relocated to io.obsagent.shaded.*; slf4j is compileOnly (uses the host's, so the agent can read host Logback/MDC). No host-classpath pollution.
No new deps for chat — uses JDK java.net.http.HttpClient (OpenAI-compatible).
Advice never throws into user code — all ByteBuddy advice is wrapped in try/catch + JUL logging.
Advice-accessed members are public static — ByteBuddy inlines advice into target classes in other packages; package-private fields/methods would IllegalAccessError (a real gotcha — see Troubleshooting).
agentmain — enables dynamic attach to a running JVM (not just -javaagent at startup).

Roadmap

✅ Phase 1 — core agent: servlet metrics + ring buffers + Source SPI + in-JVM sources + MCP server + read-only tools + scrubber/guard + shaded jar + e2e smoke test.
✅ Phase 2 — watch_method (method capture) + trace_request (call-tree) + validation demo.
✅ Built-in chat — LlmClient (OpenAI-compatible) + ChatEndpoint (agent loop) + web UI at /chat.
✅ Dynamic attach (agentmain) + two-phase trace/watch + trace reset (no stacking) + cross-class call trees.
⏳ Phase 3 (deferred) — SkyWalkingTraceSource/SkyWalkingTopologySource (query OAP) → distributed_trace/service_topology tools → cross-service business-chain root cause (order→pay). The Source SPI is already in place.
⏳ Optional tail — Spring Boot adapter (route templates/Actuator/Micrometer traceId); streaming chat (token-level SSE); obs-agent-cli; reactive/WebFlux; Java 8 bytecode broadening.

Known limitations

trace_request is pattern-based, not traceId-based — the host must set MDC traceId (e.g. Micrometer Tracing) for slow_calls/search_logs to correlate by traceId. Without it, those fields are null (route+latency diagnosis still works).
Broad .* patterns capture the entry method but may not show cross-class callees as children — trace the specific classes (with |) for cross-class trees.
Chat is non-streaming (POST /chat/completions, stream=false) — returns the full answer after the tool-calling loop. Live token streaming is a follow-up.
Servlet-only (blocking) — reactive/WebFlux not yet instrumented.
Java 17 bytecode — broadening to Java 8 (attach to older JVMs) is a follow-up.
Agent can't be hot-updated in a running JVM — re-attaching a new jar doesn't redefine already-loaded classes (system classloader caches them). To pick up code changes, restart the target JVM + re-attach (or use -javaagent at startup).
watch_method/trace_request default OFF — set enableInstrumentation=true to use them.

Troubleshooting

MCP client (Claude Code) fails to connect with "expected object, received undefined" for capabilities/serverInfo — the initialize result must include both (MCP spec). Fixed (returns {"protocolVersion","capabilities":{"tools":{}},"serverInfo":{"name":"obs-agent","version":"0.1.0"}}).

MCP tools return "content expected array, received object" / content blocks fail validation — tools/call result must be {"content":[{"type":"text","text":"<json>"}],"isError":false} (array of content blocks, each with a type). Fixed.

search_logs returns "no log source" / traceId always null — slf4j must NOT be shaded (so the agent uses the host's slf4j to reach Logback + MDC). slf4j is compileOnly + not relocated. If you see this, check the shadow config.

Browser opens /chat → {"code":-32001,"message":"unauthorized"} — browsers can't send Authorization headers on URL navigation. Open http://localhost:9911/chat?token=tok (query param accepted).

trace_request returns empty {"trees":[]} — (1) it's two-phase: call trace_request (install), generate traffic, then trace_results (read). The install call returns immediately, it does NOT capture. (2) Make sure enableInstrumentation=true. (3) Verify the class_pattern regex matches (use the FQCN; . matches any char, * is a quantifier).

trace_request shows duplicate/nested trees — fixed: repeated trace_request now reset()s the prior transformer before installing. If you still see it, you're on an old jar (rebuild + re-attach).

ByteBuddy advice silently does nothing (empty captures, no error) — the advice accesses package-private members from a cross-package target → IllegalAccessError swallowed by the try/catch → returns null. Advice logic must be in public static methods (e.g. onEnter/onExit) so the inlined advice only calls public methods. The unit test passes (same-package test class) but live cross-package targets fail — always verify on a fresh JVM with a real cross-package target.

Build fails with JDK issues — system java is JDK 8; the project needs JDK 21. Always run ./gradlew with JAVA_HOME=<path-to-jdk-21>.

Re-attaching a second agent doesn't apply new code — the first attach's classes are already loaded by the system classloader; loadAgent won't redefine them. Restart the target JVM + attach once.

Development

# build
JAVA_HOME="..." ./gradlew :obs-agent-core:build
# tests
JAVA_HOME="..." ./gradlew :obs-agent-core:test
# e2e smoke (real -javaagent attach + MCP)
JAVA_HOME="..." ./gradlew :obs-agent-core:test --tests 'io.obsagent.EndToEndSmokeTest' -Pe2e=true
# shaded agent jar
JAVA_HOME="..." ./gradlew :obs-agent-core:shadowJar

Layout:

obs-agent-core/src/main/java/io/obsagent/ — AgentPremain, config/, data/ (ring buffers + model records), source/ (SPI + injvm/ impls), instr/ (ByteBuddy: servlet/watch/trace), mcp/ (JdkHttpMcpServer), tools/ (ReadOnlyTools + InstrumentingTools), safety/ (Scrubber + Guard), chat/ (LlmClient + ChatEndpoint + ChatPage).
docs/superpowers/specs/ — design spec.
docs/superpowers/plans/ — Phase 1 / Phase 2 / chat+trace implementation plans.
.superpowers/sdd/progress.md — SDD execution ledger (gitignored).

How it was built

Designed via brainstorming → spec → implementation plans → subagent-driven development (12 Phase-1 tasks + 4 Phase-2/chat-trace tasks), each task = implementer subagent + review + fixes. ~40 commits, 29 main + 20 test Java files. Validated live on the real interview-guide Spring Boot backend (dynamic attach → MCP chat → real metrics → trace_request call-tree pinning the slow method).

See docs/superpowers/specs/ and docs/superpowers/plans/ for the full design + task breakdown.

Name		Name	Last commit message	Last commit date
Latest commit History 55 Commits
.github		.github
docs/superpowers		docs/superpowers
gradle/wrapper		gradle/wrapper
obs-agent-core		obs-agent-core
scripts		scripts
.gitignore		.gitignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
README.zh-CN.md		README.zh-CN.md
SECURITY.md		SECURITY.md
build.gradle		build.gradle
gradle.properties		gradle.properties
gradlew		gradlew
gradlew.bat		gradlew.bat
settings.gradle		settings.gradle

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

java-obs-agent

What it does

Quick start

1. Build

2. Attach to a running service (dynamic, no restart)

2b. Or attach at JVM startup (`-javaagent`, persistent across restarts)

3. Use it

Configuration

Tools

trace_request usage

Architecture

Roadmap

Known limitations

Troubleshooting

Development

How it was built

About

Uh oh!

Releases 1

Packages

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

java-obs-agent

What it does

Quick start

1. Build

2. Attach to a running service (dynamic, no restart)

2b. Or attach at JVM startup (-javaagent, persistent across restarts)

3. Use it

Configuration

Tools

trace_request usage

Architecture

Roadmap

Known limitations

Troubleshooting

Development

How it was built

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 1

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

2b. Or attach at JVM startup (`-javaagent`, persistent across restarts)

Packages