Migrate observability to Langfuse v4 SDK - #45
Open
kapitmax07 wants to merge 5 commits into
Open
Conversation
…tate gitignore The aws_secretsmanager_secret resource had been dropped locally in favor of a hardcoded secret_id on aws_secretsmanager_secret_version, which only creates a version inside an already-existing secret. A fresh terraform apply (as Guide 5 walks students through) would fail with ResourceNotFoundException since nothing actually creates the secret. Reverts the working tree back to the already-correct version committed on this branch. Also widen the tfstate gitignore pattern from *.tfstate.backup to *.tfstate.* so timestamped backup files (which contain DB credentials in plaintext) are excluded too.
Replace the old logfire-based tracing shim with the official Langfuse v4 Python SDK and OpenInference OpenAI Agents instrumentation, applied consistently to planner, tagger, reporter, charter, retirement, and the researcher App Runner service. - observability.py: idempotent setup_instrumentation(), observe()/ observation() context managers, PII masking (email/phone/card) via Langfuse's mask and mask_otel_spans hooks, and environment tagging (production/development). - Each lambda_handler.py now loads the job owner up front and tags traces with user_id/session_id/tags so agent runs group under one Langfuse session per job. - Reporter's judge/evaluator step now uses start_evaluator() with a proper NUMERIC score instead of a raw span. - Replace the old time.sleep(15) post-flush workaround with a real blocking client.flush() call before the Lambda execution environment freezes. - Add researcher/observability.py and a smoke-test script for App Runner, and wire LANGFUSE_* env vars into the 4_researcher and 6_agents Terraform modules. - Bump langfuse to >=4 and add openinference-instrumentation-openai-agents across all agent uv projects.
charter, reporter, and retirement each fetched the job once to set observe()'s user_id, then unconditionally re-fetched it again inside the handler and reassigned user_id from that second result. If the first lookup failed but the second succeeded, the Langfuse trace would be recorded with user_id=None while the agent actually ran for a known user. Reuse the already-fetched job instead of re-querying, which also drops a redundant Data API round-trip per invocation.
Fixes several issues found while debugging a failed local frontend start on Windows: a UTF-8 encoding crash when stdout isn't a real console, too-short a startup timeout for cold Next.js compiles, orphaned backend/frontend processes left behind because proc.terminate() only kills the immediate uv/cmd.exe wrapper (not its child server) on Windows, and false "process died" shutdowns triggered by that same wrapper exiting on its own while the real server keeps running fine.
…lock backend/api/main.py logs via logging.basicConfig(), which defaults to stderr. run_local.py captured that stderr with its own subprocess.PIPE but never read it, so once enough log lines accumulated the OS pipe buffer filled up and the backend's next logging call blocked forever - freezing its single-threaded event loop (and every request, including /health) with no error ever logged. Fix: merge backend stderr into stdout and continuously drain it via a background thread, matching the pattern already used for the frontend. Found and verified with py-spy: before the fix, a live hang showed the backend's only thread stuck inside logging.emit(); after the fix, 800 rapid requests left it idle in the normal asyncio event loop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan