fix(runtime-metrics): resolve runtime-id at flush time so forked workers tag their own series - #19850
fix(runtime-metrics): resolve runtime-id at flush time so forked workers tag their own series#19850UgaTheDev wants to merge 1 commit into
Conversation
When runtime metrics are enabled before a fork -- gunicorn/uWSGI preload, or single-step instrumentation enabling products in the master -- every worker inherits the parent's RuntimeWorker instance. The runtime-id tag was resolved once in __init__ and stored in _platform_tags, so all N+1 processes emitted runtime.python.* on a single series. Sum-by-runtime-id over a pod then read as the size of the master process alone; the reporter measured 0.10 of the true sum of per-process RSS with 4 workers. Resolve runtime-id per flush instead. The forksafe hook in ddtrace.internal.runtime already regenerates the id in the child, and the process metric collector already rebinds to the current process on fork, so this was the last piece still pinned to the parent. PlatformTagCollectorV2 existed only to append that snapshotted tag and is removed with it. Fixes DataDog#19526 (issue A) Signed-off-by: Kush Zingade <kush.zingade@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e6242b5e3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| self._platform_tags = self._format_tags(PlatformTags()) | ||
| self._send_runtime_id = config._runtime_metrics_runtime_id_enabled |
There was a problem hiding this comment.
Update the API test for the moved runtime-id tag
When runtime-id tagging is enabled, this now deliberately leaves runtime-id out of _platform_tags, but tests/runtime/test_runtime_metrics_api.py::test_runtime_metrics_experimental_runtime_tag still asserts at lines 218–228 that the current ID is present there. Because the runtime suite collects tests/runtime/, every parameter combination enabling either runtime-id flag will fail; update or remove that stale assertion and validate the emitted tags instead.
Useful? React with 👍 / 👎.
Fixes: #19526 (issue A)
What this fixes
#19526 reports two defects. Issue B (
dd.internal.entity_iddropped on flush) was fixed by#19602, already merged. This PR addresses what is left of issue A.
Issue A's stated root cause has two parts, and
mainhas already fixed one of them: the RSScollector no longer freezes a PID at enable time.
PSUtilRuntimeMetricCollector, which didpsutil.Process(os.getpid())in_on_modules_loadand is still what ships in 4.13.1, has beenreplaced on
mainbyNativeProcessMetricCollector, which reads the current process on everycall and re-seeds its baselines through a
forksafehook. Likewise,enable()being idempotentis no longer a defect: the worker's
PeriodicThreadauto-restarts in the child, so an inherited_instancekeeps reporting.What remains is the tag.
RuntimeWorker.__init__resolvedruntime-idonce (viaPlatformTagsV2->PlatformTagCollectorV2) and stored it inself._platform_tags. A forkedchild inherits that instance, so every worker emits
runtime.python.*under the parent'sruntime-ideven thoughddtrace.internal.runtime._set_runtime_idhas already issued the childa new one. All N+1 processes land on one series, and
sum by {runtime-id}for a pod reads as themaster process alone — the reporter's
master_onlyandsitecustomize_postrows, wheredd_sum / Σ VmRSSwas 0.10 and 0.09 against an expected 1.00.The change
runtime-idis resolved per flush rather than snapshotted:flush()already rebuildsconstant_tagson every call for exactly this reason (service, env,version), so this needs no new fork hook, no rebind, and nothing to unregister on
disable().PlatformTagCollectorV2existed only to append the snapshotted tag, so it is removed with it.This makes the documented workaround —
RuntimeWorker.disable(); RuntimeWorker.enable()ingunicorn's
post_worker_init— unnecessary onmain.Verification
tests/tracer/runtime/fork_runtime_id.pyfollows the existingfork_enable.py/fork_disable.pypattern: it constructs aRuntimeWorkerpre-fork, forks, and asserts thechild's emitted packets carry the child's
runtime-idand not the parent's, while the parentkeeps reporting under its own. Driven by
test_fork_runtime_id_tag.The replaced
test_runtime_platformv2_tagsassertedPlatformTagsV2contained aruntime-idkey;
test_runtime_worker_flush_emits_current_runtime_idnow asserts the same property one layerout, on the packets actually sent.
Fail-before / pass-after, real
fork():Caveat on how this was run. This environment has no Rust/cmake toolchain, so
maincould notbe built and the repo's test suite was not run. The fail-before/pass-after above was produced by
applying the identical patch to an installed
ddtrace==4.13.1, whoseRuntimeWorker.__init__/flush()lines being changed here are byte-identical tomain's(4.13.1 differs only in the
tracer=parameter, theddtrace.configimport style, and thepsutil-vs-native collector). The test files themselves are written against
mainand have notbeen executed by pytest — please run CI before trusting them.
Notes for the reviewer
DD_TRACE_EXPERIMENTAL_RUNTIME_ID_ENABLED/DD_RUNTIME_METRICS_RUNTIME_ID_ENABLEDdefaultsto
False, so this only affects deployments that opted intoruntime-idtagging — which isthe reporter's configuration. With the flag off, no
runtime-idis emitted, before or after.uuid-free module-global read;get_runtime_id()returns a cachedstring.
os.getpid()-vs-bound-PID warning suggested in theissue comments. On
mainnothing binds a PID any more, so there is no stale-PID state left todetect.
sum by {runtime-id}over-counting when gunicorn--max-requestsrecycles workers (a dead worker'sseries stops rather than going to zero, so a long window carries both it and its replacement).
That is a query-side aggregation concern, not something the client can fix by choosing a
different tag — a recycled worker is a new process and correctly gets a new id.