Skip to content

fix(runtime-metrics): resolve runtime-id at flush time so forked workers tag their own series - #19850

Open
UgaTheDev wants to merge 1 commit into
DataDog:mainfrom
UgaTheDev:fix/runtime-metrics-prefork
Open

fix(runtime-metrics): resolve runtime-id at flush time so forked workers tag their own series#19850
UgaTheDev wants to merge 1 commit into
DataDog:mainfrom
UgaTheDev:fix/runtime-metrics-prefork

Conversation

@UgaTheDev

Copy link
Copy Markdown

Fixes: #19526 (issue A)

What this fixes

#19526 reports two defects. Issue B (dd.internal.entity_id dropped on flush) was fixed by
#19602, already merged. This PR addresses what is left of issue A.

Issue A's stated root cause has two parts, and main has already fixed one of them: the RSS
collector no longer freezes a PID at enable time. PSUtilRuntimeMetricCollector, which did
psutil.Process(os.getpid()) in _on_modules_load and is still what ships in 4.13.1, has been
replaced on main by NativeProcessMetricCollector, which reads the current process on every
call and re-seeds its baselines through a forksafe hook. Likewise, enable() being idempotent
is no longer a defect: the worker's PeriodicThread auto-restarts in the child, so an inherited
_instance keeps reporting.

What remains is the tag. RuntimeWorker.__init__ resolved runtime-id once (via
PlatformTagsV2 -> PlatformTagCollectorV2) and stored it in self._platform_tags. A forked
child inherits that instance, so every worker emits runtime.python.* under the parent's
runtime-id even though ddtrace.internal.runtime._set_runtime_id has already issued the child
a new one. All N+1 processes land on one series, and sum by {runtime-id} for a pod reads as the
master process alone — the reporter's master_only and sitecustomize_post rows, where
dd_sum / Σ VmRSS was 0.10 and 0.09 against an expected 1.00.

The change

runtime-id is resolved per flush rather than snapshotted:

runtime_tags = self._format_tags(TracerTags()) + self._platform_tags + self._process_tags
if self._send_runtime_id:
    runtime_tags.append("runtime-id:" + get_runtime_id())

flush() already rebuilds constant_tags on every call for exactly this reason (service, env,
version), so this needs no new fork hook, no rebind, and nothing to unregister on disable().
PlatformTagCollectorV2 existed only to append the snapshotted tag, so it is removed with it.

This makes the documented workaround — RuntimeWorker.disable(); RuntimeWorker.enable() in
gunicorn's post_worker_init — unnecessary on main.

Verification

tests/tracer/runtime/fork_runtime_id.py follows the existing fork_enable.py /
fork_disable.py pattern: it constructs a RuntimeWorker pre-fork, forks, and asserts the
child's emitted packets carry the child's runtime-id and not the parent's, while the parent
keeps reporting under its own. Driven by test_fork_runtime_id_tag.

The replaced test_runtime_platformv2_tags asserted PlatformTagsV2 contained a runtime-id
key; test_runtime_worker_flush_emits_current_runtime_id now asserts the same property one layer
out, on the packets actually sent.

Fail-before / pass-after, real fork():

$ DD_RUNTIME_METRICS_RUNTIME_ID_ENABLED=true python tests/tracer/runtime/fork_runtime_id.py
AssertionError    # emitted_runtime_ids(worker) == {child_runtime_id}
EXIT=1
# with the patch applied
EXIT=0

Caveat on how this was run. This environment has no Rust/cmake toolchain, so main could not
be built and the repo's test suite was not run. The fail-before/pass-after above was produced by
applying the identical patch to an installed ddtrace==4.13.1, whose
RuntimeWorker.__init__/flush() lines being changed here are byte-identical to main's
(4.13.1 differs only in the tracer= parameter, the ddtrace.config import style, and the
psutil-vs-native collector). The test files themselves are written against main and have not
been executed by pytest — please run CI before trusting them.

Notes for the reviewer

  • DD_TRACE_EXPERIMENTAL_RUNTIME_ID_ENABLED / DD_RUNTIME_METRICS_RUNTIME_ID_ENABLED defaults
    to False, so this only affects deployments that opted into runtime-id tagging — which is
    the reporter's configuration. With the flag off, no runtime-id is emitted, before or after.
  • The per-flush cost is one uuid-free module-global read; get_runtime_id() returns a cached
    string.
  • Deliberately not included: the per-flush os.getpid()-vs-bound-PID warning suggested in the
    issue comments. On main nothing binds a PID any more, so there is no stale-PID state left to
    detect.
  • Also deliberately not addressed: the follow-up raised in the issue thread about sum by {runtime-id} over-counting when gunicorn --max-requests recycles workers (a dead worker's
    series stops rather than going to zero, so a long window carries both it and its replacement).
    That is a query-side aggregation concern, not something the client can fix by choosing a
    different tag — a recycled worker is a new process and correctly gets a new id.

When runtime metrics are enabled before a fork -- gunicorn/uWSGI preload, or
single-step instrumentation enabling products in the master -- every worker
inherits the parent's RuntimeWorker instance. The runtime-id tag was resolved
once in __init__ and stored in _platform_tags, so all N+1 processes emitted
runtime.python.* on a single series. Sum-by-runtime-id over a pod then read as
the size of the master process alone; the reporter measured 0.10 of the true
sum of per-process RSS with 4 workers.

Resolve runtime-id per flush instead. The forksafe hook in
ddtrace.internal.runtime already regenerates the id in the child, and the
process metric collector already rebinds to the current process on fork, so
this was the last piece still pinned to the parent.

PlatformTagCollectorV2 existed only to append that snapshotted tag and is
removed with it.

Fixes DataDog#19526 (issue A)

Signed-off-by: Kush Zingade <kush.zingade@gmail.com>
@UgaTheDev
UgaTheDev requested review from a team as code owners August 25, 2026 12:07

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e6242b5e3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +94 to +95
self._platform_tags = self._format_tags(PlatformTags())
self._send_runtime_id = config._runtime_metrics_runtime_id_enabled

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update the API test for the moved runtime-id tag

When runtime-id tagging is enabled, this now deliberately leaves runtime-id out of _platform_tags, but tests/runtime/test_runtime_metrics_api.py::test_runtime_metrics_experimental_runtime_tag still asserts at lines 218–228 that the current ID is present there. Because the runtime suite collects tests/runtime/, every parameter combination enabling either runtime-id flag will fail; update or remove that stale assertion and validate the emitted tags instead.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: Runtime metrics undercount gunicorn RSS after pre-fork enable; entity_id tags dropped

1 participant