Skip to content

fix(llmobs): preserve in-flight dataset mutations - #9903

Merged
mehulsonowal merged 7 commits into
masterfrom
mehul/llmobs-dataset-inflight-mutation
Aug 26, 2026
Merged

fix(llmobs): preserve in-flight dataset mutations#9903
mehulsonowal merged 7 commits into
masterfrom
mehul/llmobs-dataset-inflight-mutation

Conversation

@mehulsonowal

@mehulsonowal mehulsonowal commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Raised by this Codex review: #9762 (comment) Deeply snapshots dataset insert and update payloads before sending a batch request. This keeps the committed batch independent from live DatasetRecord values, so nested object/array mutations made while push() is in flight remain pending for a subsequent push.

Testing

  • ./node_modules/.bin/mocha packages/dd-trace/test/llmobs/experiments/experiment.spec.js
  • Targeted ESLint and syntax checks

Review notes

  • Added regression coverage for in-place mutations during insert and update pushes.
  • Switched snapshotting to vendored rfdc following BridgeAR's review comment.

@mehulsonowal
mehulsonowal requested review from a team as code owners August 20, 2026 17:16
@dd-octo-sts

dd-octo-sts Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 8.41 MB
Deduped: 9.07 MB
No deduping: 9.07 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 445.14 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 92ea840837

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/src/llmobs/experiments/dataset.js Outdated
@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Aug 20, 2026

Copy link
Copy Markdown

Tests

All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 98.58% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 600c871 | Docs | View more details | Give us feedback!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 60a842454e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/src/llmobs/experiments/dataset.js Outdated
@pr-commenter

pr-commenter Bot commented Aug 20, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-26 19:26:43

Comparing candidate commit 600c871 in PR branch mehul/llmobs-dataset-inflight-mutation with baseline commit 4d8f52e in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 2300 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:debugger-enabled-but-breakpoint-not-hit-26

  • unstable cpu_user_time [-380.522ms; +685.937ms] or [-4.357%; +7.854%]
  • unstable execution_time [-382.121ms; +683.108ms] or [-4.373%; +7.817%]
  • unstable throughput [-2038243.056op/s; +1127795.838op/s] or [-7.103%; +3.930%]

scenario:dogstatsd-with-tags-20

  • unstable cpu_user_time [-285.557ms; +361.761ms] or [-5.932%; +7.515%]
  • unstable execution_time [-287.872ms; +364.070ms] or [-5.882%; +7.439%]
  • unstable throughput [-128860.580op/s; +103243.945op/s] or [-7.519%; +6.025%]

scenario:plugin-claude-agent-sdk-compact-stream-scan-24

  • unstable cpu_usage_percentage [-4.443%; +5.626%]

scenario:plugin-graphql-long-with-depth-off-20

  • unstable max_rss_usage [-5.484MB; +7.827MB] or [-4.264%; +6.086%]

scenario:plugin-graphql-long-with-depth-off-26

  • unstable max_rss_usage [-16.665MB; +45.033MB] or [-8.984%; +24.277%]

scenario:test-optimization-large-suite-20

  • unstable max_rss_usage [-7.341MB; +2.555MB] or [-8.982%; +3.127%]

@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.57%. Comparing base (4d8f52e) to head (600c871).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff            @@
##           master    #9903    +/-   ##
========================================
  Coverage   98.57%   98.57%            
========================================
  Files         990      990            
  Lines      148942   148949     +7     
  Branches    12978    12806   -172     
========================================
+ Hits       146821   146828     +7     
  Misses       2121     2121            
Flag Coverage Δ
aiguard 57.19% <11.11%> (-0.10%) ⬇️
aiguard-integration 54.96% <11.11%> (-0.01%) ⬇️
apm-bucket-0 57.06% <11.11%> (-0.03%) ⬇️
apm-bucket-1 62.26% <11.11%> (-0.03%) ⬇️
apm-bucket-2 61.09% <11.11%> (-0.03%) ⬇️
apm-bucket-3 58.73% <11.11%> (-0.03%) ⬇️
apm-capabilities-tracing 62.31% <11.11%> (-0.01%) ⬇️
apm-integrations-aerospike 54.76% <11.11%> (-0.03%) ⬇️
apm-integrations-confluentinc-kafka-javascript 60.07% <11.11%> (-0.03%) ⬇️
apm-integrations-couchbase 55.66% <11.11%> (-0.03%) ⬇️
apm-integrations-http 60.66% <11.11%> (-0.03%) ⬇️
apm-integrations-kafkajs 60.58% <11.11%> (-0.03%) ⬇️
apm-integrations-next 58.36% <11.11%> (-0.03%) ⬇️
apm-integrations-prisma 57.24% <11.11%> (-0.03%) ⬇️
appsec 70.75% <11.11%> (-0.03%) ⬇️
appsec-express_fastify_graphql 68.19% <11.11%> (-0.03%) ⬇️
appsec-integration 49.33% <11.11%> (+<0.01%) ⬆️
appsec-kafka_ldapjs_lodash 62.16% <11.11%> (-0.03%) ⬇️
appsec-mongodb-core_mongoose_mysql 65.67% <11.11%> (-0.03%) ⬇️
appsec-next 55.69% <11.11%> (-0.03%) ⬇️
appsec-node-serialize_passport_postgres 65.07% <11.11%> (-0.03%) ⬇️
appsec-sourcing_stripe_template 63.53% <11.11%> (-0.03%) ⬇️
debugger 63.23% <11.11%> (-0.04%) ⬇️
instrumentations-bucket-0 50.75% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-1 58.58% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-10 59.69% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-11 60.34% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-12 50.66% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-13 51.52% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-14 50.76% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-2 52.01% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-3 52.64% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-4 57.68% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-5 48.41% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-6 59.18% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-7 50.95% <11.11%> (-0.03%) ⬇️
instrumentations-bucket-8 57.40% <11.11%> (-0.04%) ⬇️
instrumentations-bucket-9 56.24% <11.11%> (-0.03%) ⬇️
instrumentations-instrumentation-couchbase 49.50% <11.11%> (-0.03%) ⬇️
instrumentations-integration-esbuild 34.01% <ø> (ø)
llmobs-ai_anthropic_bedrock 61.90% <11.11%> (-0.03%) ⬇️
llmobs-bucket-1 60.45% <11.11%> (-0.03%) ⬇️
llmobs-openai 61.09% <11.11%> (-0.03%) ⬇️
llmobs-openai-agents_vertex-ai 59.16% <11.11%> (-0.03%) ⬇️
llmobs-sdk 68.21% <100.00%> (-0.02%) ⬇️
master-coverage 98.57% <100.00%> (?)
openfeature 55.00% <11.11%> (-0.01%) ⬇️
openfeature-unit 52.96% <11.11%> (-0.03%) ⬇️
platform-core_esbuild_instrumentations-misc 40.66% <11.11%> (-0.02%) ⬇️
platform-integration 59.40% <11.11%> (-0.01%) ⬇️
platform-shimmer_unit-guardrails_webpack 38.37% <11.11%> (-0.02%) ⬇️
plugins-browser-bunyan_bullmq_cassandra 60.48% <11.11%> (-0.03%) ⬇️
plugins-bucket-0 55.94% <11.11%> (-0.03%) ⬇️
plugins-bucket-1 53.03% <11.11%> (-0.01%) ⬇️
plugins-bucket-11 60.88% <11.11%> (-0.03%) ⬇️
plugins-bucket-18 60.44% <11.11%> (-0.03%) ⬇️
plugins-bucket-19 58.67% <11.11%> (-0.03%) ⬇️
plugins-bucket-20 60.56% <11.11%> (-0.03%) ⬇️
plugins-bucket-4 55.31% <11.11%> (-0.03%) ⬇️
plugins-cookie_cookie-parser_crypto 50.27% <11.11%> (-0.03%) ⬇️
plugins-fastify_fetch_fs 59.50% <11.11%> (-0.03%) ⬇️
plugins-generic-pool_google-cloud-pubsub_grpc 63.04% <11.11%> (-0.02%) ⬇️
plugins-handlebars_hapi_hono 57.52% <11.11%> (-0.03%) ⬇️
plugins-ioredis_knex_langgraph 55.98% <11.11%> (-0.03%) ⬇️
plugins-ldapjs_light-my-request_limitd-client 57.20% <11.11%> (-0.04%) ⬇️
plugins-lodash_mariadb_memcached 58.46% <11.11%> (-0.03%) ⬇️
plugins-moleculer_mongodb_mongodb-core 60.35% <11.11%> (-0.03%) ⬇️
plugins-mongoose_multer_mysql 58.20% <11.11%> (-0.03%) ⬇️
plugins-mysql2_nats_node-serialize 59.81% <11.11%> (-0.03%) ⬇️
plugins-opensearch_passport-http_pino 58.05% <11.11%> (-0.03%) ⬇️
plugins-postgres_process_pug 57.21% <11.11%> (-0.03%) ⬇️
plugins-redis_router_sequelize 60.43% <11.11%> (-0.03%) ⬇️
plugins-test-and-upstream-rhea_undici_url 60.02% <11.11%> (-0.03%) ⬇️
plugins-valkey_vm_winston 56.64% <11.11%> (+0.01%) ⬆️
plugins-ws 58.20% <11.11%> (-0.03%) ⬇️
profiling 60.73% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-aws-sdk 53.95% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-base-inject-field 49.98% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-bedrockruntime 53.71% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-client 55.20% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-dynamodb 54.52% <11.11%> (-0.04%) ⬇️
serverless-aws-sdk-eventbridge 56.04% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-kinesis 58.05% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-lambda 56.25% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-s3 54.62% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-serverless-peer-service 58.60% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-sns 58.83% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-sqs 59.25% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-stepfunctions 54.45% <11.11%> (-0.03%) ⬇️
serverless-aws-sdk-util 50.49% <11.11%> (-0.03%) ⬇️
serverless-bucket-0 52.63% <11.11%> (-0.01%) ⬇️
serverless-bucket-1 57.87% <11.11%> (-0.03%) ⬇️
test-optimization-cucumber 70.15% <11.11%> (-0.05%) ⬇️
test-optimization-cypress 64.45% <11.11%> (+0.04%) ⬆️
test-optimization-jest 71.60% <11.11%> (-0.01%) ⬇️
test-optimization-mocha 71.67% <11.11%> (+0.04%) ⬆️
test-optimization-playwright-playwright-atr 59.30% <11.11%> (-0.03%) ⬇️
test-optimization-playwright-playwright-efd 59.97% <11.11%> (-0.02%) ⬇️
test-optimization-playwright-playwright-final-status 59.58% <11.11%> (-0.03%) ⬇️
test-optimization-playwright-playwright-impacted-tests 59.69% <11.11%> (+0.11%) ⬆️
test-optimization-playwright-playwright-reporting 60.71% <11.11%> (-0.07%) ⬇️
test-optimization-playwright-playwright-test-management 60.76% <11.11%> (-0.12%) ⬇️
test-optimization-playwright-playwright-test-span 59.35% <11.11%> (-0.09%) ⬇️
test-optimization-selenium 58.53% <11.11%> (+0.32%) ⬆️
test-optimization-testopt 61.44% <11.11%> (+0.06%) ⬆️
test-optimization-vitest 72.46% <11.11%> (+0.07%) ⬆️
test-optimization-vitest-browser 58.31% <11.11%> (-0.03%) ⬇️
test-optimization-webdriverio 64.83% <11.11%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mehulsonowal

Copy link
Copy Markdown
Contributor Author

Codex's snapshotting feedback is addressed in 60a8424: batch snapshots now use a JSON round trip matching the request serialization path, including custom toJSON, with regression coverage. The latest Codex review has no new suggestions.

The remaining CI failures are unrelated to this PR: LLMObs | ai reports six parent_id mismatches in packages/dd-trace/test/llmobs/plugins/ai/index.spec.js across the latest AI SDK versions, and all-green aggregates those failures plus an xcrun/gcov environment warning. The AI job reproduced the same failures across reruns; this PR only changes the experiments dataset implementation/tests. Targeted experiments tests (31 passing), lint, syntax, and patch coverage pass.

@mehulsonowal

Copy link
Copy Markdown
Contributor Author

Follow-up for CI: the ai matrix failures were reproducible parent-span changes from ai 5.0.241/6.0.260, not dataset code. I included the existing compatibility test fix in 2db5c07 (getToolSpanParentId) so the required all-green check can validate the current dependency behavior. The local worktree lacks the version fixtures needed to run that matrix, so CI is the source of truth.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2db5c070d2

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/src/llmobs/experiments/dataset.js
@mehulsonowal

Copy link
Copy Markdown
Contributor Author

Follow-up complete: Codex feedback on JSON serialization context and direct tag mutations is addressed in 190b3a7, with regression tests. All review threads are resolved.

After rerunning the transient system-test and aggregate checks, all required checks are green: all-green and dd-gitlab/default-pipeline. The PR is only waiting for required human review.

Comment thread packages/dd-trace/src/llmobs/experiments/dataset.js Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1eacb52b8f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread packages/dd-trace/src/llmobs/experiments/dataset.js Outdated
Comment thread packages/dd-trace/src/llmobs/experiments/dataset.js
Comment thread packages/dd-trace/src/llmobs/experiments/dataset.js Outdated
@mehulsonowal
mehulsonowal merged commit deb9a44 into master Aug 26, 2026
686 checks passed
@mehulsonowal
mehulsonowal deleted the mehul/llmobs-dataset-inflight-mutation branch August 26, 2026 19:37
dd-octo-sts Bot pushed a commit that referenced this pull request Aug 27, 2026
* fix(llmobs): snapshot dataset batch payloads

* fix(llmobs): preserve JSON serialization in snapshots

* test(llmobs): support updated AI tool span parents

* fix(llmobs): reconcile inserted field and tag changes

* fix(llmobs): use rfdc for dataset snapshots

* refactor(llmobs): bind rfdc as snapshotPayload
dd-octo-sts Bot pushed a commit that referenced this pull request Aug 27, 2026
* fix(llmobs): snapshot dataset batch payloads

* fix(llmobs): preserve JSON serialization in snapshots

* test(llmobs): support updated AI tool span parents

* fix(llmobs): reconcile inserted field and tag changes

* fix(llmobs): use rfdc for dataset snapshots

* refactor(llmobs): bind rfdc as snapshotPayload
This was referenced Aug 27, 2026
pabloerhard pushed a commit that referenced this pull request Aug 28, 2026
* fix(llmobs): snapshot dataset batch payloads

* fix(llmobs): preserve JSON serialization in snapshots

* test(llmobs): support updated AI tool span parents

* fix(llmobs): reconcile inserted field and tag changes

* fix(llmobs): use rfdc for dataset snapshots

* refactor(llmobs): bind rfdc as snapshotPayload
pabloerhard pushed a commit that referenced this pull request Aug 28, 2026
* fix(llmobs): snapshot dataset batch payloads

* fix(llmobs): preserve JSON serialization in snapshots

* test(llmobs): support updated AI tool span parents

* fix(llmobs): reconcile inserted field and tag changes

* fix(llmobs): use rfdc for dataset snapshots

* refactor(llmobs): bind rfdc as snapshotPayload
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants