fix(logs-forwarder): unwrap OCI Streaming base64 envelope before formatting

[yuhuyoyo]

Summary

Customers using a cross-tenancy audit log topology — tenancy A audit logs → OCI Streaming → Service Connector Hub → logs-forwarder function in tenancy B — were receiving completely empty log messages in Datadog (only {"oci":{"service":"oci"}}, no actual log content).

Root cause

When a Service Connector Hub (SCH) source is OCI Streaming, SCH delivers each message in a base64 envelope:

{"streamPool":"...","stream":"...","partition":"0","key":"","value":"<base64-encoded log JSON>","offset":"0","timestamp":"..."}

The actual log JSON (with source, time, data, oracle, type) lives base64-encoded inside value, not at the top level.

The logs-forwarder's formatLogs decoded the body as plain log entries and handed each map straight to ProcessLogEntry, which reads those top-level fields. Since none of them exist on the streaming envelope, getFieldValue returned "" / empty maps — producing an empty log with only service: oci (the one always-emitted field).

Notably, the events-forwarder already handles this — events-forwarder/internal/formatter/formatter.go base64-decodes the value field for streaming sources. The logs-forwarder did not, so the two forwarders behaved inconsistently for the same SCH streaming topology.

Fix

Add formatter.UnwrapStreamingMessages, which:

• base64-decodes the value field of streaming envelopes into the real log entry, and
• passes plain (non-streaming) log entries through unchanged (detected via: has a value string field AND no top-level data field).

Wire it into formatLogs right after JSON decoding, before formatting. This is backward-compatible — existing Logging-source connectors (no value envelope, or entries with a data field) keep working as before.

Files changed

• logs-forwarder/internal/formatter/formatter.go — new exported UnwrapStreamingMessages
• logs-forwarder/internal/handler/handler.go — call it in formatLogs
• logs-forwarder/internal/formatter/formatter_test.go — unit tests for unwrap + end-to-end test
• logs-forwarder/internal/handler/handler_test.go — streaming-envelope case in TestFormatLogs

Why not avoid the encoding at the source?

The base64 wrapping is inherent to OCI Streaming — there's no SCH setting to deliver stream messages decoded. The stream is the only transport that bridges tenancies A→B (log groups don't cross tenancies), so decoding in the function is the correct fix rather than dropping the stream.

Test plan

• go build ./... passes
• go test ./... passes (formatter + handler packages)
• Validate against a real cross-tenancy streaming topology: confirm audit logs arrive with populated data/oracle/source/type and ddsource: oci.audit

🤖 Generated with Claude Code

[datadog-prod-us1-5]

More details

The new decoder unwraps the streaming envelope before formatting. It keeps entries with a top-level data field on the plain-log path.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit fb435ab · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}