Log backfill completion to OCI logging - #162
Closed
erics-ddog wants to merge 2 commits into
Closed
Conversation
writeResponse's "Backfill complete" message was only ever written into the detached invoke's discarded HTTP response, so a successful drain had no durable signal anywhere. The error path already logs via log.Println; this adds the equivalent for the success path.
If Backfill reaches its success return, every replay in the run already succeeded against Datadog, so intake is provably reachable at that moment — sending a completion log directly to Datadog's Logs API doesn't have the same "pipe might be down" problem sending the trigger signal would. Skipped for empty-bucket no-op runs (Replayed == 0) so it stays a signal about real backfilled data, not periodic noise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two related additions to
Backfill()'s success path, both firing only on a genuinely clean completion (no fatal replay error, no list error):log.Printflogging the summary (replayed/skipped/dropped/delete_failures) — captured by OCI Logging.https://http-intake.logs.<site>/api/v2/logs), skipped whensummary.Replayed == 0so an empty-bucket no-op run doesn't generate noise.Why
writeResponse(out, "success", "Backfill complete: "+summary.String(), nil)only writes that message into the function's HTTP response body — buthubmanager's invoke is detached, so nobody ever reads that response. A successful backfill drain previously left no durable trace anywhere. The error path already callslog.Println(err)beforewriteResponse, so failures do show up in OCI Logging; part 1 closes the equivalent gap for the success/completion case. Companion to #160, which added the equivalent "invoked" signal at the start of a run.Part 2 sends the same signal directly to Datadog. Normally we'd be wary of that — backfill exists because the customer's Datadog pipe might be degraded, so routing observability about it back through the same pipe is circular. But by the time
Backfill()reaches its success return, every object in this run has already been replayed successfully against Datadog, which proves intake is reachable right now — so notifying Datadog directly at that specific point doesn't have the same risk the "invoked" signal would.Testing
go build ./...,go vet ./..., andgo test ./...pass fordatadog-functions/lib/client.