fix(telemetry): report unpaired events, and retract migration 000155's reconciler promise - #413
Merged
Merged
Conversation
…s reconciler promise
Migration 000155 says endpoint_asset_id is nullable because "during onboarding
the agent may not yet know its asset UUID. A nightly reconciler job pairs events
with assets by agent_id."
No such job was ever written, and it cannot be. There is no join key: `agents`
has no asset column, `assets` has no agent column, and there is no join table —
so there is nothing to pair BY. Checked every promised background job in every
migration comment (15 of them); this is the only one with no implementation.
The other 14 all exist and are started.
It is also the wrong idea. Only the producer knows which endpoint an event
describes. An EDR/XDR forwarder reports on many hosts, so even the emitting
agent's own hostname is not the answer. The server cannot infer this after the
fact, at any point, by any means.
So a NULL endpoint_asset_id is permanent, not a pending state — and the
consequence is invisible. The event is stored, the response says accepted, and
the IOC correlator still matches it because it keys on values inside the event.
What silently does not happen is every asset-scoped read: Stage-4 detection
correlation's heuristic fallback (CountNearTarget, by asset + window) and the
per-asset Stage-6 dashboards. Half the feature does not apply, and nothing says
so.
Two changes, no new background job — building a reaper with no join key, for a
stream that currently has no producer at all, would be adding to the
silently-inert class rather than fixing it:
1. The migration comment retracts the promise and records why it cannot be
kept, so the next person does not go looking for a job that was never
possible.
2. The ingest response gains `unpaired`, counting ACCEPTED events that
arrived without an asset link, plus a WARN naming the impact. A producer
integrating today sees the degradation on the response it already reads
instead of discovering months later that asset-scoped correlation never
applied to its data.
Rejected counts are excluded: those events were never stored, and counting them
would send a producer looking for a configuration problem that is really a
validation error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closing the last item on the silently-inert backlog — by establishing that the promised fix is impossible, and shipping the thing that actually helps instead.
The promise
Migration
000155_runtime_telemetryexplains whyendpoint_asset_idis nullable:No such job exists. And it cannot:
There is no key to pair BY. I also swept every background job promised in every migration comment — 15 of them. This is the only one with no implementation; the other 14 all exist and are started (verified against the constructed-vs-
Start()set incmd/server/workers.go, including twotime.Tickercases my first regex wrongly flagged).It is also conceptually wrong
Only the producer knows which endpoint an event describes. An EDR/XDR forwarder reports on many hosts, so even the emitting agent's own hostname is not the answer. The server cannot infer this after the fact, at any point, by any means. (Empirically: the 3 agents with hostnames on the live DB are scanners/collectors and match 0 assets.)
Why that matters — the invisible half
A NULL here is permanent, not pending, and everything about the interaction says success:
acceptedacceptedCountNearTarget, asset + window)Half the feature does not apply and nothing says so. That is exactly the defect shape this backlog exists to kill.
What I did — and deliberately did not do
Did not build the reconciler. A reaper with no join key, for a stream that currently has no producer at all, is adding to the silently-inert class, not fixing it.
unpaired— accepted events that arrived with no asset link — plus a WARN naming the impact. A producer integrating today sees the degradation on the response it already reads, instead of discovering months later that asset-scoped correlation never applied to its data.Rejected events are excluded from the count: they were never stored, and counting them would send a producer chasing a configuration problem that is really a validation error.
Verification
Tests drive the real handler against the real database (not a mock), covering all-unpaired, all-paired, a mixed batch (the count is per-event, not a per-batch boolean), and the rejected-vs-unpaired distinction.
GOWORK=off go test ./...againstapp_test— all greenGOWORK=off make lint-ci— cleanscripts/check-migrations.shon the edited migration —✓ no destructive operations, versions uniqueNot claimed: no live producer exists, so this is proven at the handler + DB layer, not against real EDR traffic.