Skip to content

fix(telemetry): report unpaired events, and retract migration 000155's reconciler promise - #413

Merged
0xmanhnv merged 1 commit into
developfrom
fix/telemetry-unpaired-visibility
Aug 4, 2026
Merged

fix(telemetry): report unpaired events, and retract migration 000155's reconciler promise#413
0xmanhnv merged 1 commit into
developfrom
fix/telemetry-unpaired-visibility

Conversation

@0xmanhnv

@0xmanhnv 0xmanhnv commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Closing the last item on the silently-inert backlog — by establishing that the promised fix is impossible, and shipping the thing that actually helps instead.

The promise

Migration 000155_runtime_telemetry explains why endpoint_asset_id is nullable:

"during onboarding the agent may not yet know its asset UUID. A nightly reconciler job pairs events with assets by agent_id."

No such job exists. And it cannot:

agents  -> columns matching %asset%  : (none)
assets  -> columns matching %agent%  : (none)
join tables                          : agent_api_keys, agent_metrics, agent_audit_logs — none of them link an agent to an asset

There is no key to pair BY. I also swept every background job promised in every migration comment — 15 of them. This is the only one with no implementation; the other 14 all exist and are started (verified against the constructed-vs-Start() set in cmd/server/workers.go, including two time.Ticker cases my first regex wrongly flagged).

It is also conceptually wrong

Only the producer knows which endpoint an event describes. An EDR/XDR forwarder reports on many hosts, so even the emitting agent's own hostname is not the answer. The server cannot infer this after the fact, at any point, by any means. (Empirically: the 3 agents with hostnames on the live DB are scanners/collectors and match 0 assets.)

Why that matters — the invisible half

A NULL here is permanent, not pending, and everything about the interaction says success:

with an asset link without
stored yes yes
response accepted accepted
IOC correlator (keys on values inside the event) works works
Stage-4 detection heuristic (CountNearTarget, asset + window) works silently skipped
per-asset Stage-6 dashboards works silently skipped

Half the feature does not apply and nothing says so. That is exactly the defect shape this backlog exists to kill.

What I did — and deliberately did not do

Did not build the reconciler. A reaper with no join key, for a stream that currently has no producer at all, is adding to the silently-inert class, not fixing it.

  1. The migration comment retracts the promise and records why it cannot be kept, so the next person does not go hunting for a job that was never possible.
  2. The ingest response gains unpaired — accepted events that arrived with no asset link — plus a WARN naming the impact. A producer integrating today sees the degradation on the response it already reads, instead of discovering months later that asset-scoped correlation never applied to its data.

Rejected events are excluded from the count: they were never stored, and counting them would send a producer chasing a configuration problem that is really a validation error.

Verification

Tests drive the real handler against the real database (not a mock), covering all-unpaired, all-paired, a mixed batch (the count is per-event, not a per-batch boolean), and the rejected-vs-unpaired distinction.

  • GOWORK=off go test ./... against app_test — all green
  • GOWORK=off make lint-ci — clean
  • scripts/check-migrations.sh on the edited migration — ✓ no destructive operations, versions unique
  • Editing an applied migration is safe here: golang-migrate tracks version numbers, not checksums, and the change is comment-only — no schema statement was touched.

Not claimed: no live producer exists, so this is proven at the handler + DB layer, not against real EDR traffic.

…s reconciler promise

Migration 000155 says endpoint_asset_id is nullable because "during onboarding
the agent may not yet know its asset UUID. A nightly reconciler job pairs events
with assets by agent_id."

No such job was ever written, and it cannot be. There is no join key: `agents`
has no asset column, `assets` has no agent column, and there is no join table —
so there is nothing to pair BY. Checked every promised background job in every
migration comment (15 of them); this is the only one with no implementation.
The other 14 all exist and are started.

It is also the wrong idea. Only the producer knows which endpoint an event
describes. An EDR/XDR forwarder reports on many hosts, so even the emitting
agent's own hostname is not the answer. The server cannot infer this after the
fact, at any point, by any means.

So a NULL endpoint_asset_id is permanent, not a pending state — and the
consequence is invisible. The event is stored, the response says accepted, and
the IOC correlator still matches it because it keys on values inside the event.
What silently does not happen is every asset-scoped read: Stage-4 detection
correlation's heuristic fallback (CountNearTarget, by asset + window) and the
per-asset Stage-6 dashboards. Half the feature does not apply, and nothing says
so.

Two changes, no new background job — building a reaper with no join key, for a
stream that currently has no producer at all, would be adding to the
silently-inert class rather than fixing it:

  1. The migration comment retracts the promise and records why it cannot be
     kept, so the next person does not go looking for a job that was never
     possible.

  2. The ingest response gains `unpaired`, counting ACCEPTED events that
     arrived without an asset link, plus a WARN naming the impact. A producer
     integrating today sees the degradation on the response it already reads
     instead of discovering months later that asset-scoped correlation never
     applied to its data.

Rejected counts are excluded: those events were never stored, and counting them
would send a producer looking for a configuration problem that is really a
validation error.
@0xmanhnv
0xmanhnv merged commit 0a39459 into develop Aug 4, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant