Skip to content

fix: make pipeline runs self-diagnosing - #60

Merged
murdore merged 1 commit into
mainfrom
fix/pipeline-metrics-and-piggyback-timeout
Aug 18, 2026
Merged

fix: make pipeline runs self-diagnosing#60
murdore merged 1 commit into
mainfrom
fix/pipeline-metrics-and-piggyback-timeout

Conversation

@murdore

@murdore murdore commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Two fixes for signals that went silent, both surfaced while investigating why recent content appeared to be missing from the corpus. The corpus itself turned out to be healthy — these are the instruments that failed to show it.

totalItemsProcessed was structurally always 0

recordProcessed / recordSkipped / recordErrored existed in src/utils/metrics.ts but had zero call sites anywhere in the codebase. Every run reported the same totalItemsProcessed: 0 whether it collected 62 items or nothing — which is how the 2026-08-13 harvest failure stayed hidden behind 21/21 steps succeeded.

  • PipelineStep.run may now return StepCounts; runSingleStep records whatever comes back
  • Wired the two steps whose counts the runner already holds (collection, acquisition)
  • Counts reported outside a step are dropped rather than attributed to the next one
  • save() split into a pure build() plus the write, so tests can read a run's totals without clobbering the videos/pipeline-metrics.json the scheduled run owns

Verified end-to-end with START_STEP=0 END_STEP=1 (no network, no model calls): 062.

The other 19 agents still return Promise<void> and report nothing. Collection and acquisition are the two that decide whether new content actually arrived, so they carry the signal that matters.

Piggyback lost a day's harvest to a 15s wait with no diagnostics

On 2026-08-13 all three retries failed identically on Chrome DevTools port 9455 did not open within 15000ms. Three identical failures point at Chrome never starting rather than starting slowly — and stdio: "ignore" meant there was no evidence either way.

  • Wait raised 15s → 45s, exposed as IG_PIGGYBACK_PORT_TIMEOUT_MS, matching how every other piggyback knob is tuned, documented in .env.example
  • Chrome's stderr is piped and its tail attached to the failure, so a locked profile names itself
  • The wait aborts as soon as Chrome exits or is signalled, instead of blaming the port after the full timeout

Notes

Both void-in-union lint suppressions are load-bearing and carry their reason inline: the 19 wired agents are all Promise<void>, and void is not assignable to undefined — verified by compiling the alternative.

Testing

  • npm test — 488 tests across 57 files pass
  • npx tsc --noEmit clean; no new lint findings vs main
  • 12 new tests: 7 covering count aggregation and attribution, 5 covering the port wait including fail-fast on exit and on signal

…n of the work

Investigating "recent saves are missing from the corpus" turned up two failures
that reinforced each other: the harvest silently read only the top of the saved
feed, and the run summary could not express that anything was wrong.

Harvest depth. IG_PIGGYBACK_SCROLLS=4 captured ~63 of 462 saved items. Everything
below that window was never fetched, and because the run still exited 0 the loss
left no trace — metadata.json stayed internally consistent, and knowledge-base
coverage measured against it read ~99%, because items never captured were never
recorded as missing. A 40-scroll run recovered 43 unseen items, 24 of them from
2026 and two saved within the previous 48 hours. Scrolling is self-limiting: the
feed exhausted after 22 responses, so the cost is wall-clock (~100s vs ~20s),
not unbounded requests.

Run metrics. recordProcessed/recordSkipped/recordErrored had no call sites
anywhere, so summary.totalItemsProcessed was pinned at 0 — identical output
whether a run processed 884 items or none, which is how a failed harvest hid
behind "21/21 steps succeeded". PipelineStep.run may now return StepCounts, and
the two steps whose counts the runner already holds report them; counts arriving
outside a step are dropped rather than misattributed to the next one. save() is
split into a pure build() plus the write so tests can read a run's totals without
clobbering the videos/pipeline-metrics.json the scheduled run owns.

Chrome startup. On 2026-08-13 all three harvest retries died on the same 15s
DevTools-port wait, with stderr discarded there was no evidence why. The wait is
now 45s and tunable via IG_PIGGYBACK_PORT_TIMEOUT_MS, Chrome's stderr tail is
attached to the failure so a locked profile names itself, and the wait aborts as
soon as the process exits or is signalled instead of blaming the port.

Both void-in-union lint suppressions carry their reason inline: the 19 wired
agents are all Promise<void>, and void is not assignable to undefined — verified
by compiling the alternative.

Verified: 488 tests across 57 files; a live run reported totalItemsProcessed 884
against the previous constant 0, and closed the corpus gap from 7 unprocessed
videos to 0.
@murdore
murdore force-pushed the fix/pipeline-metrics-and-piggyback-timeout branch from 91d310c to deea284 Compare August 18, 2026 14:42
@github-actions

Copy link
Copy Markdown

Single Commit Policy — COMPLIANT

Status: 1 commit | Valid format | No merge commits | Ready for merge

Commit details
  • Hash: deea284c5eee293f71a7f5149ec12f3a82150bad
  • Message: fix(pipeline): stop runs from reporting success while doing a fraction of the work
  • Author: Sachin Sharma

Automated validation — dopamine single-commit enforcement

@murdore
murdore merged commit f697455 into main Aug 18, 2026
12 checks passed
@murdore
murdore deleted the fix/pipeline-metrics-and-piggyback-timeout branch August 18, 2026 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant