fix: make pipeline runs self-diagnosing - #60
Merged
Conversation
…n of the work Investigating "recent saves are missing from the corpus" turned up two failures that reinforced each other: the harvest silently read only the top of the saved feed, and the run summary could not express that anything was wrong. Harvest depth. IG_PIGGYBACK_SCROLLS=4 captured ~63 of 462 saved items. Everything below that window was never fetched, and because the run still exited 0 the loss left no trace — metadata.json stayed internally consistent, and knowledge-base coverage measured against it read ~99%, because items never captured were never recorded as missing. A 40-scroll run recovered 43 unseen items, 24 of them from 2026 and two saved within the previous 48 hours. Scrolling is self-limiting: the feed exhausted after 22 responses, so the cost is wall-clock (~100s vs ~20s), not unbounded requests. Run metrics. recordProcessed/recordSkipped/recordErrored had no call sites anywhere, so summary.totalItemsProcessed was pinned at 0 — identical output whether a run processed 884 items or none, which is how a failed harvest hid behind "21/21 steps succeeded". PipelineStep.run may now return StepCounts, and the two steps whose counts the runner already holds report them; counts arriving outside a step are dropped rather than misattributed to the next one. save() is split into a pure build() plus the write so tests can read a run's totals without clobbering the videos/pipeline-metrics.json the scheduled run owns. Chrome startup. On 2026-08-13 all three harvest retries died on the same 15s DevTools-port wait, with stderr discarded there was no evidence why. The wait is now 45s and tunable via IG_PIGGYBACK_PORT_TIMEOUT_MS, Chrome's stderr tail is attached to the failure so a locked profile names itself, and the wait aborts as soon as the process exits or is signalled instead of blaming the port. Both void-in-union lint suppressions carry their reason inline: the 19 wired agents are all Promise<void>, and void is not assignable to undefined — verified by compiling the alternative. Verified: 488 tests across 57 files; a live run reported totalItemsProcessed 884 against the previous constant 0, and closed the corpus gap from 7 unprocessed videos to 0.
murdore
force-pushed
the
fix/pipeline-metrics-and-piggyback-timeout
branch
from
August 18, 2026 14:42
91d310c to
deea284
Compare
Single Commit Policy — COMPLIANTStatus: 1 commit | Valid format | No merge commits | Ready for merge Commit details
Automated validation — dopamine single-commit enforcement |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes for signals that went silent, both surfaced while investigating why recent content appeared to be missing from the corpus. The corpus itself turned out to be healthy — these are the instruments that failed to show it.
totalItemsProcessedwas structurally always 0recordProcessed/recordSkipped/recordErroredexisted insrc/utils/metrics.tsbut had zero call sites anywhere in the codebase. Every run reported the sametotalItemsProcessed: 0whether it collected 62 items or nothing — which is how the 2026-08-13 harvest failure stayed hidden behind21/21 steps succeeded.PipelineStep.runmay now returnStepCounts;runSingleSteprecords whatever comes backsave()split into a purebuild()plus the write, so tests can read a run's totals without clobbering thevideos/pipeline-metrics.jsonthe scheduled run ownsVerified end-to-end with
START_STEP=0 END_STEP=1(no network, no model calls):0→62.The other 19 agents still return
Promise<void>and report nothing. Collection and acquisition are the two that decide whether new content actually arrived, so they carry the signal that matters.Piggyback lost a day's harvest to a 15s wait with no diagnostics
On 2026-08-13 all three retries failed identically on
Chrome DevTools port 9455 did not open within 15000ms. Three identical failures point at Chrome never starting rather than starting slowly — andstdio: "ignore"meant there was no evidence either way.IG_PIGGYBACK_PORT_TIMEOUT_MS, matching how every other piggyback knob is tuned, documented in.env.exampleNotes
Both
void-in-union lint suppressions are load-bearing and carry their reason inline: the 19 wired agents are allPromise<void>, andvoidis not assignable toundefined— verified by compiling the alternative.Testing
npm test— 488 tests across 57 files passnpx tsc --noEmitclean; no new lint findings vsmain