feat(pipeline): keep every rendered surface fresh on a 4x-daily schedule

[murdore]

Audited what the dashboard actually renders against what the pipeline regenerates. Four surfaces had stopped tracking reality — three of them silently. Running more often would not have fixed any of them, because each was write-once by construction rather than merely infrequent.

1. The video player was dead on every page

data-builder.ts never loaded .env. A standalone npm run dashboard:data therefore computed CONFIG.VIDEOS_DIR from an unset INSTAGRAM_USERNAME, looked for mp4s in videos/user_saved, found none, and emitted hasVideo:false / videoPath:null for all 478 videos. The detail page renders its <video> behind {#if detail.videoPath}, so nothing was playable anywhere.

In-pipeline runs were unaffected because runner.ts loads dotenv — which is exactly why this survived so long: the scheduled job produced a working dashboard, and any manual rebuild quietly replaced it with a broken one. Fixed with the same side-effect import runner.ts:1-5 already documents.

Verified: standalone rebuild now yields videoPath 478/478.

2. Research and verification were write-once

Both agents skipped any entry without an error, so their results never changed once written:

	Still from 2026-05
URL liveness checks	355 / 425 (84%)
Verification verdicts	355 / 473 (75%)

/tools was rendering three-month-old live/dead status, and a not_verified verdict was terminal however much the underlying links had changed.

A bare TTL would hand the next run the whole backlog at once, so src/pipeline/refresh.ts pairs it with a per-run cap. New and errored items never consume that budget, so a large intake can't starve the backlog and a run full of failures still makes progress. At ~20/run × 4 runs the May backlog drains in about five days, then holds flat.

Knobs: REFRESH_TTL_DAYS=30, REFRESH_MAX_PER_RUN=20 (0 disables).

3. /tools hid 30% of the corpus

Tools were collected only from verified videos, so 145 of 478 — and 59% of anything saved recently — contributed nothing. A new save stayed invisible until it happened to verify well on the first attempt. They're included now and carry their status in a new Verified column, so the table distinguishes them instead of the builder discarding them.

670 → 957 tools.

4. Schedule: 4×/day, with a deep/shallow harvest split

Harvest at 00/06/12/18, pipeline 30 minutes later. One deep 40-scroll sweep a day walks the whole saved feed; the other slots scroll 6, since they're only chasing the last few hours — a deep pass every slot would multiply our Instagram footprint for content the daily sweep already has.

5. Single-instance lock (required by the above)

Six-hour slots against a ~20min run leave plenty of room, but a stalled model call or a large intake can overrun. Two concurrent runners read-modify-write the same JSON, and the failure is silent: the loser overwrites with a copy it read before the winner's changes existed, so work vanishes with no error anywhere.

macOS ships no flock(1), so the guard is an atomic mkdir with stale-PID reclaim — a killed run (rc=137 has happened here) must not wedge the schedule permanently. A blocked run exits 0 so launchd records a skip, not a failure.

Testing

• npm test — 501 tests across 58 files; 13 new cover the refresh gate's TTL boundary, cap exhaustion, error-retry, budget starvation, and unparseable/missing timestamps
• Lock acquire / concurrent-refusal / release-on-exit / stale-reclaim exercised directly
• npx tsc --noEmit clean; svelte-check 490 files, 0 errors
• Both plists plutil -lint clean and load with 4 slots each; IG_PIGGYBACK_SCROLLS unpinned so the script picks depth
• The 10# hour prefix is load-bearing — bare $((08)) errors with "value too great for base", which would have broken the 08:00-era schedule

[github-actions]

Single Commit Policy — COMPLIANT

Status: 1 commit | Valid format | No merge commits | Ready for merge

Commit details

• Hash: aee7b30adbfdc445ced4055d1fd7c66c2637514f
• Message: feat(pipeline): keep every rendered surface fresh on a 4x-daily schedule
• Author: Sachin Sharma

---

Automated validation — dopamine single-commit enforcement