Video ingress pipeline for collecting and extracting frames from YouTube.
search → frames
Search finds videos and records them in a SQLite pipeline DB. Frames pulls pending URLs from the DB, fetches each stream directly, and extracts JPEGs — no full download required. Everything is resumable and idempotent.
# 1. Build the image
docker compose build
# 2. Add a grabby.yaml to your workdir (see format below)
# 3. Run the pipeline
docker compose run --rm grabby python search.py
docker compose run --rm grabby python frames.py --interval 20 --sleep 1Output lands in $GRABBY_WORKDIR/{source}/{category}/ as JPEGs, with pipeline.db and grabby.yaml at the workdir root.
The repo is bind-mounted into the container, so code edits take effect immediately — only rebuild (docker compose build) when requirements.txt or the Dockerfile changes. If the long command gets old:
alias grabby='docker compose run --rm grabby'workdir/ (or any dir set via GRABBY_WORKDIR)
grabby.yaml ← you provide this
pipeline.db ← created automatically
shards/ ← per-worker DB copies during a parallel run
youtube/
sports/ ← frames land here
auto/
gym/
search_terms:
sports:
- top sports highlights
- best sports reactions
auto:
- top cars of the year
- driving videos
gym:
- workouts
- weight lifting techniqueCategories become subdirectories. Terms are searched once and skipped on re-runs.
search_terms is a section rather than the whole file, so grabby.yaml has room
for other ingress settings later without changing format.
Ingress config stops here: grabby.yaml decides what gets collected and what
folder it lands in. What gets detected in those images is a separate question,
answered downstream by samantics via
its own coco-spec.yaml and samantics.yaml. A category here needs no
counterpart there — collect hockey frames for months before deciding what to
look for in them.
Searches YouTube for each term and records discovered URLs in pipeline.db. Already-searched terms are skipped automatically.
# Search all terms in workdir/grabby.yaml
docker compose run --rm grabby python search.py
# Single explicit query (note: files URLs under the 'query' category, not your own)
docker compose run --rm grabby python search.py --query "best gym workouts" --limit 50
# Preview what a given limit would search, without hitting the network
docker compose run --rm grabby python search.py --limit 50 --dry-run
# Force a re-search of every term
docker compose run --rm grabby python search.py --rerunsearch_log records the --limit each term ran under, so raising it re-searches
exactly the terms that need it. A term is re-searched when its previous run came
back full — results_count == limit_used, meaning YouTube had more to give — and
the new limit is higher. A term that returned fewer results than it asked for was
exhausted, so a bigger limit buys nothing and it stays skipped.
$ docker compose run --rm grabby python search.py --limit 50 --dry-run
[dry-run] SEARCH [youtube/sports] 'top sports highlights' — was capped at 25, limit now 50
[dry-run] skip [youtube/auto] 'driving videos' — exhausted at 18/25 result(s)
A term that returned zero results is always retried — yt-dlp runs with
ignoreerrors, so a transient network failure looks like an empty result set and
shouldn't be recorded as a settled answer.
Rows written before the limit was tracked have no limit_used. Their limit is
inferred as max(25, results_count) — a run that returned 499 results can't have
had a limit below 499, so old high-limit runs aren't mistaken for truncated ones.
Adjust the floor with --assume-prior-limit N.
Reads pending URLs from pipeline.db and extracts JPEG frames directly from the YouTube stream — no full video download needed. Marks each URL done/failed in the DB so runs are resumable.
# Extract a frame every 20 seconds, 1s pause between videos
docker compose run --rm grabby python frames.py --interval 20 --sleep 1
# Specific timestamps
docker compose run --rm grabby python frames.py --timestamps 1m30s 2m45s 4m10s
# Filter to one category
docker compose run --rm grabby python frames.py --interval 20 --category gym
# Retry previously failed videos
docker compose run --rm grabby python frames.py --interval 20 --retry-failed
# Single URL (bypasses DB, useful for testing)
docker compose run --rm grabby python frames.py https://youtube.com/watch?v=abc123 --interval 30Frames per category, and whether the JPEGs on disk still match the DB.
docker compose run --rm grabby python stats.py # table + disk reconciliation
docker compose run --rm grabby python stats.py --no-disk # DB only, skips the filesystem walk
docker compose run --rm grabby python stats.py --json # same numbers, machine-readablecategory frames videos files dupes missing extra
gym 48,509 4,855 46,143 2,366 0 0
pickleball 47,806 4,841 40,203 415 7,188 0
The disk columns exist because frames is an append-only log of extractions,
not an inventory. It records what was written, never what survived. Anything
that removes JPEGs afterwards — a dedup pass, a manual cull, a half-finished
copy — leaves the rows behind, so the files column is the real dataset size.
The three drift columns mean different things and have different fixes:
dupes— one JPEG, several rows. Now prevented by aUNIQUEindex (see Pipeline DB); a pre-existing pile needs clearing once.missing— rows whose file is gone. Data loss, or a transfer that never finished. Sort the basenames: a clean alphabetical cut means a copy stopped partway, which is a very different problem from frames failing to extract.extra— files with no row. Usually a merge that dropped a shard.
Matching is by basename within <workdir>/<source>/<category>/, because
frames.file_path stores container paths (/workdir/...) that don't resolve on
the host whenever GRABBY_WORKDIR points elsewhere. Filenames carry the video
id, so they're unique within a category. The DB is opened read-only, so this is
safe to run mid-extraction.
Downloads full MP4 files when you need the video itself rather than just frames. Not required for the frames pipeline.
# Download a single URL
docker compose run --rm grabby python download.py https://youtube.com/watch?v=abc123
# Clip: extract a time range only
docker compose run --rm grabby python download.py https://youtube.com/watch?v=abc123 --start 1m30s --end 2m45s
# Member-only / age-gated content: export cookies on the host first
./docker/export-cookies.sh firefox
docker compose run --rm grabby python download.py https://youtube.com/watch?v=abc123 \
--cookies /workdir/cookies.txt--browser reads the cookie store directly, which only works outside the container — inside, use --cookies with a file exported by docker/export-cookies.sh.
Frame extraction is I/O-bound — yt-dlp resolves the stream, then ffmpeg seeks into it — so N workers scale nearly linearly. The three pieces that make that safe:
Shard the queue by id, not by category. --shard i/n keeps only rows where
id % n == i. Category sharding sounds natural and doesn't work: whenever one
category dominates the backlog it becomes the floor on total runtime. With a
backlog that is 64% one category, splitting by category caps you at 1.6×, while
id % 8 gives 8.0×.
One SQLite file per worker. SQLite is single-writer; N workers on one
pipeline.db over a bind mount will corrupt it. shards.py copies the DB per
worker (via SQLite's backup API, so a live -wal comes along) and folds the
results back afterwards.
A circuit breaker. --max-consecutive-failures (default 10) stops a run
rather than draining the queue into failed rows. A bot_check or rate_limit
stops it immediately regardless — those mean YouTube is refusing you, not this
video, and continuing just converts the backlog into failures at N× speed.
# 1. split the queue
docker compose run --rm grabby python shards.py prepare 8
# 2. run 8 workers (WORKERS must match --scale)
WORKERS=8 docker compose up --scale worker=8 worker
# 3. watch from another shell
docker compose run --rm grabby python shards.py status
# 4. fold the results back into pipeline.db
docker compose run --rm grabby python shards.py mergeEach replica claims a shard for itself. Compose gives a container no way to know
which replica it is — --scale names the containers grabby-worker-1..N but
leaves $HOSTNAME as the container ID, and all replicas share one service
definition — so instead each worker races to mkdir shards/.claims/<i> and the
winner owns shard <i>. --scale alone sets the width, with no per-worker
config. shards.py prepare clears the claims; if a run dies hard and you restart
without re-preparing, clear them with rm -rf <workdir>/shards/.claims.
Workers start
WORKER_STAGGER seconds apart (default 3) so N of them don't resolve their first
video in the same second. Extraction settings come from FRAMES_ARGS:
WORKERS=4 FRAMES_ARGS="--interval 30 --sleep 2" docker compose up --scale worker=4 workerMerging is idempotent — a second merge claims nothing and inserts nothing — so
re-running it after a partial failure is safe.
The limit isn't CPU, it's how much YouTube tolerates. Only one request per video
is bot-check-sensitive (the yt-dlp metadata resolve); the ~10 range reads ffmpeg
makes afterwards go to a CDN with a signed URL. So the number that matters is
metadata resolutions per minute, which is workers ÷ seconds-per-video.
Note that --sleep is a much smaller lever than it looks: at ~115s per video,
--sleep 1 is under 1% of the cycle. What paces you is the work itself.
Start at 4, watch shards.py status for bot_check / rate_limit tags, and go
up from there. The failure tags exist so that decision is evidence rather than
vibes.
WARNING: [youtube] <id>: n challenge solving failed: Some formats may be missing.
Ensure you have a supported JavaScript runtime and challenge solver script
distribution installed.
This is not a block. yt-dlp needs a JS runtime and the EJS solver scripts to
decrypt YouTube's n parameter, and this one message covers every way either can
be missing. Two separate causes bit this image:
The runtime was too old. Debian bookworm's nodejs package is 18.x, and
yt-dlp declares MIN_SUPPORTED_VERSION = (22, 0, 0) for node
(yt_dlp/utils/_jsruntime.py). Node 18 is detected, marked unsupported and
skipped — indistinguishable from node being absent, since node --version works
fine. The Dockerfile now installs a pinned Node 22 tarball instead of the distro
package.
The solver scripts weren't in the image. yt-dlp vendors the solver core
plus deno/bun lib variants, but not the generic yt.solver.lib.js that the node
runtime needs. Without the yt-dlp-ejs package it is downloaded from GitHub on
every container start — eight downloads per run here — and any failure surfaces
only as this warning.
check-solver.py covers both, asking yt-dlp's own detection rather than looking
for binaries — a runtime can be present, on PATH and perfectly runnable while
yt-dlp still refuses it:
docker compose build --no-cache
docker compose run --rm grabby python check-solver.py [ok ] yt-dlp — 2026.07.04
[ok ] node runtime — v22.20.0 at /usr/local/bin/node (yt-dlp needs >= 22.0.0)
[ok ] yt-dlp-ejs package — 0.8.0
[ok ] yt.solver.lib.min.js — 147561 bytes, hash accepted
[ok ] yt.solver.core.min.js — 6945 bytes, hash accepted
It exits non-zero if anything is missing, so it also works as a preflight before a long run. The hash check matters as much as the presence check: if yt-dlp and yt-dlp-ejs drift apart in version, the scripts are silently rejected and yt-dlp falls back to the GitHub download — the same failure with an extra step.
It matters more than a missing-formats warning suggests: an unsolved n
parameter yields stream URLs that YouTube throttles or rejects, which comes back
as forbidden (403) tags — so a broken solver and a pile of 403s are usually the
same problem.
Failed at 20.0s: ffmpeg error -- Server returned 403 Forbidden (access denied)
Failed at 40.0s: ffmpeg error -- Server returned 403 Forbidden (access denied)
Every seek failing, rather than the occasional one, is a different problem from a signed URL expiring mid-extraction. Two distinct causes have produced it here.
Missing HTTP context (fixed). A googlevideo URL is bound to the client that
requested it. yt-dlp resolves it with its own User-Agent and cookie jar; ffmpeg
then fetched it with Lavf/... and no cookies, and Google answered 403 for every
byte range. get_direct_url now carries that context across as ffmpeg
-headers / -cookies, the same way yt-dlp's own FFmpegFD downloader does.
No fetchable URL at all (SABR). YouTube requires a GVS PO token for
playback on most player clients — web, mweb, android, ios, tv_simply
and others — and without one every range request answers 403 regardless of how
good your auth is. Only tv, web_embedded and android_vr are exempt, and of
those only tv accepts account cookies, which is why this pipeline used tv for
a long time.
Then YouTube's SABR-only rollout took tv's plain https formats away:
WARNING: [youtube] <id>: Some tv client https formats have been skipped as they
are missing a URL. YouTube may have enabled the SABR-only streaming experiment
for your account.
Under SABR there is no independently fetchable URL to hand a separate ffmpeg process — yt-dlp streams the media through its own protocol handling. New cookies, a new account and a newer yt-dlp all fail to fix it, because it isn't an auth problem.
PLAYER_CLIENTS in frames.py is now web_safari,tv,default. web_safari
serves HLS (m3u8) formats, which need no PO token and whose segment URLs are
ordinary https — so ffmpeg range-requests work exactly as before. tv is kept
behind it because it's still the better client where it works.
Cookies are load-bearing — do not delete them to "fix" a 403. tv returns
everything DRM'd without them. (An older note here said 403s get worse with
cookies; that applied only to the anonymous-ffmpeg case above, which no longer
exists.)
The discriminating test, when this recurs — does yt-dlp fetch bytes that ffmpeg can't?
docker compose run --rm grabby yt-dlp --download-sections "*00:00:20-00:00:21" \
-f "bv*[height<=1080]" --cookies /workdir/cookies.txt \
-o "/tmp/sec/%(section_start)s.%(ext)s" "<url>"Success there with 403s in the pipeline means the handoff is broken, not the auth — an architecture problem, not a credentials one.
The section-download fallback. When the fast path yields nothing — no URL
resolved, or a URL where every seek 403s — frames.py refetches through the
yt-dlp CLI: one invocation for all N sections (each call re-resolves metadata,
and metadata resolution is the bot-check-sensitive part), one frame per clip,
temp files removed afterwards. Frames land on the keyframe at or before the
requested timestamp. It is genuinely a fallback: across ~6,500 videos on the
first run after web_safari landed, it fired twice.
This area moves fast. https://github.com/yt-dlp/yt-dlp/wiki/PO-Token-Guide is
the live source; if web_safari falls too, the next levers are a PO token
provider plugin or YouTube Premium cookies (Premium needs no GVS PO token on any
client).
Every failure is recorded with a tag, so the reasons are countable afterwards instead of a wall of unique strings:
| tag | meaning | stops the run |
|---|---|---|
bot_check |
"Sign in to confirm you're not a bot" | yes |
rate_limit |
HTTP 429 | yes |
forbidden |
HTTP 403 — a signed URL expiring, or SABR (see above) | no |
age_gate, private, members_only, removed, geo, live |
this video, not you | no |
no_duration, no_format |
metadata missing | no |
other |
unrecognised — worth reading | no |
A video where every seek failed is recorded as failed, not as done with zero frames. Otherwise a block would look like success and the video would leave the queue for good.
A video that yields some frames is done — it is not going back in the queue —
but 6 of 20 is not the same as 20 of 20, and frames_count alone can't tell them
apart without knowing what was asked for. So the shortfall is recorded in
frames_error on a done row, tagged like the failure reasons are:
[partial] 6/20 frame(s): ffmpeg error -- Server returned 403 Forbidden
Countable the same way:
SELECT COUNT(*) FROM urls WHERE frames_error LIKE '[partial]%';Corrupt JPEGs count toward the shortfall too — they're deleted on write, so without that the tally would claim frames that don't exist. A later clean re-run clears the record back to NULL.
All scripts resolve paths relative to a workdir. On the host it's whatever GRABBY_WORKDIR points at; compose bind-mounts that directory to /workdir inside the container, so container-side paths are always under /workdir.
.env file (recommended, gitignored):
GRABBY_WORKDIR=/path/to/workdir
Inline (one-off override):
GRABBY_WORKDIR=/tmp/test docker compose run --rm grabby python search.py--workdir flag (explicit per-command, container path):
docker compose run --rm grabby python frames.py --workdir /workdir/other-dataset --interval 20If GRABBY_WORKDIR is unset, compose falls back to ./workdir in the repo.
All scripts accept --source (default: youtube) and --db (default: <workdir>/pipeline.db). Run any script with --help for the full list.
pipeline.db is a SQLite file at the workdir root. It tracks:
search_log— which terms have been searched, withresults_countand thelimit_used(enables resume and limit-aware re-search)urls— every discovered URL withframes_status(pending/done/failed)frames— one row per extracted JPEG with its timestamp
frames.file_path carries a UNIQUE index. Re-extracting a video is legitimate
— a retry, a resumed run — and it overwrites the same JPEG, so insert_frame
uses INSERT OR IGNORE and the second write is a no-op. Without that the ledger
inflates silently: one run left 5,352 duplicate rows behind, and the count is the
number everything downstream trusts. The index will not build on a DB that still
has duplicates, so clear them once first:
DELETE FROM frames WHERE rowid NOT IN (SELECT MIN(rowid) FROM frames GROUP BY file_path);
UPDATE urls SET frames_count = (SELECT COUNT(*) FROM frames f WHERE f.url_id = urls.id)
WHERE frames_status = 'done';
VACUUM;For counts, reach for stats.py rather than ad-hoc SQL — it reconciles against
what's on disk, which the DB alone cannot tell you. To inspect state directly:
sqlite3 "$GRABBY_WORKDIR/pipeline.db" # .headers on / .mode box / .tablesReads are safe while a run is in progress (WAL). Writes are not — do those
with the containers down, and before shards.py prepare, since the shard DBs are
copies and prepare would overwrite your edit.
Grabby's output is designed to feed directly into samantics, an auto-labelling pipeline that deduplicates frames, runs SAM3 segmentation, and exports COCO-format annotations. Point GRABBY_WORKDIR in samantics at the same directory.