samantics

Auto-labelling pipeline for vision datasets. Takes raw frames from an ingress directory, deduplicates them, auto-labels with SAM3, curates the results, and exports COCO-format annotation JSON ready for training.

dedupe → label → curate

dedupe — walks an ingress directory, runs dhash + ORB deduplication, copies accepted frames into a content-addressable image store
label — runs SAM3 over accepted images, writes bounding boxes + RLE masks to labels.jsonl
curate — auto-tiers classes from the labels (positive / hard-negative / discard), keeps everything by default, splits into train/val, exports train.json + val.json flat into the dataset dir (no config required)

Everything is resumable — pipeline.db tracks every image through all three stages so interrupted runs pick up where they left off.

Quick start

# 1. Copy .env.example and fill in your paths
cp .env.example .env

# 2. Build the Docker image
docker compose build

# 3. Deduplicate frames from ingress
docker compose run --rm dedupe

# 4. Try query terms on a few frames first — minutes, on a laptop
./notebook.sh

# 5. Auto-label (needs GPU). Sample a slice before committing the full pass.
docker compose run --rm label --limit-per-category 20
docker compose run --rm label

# 6. Look at what it produced
docker compose up browse                    # UI on :5151

# 7. Curate and export COCO JSON — both directories are literal curate.py
#    flags, EVERY run, relative to the registry root (REGISTRY_ROOT in .env is
#    mounted at /registry = the working directory, so the same command works on
#    every machine). There is deliberately no default: which labels feed which
#    dataset is the decision, and argparse refuses to start without it.
docker compose run --rm curate --label-set hotshoe-datasets-dev/labels/samantics \
  --dataset-dir hotshoe-datasets-dev/datasets/sportball --dry-run
docker compose run --rm curate --label-set hotshoe-datasets-dev/labels/samantics \
  --dataset-dir hotshoe-datasets-dev/datasets/sportball

Step 4 is the cheap one to skip and the expensive one to have skipped: a query that finds nothing costs a full labelling pass plus a relabel to discover.

Each stage tracks its own progress in pipeline.db, so any of them can be interrupted and re-run, and picks up exactly where it stopped.

To chain stages so the next starts only if the previous succeeded, use the shell — && already means "on success", and every stage exits non-zero on failure:

docker compose run --rm dedupe && docker compose run --rm label

For a run that takes hours, start it detached so it does not die with your shell (and, on a remote box, with your SSH session):

docker compose run -d --name samantics-job dedupe
docker logs -f samantics-job        # safe to Ctrl-C; the job keeps going
docker stop samantics-job           # SIGTERM, so the stage stops cleanly

For an unattended chain that survives losing your shell, put the && inside one detached container rather than running it from your terminal:

docker compose run -d --name samantics-job --entrypoint bash shell \
  -c "python dedupe.py && python label.py"

The stages are ordinary programs, so the shell is all the sequencing they need.

Configuration

All paths are set via environment variables. Copy .env.example to .env (gitignored) and edit:

LABEL_SET_DIR=/path/to/labels/<set>     # taxonomy.yaml, samantics.yaml, exemplars/, data/
GRABBY_WORKDIR=/path/to/ingress         # source frames (read-only in Docker)
REGISTRY_ROOT=/path/to/ml-registry      # curate's world, mounted at /registry
MODEL_CACHE=/path/to/models             # HuggingFace cache; keep it OUT of the data root
HF_TOKEN=hf_...                         # SAM3 is gated

LABEL_SET_DIR is picked up automatically by dedupe, label, stats and browse; --label-set overrides it for a single run. curate is the odd one out — deliberately. It reads a label set and writes a dataset, and WHICH feeds WHICH is a per-run decision, so those two paths are never defaulted anywhere. The split: .env holds the fact about the machine (REGISTRY_ROOT — where the registry lives on this disk); the per-run flags are registry-relative and therefore identical on every machine:

docker compose run --rm curate \
  --label-set hotshoe-datasets-dev/labels/<set> \
  --dataset-dir hotshoe-datasets-dev/datasets/<name> [--dry-run]

The registry is mounted at /registry, which is also the working directory — that is what makes relative paths resolve. Run it bare and curate.py's own argparse refuses with usage — both flags are required. Repeat --label-set to merge several sets; later sets win per image.

A label set is not a curated dataset. A label set is written once and read by any number of curations; a curation is rebuilt freely and owns no labels. They are separate variables so a stale path cannot quietly point labelling at a dataset directory.

notebooks/preview.ipynb is the exception: it is pre-curation and reads only GRABBY_WORKDIR, MODEL_CACHE and HF_TOKEN, finding the label set beside the ingress tree.

Label set layout

{LABEL_SET_DIR}/
  taxonomy.yaml        ← the classes that exist: id, name, grouping
  samantics.yaml       ← what to query where, and how
  exemplars/           ← cut-in reference images (optional)
    curling_stone.png        ← used by any bucket querying curling_stone
    mini_golf/
      golf_club.png          ← used by the mini_golf bucket only
    _disabled/               ← leading _ or . : never read
  data/                ← everything the machine writes
    pipeline.db        ← SQLite pipeline state (WAL mode)
    labels.jsonl       ← SAM3 output: boxes (XYXY), category_ids, RLE masks, scores

The split is by OWNER, and the boundary is a DIRECTORY rather than a list of filenames. Above data/ is authored, reviewed and versioned in git; inside it is machine-written and versioned in DVC. That means one .gitignore line per label set (/data/), one dvc add, and nothing ever tracked or ignored file by file. A curated dataset goes further: the WHOLE directory is machine-written (train.json, val.json, test.json, dataset.jsonl, curation-report.json, flat — no subdirectories) and is one dvc target (registry.sh: datasets/<name>), with only the .dvc pointer in git. There is no required config; a curation.yaml placed in the dataset dir overrides curate's defaults when a dataset truly needs it — see curate.py.

Both label-set config files are YAML for editing convenience; everything samantics writes — labels.jsonl, the dataset jsons — is still JSON.

Buckets vs supercategories

These are different things and conflating them is the usual source of confusion.

A bucket is an ingress directory: grabby writes {GRABBY_WORKDIR}/{source}/{bucket}/*.jpg, and a bucket is a place frames came from. A supercategory is a grouping of classes in taxonomy.yaml — a kind of gear.

Usually they share a name and nothing needs saying: the curling bucket runs the curling supercategory's classes. But mini_golf is a place, not a kind of object — the clubs in it are golf clubs. So the bucket borrows:

# samantics.yaml
buckets:
  mini_golf: [golf]

Now mini_golf frames are labelled golf_club and golf_ball, which makes them extra golf examples from a different venue rather than a starved mini_golf_putter class competing with golf_club for the same pixels. A bucket can list several supercategories — a "putting on curling ice" bucket would take [golf, curling].

This is why taxonomy.yaml contains no mini_golf entry. The taxonomy holds classes; where to look for them is a labelling decision and lives in samantics.yaml.

`taxonomy.yaml` — the classes that exist

One class, one id, one home. supercategory is the grouping SAM3 queries are built around and COCO's supercategory field is filled from it on export.

categories:
  person:
    - id: 1
      name: person
      keypoints: [nose, left_eye, right_eye, left_shoulder, right_shoulder]
      skeleton:
        - [1, 2]
        - [1, 3]

  golf:
    - { id: 400, name: golf_ball }
    - { id: 401, name: golf_club }
    - { id: 402, name: golf_flagstick }

  curling:
    - { id: 100, name: curling_broom }
    - { id: 101, name: curling_stone }

  # Declared with no classes — person-only by design. Its frames are hard
  # negatives.
  negatives:

Ids can be any integers; leaving gaps between groups makes it easy to add classes later. Retired ids should not be reused — an old labels.jsonl still refers to them.

There is no info or licenses block: those describe a dataset, and one label set feeds many. They live in the curation's coco: section.

What gets labelled

Ingress collects whatever grabby.yaml says. Three cases decide what labelling covers:

the bucket	what `label.py` does
a supercategory with classes, or a `buckets` entry	queries those classes, plus `always_query`
a supercategory declared with no classes	`always_query` only — hard negatives
neither	doesn't touch the images; they stay `pending`

The third case is why ingress and labelling move independently: collect hockey frames for months before deciding what to detect in them. Buckets nothing is configured for are reported at startup rather than vanishing silently:

Not in taxonomy.yaml or samantics.yaml, left unlabelled: boxing (3), cricket (2)
  5 image(s) stay pending. Add a matching supercategory to the taxonomy, or a
  'buckets' entry pointing at an existing one.

Two YAML gotchas the loader guards against: a bare 1.0 or 2025-09-17 is a float and a date, not a string (quote them), and a class or supercategory literally named no, on, or yes parses as a boolean (quote it, or the loader refuses the file). Duplicate class ids and duplicate class names are rejected too.

Converting to and from flat COCO JSON

python spec_convert.py taxonomy.json taxonomy.yaml    # and the reverse

Nesting on the way in, flattening on the way out. A supercategory declared with no classes is the one thing flat JSON cannot represent; converting out and back drops it, and the converter warns when it does.

`samantics.yaml` — what to query where, and how

Optional in principle; in practice this is where every decision that changes the output lives.

# Ingress bucket -> whose classes to look for in it. Only for buckets whose
# name disagrees with the gear in them; everything else defaults to the
# same-named supercategory.
buckets:
  mini_golf: [golf]

# Wording only. A class with no entry is queried as its own name with the
# underscores taken out — golf_ball -> "golf ball" — which is right nearly
# always, so only list the exceptions.
#
# Keyed by BUCKET, so one class can be worded differently per venue. Use a YAML
# merge key to inherit another bucket's wording along with its classes.
queries:
  golf: &golf
    golf_flagstick: flagstick with flag
  mini_golf:
    <<: *golf                 # keeps "flagstick with flag" here too
    golf_club: putter         # every mini golf club is a putter
  curling:
    curling_stone: [curling stone, overhead curling stone]   # several, unioned

# Queried on every image whatever its bucket. No default — nothing is queried
# unless you ask. This is what gives a person-only bucket something to find.
always_query:
  - person

# Detection filtering. Changing either makes previously labelled images stale.
score_thresh: 0.5
iou_thresh: 0.9

# Cut-in pass. See "Cut-in exemplars" below.
paste_when: always            # always | empty | never
dedupe_iou: 0.55
paste_scales:
  "*":           [0.15, 0.30]
  curling_stone: [0.06, 0.12]

# Per-class size floors, as a fraction of frame AREA. The most dangerous
# setting in the file: your smallest classes are also your most valuable.
limits:
  person: {min_area: 0.0004}

Everything that changes the output belongs here, not on the command line. --batch-size and --cache-dir are properties of the machine you happen to be on, so they stay flags and are rejected if you put them in the config. The reverse rule matters more: a setting that changes what gets labelled but lives only in a flag is a run you cannot reproduce and a preview you cannot trust. CLI flags still override the file for a one-off, and the resolved values are printed at startup and recorded in the provenance hash.

Cut-in exemplars

A text phrase can miss an entire camera angle — "curling stone" finds the side-on hero shot and little from the overhead sheet camera. Cut-in prompting composites a hand-cut instance of the object into the frame, boxes it, and lets the geometry encoder pool real features for the viewpoint text does not cover. The pasted object itself is detected and then excluded; labelling a fake object into a training set is the one outcome that genuinely poisons things.

Files live under exemplars/, matched to classes by longest name prefix, so curling_stone_top_1.png and curling_stone_top_2.png are both curling_stone. A file in a subdirectory named after a bucket is used only in that bucket — exemplars/mini_golf/golf_club.png is a putter, the wrong reference for a fairway club. Directories starting with _ or . are never read, which is how _disabled/ parks cutouts you are not using.

paste_scales is the setting to get right. The exemplar is a size prompt as much as a shape prompt: paste a stone three times life size and you have asked SAM3 for an object that is not in the picture. Values below 1.0 are a fraction of contain-fit, so one number means the same visible size at any resolution. Two scales per class — only two paste positions are defined — and bracket the real size rather than guessing one value.

Where a cutout lands is chosen per frame, scoring the candidate corners against what the text pass already found, so a paste falls on empty background rather than on the object you wanted labelled. Corners rather than centre: a paste over the subject both hides it and gets the real detection thrown away as is_the_paste.

Always-queried classes

always_query adds classes to every image's query set, whatever its bucket. There is no default — with it unset, a bucket declared with no classes has nothing to query, so label.py skips it and leaves those images pending:

Always queried: nothing (set always_query in samantics.yaml, or pass --always-supercategory person)
Declared with no detections: negatives -> queries: nothing
Skipping negatives (5 images): nothing to query. Left pending.

Every run prints what it resolved to, so a missing setting shows up in the first few lines rather than in the annotation counts a day later:

Taxonomy supercategories: golf, person
Bucket mini_golf <- classes of golf
Override query: mini_golf.golf_club -> putter
Always queried: person
Thresholds: score=0.5 iou=0.9 | model: facebook/sam3

Labelling is one-shot per image. An image labelled person-only under an empty supercategory is marked done and won't be revisited when you add classes for it. To pick those back up:

python label.py --relabel-supercategory hockey

What was this image actually labelled with?

Every labelled image records the query set it was produced under. pipeline.db gets a label_configs table (hash → the full resolved config) and images.label_config pointing at it, so the question is a join rather than a guess.

The recorded unit is the query set for one supercategory, not the whole file. Adding a hockey class makes hockey images stale and leaves curling alone; changing score_thresh makes everything stale. Hashing the whole config would flag every image in the dataset for any edit.

label.py reports it on every run:

Labelled under a different config: curling (13277), pickleball (25460)
  Re-run with --relabel-stale to send them back to pending.

docker compose run --rm label --relabel-stale

Images labelled before config tracking existed have no recorded config. They're reported as unknown and never counted as stale — assuming they match the current config, or that they don't, would both be wrong:

67462 image(s) labelled before config tracking — config unknown, not counted as stale.

To re-label those, target them explicitly with --relabel-supercategory <name>.

`dedupe.py` — deduplicate and ingest frames

Walks the ingress directory ({GRABBY_WORKDIR}/{source}/{category}/*.jpg), discovers all frames, and runs two-pass deduplication:

dhash — perceptual hash; images within a Hamming distance threshold are near-duplicates globally across the whole dataset
ORB — per-video feature matching; catches similar frames from the same video that dhash misses (slight motion, lighting changes)

Accepted frames are recorded in place — the ingress file is the image. This used to copy to {LABEL_SET_DIR}/images/{dhash}.jpg, which re-encoded every frame: a generation of JPEG loss and a second copy of bytes DVC could not deduplicate against ingress. Materialising a private copy is the dataset build's job, not selection's.

# Run everything (paths from .env)
docker compose run --rm dedupe

# Filter to one category
docker compose run --rm dedupe --category gym

# dhash only, no ORB (faster)
docker compose run --rm dedupe --no-orb

# Dry run — count without writing
docker compose run --rm dedupe --dry-run

# Parallelism: defaults to every usable core minus a small reserve for the OS,
# docker, and this process's own (sequential) decision pass.
docker compose run --rm dedupe # pin the worker count --jobs 32
docker compose run --rm dedupe # or just change the headroom --reserve 4

# Per-video ORB comparison window. Near-duplicates are adjacent frames, so
# comparing against every frame a video ever accepted is quadratic for no gain.
docker compose run --rm dedupe # 0 = unbounded (the old behaviour) --orb-window 80

Extraction (JPEG decode + ORB) runs across all workers; the accept/reject decisions stay sequential, because whether frame N is a duplicate depends on which of 1..N-1 were accepted. Decisions are identical at any --jobs value.

Progress prints every 200 images and at least every 15 seconds, so a slow stretch never looks like a hang:

[2400/51203]  4.7%   38.2 img/s  eta 0h21m  elapsed 0m62s  accepted=310 rej_dhash=2088 rej_orb=2 failed=0

`label.py` — SAM3 auto-labelling

Reads accepted images from pipeline.db, runs SAM3 inference in batches, appends results to labels.jsonl. Each image is marked done or failed in the DB so runs are resumable.

Requires taxonomy.yaml in LABEL_SET_DIR. samantics.yaml is optional in principle, but every setting that changes the output lives in it — thresholds, query wording, bucket borrowing, cut-in behaviour — so a real run has one.

The flags that change output (--score-thresh, --iou-thresh, --paste-when, --dedupe-iou, --always-supercategory) all default to "use the config" and exist for one-offs. Reach for them and the run stops matching the preview that validated it, with nothing on disk recording why.

# Label all pending images
docker compose run --rm label

# Filter to one category, retry previously failed
docker compose run --rm label --category animal --retry-failed

# Batch size is a property of the box, not of the dataset, so it stays a flag.
docker compose run --rm label --batch-size 32

# Person-only pass over a bucket with no classes defined yet
docker compose run --rm label --category negatives

# Saturate the GPU: N parallel workers, one per shard. Image decode,
# preprocessing and mask RLE encoding are CPU-bound and serialize with GPU
# work in a single process; shards fill the gaps. Disjoint ids, WAL-mode DB,
# lock-guarded jsonl appends — safe to run concurrently. Each worker loads
# its own model copy, so budget VRAM. Run any --relabel-* reset once,
# without --shard, before launching the fleet.
for i in 0 1 2 3; do
  docker compose run --rm label --shard $i/4 &
done; wait

`curate.py` — build a dataset from label sets

Reads label sets, selects, balances, splits and exports COCO. Everything is driven by <dataset-dir>/curation.yaml; there are no tuning flags.

# needs only PyYAML — no GPU, no torch
python curate.py --label-set .../datasets/<name> \
                 --label-set .../<registry> --dry-run
python curate.py --label-set .../datasets/<name> \
                 --label-set .../<registry>

It never writes to a label set, so the same labels can feed any number of datasets and re-labelling never clobbers a curation decision.

`browse.py` — look at labels in FiftyOne

Reads data/labels.jsonl directly, so you are inspecting raw labeller output rather than a curated export — which is where the failures worth catching live.

docker compose up browse                    # UI on :5151
docker compose up browse -- --class pickleball --max-score 0.65   # the false-positive band

up, not run, and not only because it serves a UI: FiftyOne's database is the mongo service, and up starts it. The compose file waits on its healthcheck, because FiftyOne connects on import and does not retry.

Why a mongo service rather than FiftyOne's own embedded one. fiftyone-db publishes wheels for macOS and Windows only, so every Linux install falls through to its sdist, which downloads a MongoDB build for the detected distro — and MongoDB publishes none for Debian arm64. On the GH200 that surfaces as MongoDB could not be installed on your system, or as a mongod that cannot link libcurl.so.4. Chasing it with the base image works but pins Dockerfile.tools to whatever distro fiftyone-db currently likes, and drags curate onto it too. The official mongo image is properly multi-arch and version-pinned.

On a remote box, forward the port:

ssh -L 5151:localhost:5151 lambda -t 'cd ~/src/samantics && docker compose up browse'

`notebooks/preview.ipynb` — try query terms before a run

Labelling 53k frames under a bad query costs GPU hours and then a relabel. This tries terms on a handful of real frames first, in seconds.

./notebook.sh          # venv if needed, deps if stale, then serves JupyterLab

Native, not docker, and on a Mac that is the point: Docker Desktop has no GPU passthrough, so a container cannot reach MPS and SAM3 drops to CPU. The venv gets Metal.

Past a few dozen frames per category, run it on the GPU box instead. Same notebook, a kernel that runs inside this repo's image — see "Notebooks on the GPU box" under Docker below.

The kernel it serves is registered as samantics, displayed Samantics — not "Python 3 (ipykernel)". notebook.sh installs it into .venv on first run, so it is invisible to every other Jupyter on your machine, and KERNEL_DISPLAY=… in the environment renames it.

The name matching the box's kernelspec is the part that matters. A notebook records the kernel it ran on, so preview.ipynb says samantics and resolves at both ends — here to the venv, on the box to the container — with only the display name telling you which. Without that, a notebook saved on the box would open here asking you to pick a kernel. (The venv's stock Python 3 (ipykernel) entry is still in the picker; it belongs to ipykernel itself and is harmless.)

Once you are working on both, the same script carries the file between them:

./notebook.sh --pull            # what would come back from the box
./notebook.sh --pull --force    # bring it back
./notebook.sh --push --force    # send yours up

notebooks/ only, over the lambda ssh alias (override with LAMBDA_HOST in .env). Nothing writes without --force — a bare --pull/--push prints the files it would replace and stops, because both copies are hand-edited and either direction can eat an afternoon's work.

Pull before you deploy. lambda-deploy.sh rsyncs the whole tree with --delete, so it will happily overwrite the box's notebook with yours.

Two cells are yours. §1 is the dials, §4 is the terms; nothing else needs touching.

# §1 — what to look at
CATEGORIES = ["pickleball", "baseball"]   # ingress category dirs; §2 lists them
N, SEED, PAGE = 10, 0, 0                  # images PER category, then which ones
DETECT_THRESH, PROD_THRESH = 0.15, None   # None -> samantics.yaml's score_thresh
WITH_PERSON, LABEL_SET = False, None
SHOW_MASKS, MAX_DETS = True, 12

It calls the same code the GPU run calls. §3 invokes detect_batch — the function run_labeling invokes — and then postprocess_results, in the same order with the same arguments, so the boxes in the KEPT panel are the boxes that would reach labels.jsonl. Nothing in the notebook reimplements pasting, scaling, placement or class resolution; every one of those is a call into label.py or common/paste.py.

Settings come from samantics.yaml, not from the notebook. Every knob in §1 defaults to None, meaning "use the config": SCORE_THRESH, PASTE_SCALES, PASTE_WHEN, DEDUPE_IOU. Put a number in one and the preview stops being a rehearsal of the run — which is occasionally what you want, but should be a deliberate act. When a value wins, move it into samantics.yaml and set the knob back to None.

CATEGORY in §1 is an ingress bucket, not a supercategory, so previewing mini_golf resolves through buckets: exactly as labelling does and pulls in exemplars/mini_golf/.

Frames are read straight from {GRABBY_WORKDIR}/{source}/{bucket}/*.jpg — no pipeline.db, no dedupe run in the way. The label set is found beside ingress at {data root}/labels/{set}/; set LABEL_SET in §1 only if you keep more than one.

Three panels per frame. KEPT is what would be labelled. REJECTED is what was thrown away and why — <thr below the score floor, PASTE the pasted object itself, dup N.NN an overlapping box that a higher-scoring one beat. AS PASTED is the frame SAM3 was actually handed, one panel per cutout that ran, titled with what that cutout kept and had rejected — a cutout showing 0 kept is one earning nothing but GPU time.

Solid boxes came from the text pass, dotted from a cut-in. A dup on a solid box means a text detection lost to a more confident cut-in and was replaced by it: the object is still labelled, but the box, mask and recorded score came from the cut-in.

§3b answers "is the paste the right size?" It crops each pasted object at native resolution next to the real detections' size range. The exemplar is a size prompt as much as a shape prompt, and a stone pasted several times life size asks for an object that is not in the frame.

Short and generic beats specific. Queries are sent per bucket and ingress already filtered these frames by search, so a pickleball frame is a pickleball scene — the round thing in it is the pickleball, and balls finds it. plastic ball with holes detects nothing: SAM3 goes looking for the literal description. The scene supplies the context; the query only has to name the shape.

Changing a term or a threshold changes the config fingerprint, so anything labelled under the old set goes stale — see "What was this image actually labelled with?" above. A supercategory gaining its first classes needs --relabel-supercategory <name> instead, since its images are already done from a person-only pass.

This is for choosing a term, not for reviewing output. It runs SAM3 live on a handful of frames; it has no notion of a labelling run, and it is not where you judge one. That is browse.py — FiftyOne holds several label fields per sample, so ground truth and predictions sit side by side on the same image, filterable and sortable, with evaluate_detections() for TP/FP/FN when it comes to that. Reviewing 53k frames is a database problem, and FiftyOne is the database.

`import_existing.py` — migrate from a previous pipeline

If you have data from an older separate-directory pipeline layout (01-selection/, 02-labels/, 03-curation/), import it into pipeline.db without re-running anything:

python import_existing.py --data-root /path/to/old-dataset

# Dry run first to verify counts
python import_existing.py --data-root /path/to/old-dataset --dry-run

Then run curate pointing at the existing labels file:

docker compose run --rm curate --labels-jsonl /path/to/old-dataset/02-labels/labels.jsonl

Pipeline DB

pipeline.db is a SQLite database tracking every image through all three stages:

Stage	Column	Values
select	`select_status`	`pending` / `accepted` / `rejected`
label	`label_status`	`pending` / `done` / `failed`
curate	`curate_status`	`pending` / `included` / `excluded`

Useful queries:

# What failed labelling and why?
sqlite3 $LABEL_SET_DIR/data/pipeline.db \
  "SELECT source_path, label_error FROM images WHERE label_status='failed' LIMIT 20"

# Which videos contributed the most accepted frames?
sqlite3 $LABEL_SET_DIR/data/pipeline.db \
  "SELECT video_id, COUNT(*) n FROM images WHERE select_status='accepted'
   GROUP BY video_id ORDER BY n DESC LIMIT 10"

# Accepted but not yet labelled (work remaining before curate)
sqlite3 $LABEL_SET_DIR/data/pipeline.db \
  "SELECT COUNT(*) FROM images WHERE select_status='accepted' AND label_status != 'done'"

Ingress layout

Frames are sourced from grabby, which organises output by source and category:

{GRABBY_WORKDIR}/
  {source}/              e.g. youtube/
    {category}/          e.g. gym/
      *.jpg              frames named {video_id}_{timestamp_ms}.jpg

Set GRABBY_WORKDIR in .env to point at grabby's output directory.

Docker

compose.yaml loads .env and mounts:

host (from `.env`)	container	notes
`GRABBY_WORKDIR`	`/grabby-workdir`	ingress, read-only — nothing here writes frames
`LABEL_SET_DIR`	`/dataset`	the label set or dataset being written
`MODEL_CACHE`	`/model-cache`	every service that can load SAM3 — `label`, `kernel`, `shell`; sets `HF_HUB_CACHE`
`KERNEL_CONNECTION_DIR`	`/connection-spec`	`kernel` only; set by `docker/kernel.sh`, not `.env`
`.`	`/workspace`	live tree, so edits apply without a rebuild

LABEL_SET_DIR, GRABBY_WORKDIR and MODEL_CACHE are re-exported inside the container as the mount points above, shadowing the host paths .env supplies. Without that, label.py reads MODEL_CACHE as a host path, hands it to from_pretrained(cache_dir=...), and re-downloads SAM3 into a discarded layer on every run while the real cache directory stays empty.

The mount and that shadowing travel together, as the x-model-paths / x-model-env anchors, and every service that can reach SAM3 takes both — which includes shell, because the documented way to chain stages (--entrypoint bash shell -c "python dedupe.py && python label.py") runs label.py through it. A service that loads a model without them does not fail; it quietly re-downloads gigabytes into a container that then exits. dedupe, curate and browse are left plain: ORB/dhash, PyYAML and FiftyOne respectively, no model between them.

mongo has no host mount: it is FiftyOne's database, used only by browse, and its data lives in the fiftyone-db named volume. browse rebuilds its dataset from labels.jsonl each run, so nothing there is worth keeping.

Each service is a command, so arguments pass straight through:

docker compose run --rm dedupe --category curling
docker compose run --rm label --limit-per-category 200
docker compose run --rm curate --label-set <registry-relative> --dataset-dir <registry-relative> --dry-run
docker compose up browse                        # UI on :5151
docker compose run --rm shell python stats.py   # anything else
docker compose run --rm shell                   # interactive bash

Only label and kernel reserve a GPU. curate and browse build from docker/Dockerfile.tools — a slim image without the CUDA stack, so rebuilding after a curation change takes seconds rather than minutes.

Notebooks on the GPU box

kernel is the one service you never run by hand. It is notebooks/preview.ipynb on the GH200: a Jupyter kernel inside this repo's image, started by a notebook server that is not ours — the JupyterLab behind Lambda's UI, running as ubuntu on the host.

Lambda UI  ->  kernelspec  ->  docker/kernel.sh  ->  docker compose run kernel
               (generated by your deploy script, step 5)

Both ends name the kernel samantics, so a notebook carried between them finds a kernel at both without prompting.

docker/Dockerfile's kernel target is the labelling image plus requirements-kernel.txt (ipykernel, matplotlib, ipywidgets) — one pip layer on top of shared ones. Sharing the image is the reason to do it this way at all: the notebook imports label.py's own load_sam3 / run_batch against the same torch, the same CUDA and the same model cache a run uses, so a term that scores in the notebook scores the same in the run. A venv on the host would be a second dependency path and a weaker claim — and that claim is the notebook's whole value.

Two things differ from every other service here:

network_mode: host. The server picks five ZMQ ports and writes them to a connection file before the container exists. On a bridge network the kernel binds them where nothing can reach it, and the UI sits at "Kernel starting" forever. ${KERNEL_CONNECTION_DIR} — exported by kernel.sh — mounts the directory that file landed in.
The kernel image must be pre-built. BUILD_SERVICES="label kernel" in your deployment repo does it. "Builds on first use" means a server starting a kernel against a startup timeout measured in seconds.

When a kernel dies at startup the UI says only that it did. kernel.sh tees every launch to ~/.cache/samantics/kernel.log, which is where the reason is:

ssh lambda 'tail -f ~/.cache/samantics/kernel.log'
docker ps --filter name=samantics-kernel

The box's notebooks/ is downstream of yours: rsync --delete overwrites it on every deploy, so scp back anything you edited in the Lambda UI before redeploying.

Both images run as uid 1000, the ubuntu account on the Lambda box, so what a run writes to the dataset mount is owned by the user who has to clean it up rather than by root. The kernel compares numbers, not names, so matching the uid is the whole mechanism. On a host that numbers its users differently:

docker compose build --build-arg UID=$(id -u) --build-arg GID=$(id -g)

Navigation Menu

Repository files navigation

samantics

Quick start

Configuration

Label set layout

Buckets vs supercategories

`taxonomy.yaml` — the classes that exist

What gets labelled

Converting to and from flat COCO JSON

`samantics.yaml` — what to query where, and how

Cut-in exemplars

Always-queried classes

What was this image actually labelled with?

`dedupe.py` — deduplicate and ingest frames

`label.py` — SAM3 auto-labelling

`curate.py` — build a dataset from label sets

`browse.py` — look at labels in FiftyOne

`notebooks/preview.ipynb` — try query terms before a run

`import_existing.py` — migrate from a previous pipeline

Pipeline DB

Ingress layout

Docker

Notebooks on the GPU box

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
common		common
docker		docker
notebooks		notebooks
tests		tests
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
browse.py		browse.py
compose.yaml		compose.yaml
curate.py		curate.py
dedupe.py		dedupe.py
import_existing.py		import_existing.py
label.py		label.py
notebook.sh		notebook.sh
requirements-browse.txt		requirements-browse.txt
requirements-kernel.txt		requirements-kernel.txt
requirements-label.txt		requirements-label.txt
requirements-notebook.txt		requirements-notebook.txt
requirements.txt		requirements.txt
spec_convert.py		spec_convert.py
stats.py		stats.py

Latest commit

History

Folders and files

Repository files navigation

samantics

Quick start

Configuration

Label set layout

Buckets vs supercategories

taxonomy.yaml — the classes that exist

What gets labelled

Converting to and from flat COCO JSON

samantics.yaml — what to query where, and how

Cut-in exemplars

Always-queried classes

What was this image actually labelled with?

dedupe.py — deduplicate and ingest frames

label.py — SAM3 auto-labelling

curate.py — build a dataset from label sets

browse.py — look at labels in FiftyOne

notebooks/preview.ipynb — try query terms before a run

import_existing.py — migrate from a previous pipeline

Pipeline DB

Ingress layout

Docker

Notebooks on the GPU box

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`taxonomy.yaml` — the classes that exist

`samantics.yaml` — what to query where, and how

`dedupe.py` — deduplicate and ingest frames

`label.py` — SAM3 auto-labelling

`curate.py` — build a dataset from label sets

`browse.py` — look at labels in FiftyOne

`notebooks/preview.ipynb` — try query terms before a run

`import_existing.py` — migrate from a previous pipeline