Snakemake orchestration for the real-data pipeline - #852
Open
cailmdaley wants to merge 71 commits into
Open
Conversation
) Implements the round-2 design in #848: static two-phase execution driven by a parse-time SQLite run index (never a rule input, so appending tiles never invalidates completed work); rules wrap the existing per-unit shapepipe_run config chains at tile/exposure granularity with deterministic RUN_DATETIME=False directory() outputs; in-job completeness count-floor (the ported bash complete_check table, workflow/scripts/completeness.py); one-allocation execution profile for nibi with apptainer software-deployment (the workflow never calls apptainer itself); out-of-DAG run_report verb + onsuccess/onerror hooks; workflow/bin/sp as the committed launcher. First pass touches no module code. Validated by dry-run against a 4-tile P3 subset (219-job compute DAG); prepare phase runs green on nibi. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s.cmd
Snakemake formats params once, so a {threads} placeholder inside params.cmd
survives literally and SMP_BATCH_SIZE never resolves (fork width silently 1,
serial modules — the p3-first10 failure mode). Move '--threads {threads}' into
each rule's shell: string; tile_exp_forest takes none (build_forest.py is
single-threaded symlinking and has no such flag). Completeness table doc
touch-ups ride along.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NA8M1oLZLbWJxwoyTNrTAi
…star-cat store naming Three P0-blocking fixes found by running against real data: - exposures(name): the fabricated per-unit exp_numbers list must carry the original exposure name verbatim (2605805p — get_images matches <name>.fits.fz; the bare dedup id matches nothing). The index now stores both; exp_get_images passes --exp-name from the parse-time dict. - tile_star_cat: the pre-generated store names tiles in ShapePipe's image-number convention (dots->dashes); translate when linking. - profile: PYTHONPATH pinned to this branch's src/ (identical to develop@97e16d50 — orchestration commits never touch src/); NOT shapepipe-prod (drifted to a PR branch mid-run, and the live p3-batch1 job reads it) and not the sif default (frozen pre-#843). Run scoped to the 210/211 overlap quad (19 unique exposures, 15 shared — exercises structural dedup; the zero-overlap 196 quad is kept for the append-invariant test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NA8M1oLZLbWJxwoyTNrTAi
… config - get_images looks up <name>.fits.fz: the store keeps the CFHT 'p' suffix the index's base-id key strips. exposures gains a name column; rules pass --exp-name; the wrapper writes the ORIGINAL name into the fabricated exp_numbers list — and writes it unconditionally (an exists-guard pinned a stale pre-fix file). - Star cats are consumed as DIRECTORIES via the wrapper's $SP_RUN symlinks (the v2.0 mechanism; exp cats are per-CCD, 40/exposure) — dropped the per-unit star-cat file rules that linked nonexistent names nothing read. - get_images completeness counts follow the nibi symlink configs (tile 2, exp 3; the v2.0 4/6 were the canfar vos flavor), verified against p3-batch1. - P0 run config: 210/211 quad (19 unique exps, 13 shared across tiles — exercises structural dedup; the 196 quad had zero overlap), PYTHONPATH pinned to this branch's src (= develop@97e16d50; shapepipe-prod drifted mid-run). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…iterally
Same single-format trap as {threads} (187950c): Snakemake formats the shell
string once, so the escaped {{output}} in tile_exp_forest's params reached
build_forest as a literal './{output}' — all four forest jobs racing one
garbage dir (FileExistsError at 50% of the P0 run). Move --forest {output}
into the shell string. Swept the rules for further escaped placeholders: none.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ac/code/session_01NA8M1oLZLbWJxwoyTNrTAi
…pe is not cosmetic exp_utils.get_exp_output_files hardwires the v2.0 sharded layout into its $SP_EXP glob (<SP_EXP>/<prefix>/<base>/output/run_sp_*/...); a flat forest fails every tile gather stage with 'No split_exp_runner output found'. The exposure STORE stays flat (nothing globs it — the index drives every path); only the module-facing forest view carries the 2-digit shard. Verified by hand-running tile_merge_headers on 210.296 (1/1 OK). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ule's params The default rerun-triggers' code trigger hashes only the rule's own shell string — NOT external scripts it calls. Bitten live: the forest-shard fix changed build_forest.py but no rule text, so stale flat forests survived a rerun and tile_merge_headers kept failing. SCRIPT_HASH (md5 of sp_rule.py + completeness.py + build_forest.py, computed at parse time) rides into params via an accept-and-ignore --script-hash flag, so script edits propagate exactly like code edits under the full default triggers. One-time cost: every rule's params change once, re-running completed work — the same tradeoff as any code change under full triggers, and the P0-scale rerun doubles as a live test of rerun semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-full quota) def-mjhudson /project sits at its 27/27 TiB ceiling: a .snakemake metadata write died mid-run (killing an otherwise-green invocation after all 8 ngmix chunks of the first tile passed), and by end of night even git and file edits on /project failed. bin/sp now runs snakemake with --directory <run_dir>-state on /scratch (state only — data paths are absolute), the Snakefile resolves its configfile relative to itself so --directory can't break it, and the run index lives with the run on /scratch until project space is reclaimed (finding 15's /project placement stays the design intent; noted in config). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-unit star_cat file rules were removed (the store is per-CCD and the
mask config reads it as a dir symlink), but exp_mask kept declaring
exp/{exp}/star_cat-{exp}.fits as input. No producer exists, so the main DAG
became unbuildable the moment the tile list grew past the prepared set —
MissingInputException on every new exposure. Dormant for P0 (all 19 exposures
pre-staged); caught live by the append-invariant test (196 quad appended:
DAG build died before scheduling anything).
With the fix the append dry-run schedules exactly the appended cone (185 jobs:
30 new exposures x4 stages once each, 4 tile chains) and nothing for the
finished tiles; restored-state dry-run returns to 'Nothing to be done'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0188niYp9MmhLmZou1KXWbye
Contributor
Author
|
P0 delivered: 4/4 tiles through — Claude (Fable) on behalf of Cail 🤖 Generated with Claude Code |
…tion reporting sextractor writes 3 files/CCD on nibi (120/exp, not v2.0's 80); vignetmaker run_2 writes 5 sqlites/tile (not canfar's 4) — both verified against the P0 tree and the bash baseline. run_report now aggregates found-vs-expected at file granularity per runner, so warn-runner attrition (psfex_interp) is reported instead of hidden behind unit-level completeness. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRBKnHV6oUXEqRpsjqJpex
…he only mode The three module touches named in #848: - ID_OBJ_MIN/MAX read via getexpanded, so the committed ngmix template can interpolate $NGMIX_ID_MIN/$NGMIX_ID_MAX per chunk. Unset vars fail loudly. - SEED_FROM_POSITION is retired as an option: each object's RNG derives from its own (ra, dec, ccd), so results are bit-identical under any chunking. Tile-seed path, Ngmix.get_prior, and self._rng removed; a config that still sets SEED_FROM_POSITION=False raises with a clear message. - RUN_NAME read via getexpanded, so per-chunk run dirs can come from $SP_NGMIX_CHUNK in the committed template. Tests: tests/module/test_ngmix.py 24 passed / 1 pre-existing failure (azgauss needs ngmix>=2.4.1, absent from the runtime image; fails on clean tree too); test_ngmix_weight_validation.py 14 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRBKnHV6oUXEqRpsjqJpex
…ime rewriting (#848 D2) New workflow/config/cfis/ (example/cfis + p3-batch1 deltas, rewritten; example/cfis untouched for the bash flow): - RUN_DATETIME=False committed; every 'last:' INPUT_DIR replaced by the fixed $SP_RUN/output/run_sp_<X>/<runner>/output path of its producer. - NUMBER_LIST=$SP_UNIT_NUM on unit-ID-numbered stages (getlist env-expands; SP_UNIT_NUM carries the dashed form, e.g. -210-282 — rules do the transform). - ngmix template: $NGMIX_ID_MIN/$NGMIX_ID_MAX, SEED_FROM_POSITION=True, per-chunk RUN_NAME=run_sp_tile_ngmix_Ng${SP_NGMIX_CHUNK}u, SAVE_BATCH=250. - merge_sep_cats: N_SPLIT_MAX=$NGMIX_N_CHUNKS; module reads it env-expanded (same getexpanded treatment as the ngmix chunk fields). - make_cat from the nosm variant (what the P3 runs used) — no spread_model dependency. - p3 deltas folded: real INPUT_PATH/RETRIEVE=symlink, exp_Ma external star cat ($SP_RUN/star_cat_exp), PiViVi RUN_DATETIME fix. Rejected: SMP_BATCH_SIZE deltas (superseded by -b {threads}). - tile_Sx from the nomask variant (plain Sx needs an unlisted tile_Ma stage); tile_Uc kept but its run_sp_tile_Gic input path is unverified (Uc is not the production detect variant). All 14 configs pass a strict $VAR-expansion round-trip with the workflow's env stubbed. workflow/config.yaml config_src now points in-repo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NRBKnHV6oUXEqRpsjqJpex
…ss check CLI + manifest-driven report (#848 D2/D3) completeness.py gains the 'check <stage> <manifest>' CLI every rule's shell line calls after shapepipe_run: counts products under $SP_RUN against the floor table, writes a byte-stable JSON manifest (per-runner found/expect/ floor/warn, log-scraped failure reasons, no wall-clock), exits nonzero iff a mandatory runner is below floor — manifest always written first. STAGE_DIR maps stages to the committed configs' fixed RUN_NAMEs; the ngmix entry env-expands ${SP_NGMIX_CHUNK} so chunk K checks chunk K's dir. run_report.py rewritten to read manifests instead of disk-scanning: per-tile / per-exposure tables with failure reasons, attrition aggregate (failed units excluded), tile-blocked-by-exposure join, 'not run' for missing manifests. CLI interface unchanged (--run-dir --index --status; --out/--limit added). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5KZRTAZKSBAnpdDhE7yzK
…harded stores, parse-time index, native scattergather (#848 D1–D5) sp_rule.py is deleted; its responsibilities dissolve into the design: config normalization -> the committed S2 configs; log sync -> dead (zero 'last:'); unit furniture -> inline shell (tile_numbers.txt, pseudo-Fe exp_numbers with the ORIGINAL exposure name, star-cat symlinks); count floor -> completeness.py check writing the manifest each rule declares as its only output. SP_UNIT_NUM carries the leading-dash dashed form; the rules do the transform. Stores shard to tiles/<2ch>/<ID>/ and exp/<2ch>/<base>/; build_index accumulates (no DROP TABLE) and is built at parse time of the compute invocation; bin/sp reduces to run (two invocations) + report + cancel. ngmix scatters via the scattergather knob with in-job closed ID ranges (scripts/ngmix_range.py); retries with attempt-scaled mem on transient- exposed rules; protected() dropped; temp(directory()) rides as a scoped secondary output on the vignette store and chunk dirs — the one exception to no-directory-outputs, so native temp() reclaims intra-tile bulk. Deviations proven by implementation: merge_sep_cats INPUT_DIR must stay relative (MergeSep re.sub's the first '1' in the path — absolute sharded paths break chunk discovery); failed jobs need keep-incomplete in the profile or Snakemake deletes the failure manifests sp report exists to read (-> S6). Dry-runs: prepare 13 jobs, compute 141 on the 210/211 quad (19 exposures, 8 ngmix chunks); --set-scatter ngmix=1 -> 113. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5KZRTAZKSBAnpdDhE7yzK
All eleven findings from the fresh-eyes review, repro-verified: keep-incomplete:true (failure manifests are the post-mortem record — Snakemake otherwise deletes exactly what sp report reads); sp run survives a partial prepare (the missing-threshold is the gate, not set -e); the parse-time index build runs only under SP_PHASE=compute, always evaluates the threshold there, and refuses a silent empty run (WorkflowError on zero ready tiles) while passthrough invocations (--unlock, --dag, targets) never build; profile set-threads/set-resources blocks deleted — they silently replaced the rules' attempt-scaled lambdas and fork widths; manifests write only on content change (a byte-identical rewrite churned the whole cone via mtime); per-tile edge refresh in build_index (stale tile→exposure edges are unrepresentable-shrink no more); blame join blocks on failed/missing only, never warn (else all-tiles-blocked-by-all- exposures at production attrition); ngmix range eval no longer swallows script failure; compute DAG gets a regeneration edge to tile_find_exposures; threshold check precedes any durable index write. Manifest 'unit' is the human ID (basename of SP_RUN); build_forest handles a real-dir dst; stale comments (N_SPLIT_MAX expansion, && check form, sp_rule/protected refs) corrected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B5KZRTAZKSBAnpdDhE7yzK
…tones (#848 D5, S5) Exposure-store reclamation moves from a standalone script you remember to run into the DAG itself. One `clean_exposure` job per exposure takes every consuming tile's `tile_vignets` manifest as `input:` — vignets is the last stage that reads exposure products — deletes the store, and leaves a `cleaned.json` tombstone as its output. Writer, then readers, then cleaner: DAG-ordered, race-free, rolling mid-run while ngmix still grinds. The consumer set comes from the accumulating index (new EXP_TILES inverse edge), so it spans the campaign, not one invocation. The tombstone / late-append interaction, which is the whole design problem here: * The job deletes the exposure's `manifests/` along with its `output/`. That is load-bearing. The manifests are the exposure rules' DECLARED outputs; had they survived, a tile appended after the clean would find the exposure chain "up to date" and run its vignets against a store that is no longer on disk. With them gone the DAG sees an unbuilt chain and regenerates it — the accepted cost of a late append, expressed as ordinary Snakemake bookkeeping rather than a special case. The tombstone deliberately does NOT stand in for those manifests: it is input to nothing, so it can never mask their absence. * Tiles already finished are not disturbed: Snakemake demands a missing intermediate only when something downstream of it must run. * `params.consumers` carries the consumer set, so growing it makes the tombstone stale under the default `params` rerun-trigger; the clean job then reruns after the new tile's vignets, against the enlarged set. Cleaned once per consumer set, not once per campaign. * Nothing is lost to the report: each manifest's content is copied verbatim into the tombstone before deletion. Scope guard: an exposure is only eligible when every consuming tile is either in this run's scope or already has its vignets on disk. Without that parse-time test, requesting a tombstone for an exposure shared with a later batch would drag that batch's whole tile chain into this DAG through the clean rule's input — scope expansion by cleanup. Ineligible exposures are deferred, never lost: the invocation that finishes their last consumer picks them up. Gated by `clean:` in workflow/config.yaml (default false for the 4-tile P0 run, where the exposure store is exactly what you want to inspect) and by SP_PHASE=compute, so prepare and passthrough parses schedule no deletions. Flipping it on later reclaims retroactively — the missing tombstones schedule exactly the outstanding clean jobs. Invariants held: no temp() on exposure-level outputs; the loud WorkflowError on zero ready tiles; keep-incomplete untouched. Dry-runs on the 210/211 quad (19 exposures): prepare 13; compute 145 with clean off (identical to b149e18 under the same disk state); compute 164 with clean on = 145 + 19 clean_exposure. Prepare and passthrough parses show no clean rules and leave the index untouched. Deviation from the gap-map wording: one wildcarded rule with an index-driven input function, not literally one generated rule object per exposure. The DAG is identical, and at DR6 scale ~20k rule objects is a parse cost with nothing bought. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxXRKzfu6RUGNLv7ytNsHw
The profile flips from the P0 one-allocation/local-scheduler mode to the PRD's executor: slurm. jobs: 800 = 80% of the queried MaxSubmitPU=1000 on def-mjhudson (sacctmgr, 2026-07-30). default-resources carry mem_mb, runtime, slurm_account: def-mjhudson; keep-incomplete stays (a failed job's manifest is its only post-mortem record). No set-resources/set-threads — rules own their attempt-scaled lambdas. Env pins recorded in the README: snakemake>=9,<10, snakemake-executor-plugin-slurm>=2.7,<3 (installed: 9.23.1 / 2.7.1). group: fusion deferred — it needs rule edits, tracked in the README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxXRKzfu6RUGNLv7ytNsHw
…bstones (#848 D5) An adversarial review of the in-DAG clean_exposure design found eight problems. This fixes all eight. The central one is a rerun cascade. THE CASCADE, AND WHAT ACTUALLY CUT IT clean_exposure deletes an exposure's manifests on purpose: that is what makes a tile appended later rebuild the chain instead of running against an empty store. But those manifests are the tile side's inputs, and an exposure is read by ~7-10 tiles. So one rebuilt exposure made every other tile that reads it rerun, and that rerun rebuilt ITS exposures, spreading across the whole exposure-overlap connected component. On fixture t4 (four tiles chained by shared exposures, all complete, all exposures cleaned, one tile's catalogue deleted), asking for the one damaged tile scheduled all four tiles' chains. ancient() alone does NOT fix it, and the measurement says why. Snakemake suppresses the pending-producer propagation only when every shared file is ancient AND EXISTS (dag.py); reclamation has deleted these. Job counts with ancient() alone were identical to before. Three changes cut it, and t4 proves each: * a FINISHED tile (final_cat on disk) drops the exposure manifests that are gone from its input list, and holds the rest through ancient(). It has already extracted everything it will ever read from them, so the propagation has nowhere to go. Only missing manifests are dropped, so a campaign that has cleaned nothing declares exactly the edges it always did. * clean_exposure declares only IN-SCOPE consumers' vignets manifests. An out-of-scope consumer is checked for existence at parse time instead; declaring it pulled that finished tile's whole chain into the DAG. * the profile drops the `input` rerun-trigger. With it on, the conditional edge reads as "set of input files has changed" and reruns the finished tiles anyway — against a store that is gone. Nothing is lost: a changed exposure list still arrives through the find_exposures manifest's mtime, and the ngmix chunk count rides in params. t4, one damaged tile of four: 82 jobs before, 70 with the edge cut but the trigger on, 28 with all three — exactly the damaged tile's chain, its own two exposures, and the clean jobs. Same 28 whether the tile list is the one tile or all four. Appending a tile that shares a cleaned exposure still rebuilds that exposure and reschedules its clean, and touches no other tile (25 jobs). A complete campaign schedules nothing, clean on or off. The cost to know: --forcerun on a tile whose final_cat exists will not rebuild its reclaimed exposures. Delete the final_cat first. THE OTHER SEVEN * sp report read no tombstones, so every cleaned exposure came back "not_run" and blocked the complete tiles whose completion had authorised the deletion. It now reads the absorbed manifests out of cleaned.json, reports the unit as `cleaned` with its warn counts intact, and never blames it for a tile. Fixture t3: 3 exposures "not run" + 2 tiles blocked, before; 3 cleaned + 0 blocked, after. * clean_ignore_tiles: in config.yaml. A permanently-failed tile pinned its ~80 exposures forever; listed there, it leaves the consumer sets. Retrying it later rebuilds those chains from scratch, and the config says so. * the exp_psf benchmark tsv moved out of manifests/, which reclamation deletes. It is the measured-memory feed for mem_mb sizing (D4). * clean_exposure is a localrule. At DR6 scale it was ~20k sbatch submissions to run rmtree. * `--config clean=false` was truthy, so switching reclamation off switched it on. Config booleans now parse strings, and reject what they cannot read. * the tombstone is written, complete and atomically, BEFORE anything is deleted. The crash window now costs disk, never the record. * the false comments in clean_exposure.py and config.yaml are true again. Dry-runs only; no jobs were run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxXRKzfu6RUGNLv7ytNsHw
…trigger gate Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxXRKzfu6RUGNLv7ytNsHw
…harded, pre-manifest) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UxXRKzfu6RUGNLv7ytNsHw
…on_cpu)
The executor plugin validates against sacctmgr associations, which on nibi carry
the _cpu/_gpu suffix — the bare account that sbatch scripts accept is rejected
at submission ('appears to be invalid'). First-contact finding from the first
real slurm-executor run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UxXRKzfu6RUGNLv7ytNsHw
The SLURM executor re-invokes snakemake inside every job, and that re-invocation inherits SP_PHASE=compute — so every job's parse re-ran build_index.build. Hundreds of concurrent sqlite writers on Lustre produced the 'database is locked' storm that killed the first real run (smk-g3, 2026-07-31): the top-level compute parse died first, and the compute jobs a later invocation submitted kept dying in their own parses. - gate the build on workflow.is_main_process as well as SP_PHASE; job parses now only LOAD the index - timeout=60 on every sqlite3.connect (Snakefile loader, build_index, run_report, build_forest): Lustre lock handoffs outlive the 5 s default even under pure reader traffic - config.yaml: the quad is 186/187 (real files); 210/211's tiles are broken symlinks into anaennis' moved processed_tiles tree, as are ~8.5k of the 10.3k staged tiles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An sbatch head job starts in the submission directory; snakemake's Snakefile discovery is cwd-relative and found nothing. The launcher already promised cwd-independence — now it delivers it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
P3 fixture measures 7.9 exposures/tile (50 tiles / 181 exposures, sp_p3_batch1-17716364.log). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A manifest was written on failure too, and `keep-incomplete: true` kept it. A head process that died without recording job failures therefore left failed stages looking complete, and a resume scheduled downstream work on top of them (exp_psf on an empty exp_mask output dir, live). completeness.py now forks on the result: success writes `<stage>.json` and removes a stale `<stage>.failed.json`; failure writes `<stage>.failed.json` and removes a stale `<stage>.json`. Exactly one exists, so `<stage>.json` means the stage succeeded. Both keep the write-only-on-change behaviour. The failed manifest is never a declared output, so snakemake neither tracks nor deletes it — which lets the profile drop `keep-incomplete` and get the wanted default back: a failed job's declared output is removed. run_report.py reads either name and takes the status from the manifest body, so manifests written under the old convention still report as failed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The PRD assumed mask star catalogues were pre-generated offline. The pre-staged
store covers 1 of the current quad's 18 exposures, and symlinking the whole store
into a unit breaks the mask module outright: the file handler intersects the
image numbers found across a config's INPUT_DIRs, and the store contributes 7240
foreign numbers, so the intersection is empty ("numbers ... do not intersect").
New rule `exp_star_cat`, one per exposure: it runs create_star_cat.py in the
container against the exp_get_images symlink farm (one Vizier cone query over the
focal plane, ~6 s), then builds a REAL per-unit `star_cat_exp/` of 40 per-CCD
symlinks — exactly this exposure's numbers, so the intersection is right by
construction and config_exp_Ma.ini needs no change. exp_mask takes its manifest
as an input.
It is a localrule, so the queries run serially in the head process (an sbatch job
on a compute node, which has internet): CDS is protected at any campaign scale
and the scheduler never sees a six-second job. The catalogue itself is cached
run-independently under `star_cats:`, now the cache root on /scratch, and
create_star_cat.py skips an existing cat, so reruns and later campaigns are free.
unit_pre no longer materialises the star_cat_exp/star_cat_tiles store symlinks.
tile.smk records what a future tile_mask needs on the tile side.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
completeness.py forked on its count verdict alone. A job whose floors were met but whose shapepipe_run exited nonzero therefore wrote the SUCCESS manifest and still exited nonzero. Snakemake then deleted that manifest as a failed job's output, the check had already unlinked any prior failed.json, and the failure left no trace on disk at all. The two statements are now composed. sp_shell passes `--job-rc "$rc"`; a nonzero rc forces the failure branch, records job_rc and a shapepipe_run failure entry that run_report surfaces, and the fork writes failed.json and exits 1 as usual. The rc is recorded only when nonzero, so every existing success manifest stays byte-identical and the mtime rerun-trigger does not churn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four ways the star-catalogue stage lied about its own state, plus one comment that lied about the scheduler. The farm loop wrote into the shared pool. Legacy unit dirs carry star_cat_exp as a symlink into the old shared star-cat pool, `mkdir -p` no-ops on it, and the 40-link loop then followed the link and wrote this exposure's links into the pool (520 stray links found live). The rule now unlinks a symlinked farm before the mkdir. It never rm -rf's a real farm: that farm is its own output, and ln -sfn refreshes it in place. A killed cache write stayed poisonous forever. create_star_cat.py wrote the cache FITS in place, so an interrupted write left a truncated file that the `isfile -> continue` skip trusted for the rest of the campaign — and `test -s` passes on partial bytes. Both branches now write a hidden, PID-tagged temp with a .fits suffix (astropy picks its writer from the extension) and os.replace it into position; a leftover temp is removed in a finally. The dot-prefix keeps the temp out of the `star_cat*` globs. The manifest attested forever. It was the rule's only declared output, so a purged cache (/scratch, 60 days) or a deleted farm left the stage looking complete while exp_mask had nothing to read. The ccd-0 farm link is now declared alongside it. Snakemake's existence test follows symlinks, so a purge leaves that link dangling, the rule reruns, and it re-queries or re-links — the recovery config.yaml already promised. The script was outside the fingerprint. exp_star_cat's params held only paths, so an edit to create_star_cat.py never reran it. STAR_CAT_HASH follows the existing completeness/forest/clean pattern. The hash deliberately does NOT key the cache path: a rerun still takes the skip-if-exists branch, so a semantic change to the query needs the cache cleared by hand. config.yaml now says so. And the localrule comment claimed local execution serialises the queries. It does not: local jobs run in a pool sized by --local-cores (unset here, so the head process's CPU count, 5 as measured), which is the real concurrency cap. The localrule stays — 5-way is ordinary CDS politeness, and the alternative is ~20k sbatch submissions for six seconds of work each. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
clean_exposure targeted only output/ and manifests/, so the star_cat_exp link farm outlived the exp_star_cat manifest that owned it — 40 symlinks belonging to no rule. The farm and the legacy star_cat_tiles link are now targets too. This is consistency, not bytes: the catalogue stays in the run-independent cache, so rebuilding the farm is a relink and no query. Deletion is symlink-safe. Legacy unit dirs carry star_cat_exp as a link into the shared pool, and rmtree would recurse through it and empty that pool for every other exposure in the campaign. A symlink target is unlinked; only a real directory is rmtree'd. Existence is tested as `is_symlink() or exists()`, because exists() follows the link and would skip a dangling legacy one. Separately, astroquery pickled every Vizier response into $HOME/.astropy/cache/astroquery/Vizier at ~2 MB a query (10 queries, 10 pickles, observed live). The workflow already caches the result as a FITS on scratch and skips the query on a hit, so the pickle is a duplicate — and at campaign scale (~25k exposures) it is ~50 GB against a 50 GB home quota. query_vizier now passes cache=False. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… 31 tiles The rule comment asked for a measurement across several tiles before this number moved, and the smk-g4 campaign is it: cgroup high-water 27.40 GiB at worst and 22.14 GiB on average across 31 fused groups, against the 109.4 GiB the group was reserving. 5000 per chunk takes the group to 40000 MiB, a 1.43x margin over the worst tile observed and about 4x the anonymous footprint -- snakemake's psutil benchmark puts each chunk at ~1.25 GiB, so the eight chunks are ~10 GiB and everything above that is reclaimable page cache. This is the campaign's largest single cost saving, and it is a schedule change rather than a bookkeeping one. nibi bills max(cores, mem_GB/4), so at 112 GB the group billed 28 core-equivalents for 8 real cores -- measured live at 905 billed against 266 allocated, a factor of 3.40. At 40 GB it bills 10. The account's fairshare target is ~250 CE, which buys 9 tiles in flight at 28 and 25 at 10, and a DR6 campaign's wall clock is (tiles / tiles-in-flight) x per-tile elapsed. On the campaign's measured per-tile cost that is ~190 days against ~68. Not 4000, which is the arithmetic floor: it would take the group to 32000 where tile_vignets' own 32000 binds and the group finally bills its 8 real cores. That is a 1.17x margin over the worst tile measured, and the first thing to give under pressure would be the page cache holding the node-local store -- which is what makes the fused tile fast. Worth taking later, with a measurement of cache behaviour under pressure rather than on arithmetic. `resources` is not a rerun trigger, so no completed work is invalidated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NYcxuAbZbQivC5UFtgecxH
…laims Adversarial review of the two preceding commits, against the code rather than against their messages. Three things came back. STAGING WAS NOT ATOMIC, and `cp -u` made that permanent. `cp` writes in place, so an interruption -- ENOSPC on /local/scratch, an OOM-kill (which skips the EXIT trap), node death -- leaves a truncated destination whose mtime is NEWER than the NFS source. `cp -u` then skips it forever, and nothing downstream revalidates: TILE_VIGNET_FRESH and TILE_VIGNET_REQUIRED both look only at the vignette store. A truncated WCS store reaches ngmix as wrong astrometry rather than as an error, which is the worst failure mode this change could have had. It is now an unconditional copy to a temp name plus `mv -f`, atomic within the filesystem, with the source spelled out by tile id instead of globbed so a missing file is one precise error. Unconditional also closes the race the eight concurrent chunks had on a shared destination in the tile_vignets-not-in-group recovery path; renaming over a file a sibling already has open is safe, since the open descriptor keeps the old, identical inode. Nine 11 MB copies per tile at job start is not a cost worth a staleness rule. THE MEMORY MARGIN WAS OVERSTATED. The previous comment took snakemake's psutil benchmark (~1.25 GiB per chunk) as the anonymous footprint and claimed ~4x headroom. This repo carries a second estimate that disagrees by 1.8x -- config_tile_Ng_template.ini's own SAVE_BATCH note puts a worker at ~2.3 GB peak from an A/B test -- and the psutil figure is sampled on a 30-second grid, which is exactly what a SAVE_BATCH = 250 flush sawtooth hides. Prefer the larger: ~18.4 GiB anonymous for eight chunks, so the margin is ~2.1x with ~20 GiB left for cache against an 8.1 GiB store. Still comfortable, and now honest. The 4000 floor's margin is 1.14x, not the 1.17x claimed (MiB/GiB slip). EDITING tile_local() INVALIDATES FINISHED TILES, and the consequences on a resume are destructive rather than merely expensive: the reruns are unsatisfiable because clean_exposure has reclaimed the exposure stores, and a failed group job's postprocess deletes every member's existing outputs -- including tile_make_cat's final_cat on the persistent root. The full mechanism is now in the TILE_LOCAL comment, with the `--rerun-triggers` escape hatch for the case where a resume is unavoidable. Nothing about the change is wrong; it just must land between campaigns. Also corrected: the prologue does not run in tile_merge_cats (no pre_run), and config_tile_Mc.ini reads $NGMIX_VIGNET_DIR but not $SP_WCS_DIR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NYcxuAbZbQivC5UFtgecxH
smk-g4 landed two changes after its campaign finished and neither has ever run: the tile-level WCS sqlite staged node-local, and tile_ngmix's mem_mb cut 14000 -> 5000. Point the campaign roots at a fresh smk-g5 tree holding one tile, 186.307. That tile is the control because it ran ALONE as smk-g4 job 20799387 rather than inside the 34-tile campaign, so its baseline has no concurrency confound: 8063 s elapsed, chunk median 6813 s at 59% mean load, MaxRSS 26.6 GiB. The prediction, registered in job_head.sh before the run: load above 90%, chunk median near its own 4050 s of CPU, group near 83 min, MaxRSS unmoved (it is tile_vignets' peak, not ngmix's), and final_cat bit-identical to smk-g4's -- which #873's position-seeded split makes checkable for the first time. Fresh roots rather than a resume, because tile_local() changed and `params` is an active rerun trigger: pointing this at smk-g4 would delete the 34 finished tiles' catalogues rather than merely re-run them. clean is off -- ~54 GiB of exposure stores against 979 GiB free buys a re-measurement that re-runs only the shape chain.
…bject count ngmix fits an object jointly across every exposure it lands on, so a chunk's cost is set by its (object, epoch) pairs, not its objects. The campaign measured 0.2714 CPU-s per pair with an intercept consistent with zero, plus ~0.05 CPU-s of per-object setup, and equal-object-count chunks of the SAME tile still varied 1.6x in cost, tracking their epoch counts at R^2 0.91-1.000. The eight chunks are siblings in the fused tile_shape group, which ends when its slowest member does, so that spread was pure wall clock: 7.7 hours across the 34-tile smk-g4 campaign, and on 196.307 it took the group to 97.4% of its wall limit. The epoch count is already on disk. tile_detect's SExtractor post-process writes one EPOCH_<k> extension per overlapping exposure into the same sexcat this script already opens -- n_obj rows of NUMBER/EXP_NAME/CCD_N, 529,470 B each -- and summing CCD_N >= 0 across them reproduces the final catalogue's N_EPOCH column exactly, checked row by row against 186.307's final_cat, all 35,298 of them. The extension count is the tile's exposure count, so they are found by name prefix. LDAC_OBJECTS is never read: at 10,662 B per row it is 376,347,276 B of the 380 MB file, more than the straggle this exists to remove. Weights are w_i = 1000*epochs_i + 184, the 184 being 0.05/0.2714 epochs of per-object setup -- it is what stops the 13 zero-epoch objects on 186.307 from weighing nothing. The split minimises the MAXIMUM chunk weight exactly (bisection on capacity, then a left-to-right fill), because the group waits for the slowest chunk and not the average one. On 186.307's real sexcat the old ranges give per-chunk epochs 12,783-16,881, max/median 1.174; the new ones 14,509-14,704, max/median 1.008. Predicted slowest-chunk cost 4802 -> 4181 CPU-s, -12.9%. Across all 34 sexcats the slowest chunk goes from 1.131x-1.627x its tile's median to 1.0000x-1.0002x, and the sum of slowest-chunk cost falls 132,875 -> 113,710 predicted CPU-s, 14.4%. The predictions track that tile's eight measured chunk CPU times at R^2 0.974, over-predicting by a uniform ~3.4%, and the residual has a named cause: N_EPOCH is GEOMETRIC, while ngmix drops epochs it cannot fit. NGMIX_N_EPOCH in the same final catalogue is lower for 2,203 of the 35,298 objects and never higher -- 113,947 pairs against 116,727, 2.4% -- and refit on it the law is 0.2681 CPU-s per pair at R^2 0.997, i.e. the campaign's 0.2714 is right and the geometric proxy is what is slightly off. The oracle cannot be had before ngmix runs and is barely worth having: splitting on it moves boundaries by up to 177 objects and improves the true slowest chunk by 1.07%. Chunk boundaries cannot change a result -- ngmix seeds its RNG from object position, which is boundary-invariant by construction, and SEED_FROM_POSITION is no longer even optional (ngmix_runner.py raises on False). The hazard this is designed against is that all eight chunks compute the ranges INDEPENDENTLY and must agree; a disagreement double-measures or drops objects and merge_sep_cats would concatenate the result without complaint. Hence integer arithmetic end to end, NUMBER = 1..N asserted rather than assumed, and no fallback: an unreadable EPOCH extension exits non-zero, because a fallback would apply only to the chunks that hit it. NGMIX_RANGE_HASH is the same hazard across TIME. tile_ngmix's only fingerprint was SCRIPT_HASH, which is completeness.py; params.pre carries the range script's INVOCATION, and the invocation is invariant to the script's body. So a tile resumed across an edit to the split would have kept the old ranges on its finished chunks and taken the new ones on its reruns -- some objects measured twice, others by nobody, the tile green. The hash joins SCRIPT/FOREST/CLEAN/ STAR_CAT in the Snakefile and lands on tile_ngmix alone. It also means this commit correctly invalidates in-flight ngmix chunks rather than mixing splits. tests/unit/test_ngmix_range.py pins the partition invariant with hypothesis over arbitrary epoch distributions, and checks the split against a brute-force minimax on small cases. It imports the script by path, so it needs neither the container nor the shapepipe package. n_obj < n_chunks is now defined instead of accidental: the surplus chunks get (n_obj + 1, n_obj), the canonical empty closed interval. The old equal-count split emitted (1, 0) there, and ngmix reads ID_OBJ_MAX = 0 as unbounded (ngmix.py:804-806) -- each empty chunk would have re-measured the entire tile. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017gNJUBiHxeYVBPpsGxRVKL
clean_exposure reclaims exposure stores; nothing reclaimed tiles, so a
FINISHED tile held 1.19 GiB across 137 inodes on scratch forever (measured on
186.307: run_sp_tile_Sx 745 MB, run_sp_tile_Uz 382 MB, run_sp_Mc 46 MB,
run_sp_Ms 39 MB, run_sp_tile_Mh_exp 11 MB, plus 62 non-output inodes of 15,726
bytes). At DR6's 23,114 tiles that is 26.9 TiB against a 1 TiB quota and 3.17M
inodes against a 1M one; the byte bound binds first, so until this rule exists
no batch may exceed ~859 tiles. Deleting only output/ does not fix it either --
62 inodes per tile is still 1.43M at DR6, over quota on its own -- so the whole
directory goes.
clean_tile is the symmetric rule: a localrule keyed on the tile's own final_cat
on the persistent root, ordered after the tile so reclamation is ROLLING (the
scratch high-water tracks tiles in flight, not tiles in the batch). Eligibility
is trivial where the exposure case is hard: a tile's store has no consumer
outside that tile, so there is no consumer set, no staleness to detect, and no
edge to clean_exposure at all. Scope runs the other way instead -- TILES_READY
only, because an out-of-scope tile's tombstone would drag its final_cat and the
whole fused tile_shape group into the DAG, where the params trigger can rerun a
finished tile against reclaimed exposure stores.
FOUR SURVIVORS, and two of them were found by working out what else reads a
tile's store rather than by looking at the tile:
* manifests/tile_vignets.json -- clean_exposure keys campaign-wide eligibility
on exactly this path. Removing it stranded all 7 fixture exposures (dry-run
job counts: 7 clean_exposure jobs with it, 0 without), because every
exposure had an out-of-scope consumer whose vignets manifest is the test.
* manifests/tile_find_exposures.json -- rule prepare_all_tiles declares it for
EVERY tile in the list, and the list accumulates. Without it the prepare
parse schedules the whole tile_prep group per cleaned tile per invocation
(measured on the fixture: 4 jobs where there should be none), and
tile_uncompress writes 382 MB back into the store just emptied.
* output/run_sp_tile_Fe/.../exp_numbers-<IDra>-<IDdec>.txt -- build_index reads
it at the parse of EVERY compute invocation. Without it the tile counts as
missing and SP_MISSING_THRESHOLD=0.0 makes the parse fatal: "Missing
exposure lists for 1/2 tile(s) (fraction 0.500 > threshold 0.0)". Not a
degradation, a stop -- the campaign cannot resume at all.
* cleaned.json -- the tombstone, absorbing all 16 manifests verbatim plus the
8 ngmix benchmark rows, so sp report still reports the tile and D4's
measured-memory feed is not lost with the store.
Ten inodes and 12.9 KB survive per tile (231k inodes, ~300 MB at DR6). The
three pre-existing survivors are checked BEFORE any deletion and the job
refuses loudly if one is absent: a drift in those paths is invisible at
deletion time and shows up much later as a campaign that will not parse.
KEEPING tile_vignets.json DOES NOT TRIP TILE_VIGNET_REQUIRED, and the reason is
what gets deleted rather than what gets kept -- worth stating because a cleaned
tile is on its face the exact state that guard exists to catch. Forcing a
cleaned tile back (final_cat deleted) reschedules tile_vignets inside the
tile_shape group, so the store is rebuilt: 19 jobs, whole chain from
tile_get_images. Snakemake's own reason is "Input files updated by another job:
exp_forest, tile_find_exposures.json, tile_detect.json" -- structural
propagation, NOT the mtime trigger, which matters because it is not in
profiles/nibi's rerun-triggers and cannot be switched off there. The
counterfactual pins the mechanism: with every upstream manifest restored and
mtimes controlled (the "delete only output/" design), the same force schedules
8 x tile_ngmix + merge_cats + make_cat with tile_vignets ABSENT and every chunk
trips the guard. So tile_detect.json must never join the survivor list.
smk-g4's job_head.sh runbook sweep for this hazard already skips tiles with a
final_cat, i.e. every tile this rule touches.
Absorption is ADDITIVE over an existing tombstone. A naive re-absorption cut a
16-manifest tombstone down to the two surviving manifests on the second clean,
found by running the fixture clean twice; script_hash reruns this job on any
edit to clean_tile.py, so that rerun is not hypothetical. (clean_exposure.py
has the same latent hole and is deliberately left alone here.)
Deletion is a whitelist walk, not an rmtree with exceptions, and is
symlink-safe for a sharper reason than the exposure case: exp_forest holds ~8
links into the shared exposure stores. Verified on the fixture that all 7
exposure stores keep their content across a clean, and that a dangling link
left by an already-reclaimed exposure is unlinked rather than tripped over.
clean_tiles is a NEW config key, default false, deliberately not `clean:`.
Exposure reclamation costs a rebuild if a tile is appended later; tile
reclamation destroys the per-tile audit trail and nothing rebuilds it --
sp_tilecost.py reads each tile's 380 MB sexcat from the scratch store to
attribute the fused group job's cost, and that is how every mem_mb and runtime
number in tile.smk was derived. Required for any batch over ~850 tiles, and it
turns off per-tile cost attribution. The eight benchmark TSVs it also reads
survive inside the tombstone but not at their on-disk paths; teaching that tool
to fall back to cleaned.json is a named follow-up, not done here.
run_report reads tile tombstones back the way it reads exposure ones, with one
asymmetry: clean_tile cannot empty manifests/, so absorb_tombstones takes the
survivor set explicitly -- otherwise the "records on disk mean a rebuilt chain"
guard fires on every cleaned tile and a fully reclaimed campaign reports as one
that ran two stages and stopped. Verified: 2/2 final cats, all nine stages
complete, the reclaimed tile listed as cleaned.
Verified by construction on a throwaway fixture (structural copy of 186.307,
bulk files stubbed, now deleted); no SLURM job was submitted and neither smk-g4
nor smk-g5 was touched. Dry-run counts: clean_tiles=false -> 0 clean_tile jobs;
true with one finished tile -> exactly 1, and tile_make_cat is NOT dragged back
in; true with the tile unfinished -> tile_make_cat then clean_tile, in that
order. A real clean of the fixture tile took it from 137 inodes to 10.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017gNJUBiHxeYVBPpsGxRVKL
…ndary Adding NGMIX_RANGE_HASH protects a resume from a corrupt partition by forcing the chunks to rerun -- and on a root that already holds finished tiles, that rerun is destructive rather than merely expensive. The hash and the rule now say so where each is read. Snakemake compares the SET of recorded param values. smk-g4's metadata for a finished chunk (record_format_version 6) holds two, the SCRIPT_HASH digest and the pre string, so a third entry is new and the job replans with "Params have changed since last execution". Measured on snakemake 9.23.1 under profiles/nibi's trigger set, on a fixture mirroring the group with everything up to date: the eight chunks, tile_merge_cats and tile_make_cat reschedule; tile_vignets and tile_detect do NOT, because their manifests exist, so missing_output never queues them and nothing propagates down. A group replanned without tile_vignets is exactly the state TILE_VIGNET_REQUIRED exists to catch. Every chunk trips the guard, the group fails, and GroupJob.postprocess(error=True) fans out over every member and removes its existing outputs -- tile_make_cat's final_cat on the persistent root among them. The TILE_LOCAL warning does not already cover this, and the distinction is the reason the note is worth its length: tile_local() sits in the params.pre of tile_vignets, tile_ngmix AND tile_make_cat, so an edit there drags tile_vignets back in and the store is rebuilt. range_hash is on tile_ngmix ALONE. It is the one-rule fingerprint change that is dangerous, not the three-rule kind that warning describes, and the warning reads as though it covered both. Nor does job_head.sh's runbook sweep help: it skips any tile with a final_cat, which is exactly the set this breaks. clean_tiles inverts it. With reclamation ON the tombstoned tile has lost tile_detect.json, and the structural "Input files updated by another job" propagation puts tile_vignets back in the group, so the store is rebuilt and nothing trips. The shipped default clean_tiles: false is the dangerous configuration -- the opposite of how reclamation reads everywhere else in tile.smk. Recommendation is tile_local()'s, unchanged: land the hash, and every later edit to ngmix_range.py, at a campaign boundary on a fresh root, with the same `--rerun-triggers mtime code software-env` escape hatch and the same caveat if a resume is unavoidable. Separately, ngmix_range.py's docstring now carries the CROSS-TIME hazard beside the cross-process one it already argued. The hash covers resume across an edit; nothing covers an edit DURING flight, and a fused group holds its eight chunks open over the live checkout for hours (smk-g4: 6,236-7,762 s elapsed per chunk), each reading the file when its own shell starts. Reproduced live: an invocation four seconds after a rewrite returned chunk 5 of 186.307 as 17649..22060 where the other seven had been split 17129..21229 -- 520 objects orphaned, 831 double-measured -- while twelve sequential and thirty-six concurrent runs against a stable checkout gave the identical correct partition. Not a defect in the script; the same between-campaigns rule, binding harder because tile_local() fails loudly and this fails green. Comments only. No code, no behaviour, no rerun trigger moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017gNJUBiHxeYVBPpsGxRVKL
Adversarial review of the merged branch. It cleared the survivor list — swept workflow/ and sp-products/tools/ for every path built under tiles/<shard>/<tile>/ and found nothing missing — and returned four things. One is a real bug. A RECLAIMED TILE'S `warn` SILENTLY BECAME `complete`. All eight chunk manifests carry stage "tile_ngmix". load_manifests resolves that collision worst-status- wins; absorb_tombstones resolved it with setdefault over sorted stems, so only tile_ngmix_1 survived and a warn on any other chunk vanished the moment the tile was cleaned. Reproduced on the real 186.307 manifests with chunk 5 flipped: tile_ngmix ok 0 warn 1 before, ok 1 warn 0 after, and the tile left the "tiles not complete" table. The data was in the tombstone all along; only the reader dropped it. The rule is now one function, keep_worst(), used by both readers, and the tombstone's per-stem records are collapsed to one per stage BEFORE merging rather than after. End-to-end on the fixture: every stage's counts and per-runner products identical before and after reclamation. The other half of that finding does not reproduce as a reclamation bug. A stage's products aggregate comes from its one surviving record either way, so tile_ngmix attrition is counted over one chunk of eight — on disk as well as in the tombstone, before this change and after. That is the price of collapsing the chunks to one stage row; the tally_level docstring claimed the opposite and now states it, including the 8x over-statement of completeness it implies. THE SYMLINK COMMENT NAMED THE WRONG BLAST RADIUS. A finished tile holds nine links, not seven. The two unnamed ones are in run_sp_tile_Git/get_images_runner/ output/ and point into /project/def-mjhudson/unions-wl/tiles — 621 GB across 2,536 files of staged survey imaging on the backed-up, group-shared filesystem, put there by get_images RETRIEVE=symlink, so every tile in every campaign carries a pair. Behaviour was already correct, but the comment told the next person editing prune() that the worst case was a rebuildable scratch store. It also reasoned about prune's own is_symlink() test, which is not what protects either class: both are handed wholesale to shutil.rmtree, exp_forest/ as a top-level entry and run_sp_tile_Git/ inside output/, which prune descends only because the Fe survivor lives there. Said explicitly now. A PARTIALLY REBUILT CLEANED TILE LOSES ITS TOMBSTONE RECORD. Written down, not fixed, and the reason is that the obvious fix is wrong rather than long: filling per-stage from the tombstone would report the previous generation's "complete" for a stage whose manifest is absent because it is mid-rerun or failed. A stale complete is invisibly wrong where "not run" is visibly incomplete, and the same hazard reaches exposures campaign-wide. The all-or-nothing guard is a generation boundary and is worth more than the cosmetics. sp_costmodel.py joins sp_tilecost.py in the clean_tiles: doc — it reads the same sexcat (plus its EPOCH_k extensions) and the same benchmark TSVs. Their third artifact, run_sp_tile_ngmix_Ng<k>u/, is NOT lost to this flag: tile_ngmix declares its chunk dir temp(), so snakemake reclaims it when tile_merge_cats runs and it is already absent from every finished tile. Two test modules, following tests/unit/test_ngmix_range.py's convention (import the script by path, container-free). test_run_report_tombstones pins the invariant that catches the whole class — the tombstone path and the on-disk path agree, stage for stage — parametrised over all eight chunks because the bug was invisible on chunk 1, the one stem first-key-wins happened to keep and the one anybody would test by hand. Against the pre-fix reader: 8 failed, 3 passed. test_clean_tile_prune pins symlink safety at the depths rmtree actually walks, because that failure mode is silent, immediate and unrecoverable. 16 new tests; tests/unit is 258 passed, 3 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017gNJUBiHxeYVBPpsGxRVKL
…rved The comment asserted that a failed group job removes final_cat. The replanning half of that chain was reproduced; the deletion half never has been, here or in the tile_local warning it builds on. Two independent reviews failed to reach it for the same reason -- snakemake reports "Group jobs: inactive (local execution)", so no login-node fixture exercises the group path and only a submitted job against a finished tile can settle it. The mechanism stays as written, because reading it off snakemake's own cleanup path is what makes the hazard credible. What changes is that a comment which confines a whole class of change to campaign boundaries now says that its last step is derived, and points at the experiment that would make it measured -- designed on a tile whose final_cat is bit-reproducible from another campaign's independent copy, so the observation costs nothing to make. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017gNJUBiHxeYVBPpsGxRVKL
… derived cfe62ed separated the observed half of the range_hash hazard from the derived half in tile.smk, but the NGMIX_RANGE_HASH comment in the Snakefile still asserted the deletion flatly: "the failed group's postprocess removes each member's existing outputs, final_cat included". A reader who stops at the hash definition -- which is where the other four hashes are argued, so a plausible place to stop -- would take the destruction as measured. Same split as tile.smk's, one clause: the replanning and the guard trip are observed, the removal is DERIVED from snakemake's cleanup path and has never been seen. Points at tile.smk for the source lines and at sp-products/smk-g5/EXPERIMENT_postprocess_deletion.md for the run that would settle it, rather than restating either. A separate commit and not an amend of f196fa1: two commits now sit on top of it in this worktree, one of them another session's report fix, and rewriting them to fold in a comment is not a trade worth making. Comments only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017gNJUBiHxeYVBPpsGxRVKL
The merge that precedes this adds range_hash to tile_ngmix's params, which on a root holding finished tiles replans the fused group WITHOUT tile_vignets -- the state TILE_VIGNET_REQUIRED catches, after which a failed group job removes every member's outputs, final_cat included. So the merge and this switch are one operation: leaving the config pointing at smk-g5, which now holds a finished tile, would ship a branch whose next `sp run` destroys it. smk-g6 is 64 tiles chosen by batch geometry (sp-products/smk-g6/batch.md), seeded on the 34 smk-g4 tiles so the campaign is a paired contrast at scale rather than a fresh unknown -- growing that seed set is nearly free, because it is sparse and the first 18 tiles added introduce no new exposure stores. 127 distinct stores, ~838 GiB projected peak with clean_tiles on. Both reclamation mechanisms are on: exercising clean_tile at scale is one of the things this campaign exists to do, and without it 64 tiles leave 77 GiB of residue that never goes away. Preconditions and the launch order are in sp-products/smk-g6/LAUNCH.md.
Job 20818649 against smk-g5, 32 seconds. A params change on tile_ngmix alone replanned the fused group without tile_vignets -- the SLURM group label itself came back as tile_shape_tile_make_cat_tile_merge_cats_tile_ngmix -- every chunk tripped TILE_VIGNET_REQUIRED, the group failed, and final_cat-186.307.fits was gone from the persistent products root afterwards. The tile's directory was empty. Restored from a copy taken first; the catalogue is bit-reproducible from smk-g4's independent copy of the same tile, which is why smk-g5 was the safe target. This had been derived twice from snakemake's cleanup path and never seen, because "Group jobs: inactive (local execution)" puts the group path out of reach of any login-node fixture. It is the constraint that confines a whole class of change to campaign boundaries, so it was worth 32 seconds to stop guessing. One thing the run makes plain that the derivation did not: the missing-store guard is what turns this from a silently wrong catalogue into a failed job, and the failed job is exactly what deletes the science product. The guard is still the right trade. But on a fused group "fails loudly" is not a cheap outcome, which argues for never reaching the state rather than for softening the guard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017gNJUBiHxeYVBPpsGxRVKL
Contributor
Author
Orchestration benchmark: 64 tiles end-to-end on nibiOne command took 64 tiles from raw images to Compute efficiency. The ngmix phase — just under 90% of all compute — runs at 98% CPU utilisation. The 8 parallel chunks of a tile finish within ~4% of each other. Total campaign cost: 711 billed core-hours, ~11 per tile. The limiting resource is scratch space, not compute. Intermediate products cost ~15 GB per tile in flight, cleaned up automatically as tiles finish. Given nibi's 1 TB scratch quota, this caps us at ~68 concurrent tiles; we plan to request a temporary quota increase (~10 TB) for the production run, after which throughput is set by available cores. Scaling to DR6 (dec > 30, ~20,200 tiles): ~225,000 billed core-hours (~26 core-years) at the measured per-tile cost. Calendar time scales with concurrency: ~7 weeks at the current quota, ~1–2 weeks with the increase (assuming the cluster grants cores at the rate we've seen). Final catalogues need ~0.8 TB of project space. Over the course of tuning this workflow, wall time per tile fell ~1.7× and billed cost ~4.9×, mostly from moving the WCS database node-local and right-sizing memory requests. Outputs. The 64 final catalogues are on nibi at — Claude on behalf of Cail |
cailmdaley
marked this pull request as ready for review
August 30, 2026 20:50
…e onto position-seeded RNG Take-both resolution in the ngmix module: develop's blend_handling/seg/dilate parameters and pixel-scale-from-WCS derivation are kept; the branch's retirement of the tile-seeded RNG mode (position-seeded per-object RNG as the only mode) stands, so develop's seed_from_position plumbing is dropped rather than merged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
…ar-cats use the resolved image Five surgical de-duplications in the workflow layer, all behaviour-preserving: * the six copy-pasted md5(...)[:12] blocks collapse to one script_hash(name) helper and six one-line constants (identical digests); * unit_pre() drops its `level` parameter and takes it from STAGE_DIR, which already maps stage -> (level, subdir); the 13 call sites lose an argument they could only get wrong. The generated params.pre strings are unchanged; * exposure.smk's in_container() stops hand-building an apptainer line from config['container'] and uses the Snakefile's resolved _image plus container.profile_apptainer_args(). This removes a real divergence: star_catalogue and exp_star_cat were the ONLY rules that ignored a user's dev sandbox. It fails loudly if the profile is unreadable rather than silently dropping --cleanenv and the PYTHONPATH pin. REPO_DIR is now unused and goes with it; * tile_local() calls unit_num() instead of re-spelling the dot->dash convention inline (unit_num supplies the separating dash, so the literals lose theirs); the produced path is byte-identical, which matters because it lands in params.pre; * clean_targets()/clean_tile_targets() and the sqlite slurp are gated on workflow.is_main_process. Both fed `rule all` at module level and so ran on every one of the ~800 per-job re-parses, none of which can schedule `all`; the slurp read the whole campaign's tile_exposures there. Job parses now get the same three lookups (exp_name / tile_exposures / exp_consumers) backed by memoised single-row queries, ORDER BY rowid so the lists match the eager path's insertion order exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
tile_vignets and tile_make_cat carried near-copies of sp_shell's rc-composition body, each with its own copy of the rationale comment. sp_shell now takes check_args (extra completeness-check flags) and post (bash between the check and the exit), and all three rules go through it. Verified byte-identical shell strings for both rules, which is what keeps the `code` rerun-trigger quiet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
…stat less Four small fixes in the scripts the rules call: * completeness.py: the COMPLETENESS/STAGE_DIR rows for `tile_mask` and the STAGE_DIR row for `tile_detect_uc` describe stages no rule declares (tile.smk:44 says why), so they are removed and named in the table comment as what the masked variant would add back. workflow/config/cfis/config_tile_Uc.ini goes with them — no rule reads it; the legacy bash reads example/cfis/, which keeps its copy. * completeness.py count_products: os.scandir with a follow-stat only on symlink entries, instead of a p.exists() stat on every file. The dead-symlink filter is the point of that test and its semantics are unchanged. * run_report.py: `**` recursive globs become fixed-depth `*/*/manifests/*.json`, `*/*/logs/*.json` and `*/*/cleaned.json`. The sharded layout is exactly two levels; `**` walked every unreclaimed output/ tree to find records that can only be at one depth. The docstring's "the sharding depth is not this script's business" claim goes with it. * run_report.py tally_level: the per-stage record's "complete" becomes a list of unit ids like the other four status keys, rather than a bare int. The two consumers take len(); the emitted JSON gains the complete-unit list. * star_cats.py focal_plane_disc: one fits.open pass over the 40 CCDs instead of 40 fits.getheader calls, each of which reopened the file and re-walked the HDU list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
…n the library workflow/scripts/star_cats.py and scripts/python/create_star_cat.py carried the same atomic FITS write twice (same docstring, same argument) and the same manual-CD WCS construction and focal-plane footprint twice — star_cats.py's docstring even said "Same construction as create_star_cat.py". Both pairs move into src/shapepipe/utilities, beside vizier.py and reached the same way (the PYTHONPATH pin the star-cat rules already run under): * file_io.write_atomic(table, path) — the create_star_cat signature loses output_dir and img_number, which only ever reconstructed the destination's own directory and basename; * focal_plane.get_wcs / ccd_center_and_radius / focal_plane_disc — the linear-terms-only WCS workaround is unchanged, and the shared disc does one fits.open pass instead of a getheader per CCD. create_star_cat's tile branch now calls ccd_center_and_radius rather than inlining it, and both branches convert the shared degrees to the arcmin Vizier wants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
…non-portable line bin/sp's cfg() was a sed pattern-match over YAML; the venv it activates two lines earlier ships PyYAML (snakemake depends on it), so it parses the file instead. Same values for the two keys it reads. profiles/nibi's `--home /home/cdaley` cannot be made portable in place: YAML cannot splice an env var, snakemake escapes a literal $ before any shell sees it, and --apptainer-args REPLACES the profile value rather than appending — so injecting from bin/sp would mean restating the whole string, which container.py reads from this file precisely so `sp container exec` and a job cannot diverge. Marked as the one line a new user must edit, with the argument for why it stays a line and not a mechanism. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
…mple/cfis Only the .ini files are a genuine fork (fixed INPUT_DIRs, NUMBER_LIST); the SExtractor/PSFEx parameter files and the mask_default data directory were byte-for-byte copies. Symlinks keep $SP_CONFIG resolution unchanged while making the single source of truth visible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
Each explanation now lives where the mechanism is introduced; the other sites carry a one-line pointer. Consolidated: the manifest-vs-log contract (completeness.py), the params-formatted-once hazard (unit_pre), the localrule argument (Snakefile localrules), the exposure/tile reclamation asymmetry (clean_tile.py, clean_tile_targets), group resource composition (prepare.smk), the ngmix_range resume hazard (tile.smk range_hash), and what can be carried through apptainer-args (profiles/nibi --bind /local). Also drops two stale claims: tile_vignets no longer declares a temp() directory output, and the profile's group label list said tile_finish. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
…home clean_tile.py stops re-arguing clean_exposure.py's symlink safety, log deletion and tombstone-first ordering; run_report.py stops restating the manifest-vs-log contract and the ngmix products caveat; build_index.py's module docstring stops duplicating build(); ngmix_range.py points the RNG claim at position_seed and the resume hazard at NGMIX_RANGE_HASH. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
The physics lived in four places (module docstring, class Notes, the runner guard, the process() loop). It now lives in position_seed's docstring; the others carry a line and a pointer. Drops two stale claims there: the seed is no longer "off in production", and the noise-realisation rng is documented as per-object rather than per-tile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
The eight ngmix chunks of a tile each ran ngmix_range.py in their own shell and trusted the others to have landed on the same boundaries. That trust was misplaced once: a fused group holds its chunks open over the live checkout for hours, so an edit landed mid-flight was read by some chunks and not others — tile 186.307, chunk 5 split 17649..22060 where its siblings had 17129..21229, 520 objects orphaned and 831 measured twice, and merge_sep_cats would have concatenated it green. Single writer instead of eight agreeing readers. ngmix_range.py gains --write (compute the whole partition, publish it as JSON via temp-name + rename) and --read (look one chunk's row up). tile_vignets — the first member of the fused group, strictly preceding every chunk in its DAG — writes $SP_LOCAL/ngmix_ranges.json in its pre_run; each chunk reads its row. The partition math is untouched, so the boundaries are identical for identical inputs. Read mode does not fall back to recomputation when the file is absent; it exits non-zero pointing at tile_vignets. The fallback would be the race again. The file is group-internal plumbing, not DAG currency: it lives and dies with the group job exactly as the vignette store does, so it is no rule's output. NGMIX_RANGE_HASH now fingerprints the script that PRODUCES the file and rides on tile_vignets as well as tile_ngmix. It guards reproducibility across a resume, not sibling agreement — and putting it on both rules also removes the one-rule-fingerprint failure mode that deleted a final_cat on smk-g5, since an edit to the split now drags tile_vignets back into the group. Tests: the invariance property — write the partition, read every row back — reproduces the per-chunk computation exactly, plus the index-validation and no-fallback cases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
… campaign The SLURM executor re-invokes snakemake on every job's node, so jobs re-parsed the Snakefile and read workflow/scripts/* and the ini chain out of the LIVE checkout, hours after launch -- how the 186.307 partition split into two disagreeing halves. `sp run` now copies workflow/ (config symlinks dereferenced), src/ and the profile into <state dir>/code, records HEAD and a dirty flag in snapshot.json, and runs the campaign entirely out of that tree. Refreshed wholesale on every `sp run`; the other verbs run out of the existing snapshot, `sp container` excepted. workflow.basedir IS the snapshot, so every internal path follows for free. The profile's PYTHONPATH pin is the one that cannot (YAML splices nothing, and snakemake escapes a literal $): the snapshot's copy of the profile has that one path rewritten and --profile points at the copy, so container.py keeps reading one apptainer-args line -- the snapshot's, from inside a job; the checkout's, for interactive `sp container exec`. Docs follow: workflow/README.md gains the "launch code snapshot" section as the one home, and the mid-campaign-edit warnings in the Snakefile header, tile.smk and ngmix_range.py shrink to pointers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
Each vignet.<store>[str(obj_id)] access unpickles the object's whole all-epoch dict; six of them sat inside the per-epoch loop, so an object with n epochs paid n full unpickles per store where 1 suffices. Read each store's per-object dict once before the loop and index the locals inside it. The psf/gal dicts read for the empty-check in process() are now passed down instead of being re-read. Also cache f_wcs_file[exp_name] in a per-call dict: it was fetched twice per epoch (WCS, then header), each time unpickling all CCDs of the exposure. No numerical change: same values, fewer unpickles. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PsP9yhF9okPYrWKydmnpcS
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #848
Closes #873
Closed #844
The PRD in #848 is the design reference; this PR realizes it.
What this replaces: the three-level bash orchestration of real-data runs (
run_job_sp_canfar_v2.0.bash→job_sp_canfar_v2.0.bash→ per-site sbatch wrappers). The bash path is left in place until this workflow is validated and adopted; removing it is a follow-up PR.Shape:
workflow/at top level — Snakefile + three rule files, six stdlib scripts, committed configs with fixed paths, one nibi profile.workflow/bin/spis the entry point:sp run(two snakemake invocations, prepare then compute, joined by a sqlite tile↔exposure index),sp report(standalone, runs mid-flight),sp container(pull/status/sandbox/exec — per-user image cache, resolution sandbox → SIF → configured default),sp cancel.Additional changes not described in #848. Three change scientific behavior for all users, bash path included:
SEED_FROM_POSITION = Falsenow fails loudly instead of running the retired mode.rand_splitis now seeded from the file number (previously unseeded), making the PSF train/validation split reproducible — and different from any previous draw.STAR_THRESH20 → 22 in the twoconfig_tile_PiViVi_canfar_*.ini, aligning with the 80% convention already in the exposure PSF configs.Mechanical enablers and hardening, no output changes: ngmix ID range /
RUN_NAMEread env-expanded;merge_sep_catschunk-path derivation fixed for absolute paths with digits in parent dirs; Vizier responses no longer cached to$HOME(~50 GB at campaign scale); star catalogues written atomically; unit tests for the chunk partition, tile reclamation, and run report.Reviewing: read the PRD's D1–D6 first; each rule file's header comments carry the local why. Run results and DR6 projections are in the benchmark comment below.
🤖 Generated with Claude Code