LCR-179: Implement asynchronous Slurm job support

[dkn16]

Summary

Adds v1 asynchronous Slurm execution to lightcone-cli while preserving the
existing synchronous pilot-job model.

Long-running work can now be submitted with:

lc run --async <output> --universe <universe>

The submitted batch script re-enters plain lc run inside the new allocation.
Container wrapping, including podman-hpc, Dask dispatch, Snakemake execution,
validation, locking, and manifest generation therefore remain identical to
synchronous execution.

User-facing changes

lc run --async

• Allowed from both Perlmutter login and compute nodes.
• Resolves the requested output's complete upstream sub-DAG.
• Requires resources.time_limit on every recipe in that sub-DAG.
• Aggregates:
  • CPU, memory, and GPU shape using the element-wise maximum;
  • walltime using the serial sum multiplied by configurable padding.
• Selects Perlmutter shared or regular deterministically.
• Renders .lightcone/jobs/<timestamp>-<target>.sbatch.
• Activates the environment containing the current Python executable.
• Executes plain lc run inside the batch allocation.
• Submits with sbatch --parsable.
• Records the job in .lightcone/jobs/<job-id>.json.
• Submits one job per discovered universe when --universe is omitted.

Configuration

The first async submission requires a Slurm account:

# ~/.lightcone/config.yaml
container:
  runtime: podman-hpc

slurm:
  account: m1234
  time_padding: 1.5

The account can also be overridden per submission:

lc run --async heavy_fit --account m1234

lc status

• Batch-queries squeue for active jobs and sacct for completed or failed
  jobs.
• Annotates affected outputs with:
  • queued;
  • running;
  • failed;
  • cancelled;
  • unknown.
• Includes job id, QoS, raw Slurm state, and failure log path.
• Preserves manifests as the source of truth for completed outputs.
• Adds job data to lc status --json.

lc cancel

Adds:

lc cancel <job-id | output-id>

The command resolves one active recorded job, calls scancel, and updates its
local record.

Synchronous resource preflight

Plain lc run now checks whether every selected recipe fits a worker in the
current Slurm allocation.

An impossible CPU, memory, or GPU request fails immediately with a hint to
request a larger interactive allocation or use lc run --async, instead of
leaving Dask waiting indefinitely for an eligible worker.

Resource plumbing

Adds one canonical ASTRA resource parser shared by sync and async execution:

ASTRA	Snakemake
cpus	threads, cpus_per_task
memory	mem_mb
gpus	gpus_per_task
time_limit	runtime

This fixes the previous gap where recipe.resources existed in ASTRA but was
not emitted into generated Snakemake rules.

Perlmutter policy

Site policy lives in engine/site_registry.py.

• CPU shared jobs support up to half a CPU node.
• GPU shared jobs support one or two GPUs and request the corresponding CPU
  and host-memory allocation.
• Larger single-node jobs use regular.
• Padded walltime beyond the 48-hour regular cap is rejected.
• A recipe larger than one node is rejected in v1.

Execution invariant

The asynchronous layer only allocates and tracks work:

lc run --async
    -> sbatch
        -> plain lc run inside SLURM_JOB_ID
            -> existing Dask/Snakemake/container/manifest path

It does not introduce a second recipe execution implementation.

Documentation

Updates:

• CLI references for run, status, and cancel.
• First-run configuration instructions.
• Cluster and Perlmutter workflow documentation.
• Architecture and Python API references.
• Site registry and resource mapping documentation.

Test plan

• pytest -q -m "not slow" — 369 passed, 1 deselected.
• ruff check src tests
• mypy src
• Generated Snakefile parses with Snakemake.
• Documentation navigation configuration parses.
• Tests cover resource parsing, DAG resolution, QoS mapping, script
  rendering, submission records, polling, cancellation, login-node guidance,
  compute-node detection, and multi-universe submission.
• Run the existing real-Dask smoke test in an environment permitting local
  network binding.
• Submit a small real Perlmutter shared CPU job.
• Submit a small real Perlmutter shared GPU job.
• Verify status transitions and cancellation against live Slurm.

V1 limitations

• Perlmutter is the only configured async site.
• One coarse, single-node sbatch allocation is submitted per universe.
• No job arrays, per-rule Slurm dependencies, automatic sharding,
  checkpointing, or daemon.
• The batch script uses the project checkout at execution time; working-tree
  snapshotting is not included in this PR.

Implementation

• Branch: feat/async-jobs
• Commit: 04be9de — Implement asynchronous Slurm job support
• Linear: LCR-179

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	lightcone-cli	04be9de	Commit Preview URL Branch Preview URL	Jul 16 2026, 09:28 AM