LightconeResearch · dkn16 · Jul 16, 2026
diff --git a/docs/api/async_jobs.md b/docs/api/async_jobs.md
@@ -0,0 +1,26 @@
+# lightcone.engine.async_jobs
+
+Coarse-grained SLURM submission, polling, and cancellation for
+`lc run --async`.
+
+Source: `src/lightcone/engine/async_jobs.py`.
+
+The module resolves a requested ASTRA sub-DAG, aggregates its recipe
+resources, selects a site policy, and renders one sbatch script. The script
+re-enters ordinary `lc run`, so the async layer never duplicates recipe,
+container, Dask, manifest, or validation logic.
+
+Important entry points:
+
+- `resolve_subdag_outputs()` — requested materializable outputs plus upstream
+  recipe dependencies.
+- `aggregate_job_resources()` — element-wise maximum node shape and padded
+  serial walltime.
+- `select_slurm_policy()` — deterministic site profile to `shared` or
+  `regular`.
+- `submit_job()` — render, `sbatch --parsable`, and persist a `JobRecord`.
+- `refresh_job_records()` — batch-query `squeue`, then `sacct`.
+- `cancel_job()` — resolve an active record, call `scancel`, and update it.
+
+Project records live in `.lightcone/jobs/<job-id>.json`; logs live under the
+resolved scratch root.
diff --git a/docs/api/cli.md b/docs/api/cli.md
@@ -1,7 +1,8 @@
 # lightcone.cli.commands
 
-The Click surface. Defined in `src/lightcone/cli/commands.py`. Six
-public commands: `init`, `run`, `status`, `verify`, `build`, `setup`.
+The Click surface. Defined in `src/lightcone/cli/commands.py`. Public
+commands include `init`, `run`, `status`, `cancel`, `verify`, `build`, and
+`export`.
 
 The user-facing reference is in [CLI Overview](../cli/index.md). This
 page is a tour of the module internals.
@@ -14,10 +15,7 @@ page is a tour of the module internals.
 @click.pass_context
 def main(ctx: click.Context) -> None:
     ctx.ensure_object(dict)
-    if ctx.invoked_subcommand in ("setup", "init", "eval"):
-        return
-    if not _config_path().exists():
-        # print friendly error, sys.exit(1)
+    _ensure_global_config()
 ```
 
 `main` is exposed as `lightcone.cli.main` (re-exported from
@@ -44,8 +42,8 @@ it directly. To add a new tier, edit this dict and update the
 
 ### `_config_path() → Path`
 
-Returns `~/.lightcone/config.yaml`. Used by the `main` group's
-auto-init check and by `setup`.
+Returns `~/.lightcone/config.yaml`. The main group creates it on first use
+with container and SLURM defaults.
 
 ### `_project_root(start: Path | None = None) → Path`
 
@@ -76,6 +74,9 @@ Map a status literal to the Rich-formatted display label:
 | `stale` | `[yellow]✸ stale[/yellow]` |
 | `missing` | `[red]✗ miss[/red]` |
 | `alias` | `[dim]→ alias[/dim]` |
+| `queued` | `[blue]◷ queued[/blue]` |
+| `running` | `[cyan]▶ running[/cyan]` |
+| `failed` | `[red]✗ failed[/red]` |
 
 ## Boilerplate text
 

diff --git a/docs/api/index.md b/docs/api/index.md
@@ -7,17 +7,19 @@ is a thin Click wrapper around these modules.
 
 | Module | Role |
 |--------|------|
-| [`lightcone.cli.commands`](cli.md) | Click CLI: `init`, `run`, `build`, `status`, `verify`, `setup`. |
+| [`lightcone.cli.commands`](cli.md) | Click CLI: `init`, `run`, `build`, `status`, `cancel`, `verify`, `export`. |
 | [`lightcone.engine.manifest`](manifest.md) | Per-output `.lightcone-manifest.json` write/read; `code_version`, `sha256_dir`. The integrity layer. |
 | [`lightcone.engine.snakefile`](snakefile.md) | Generate `.lightcone/Snakefile` and `snakefile-config.json` from `astra.yaml`. |
 | [`lightcone.engine.container`](container.md) | Runtime detection, content-addressed image tags, `wrap_recipe`. |
 | [`lightcone.engine.dask_cluster`](dask_cluster.md) | Cluster lifecycle for `lc run` (local / SLURM / external). |
+| [`lightcone.engine.resources`](resources.md) | Canonical ASTRA resource parsing and Snakemake-name mapping. |
+| [`lightcone.engine.async_jobs`](async_jobs.md) | Coarse SLURM submission, polling, records, and cancellation. |
 | [`lightcone.engine.status`](status.md) | Manifest-driven status walker. |
 | [`lightcone.engine.verify`](verify.md) | Recompute hashes; walk the input chain. |
 | [`lightcone.engine.tree`](tree.md) | Sub-analysis tree helpers — outputs, decisions, `from:` resolution. |
 | [`lightcone.engine.validation`](validation.md) | Post-recipe sanity checks (empty dir, all-NaN columns, …). |
 | [`snakemake_executor_plugin_dask`](dask_executor.md) | Snakemake executor plugin → `dask.distributed`. |
-| `lightcone.engine.site_registry` | Vestigial — no active code path imports it. See [api/site_registry](site_registry.md). |
+| [`lightcone.engine.site_registry`](site_registry.md) | Site detection, scratch/runtime defaults, and async SLURM policies. |
 
 ## Common entry points
 

diff --git a/docs/api/resources.md b/docs/api/resources.md
@@ -0,0 +1,11 @@
+# lightcone.engine.resources
+
+Canonical parsing for ASTRA `Recipe.resources`.
+
+Source: `src/lightcone/engine/resources.py`.
+
+`parse_recipe_resources()` normalizes portable ASTRA values to CPUs, MB,
+GPUs, and seconds. `RecipeResources.snakemake()` maps those to
+`cpus_per_task`, `mem_mb`, `gpus_per_task`, and minute-based `runtime`.
+Synchronous Snakefile generation and asynchronous SLURM sizing both use this
+module, preventing the two paths from interpreting a recipe differently.
diff --git a/docs/api/site_registry.md b/docs/api/site_registry.md
@@ -1,21 +1,18 @@
 # lightcone.engine.site_registry
 
-> **Status: orphaned.** Nothing in the active code path imports this
-> module. It's residue from the (now-removed) target system. Keep
-> reading if you want to know what's still there; otherwise skip to
-> [api/container](container.md) and [api/dask_cluster](dask_cluster.md)
-> for what actually drives execution today.
+Site detection and deterministic per-site execution defaults. The active
+container, scratch, and asynchronous SLURM paths all consume this registry.
 
 Source: `src/lightcone/engine/site_registry.py`.
 
-## What's still in the file
+## Public data and helpers
 
 The module exposes:
 
 - `SITE_DEFAULTS` — a dict mapping site keys (`"perlmutter"`, `"local"`)
   to a structured defaults dict (display name, hostname patterns,
-  suggested QoS / constraint / time-limit options, scratch deny paths,
-  container runtime).
+  suggested QoS / constraint / time-limit options, scratch paths, container
+  runtime, and `async_slurm` policies).
 - `detect_site(hostname_or_name) → str | None`
 - `get_site_defaults(site_key) → dict | None`
 - `list_known_sites() → list[tuple[str, str]]`
@@ -24,25 +21,10 @@ The module exposes:
 The functions still work on their own; they just don't have a caller
 inside lightcone-cli right now.
 
-## What's actually used today
+## Async SLURM policy
 
-The Perlmutter scratch deny rules used to be merged into
-`.claude/settings.json` automatically when a non-local target was
-configured. With the target system gone, the equivalent rules are
-hard-coded inline in `PERMISSION_TIERS` (see
-`src/lightcone/cli/commands.py`):
-
-```python
-"Edit(//scratch/**)",
-"Edit(//pscratch/**)",
-```
-
-If you want richer per-site rules without rebuilding the target
-system, point `lc init`'s `_install_claude_plugin` at
-`get_site_scratch_deny_rules(detect_site(socket.gethostname()))` and
-merge the result into the deny list. Two lines of code.
-
-## Recommendation
-
-Either delete this module (no callers) or revive it for the use case
-above. Leaving it as-is encourages the drift this audit is fighting.
+Perlmutter declares CPU and GPU node shapes, fractional shared limits, and
+the 48-hour `shared`/`regular` caps under `async_slurm`. The submission
+engine selects the CPU profile when no GPU is declared and the GPU profile
+otherwise. Adding another site is a table extension rather than a new branch
+in the renderer.
diff --git a/docs/api/snakefile.md b/docs/api/snakefile.md
@@ -42,20 +42,20 @@ rule <name>:
     output:
         data=directory("<output_dir>"),
         manifest="<output_dir>/.lightcone-manifest.json",
+    threads: <recipe.resources.cpus>
+    resources:
+        cpus_per_task=<cpus>,
+        mem_mb=<memory in MB>,
+        gpus_per_task=<gpus>,
+        runtime=<time_limit in minutes>,
     params:
         cfg=lambda wc: CFG["<rule_key>"][wc.universe],
     run:
-        shell('printf "▶ <rule_key> [%s]\\n" "{wildcards.universe}" >&2')
-        shell(params.cfg["shell_command"])
-        write_manifest(
-            output_dir=Path(output.data),
-            inputs={"<inp_id>": Path(input.<inp_id>), ...},
-            cfg=params.cfg,
-        )
-        for _w in validate_output(Path(output.data), params.cfg.get("output_type"), params.cfg["output_id"]):
-            print(f"\033[33m⚠\033[0m {_w}", file=sys.stderr)
+        run_rule(...)
 ```
 
+Unset optional memory/GPU/time values are omitted. CPU defaults to one.
+
 ## `cfg` content
 
 Per-`(rule_key, universe)` entry written into

diff --git a/docs/api/targets.md b/docs/api/targets.md
@@ -1,11 +1,15 @@
 # lightcone.engine.targets (removed)
 
-The target configuration module is gone. The only remaining global config
-is `~/.lightcone/config.yaml`, which today carries one key:
+The target configuration module is gone. The remaining global config is
+`~/.lightcone/config.yaml`:
 
 ```yaml
 container:
   runtime: auto   # auto | docker | podman | podman-hpc | none
+slurm:
+  account: null
+  time_padding: 1.5
 ```
 
-It is read by [`lightcone.engine.container.load_runtime`](container.md).
+It is read by [`lightcone.engine.container.load_runtime`](container.md) and
+[`lightcone.engine.async_jobs`](async_jobs.md).
diff --git a/docs/architecture.md b/docs/architecture.md
@@ -143,8 +143,9 @@ For each declared recipe input:
 - **`alias`** — output declared without a recipe; materialized only as a
   side effect of an upstream.
 
-`status` reads only manifests. No Snakemake import, no `.snakemake/`
-directory required, works on a fresh clone or frozen archive.
+Recorded async jobs can temporarily overlay `queued`, `running`, `failed`,
+`cancelled`, or `unknown`. `status` still does not import Snakemake or require
+`.snakemake/`; it polls Slurm only when project job records exist.
 
 ---
 
@@ -163,12 +164,25 @@ across three branches:
    one `dask worker` per node across the allocation. Workers advertise
    the node's resources via Dask abstract resources (`cpus`, `memory`,
    `gpus`). The Snakemake executor plugin maps per-rule
-   `cpus_per_task` / `mem_mb` / `gpus_per_task` to per-task constraints.
+   ASTRA's `cpus` / `memory` / `gpus` values are normalized to
+   `cpus_per_task` / `mem_mb` / `gpus_per_task` per-task constraints.
 3. Neither → `LocalCluster()` sized to the local machine.
 
 The scheduler is always in-process so its lifetime equals the run's
 lifetime: no service to manage, no orphaned schedulers.
 
+### Asynchronous allocations
+
+Module: [`lightcone.engine.async_jobs`](api/async_jobs.md).
+
+`lc run --async` resolves one requested universe's sub-DAG, aggregates
+portable ASTRA resources, selects a site policy, and submits one coarse
+sbatch job. The rendered script activates the submitting Python environment
+and invokes plain `lc run`; once running, it follows branch 2 above and keeps
+all container and provenance behavior identical to synchronous execution.
+Job records are local caches; Slurm owns job state and manifests own output
+state.
+
 ### The Snakemake executor
 
 Module: [`snakemake_executor_plugin_dask`](api/dask_executor.md).
@@ -270,18 +284,20 @@ warnings.
 src/lightcone/                  # PEP 420 namespace package — NO __init__.py
 ├── cli/                        # Click surface
 │   ├── __init__.py             # exposes main()
-│   ├── commands.py             # init, run, status, verify, build, export
+│   ├── commands.py             # init, run, status, cancel, verify, build, export
 │   └── plugin.py               # plugin source-dir discovery
 ├── engine/                     # execution substrate
 │   ├── manifest.py             # write_manifest, sha256_dir, code_version
 │   ├── snakefile.py            # generate .lightcone/Snakefile from astra.yaml
 │   ├── container.py            # docker/podman/podman-hpc build + recipe wrap
 │   ├── dask_cluster.py         # cluster lifecycle (local/SLURM/external)
+│   ├── resources.py            # ASTRA resource normalization
+│   ├── async_jobs.py           # SLURM submit/status/cancel
 │   ├── status.py               # manifest-driven status walker (no Snakemake)
 │   ├── verify.py               # recompute hashes, walk the chain
 │   ├── tree.py                 # sub-analysis tree helpers
 │   ├── validation.py           # post-recipe output sanity checks
-│   └── site_registry.py        # vestigial; not imported by active code
+│   └── site_registry.py        # site runtime, scratch, and SLURM policy
 └── eval/                       # evaluation harness for the agent loop
     ├── cli.py harness.py sandbox.py graders.py build.py report.py models.py
 
@@ -344,7 +360,7 @@ each rule to a Dask scheduler.
 | `.lightcone/Snakefile` | Project (generated) | Auto-generated by `lc run`. Don't edit. |
 | `.lightcone/snakefile-config.json` | Project (generated) | Per-`(rule, universe)` config. |
 | `.lightcone/lightcone.yaml` | Project | Tiny scratchpad — currently writes only `target: local`. Not consumed by today's code. |
-| `~/.lightcone/config.yaml` | User | `container.runtime`. |
+| `~/.lightcone/config.yaml` | User | `container.runtime`, `slurm.account`, async walltime padding. |
 | `.claude/settings.json` | Project | Claude Code permissions. |
 
 The `dagster.yaml` and `~/.lightcone/targets/*.yaml` files referenced in

diff --git a/docs/cli/cancel.md b/docs/cli/cancel.md
@@ -0,0 +1,23 @@
+# lc cancel
+
+Cancel a recorded asynchronous SLURM job.
+
+## Synopsis
+
+```text
+lc cancel <JOB-ID | OUTPUT-ID>
+```
+
+`lc cancel` refreshes recorded state, resolves the reference to one active
+job, calls `scancel`, and stores `CANCELLED` in the job record. An output id
+may refer to either an explicitly requested target or one of its resolved
+upstream dependencies. If it matches multiple active jobs, cancel by job id
+to disambiguate.
+
+```bash
+lc cancel 1234567
+lc cancel heavy_fit
+```
+
+Only jobs recorded under the current project's `.lightcone/jobs/` directory
+are managed.
diff --git a/docs/cli/index.md b/docs/cli/index.md
@@ -19,6 +19,7 @@ Code skills, with the CLI as the durable, scriptable backstop.
 | [`lc run`](run.md) | Generate the Snakefile and dispatch through Snakemake + Dask. |
 | [`lc build`](build.md) | Build container images declared in `astra.yaml`. |
 | [`lc status`](status.md) | Manifest-driven status report. No Snakemake import needed. |
+| [`lc cancel`](cancel.md) | Cancel a recorded asynchronous SLURM job. |
 | [`lc verify`](verify.md) | Recompute hashes, walk the input chain, surface tampering. |
 | [`lc export`](export.md) | Emit interoperable bundles (Workflow Run RO-Crate) for publication. |