Fix env cache churn from defaulted exclude-newer and race in concurrent env creation

[OwenPriceSkelly]

Problem

Found 2026-09-01 while launching several concurrent tasks of one function: 5 of 6 concurrent .local() tasks failed with a mix of uv venv: "A directory already exists", "failed to open file ...uv.toml: No such file or directory", and ModuleNotFoundError from a half-installed environment. Two underlying bugs:

1. A defaulted exclude-newer churns the env hash every second. When a script's PEP 723 header has no [tool.uv] exclude-newer, Pep723Metadata defaults it to datetime.now(timezone.utc) at parse time, and compute_env_hash includes the whole [tool.uv] table. So the env hash changes every second: every run (and every Function whose shell command is templated in a different second) gets a fresh ENV_DIR and a full uv rebuild, defeating the per-site environment cache entirely. Meanwhile N tasks of one function submitted together do share a hash — and since their env never pre-exists, they all take the create path at once.

2. Environment creation isn't safe under concurrency. The shell template guards creation with a bare [ -d "$ENV_DIR" ] check and builds directly at the final path, so concurrent same-hash tasks race: two uv venv calls collide, the shared ${ENV_DIR}.uv.toml staging file gets mv'd out from under the others, and a task can start running against an env another task is still installing into.

Fix

• compute_env_hash now drops exclude-newer from the hashed [tool.uv] table when the user didn't set it explicitly (checked via pydantic model_fields_set). An exclude-newer pinned in the script header still affects the hash exactly as before, and the defaulted value is still written to the env's uv.toml, so fresh builds remain pinned to their build time.
• The shell template builds the env in a unique temp dir ($ENV_DIR.tmp.$(hostname).$$) — including its staged uv.toml and groundhog-meta.json — then publishes it with an atomic rename. A task that loses the publish race discards its build and uses the winner's. uv venv gets --relocatable so entry-point script shebangs stay valid after the rename. Net effect: an existing ENV_DIR always means a complete environment.

A task killed mid-build can still leave a stale *.tmp.* dir behind, but stale temp dirs are inert (never reused, unlike the half-built ENV_DIRs the old code could leave and then happily reuse).

Verification

• Existing suite passes (357 tests), plus new tests: defaulted exclude-newer hashes identically across parses at different times, user-pinned exclude-newer still changes the hash, and the template builds in the temp dir / publishes by rename / never touches $ENV_DIR/ before publish.
• End-to-end: rendered a real shell command from a script with no exclude-newer; the env hash is now stable across seconds, and 6 concurrent executions of the env-creation section against a cold cache all succeeded (previously 5/6 failed), leaving exactly one complete env dir — with uv.toml and groundhog-meta.json inside, and its python able to import the installed dependency.

One adjacent issue observed but left alone: on hosts with no uv on PATH, concurrent tasks also race in the pip install uv bootstrap block. That's pre-existing and much rarer in practice.

🤖 Generated with Claude Code