Skip to content

fix(DEPLOY-005): resolve pinned universe sources from the repo, not DATA_DIR - #116

Open
DoRmAmMu1997 wants to merge 2 commits into
mainfrom
fix/deploy-005-universe-source-path
Open

fix(DEPLOY-005): resolve pinned universe sources from the repo, not DATA_DIR#116
DoRmAmMu1997 wants to merge 2 commits into
mainfrom
fix/deploy-005-universe-source-path

Conversation

@DoRmAmMu1997

Copy link
Copy Markdown
Owner

The defect

refresh_universe_files() could not rebuild the three Hemant universes in any deployment that sets DATA_DIR.

Their pinned source lists were resolved from UNIVERSE_DIR, which follows DATA_DIR. But COPY . . puts those lists at /app/data/universes/, so a container with DATA_DIR=/data looked on the data volume instead and raised:

FileNotFoundError: Universe source CSV not found: /data/universes/hemant_super_45.csv
Step Value Evidence
Image layout CSVs at /app/data/universes/ WORKDIR /app + COPY . . (Dockerfile:29,52); .dockerignore whitelists the four tracked CSVs
Repo-root default DEFAULT_DATA_DIR = /app/data PROJECT_ROOT = parents[2] (settings.py:27,30)
Runtime override DATA_DIR=/datauniverse_dir = /data/universes render.yaml:60,104; settings.py:341,122
Source path (before) /data/universes/<key>.csv built at import from UNIVERSE_DIR

Two blast radii

  1. The daily-scan cron never ran. render.yaml runs it as sh -c "python -c '...refresh_universe_files()' && run_daily_scan" on an ephemeral filesystem with no disk at all (render.yaml:95). So /data was empty every run, the refresh raised, and && meant the scan was skipped. Every night.
  2. First-deploy seeding failed. docs/operations.md:912 tells the operator to run that exact command in a Render Shell to populate the empty disk.

The blueprint comment shows how the gap was reasoned into existence — "the nifty/fno lists are downloaded, not baked into the image". True for NIFTY/F&O; the Hemant lists are baked in, just at a path nothing looked at. That comment is corrected here.

The fix

Separate the two concerns onto different anchors:

  • inputsUNIVERSE_SOURCE_DIR, anchored to PROJECT_ROOT: pinned, reviewed, shipped inside the image. New data/universes/sources/*.csv, extracted verbatim from the existing source_symbol values, so counts and ordering are unchanged (43 / 43 / 262, and the existing pinned-snapshot assertions on the last three tokens still hold).
  • outputsUNIVERSE_DIR, still following DATA_DIR onto the data volume.

This also closes a second-order smell the coupling caused: source and output being one path meant every refresh rewrote the file it had just read, so the working tree went dirty after any app run, and every committed row carried an absolute developer path in its source column. repo_relative_source_label() now stores a repo-relative POSIX label, and the three committed CSVs are rewritten accordingly — that diff is the source column only.

On the regression test

It runs in a subprocess with a relocated DATA_DIR, and that is deliberate rather than lazy: the faulty binding happened at import time, so monkeypatching the environment inside an already-imported process would have proved nothing. A fresh interpreter is exactly what the Render cron and docker compose give us.

Verified failing before this change with the exact production error above, and passing after.

Verification

Caveat, stated plainly: this was traced statically through committed files and reproduced in-process. Docker is not installed on the development machine, so the container-level run was not performed. If the Render blueprint was never applied, the defect was latent rather than live.

Docs updated: universe-management LLD (including a new design-decision row), render.yaml, docs/operations.md, README.md.

🤖 Generated with Claude Code

DoRmAmMu1997 and others added 2 commits September 4, 2026 19:04
…ATA_DIR

`refresh_universe_files()` could not rebuild the three Hemant universes in any
deployment that sets DATA_DIR, because their pinned source lists were resolved
from UNIVERSE_DIR (which follows DATA_DIR) rather than from the code.

In the image, `COPY . .` puts the lists at /app/data/universes/, but Render and
docker-compose set DATA_DIR=/data, so the builder looked on the data volume and
raised `FileNotFoundError: Universe source CSV not found:
/data/universes/hemant_super_45.csv`. Two consequences:

  * The daily-scan cron never ran. render.yaml runs
    `sh -c "python -c '...refresh_universe_files()' && run_daily_scan"` on an
    ephemeral filesystem with no disk at all, so the refresh raised every night
    and `&&` meant the scan was skipped.
  * First-deploy seeding failed. docs/operations.md tells the operator to run
    that exact command in a Render Shell to populate the empty disk.

The same coupling meant a generated universe CSV was also its own input: each
refresh rewrote the file it had just read, so the working tree went dirty after
every app run and every committed row carried an absolute developer path in its
`source` column.

Split the two concerns onto separate anchors:

  * inputs  - `UNIVERSE_SOURCE_DIR`, anchored to PROJECT_ROOT, pinned, reviewed,
    shipped in the image (new `data/universes/sources/*.csv`, extracted verbatim
    from the existing `source_symbol` values - counts and ordering unchanged at
    43 / 43 / 262).
  * outputs - `UNIVERSE_DIR`, still following DATA_DIR onto the data volume.

Also store `source` as a repo-relative POSIX label via
`repo_relative_source_label()`, and rewrite the three committed CSVs so they no
longer embed a developer's home directory. That diff is the source column only.

The regression test runs in a subprocess with a relocated DATA_DIR, because the
faulty binding happened at import time and monkeypatching an already-imported
process would prove nothing. Verified failing before this change with the exact
production error, and passing after.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant