Retail DC demo — Shoprite-style data generator + 18-model dbt pipeline - #2
Open
imbilawork wants to merge 13 commits into
Open
Retail DC demo — Shoprite-style data generator + 18-model dbt pipeline#2imbilawork wants to merge 13 commits into
imbilawork wants to merge 13 commits into
Conversation
- Bump pyjwt pin to >=2.10.1,<3.0 (closes Dependabot alert #1 — PyJWT 2.9.0 accepts unknown `crit` header extensions, a high-severity issue for any service validating JWTs from untrusted parties). - Add sql/neon-init.sql — one-time CREATE SCHEMA for superset_meta and analytics. Kept out of dbt/seeds/ where dbt would try to load it as tabular data. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- scripts/setup-env.sh: reads a Neon pooled URI (stdin, secrets.txt, or interactive prompt), derives the direct endpoint by stripping `-pooler`, generates the three required secrets with openssl, writes a mode-600 .env. - docs/quickstart.md: eight-step sequence from bare MacBook to a running Superset with seeded SilverGro marts. Includes dbt setup, dataset registration, and dashboard authoring hints. - README: point at the quickstart instead of duplicating setup steps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- add flask-cors: ENABLE_CORS=True requires the dep; without it superset-init crashes on ModuleNotFoundError before metadata migrations run. - add flask-appbuilder>=4.5.4: FAB 4.5.0 emits int JWT subjects; pyjwt>=2.10 rejects those on decode, breaking every authenticated API call after login. - add TCP keepalives to SQLALCHEMY_ENGINE_OPTIONS: superset init creates hundreds of permission-view rows in one transaction; Neon's proxy silently drops idle connections mid-transaction without keepalives (observed: SSL SYSCALL error: EOF after ~17 min into init). - celery beat --schedule=/tmp/celerybeat-schedule: default schedule path is /app which is read-only for the superset user, beat restart-loops otherwise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eline
Populates a second analytics schema (analytics_retail_dc) alongside the SAFEX
commodity model. Demonstrates what the BI stack looks like against a large-
retailer inventory footprint:
- 8 DCs (Cilmor, Canelands, Basfour, Montague Gardens, Brackenfell Liquor,
Midrand, East London, Bloemfontein) with realistic capacity 14k-65k pallets
- 2,493 SKUs across 18 categories / 122 subcategories
- 80 SA-flavoured suppliers (Tiger Brands, AVI, Clover, Unilever SA, etc.)
+ three own-label flags (Ritebrand, Oh My Goodness!, Simple Truth)
- 820 stores across 6 banners with province-weighted distribution
- 183k current pallet positions (util 52-72% across DCs)
- 553k movements over 60 days with Pareto-weighted SKU velocity
(top 20% of SKUs do ~80% of outbound volume)
scripts/generate_retail_dc.py — deterministic (seed=42), standard-library only.
Dims land as dbt seeds; fct_pallet_stock regenerates locally (gitignored);
fct_stock_movements loads via psycopg2 COPY (scripts/load_movements.py) because
dbt seed would take an hour for 550k rows over Neon's latency.
dbt models in dbt/models/dc/: 8 staging views + 10 marts covering DC
utilisation, stock cover days, expiry risk, category mix, store/DC flow,
supplier inbound and top movers.
scripts/superset_bootstrap.py — idempotent API driver: registers the Neon DB,
creates datasets, builds charts and wires them into a "Retail DC — Operations
Overview" starter dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single self-contained HTML file covering the whole stack in one place: architecture diagram (SVG), both dbt domains + row counts, local setup, retail DC demo bring-up, GCP Cloud Run deploy, the four first-run bring-up gotchas, operations (refresh / redeploy / teardown), and a reference table of env vars, deployed services and repo layout. No external dependencies. Open in a browser to read. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dbt-project-yml schema override concatenates with profile target, so the actual landing schema for SAFEX models is analytics_analytics. Added a callout flagging the issue + the fix path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GitHub secret-scanning flagged a Postgres connection-string-shaped placeholder; it was docs/example values, not real credentials, but the document also carried real infrastructure identifiers (Neon endpoint, Upstash endpoint, Cloud Run URL, project number) that should not be in a public-repo doc. Replaced each with angle-bracket placeholders and reworded the pooled-URI example so it no longer parses as a valid URL. No actual credentials were ever committed. This changes HEAD only; the previous commit 8bab97f still contains the identifiers in its diff until either the alert is dismissed as a false positive, or the branch is rewritten with git filter-repo and force-pushed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents the Global External HTTPS LB + Serverless NEG workaround for custom domains in africa-south1 (where managed run domain-mappings returns 501 UNIMPLEMENTED). Covers: - why native domain-mappings is blocked - SVG of the request path: browser → DNS → anycast IP → forwarding rule → HTTPS proxy → URL map → backend service → serverless NEG → Cloud Run - 7-command gcloud provisioning sequence - DNS record to add (Cloudflare grey-cloud) - three gotchas: portName on backend service with Serverless NEG; cert stuck at PROVISIONING behind Cloudflare proxy; and the 501 UNIMPLEMENTED on africa-south1 that started this whole dance - verify + teardown All placeholder values; no real anycast IP, hostname, or project number. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Complete inventory of what the GCP deploy creates, with IAM graph and day-two material: - resource inventory table: every resource in nth-bi-stack, by layer - IAM graph SVG: principals → identities → roles/resources - Secret Manager table: each of the 8 secrets with purpose + consumer - Cloud Run service flags table with the reasoning behind every value (CPU/memory/concurrency/min-max/timeout) - cost expectations at demo scale (~$60-70/mo, itemised) - observability: where logs / metrics / audit live + a gcloud logging tail - teardown: surgical cleanup of the bi-stack resources vs full project deletion All placeholder values; no real identifiers or credentials. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The custom domain (Global HTTPS LB) and min-instances=1 are the two
costs that make the stack ~$60/mo at idle. Neither is necessary for
dev/testing, but both are nice when showing to a customer.
- Add gcp/deploy-lb.sh: idempotent, 7-resource LB provisioning. Takes
GCP_PROJECT + BI_DOMAIN env vars. Prints the anycast IP at the end.
- Add gcp/teardown-lb.sh: reverse-order delete for clean removal.
- Default gcp/deploy-web.sh to --min-instances=0 (was 1). Override with
BI_MIN_INSTANCES=1 for an always-warm instance ahead of a demo.
- Update docs/overview.html:
- new "scripted provisioning + teardown" subsection for the LB
- extended cost table showing both configurations side by side
- callout explaining the default shape and how to flip to warm+LB
Steady-state cost on default settings: ~$0-2/mo. For a demo, two
commands get to ~$60/mo polish: deploy-lb.sh + BI_MIN_INSTANCES=1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Old string-splitting code:
port=int(REDIS_URL.rsplit(":", 1)[-1].split("/")[0])
crashes on ValueError when the URL has a query component, e.g.
rediss://default:pw@host:6379?ssl_cert_reqs=CERT_REQUIRED
because the "port" piece becomes "6379?ssl_cert_reqs=CERT_REQUIRED".
Upstash + python-redis require ssl_cert_reqs for TLS connections, so
that query param is mandatory for the Cloud Run deploy. Caused a web
startup crash the moment Cloud Run picked up a new secret version with
the param appended.
Fix: use urllib.parse.urlparse. Also enables ssl=True + ssl_cert_reqs
on RESULTS_BACKEND automatically when the URL scheme is rediss://.
Local docker-compose (redis://redis:6379/0) behaves identically.
Fifth first-run bring-up bug — should have caught this when I added the
Upstash ssl_cert_reqs query param, didn't, cost us a surprise 503 the
next day when Cloud Run re-read the secret.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-click links into Cloud Run, logs, monitoring, IAM, secrets, Artifact Registry, billing, org policies, and load balancer lists. Saves the tab-hunt every time something needs checking. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Added 'Neon compute topology — read replica split' subsection under Data model. Covers: - why split: offload dashboard SELECTs from the primary that takes Superset metadata writes + dbt builds - URI-to-endpoint mapping (metadata → primary direct, analytics → replica direct, dbt → primary pooled) - Neon-specific note: replicas share credentials with the primary; endpoint URL is a literal string-swap of the ep- prefix - snippet: swap via one gcloud secrets versions add + one PUT against Superset's database API Also flagged the change in the Secret Manager inventory: ANALYTICS_DB_URI now points at the read replica, not the primary. Companion to the runtime change: ANALYTICS_DB_URI secret bumped to v2 with the replica URI; Superset's stored 'Analytics (Neon)' database record updated via API to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a second analytics schema (
analytics_retail_dc) alongside SAFEX to show what the BI stack looks like against a large-retailer inventory footprint.scripts/generate_retail_dc.py): 8 DCs, 2,493 SKUs across 18 categories, 80 SA-flavoured suppliers (Tiger Brands, AVI, Clover, Unilever SA, P&G, Distell, SAB + 3 own-label), 820 stores across 6 Shoprite-group banners, 183k pallet positions, 553k movements over 60 days with Pareto-weighted SKU velocity. Deterministic (seed=42).dbt/models/dc/: 8 staging + 10 marts covering DC utilisation, stock cover days, expiry risk, category mix, store/DC flow, supplier inbound, top movers.scripts/load_movements.py): bulk-loads the 553k movement rows viapsycopg2.copy_expert(dbt seed would take ~1 hour over Atlantic latency).scripts/superset_bootstrap.py): idempotent REST-API driver that registers the Neon DB, creates 10 datasets, builds charts, and wires them into a "Retail DC — Operations Overview" starter dashboard.Row counts
dim_skudim_storefct_pallet_stockfct_stock_movementsfct_stock_cover_daysfct_store_dc_flowStock on hand: R14.0 bn. 28-day outbound: R25.8 bn. Cover-days bucket split: 22% stockout risk / 21% healthy / 16% buffer / 31% overstock / 9% non-moving.
Depends on
Test plan
python3 scripts/generate_retail_dc.pyregenerates all CSVs deterministicallydbt seed --select path:seeds/dcloads 7 seeds (~180s for pallet_stock over Neon)scripts/load_movements.pyCOPYs 553k rows cleanlydbt run --select path:models/dcbuilds 18 models greendbt test --select path:models/dcpasses 8 source testsscripts/superset_bootstrap.pyregisters DB + datasets + dashboard against a running Superset🤖 Generated with Claude Code