Create reusable anonymized Project Ledger template

.dockerignore

@@ -0,0 +1,9 @@
+.git
+.github
+.venv
+__pycache__
+data
+logs
+tests
+infra
+dbt/target

.env.example

@@ -0,0 +1,8 @@
+# Required only for live SEC requests.
+LEDGER_SEC_USER_AGENT=YourCompany Data Engineering your-team@example.com
+
+# Optional overrides.
+LEDGER_OUTPUT_DIR=data
+LEDGER_REQUEST_TIMEOUT_SECONDS=30
+LEDGER_MAX_RETRIES=3
+LEDGER_REQUESTS_PER_SECOND=5

.github/workflows/ci.yml

@@ -0,0 +1,49 @@
+name: CI
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.12"
+      - run: uv sync --group dev
+      - name: Ruff
+        run: |
+          set -o pipefail
+          uv run ruff check src tests --output-format=full 2>&1 | tee ruff-output.txt
+      - name: Upload Ruff diagnostics
+        if: failure()
+        uses: actions/upload-artifact@v4
+        with:
+          name: ruff-diagnostics
+          path: ruff-output.txt
+          retention-days: 7
+      - name: Mypy
+        run: uv run mypy src
+      - name: Pytest
+        run: uv run pytest
+      - name: Fixture pipeline smoke test
+        run: |
+          cp config/ledger.example.yml config/ledger.yml
+          uv run python -m ledger.cli --config config/ledger.yml --fixture-dir tests/fixtures
+      - name: dbt build
+        run: uv run dbt build --project-dir dbt --profiles-dir dbt
+      - uses: hashicorp/setup-terraform@v3
+        with:
+          terraform_version: "1.9.8"
+      - name: Terraform format
+        run: terraform -chdir=infra/gcp fmt -check -recursive
+      - name: Terraform validate
+        run: |
+          terraform -chdir=infra/gcp init -backend=false
+          terraform -chdir=infra/gcp validate

.gitignore

@@ -0,0 +1,18 @@
+.venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.env
+config/ledger.yml
+data/
+logs/
+target/
+dbt/target/
+dbt/logs/
+*.duckdb
+.terraform/
+*.tfstate
+*.tfstate.*
+.DS_Store

CONTRIBUTING.md

@@ -0,0 +1,16 @@
+# Contributing
+
+Create a focused branch, add or update tests with each behavior change, and run:
+
+```bash
+bash scripts/validate.sh
+```
+
+Pull requests should state:
+- the observable behavior changed
+- the data grain and contract affected
+- fixture coverage
+- full-refresh and repeat-run behavior
+- deployment or migration implications
+
+Do not include personal identifiers, private infrastructure names, credentials, or generated runtime evidence in commits.

Dockerfile

@@ -0,0 +1,16 @@
+FROM python:3.12-slim
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    UV_COMPILE_BYTECODE=1
+
+WORKDIR /app
+
+RUN pip install --no-cache-dir uv
+COPY pyproject.toml README.md ./
+COPY src ./src
+COPY config ./config
+RUN uv sync --no-dev
+
+ENTRYPOINT ["uv", "run", "python", "-m", "ledger.cli"]
+CMD ["--help"]

Makefile

@@ -0,0 +1,17 @@
+.PHONY: install test validate fixture dbt
+
+install:
+	uv sync --group dev
+
+test:
+	uv run pytest
+
+validate:
+	bash scripts/validate.sh
+
+fixture:
+	cp -n config/ledger.example.yml config/ledger.yml || true
+	uv run python -m ledger.cli --config config/ledger.yml --fixture-dir tests/fixtures
+
+dbt:
+	uv run dbt build --project-dir dbt --profiles-dir dbt

README.md

@@ -0,0 +1,178 @@
+# Project Ledger
+
+Project Ledger is a reusable reference implementation for building a production-oriented SEC/XBRL data platform. It was extracted from a broader engineering implementation and generalized so another engineer can run, study, and extend the architecture without inheriting private infrastructure, personal identifiers, credentials, or project-specific history.
+
+The repository demonstrates how to move from deterministic local development toward managed cloud execution while preserving stable ingestion, storage, modeling, testing, and delivery boundaries.
+
+## What this project demonstrates
+
+- fixture-first API integration and deterministic CI
+- governed issuer selection and CIK validation
+- raw-source preservation before transformation
+- normalization of semi-structured XBRL observations
+- repeatable and idempotent pipeline behavior
+- dbt staging, analytical grain definition, and deduplication
+- strict typing, linting, unit tests, data tests, and smoke tests
+- Airflow orchestration boundaries
+- Docker packaging
+- parameterized GCP infrastructure as code
+- separation of code, configuration, identity, and secrets
+- extraction of a project-specific implementation into a reusable engineering template
+
+## Architecture
+
+```text
+SEC Company Facts API or fixtures
+            |
+            v
+Python ingestion and validation
+            |
+       +----+--------------------+
+       |                         |
+       v                         v
+raw JSON per issuer       normalized JSONL facts
+       |                         |
+       +------------+------------+
+                    v
+             dbt + DuckDB
+                    |
+                    v
+             analytical marts
+```
+
+Optional managed path:
+
+```text
+GitHub Actions -> container image -> Cloud Run Job -> GCS/BigQuery
+                                      ^
+                                      |
+                               Airflow/Composer
+```
+
+The local implementation is intentionally runnable with DuckDB and committed fixtures. The cloud layer is parameterized so storage, execution, IAM, and orchestration can be replaced without rewriting the core ingestion contract.
+
+See [Architecture](docs/ARCHITECTURE.md) for component responsibilities and extension points.
+
+## Engineering scope
+
+The reusable template includes:
+
+- Python SEC Company Facts ingestion
+- live and fixture-backed clients
+- raw JSON and normalized JSONL outputs
+- atomic local writes
+- dbt/DuckDB staging and mart models
+- Python and dbt tests
+- GitHub Actions validation
+- Docker packaging
+- an Airflow DAG example
+- Terraform for a GCP deployment skeleton
+
+The broader implementation from which this repository was extracted also explored managed GCP execution, BigQuery serving, Composer or Airflow orchestration, workload identity, monitoring, market-price enrichment, machine-learning extensions, language-model-assisted analysis, and earnings-event processing. Those later extensions are documented as integration directions rather than represented here as completed template features.
+
+## Key design decisions
+
+1. **Determinism before live integration.** The complete transformation path runs from committed fixtures.
+2. **Raw preservation before normalization.** Original payloads remain available for replay and debugging.
+3. **Configuration over identity.** Issuers, paths, cloud resources, and runtime contacts are externalized.
+4. **Stable local-to-cloud contracts.** Core ingestion and modeling boundaries remain consistent as infrastructure changes.
+5. **Validation across layers.** CI checks application code, pipeline behavior, analytical models, and infrastructure syntax.
+
+See [Engineering Decisions](docs/DECISIONS.md) for the rationale and trade-offs behind these choices.
+
+## Quick start
+
+### 1. Install prerequisites
+
+Install Python 3.12 and `uv`.
+
+### 2. Create local configuration
+
+```bash
+cp .env.example .env
+cp config/ledger.example.yml config/ledger.yml
+```
+
+Set a descriptive SEC user agent in `.env`:
+
+```text
+LEDGER_SEC_USER_AGENT=YourCompany Data Engineering your-team@example.com
+```
+
+### 3. Install and validate
+
+```bash
+uv sync --group dev
+uv run pytest
+bash scripts/validate.sh
+```
+
+### 4. Run from deterministic fixtures
+
+```bash
+uv run python -m ledger.cli \
+  --config config/ledger.yml \
+  --fixture-dir tests/fixtures
+```
+
+### 5. Run against the live SEC API
+
+Use live mode only after configuring a valid, monitored user agent:
+
+```bash
+uv run python -m ledger.cli --config config/ledger.yml --live
+```
+
+### 6. Build the local warehouse
+
+```bash
+uv run dbt build --project-dir dbt --profiles-dir dbt
+```
+
+## Validation
+
+The delivery gate verifies:
+
+- Ruff formatting and linting
+- strict mypy checks
+- Python unit and pipeline tests
+- a deterministic fixture ingestion smoke test
+- dbt model builds and data tests
+- Terraform formatting and validation
+
+A passing CI run establishes that the reference implementation is internally consistent. It does not by itself prove production scale, cloud deployment, or long-running operational reliability.
+
+## Documentation
+
+- [Case Study](docs/CASE_STUDY.md): origin, engineering objective, implemented scope, failure modes, evidence, limitations, and next steps
+- [Architecture](docs/ARCHITECTURE.md): design goals, components, and production extension points
+- [Engineering Decisions](docs/DECISIONS.md): major choices, rationale, and trade-offs
+- [Operations Guide](docs/OPERATIONS.md): operating modes, monitoring, failure handling, replay, security, and deployment checks
+- [Customization Guide](docs/CUSTOMIZATION.md): adapting the template to another organization or source system
+
+## Template boundaries
+
+This repository deliberately excludes:
+
+- personal sprint journals and career-development notes
+- generated dashboards, runtime logs, and private operational evidence
+- real cloud project IDs, bucket names, service accounts, personal emails, and secrets
+- issuer selections tied to a specific person or organization
+- claims that the template itself is a currently hosted production service
+
+These exclusions make the repository safe to reuse and keep its claims precise. The project should be evaluated as a production-oriented reference architecture and portfolio case study, not as proof of an active public deployment.
+
+## Limitations and next steps
+
+The current template does not include a complete managed warehouse loader, production alert routing, schema-migration automation, distributed backfill controls, or production cost benchmarks.
+
+The highest-value extensions are:
+
+1. implement GCS and BigQuery storage adapters
+2. add explicit schema contracts and migration checks
+3. add freshness, volume, error-rate, and cost observability
+4. add ephemeral cloud integration tests
+5. add bounded replay and backfill controls
+6. publish synthetic-data dashboards over the governed mart
+
+See [Customizing the template](docs/CUSTOMIZATION.md) before deploying it in a real environment.

config/ledger.example.yml

@@ -0,0 +1,20 @@
+project:
+  name: project-ledger
+
+sec:
+  # The CLI reads the actual value from LEDGER_SEC_USER_AGENT.
+  user_agent_env: LEDGER_SEC_USER_AGENT
+  issuers:
+    - symbol: AAPL
+      cik: "0000320193"
+    - symbol: MSFT
+      cik: "0000789019"
+
+storage:
+  output_dir_env: LEDGER_OUTPUT_DIR
+  default_output_dir: data
+
+pipeline:
+  request_timeout_seconds_env: LEDGER_REQUEST_TIMEOUT_SECONDS
+  max_retries_env: LEDGER_MAX_RETRIES
+  requests_per_second_env: LEDGER_REQUESTS_PER_SECOND

dags/ledger_daily_pipeline.py

@@ -0,0 +1,37 @@
+"""Example Airflow DAG for the reusable Project Ledger template."""
+
+from __future__ import annotations
+
+import os
+from datetime import datetime, timedelta
+
+from airflow import DAG
+from airflow.operators.bash import BashOperator
+
+PROJECT_DIR = os.environ.get("LEDGER_PROJECT_DIR", "/opt/project-ledger")
+CONFIG_PATH = os.environ.get("LEDGER_CONFIG_PATH", "config/ledger.yml")
+
+with DAG(
+    dag_id="ledger_daily_pipeline",
+    start_date=datetime(2025, 1, 1),
+    schedule="0 6 * * *",
+    catchup=False,
+    default_args={"retries": 2, "retry_delay": timedelta(minutes=5)},
+    tags=["ledger", "sec", "template"],
+) as dag:
+    ingest = BashOperator(
+        task_id="ingest_companyfacts",
+        bash_command=(
+            f"cd {PROJECT_DIR} && uv run python -m ledger.cli "
+            f"--config {CONFIG_PATH} --live"
+        ),
+    )
+
+    build = BashOperator(
+        task_id="build_dbt_models",
+        bash_command=(
+            f"cd {PROJECT_DIR} && uv run dbt build --project-dir dbt --profiles-dir dbt"
+        ),
+    )
+
+    ingest >> build

dbt/dbt_project.yml

@@ -0,0 +1,20 @@
+name: ledger
+version: "1.0.0"
+config-version: 2
+profile: ledger
+
+model-paths: ["models"]
+clean-targets: ["target", "dbt_packages"]
+
+vars:
+  normalized_facts_path: "data/normalized/financial_facts.jsonl"
+
+models:
+  ledger:
+    +persist_docs:
+      relation: true
+      columns: true
+    staging:
+      +materialized: view
+    marts:
+      +materialized: table