News Tracker Template

Reusable starter repo for building a domain-specific news or newsletter tracker. It keeps the production plumbing from the education newsletter system, but removes the original project history, trained model artefacts, live source roster, private notes, and client-specific governance write-ups.

Use this as the base for a new tracker when you want the same shape of system: source ingestion, Supabase storage, classification, ranking, summary enrichment, monitoring jobs, and a curator dashboard.

What is included

• Python scraping and ingestion modules under src/scraping/.
• Sentence-transformer embedding and scikit-learn classifier training under src/classify/.
• FastAPI model serving under src/serving/.
• Streamlit curator dashboard under dashboard/.
• Supabase migrations under migrations/.
• Manual GitHub Actions workflows under .github/workflows/.
• Reusable governance templates under docs/templates/.

What is intentionally not included

• No trained model files.
• No original labelled training data.
• No real source roster or Google Alert feed URLs.
• No previous project history.
• No client/stakeholder notes.

Create a new tracker from this template

1. Create a new GitHub repository from this template.

2. Edit config/domain.yml with the tracker name, labels, colours, relevance terms, and table names.

3. Copy config/sources.example.yml to src/scraping/sources.yml and replace the example sources.

4. Set up a fresh Supabase project and run the migrations in migrations/.

5. Add labelled training data as data/modelling/train.csv and data/modelling/val.csv.

6. Train a model:

   python src/pipeline.py --training

7. Deploy the FastAPI classifier and set the app secrets.

8. Turn on the scheduled GitHub Actions once secrets and sources are ready.

Required labelled data schema

The training scripts expect:

• data/modelling/train.csv
• data/modelling/val.csv

Each file should contain at least:

• text_clean: text used for classification.
• target: class label matching one of the label keys in config/domain.yml.
• title and url are optional but useful for evaluation outputs.

Local setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt
python -m pytest tests/ -q

Run the dashboard locally:

streamlit run dashboard/app.py

Run the API locally after training a model:

uvicorn src.serving.api:app --host 0.0.0.0 --port 8000

Secrets

Start from .env.example. For a deployed tracker, configure secrets in the hosting provider and in GitHub Actions rather than committing local .env files.

Typical runtime secrets:

• SUPABASE_URL
• SUPABASE_SERVICE_KEY
• CLASSIFIER_API_URL
• CLASSIFIER_API_KEY
• OPENAI_API_KEY
• ANTHROPIC_API_KEY if you use Anthropic fallback enrichment

Automation defaults

The included workflows are templates. Keep them manual until a new tracker has:

• a fresh Supabase project,
• a deployed classifier,
• configured secrets,
• reviewed schedules,
• curated source and relevance config.

Governance docs

Use the files in docs/templates/ as starting points for a new tracker:

• model card,
• dataset datasheet,
• threat model,
• deployment checklist.