Ship evaluated T-24h delay prediction model

.gitignore

@@ -13,5 +13,3 @@ build/
 .vercel
 data/raw/
 data/processed/
-artifacts/models/
-artifacts/reports/

README.md

@@ -19,23 +19,24 @@ FlightOps AI is a tested vertical slice of an operations platform: it accepts re
 | Backend engineering | Typed FastAPI routes, validation, layered service/repository design, OpenAPI docs |
 | Data engineering | Normalized UTC events, indexed SQLite persistence, latest-event queries, aggregate summaries |
 | Reliability | Idempotent writes, bounded inputs, typed 404s, health endpoint, retry-safe behavior |
-| Explainable AI | Versioned baseline risk score with per-factor contributions—clearly identified as rules, not a trained model |
+| Applied ML | Evaluated T-24h delay model trained on 90,000 official BTS records, portable JSON inference, calibration, model card, and error slices |
 | Observability | Prometheus-compatible request and ingestion counters |
 | Cloud delivery | Non-root Docker image, Compose setup, Vercel entrypoint, automated GitHub Actions tests |
 
 ### At a glance
 
-- **5 operational endpoints** for ingestion, risk, summaries, health, and metrics
+- **7 operational endpoints** for ingestion, risk, trained prediction, model evidence, summaries, health, and metrics
 - **4 scored risk signals** plus an explicit cancellation adjustment
 - **Idempotency by event ID**, so producer retries do not duplicate records
-- **6 automated tests** covering API behavior, validation, scoring, summaries, and metrics
+- **8 automated tests** covering API behavior, validation, rules, trained inference, summaries, and metrics
+- **90,000 official BTS records** in a checksum-tracked, reproducible chronological evaluation
 - **One-command local start** with Docker Compose
 
 ## Two-minute recruiter tour
 
-1. Read the [risk engine](app/risk.py) to see transparent feature contributions and model versioning.
-2. Read the [API test](tests/test_api.py) to see idempotency, validation, risk, summaries, and metrics working together.
-3. Read the [architecture notes](docs/architecture.md) for current tradeoffs and the production evolution path.
+1. Read the [model card](docs/ml/model-card.md) for the chronological evaluation, calibration, operating threshold, error slices, and limitations.
+2. Read the [portable inference code](app/ml_model.py) to see how inspectable JSON coefficients serve predictions without unsafe pickle loading.
+3. Read the [API tests](tests/test_api.py) to see idempotency, validation, rules, trained inference, summaries, and metrics working together.
 4. Open the [live API explorer](https://flightops-ai-mu.vercel.app/docs) to exercise every endpoint without local setup.
 
 ## Architecture
@@ -46,9 +47,11 @@ flowchart LR
     A --> S["Operations service"]
     S --> R["SQLite event repository"]
     S --> E["Explainable risk engine"]
+    A --> P["T-24h schedule model"]
     A --> M["Service metrics"]
     R --> Q["Operational summary"]
     E --> O["Risk score + factor explanations"]
+    P --> O2["Delay probability + model version"]
 ```
 
 The repository boundary keeps persistence replaceable. SQLite makes the demo reproducible with zero infrastructure; PostgreSQL is the planned durable production store. See [docs/architecture.md](docs/architecture.md) and [ADR 0001](docs/adr/0001-start-with-a-tested-vertical-slice.md).
@@ -110,6 +113,15 @@ curl http://127.0.0.1:8000/v1/operations/summary
 curl http://127.0.0.1:8000/metrics
 ```
 
+Request a schedule-only trained-model prediction:
+
+```bash
+curl -X POST http://127.0.0.1:8000/v1/predictions/delay \
+  -H 'content-type: application/json' \
+  -d '{"flight_date":"2026-08-10","reporting_airline":"DL","origin":"IAD","destination":"ATL","crs_departure_time":815,"crs_elapsed_time":115,"distance":534,"distance_group":3}'
+curl http://127.0.0.1:8000/v1/models/delay/metadata
+```
+
 Posting the same `event_id` again returns `200` with `"created": false`; the database keeps one event. That makes upstream retries safe.
 
 ## API surface
@@ -120,8 +132,25 @@ Posting the same `event_id` again returns `200` with `"created": false`; the dat
 | `POST` | `/v1/events` | Validate and idempotently store a flight event |
 | `GET` | `/v1/flights/{flight_id}/risk` | Score the latest event and explain each contribution |
 | `GET` | `/v1/operations/summary` | Return fleet-level event and high-risk counts |
+| `POST` | `/v1/predictions/delay` | Predict T-24h arrival-delay probability from schedule-only fields |
+| `GET` | `/v1/models/delay/metadata` | Expose model version, threshold, evaluation metrics, lineage, and limitations |
 | `GET` | `/metrics` | Export Prometheus-compatible counters |
 
+## Trained-model evidence
+
+The deployed `bts-schedule-logistic-v1` model was trained on a deterministic 90,000-row sample from official BTS monthly files. Training uses Jan-Dec 2024, threshold selection uses Jan-Mar 2025, and the untouched test period is Apr-Jun 2025.
+
+| Test metric | Result |
+| --- | ---: |
+| ROC-AUC | 0.640 |
+| PR-AUC | 0.335 |
+| Precision | 0.304 |
+| Recall | 0.748 |
+| F1 | 0.432 |
+| Brier score | 0.176 |
+
+These are retrospective public-data results, not production accuracy claims. Reproduce the download and training with the [ML runbook](ml/README.md), then inspect the [model card](docs/ml/model-card.md), [data manifest](artifacts/data/bts-sample-manifest.json), and [error slices](artifacts/reports/error-slices.csv).
+
 ## Quality checks
 
 ```bash
@@ -132,7 +161,7 @@ GitHub Actions installs the project in a clean Python 3.12 runner and executes t
 
 ## Decisions and honest limitations
 
-- The current score is a deterministic, versioned baseline—not a trained ML model. This makes the behavior testable and gives a future model a measurable benchmark.
+- The event-risk endpoint remains a deterministic rules fallback; the separate schedule-prediction endpoint uses the evaluated trained model and never mixes post-departure fields into T-24h inference.
 - SQLite is appropriate for a local demonstration. On Vercel it uses ephemeral `/tmp` storage, so the public demo is not a durable system of record.
 - Metrics are process-local. A production deployment would export them to managed observability infrastructure.
 - Authentication, rate limiting, a durable event queue, and infrastructure-as-code belong in a later production milestone.
@@ -147,13 +176,13 @@ GitHub Actions installs the project in a clean Python 3.12 runner and executes t
 - [x] Docker packaging and CI
 - [ ] PostgreSQL migrations and query-performance evidence
 - [ ] React + TypeScript operations dashboard
-- [ ] Versioned delay-prediction model with an evaluation report
+- [x] Versioned delay-prediction model with an evaluation report
 - [ ] Queue, retries, caching, and failure-injection tests
 - [ ] Evaluated incident/runbook assistant with citations
 - [ ] AWS deployment with Terraform and a cost estimate
 - [ ] Load-test report, SLO, and incident write-up
 
-The trained-model milestone is specified in [Issue #2](https://github.com/mitulpatel123/flightops-ai/issues/2) and begins with a leakage-resistant [data contract](docs/ml/data-contract.md).
+The trained-model milestone is specified in [Issue #2](https://github.com/mitulpatel123/flightops-ai/issues/2) and governed by a leakage-resistant [data contract](docs/ml/data-contract.md).
 
 ## Project integrity
 

app/main.py

@@ -7,7 +7,15 @@
 from fastapi.responses import RedirectResponse
 
 from app.db import EventRepository
-from app.models import DelayRisk, FlightEvent, IngestResponse, OperationsSummary
+from app.ml_model import get_delay_model
+from app.models import (
+    DelayRisk,
+    FlightEvent,
+    IngestResponse,
+    ModelPrediction,
+    OperationsSummary,
+    ScheduledFlight,
+)
 from app.service import FlightNotFoundError, OperationsService
 
 
@@ -41,8 +49,8 @@ async def lifespan(_: FastAPI):
 
     api = FastAPI(
         title="FlightOps AI",
-        version="0.1.0",
-        description="Airline operations event ingestion and explainable delay-risk API.",
+        version="0.2.0",
+        description="Airline operations ingestion, explainable risk, and evaluated T-24h delay prediction API.",
         lifespan=lifespan,
     )
 
@@ -52,7 +60,7 @@ def index() -> RedirectResponse:
 
     @api.get("/health")
     def health() -> dict[str, str]:
-        return {"status": "ok", "version": "0.1.0"}
+        return {"status": "ok", "version": "0.2.0"}
 
     @api.post("/v1/events", response_model=IngestResponse, status_code=status.HTTP_201_CREATED)
     def ingest_event(event: FlightEvent, response: Response) -> IngestResponse:
@@ -77,6 +85,15 @@ def get_summary() -> OperationsSummary:
         metrics.increment("summary_requests")
         return service.summary()
 
+    @api.post("/v1/predictions/delay", response_model=ModelPrediction)
+    def predict_delay(flight: ScheduledFlight) -> ModelPrediction:
+        metrics.increment("delay_predictions")
+        return get_delay_model().predict(flight)
+
+    @api.get("/v1/models/delay/metadata")
+    def delay_model_metadata() -> dict:
+        return get_delay_model().metadata()
+
     @api.get("/metrics", response_class=Response)
     def get_metrics() -> Response:
         return Response(metrics.render(), media_type="text/plain; version=0.0.4")

app/ml_model.py

@@ -0,0 +1,86 @@
+from __future__ import annotations
+
+import json
+import math
+from functools import lru_cache
+from pathlib import Path
+from typing import Any
+
+from app.models import ModelPrediction, ScheduledFlight
+
+
+MODEL_PATH = Path(__file__).resolve().parent.parent / "artifacts" / "models" / "delay-logistic-v1.json"
+
+
+class PortableLogisticModel:
+    """Small, inspectable JSON logistic model with no pickle execution risk."""
+
+    def __init__(self, payload: dict[str, Any]) -> None:
+        self.payload = payload
+        self.threshold = float(payload["threshold"])
+        self.version = str(payload["model_version"])
+
+    @classmethod
+    def from_path(cls, path: Path = MODEL_PATH) -> "PortableLogisticModel":
+        return cls(json.loads(path.read_text(encoding="utf-8")))
+
+    def predict(self, flight: ScheduledFlight) -> ModelPrediction:
+        values = _model_values(flight)
+        linear = float(self.payload["intercept"])
+
+        numeric = self.payload["numeric"]
+        for name, value in values.items():
+            if name not in numeric:
+                continue
+            spec = numeric[name]
+            scale = float(spec["scale"]) or 1.0
+            standardized = (float(value) - float(spec["mean"])) / scale
+            linear += standardized * float(spec["coefficient"])
+
+        categorical = self.payload["categorical"]
+        for name in ("Reporting_Airline", "Origin", "Dest"):
+            category = str(values[name])
+            linear += float(categorical[name].get(category, 0.0))
+
+        probability = 1.0 / (1.0 + math.exp(-max(-35.0, min(35.0, linear))))
+        probability = round(probability, 4)
+        return ModelPrediction(
+            probability=probability,
+            predicted_delayed=probability >= self.threshold,
+            threshold=self.threshold,
+            model_version=self.version,
+            caveat="Retrospective BTS schedule-data estimate; not a live operational guarantee.",
+        )
+
+    def metadata(self) -> dict[str, Any]:
+        return {
+            "model_version": self.version,
+            "prediction_time": self.payload["prediction_time"],
+            "threshold": self.threshold,
+            "trained_at": self.payload["trained_at"],
+            "data_manifest_sha256": self.payload["data_manifest_sha256"],
+            "test_metrics": self.payload["test_metrics"],
+            "limitations": self.payload["limitations"],
+        }
+
+
+def _model_values(flight: ScheduledFlight) -> dict[str, float | str]:
+    hour = flight.crs_departure_time // 100
+    angle = 2.0 * math.pi * hour / 24.0
+    return {
+        "Month": float(flight.flight_date.month),
+        "DayOfWeek": float(flight.flight_date.isoweekday()),
+        "DepHourSin": math.sin(angle),
+        "DepHourCos": math.cos(angle),
+        "CRSElapsedTime": float(flight.crs_elapsed_time),
+        "Distance": float(flight.distance),
+        "DistanceGroup": float(flight.distance_group),
+        "Reporting_Airline": flight.reporting_airline,
+        "Origin": flight.origin,
+        "Dest": flight.destination,
+    }
+
+
+@lru_cache(maxsize=1)
+def get_delay_model() -> PortableLogisticModel:
+    return PortableLogisticModel.from_path()

app/models.py

@@ -1,4 +1,4 @@
-from datetime import datetime, timezone
+from datetime import date, datetime, timezone
 from typing import Literal
 
 from pydantic import BaseModel, ConfigDict, Field, field_validator
@@ -61,3 +61,39 @@ class OperationsSummary(BaseModel):
     delayed_events: int
     cancelled_events: int
     high_risk_flights: int
+
+
+class ScheduledFlight(BaseModel):
+    """Schedule-only fields available at least 24 hours before departure."""
+
+    model_config = ConfigDict(str_strip_whitespace=True)
+
+    flight_date: date
+    reporting_airline: str = Field(min_length=2, max_length=8)
+    origin: str = Field(min_length=3, max_length=4)
+    destination: str = Field(min_length=3, max_length=4)
+    crs_departure_time: int = Field(ge=0, le=2359)
+    crs_elapsed_time: float = Field(gt=0, le=1440)
+    distance: float = Field(gt=0, le=12000)
+    distance_group: int = Field(ge=1, le=25)
+
+    @field_validator("reporting_airline", "origin", "destination")
+    @classmethod
+    def normalize_code(cls, value: str) -> str:
+        return value.upper()
+
+    @field_validator("crs_departure_time")
+    @classmethod
+    def validate_clock_time(cls, value: int) -> int:
+        if value % 100 >= 60:
+            raise ValueError("crs_departure_time must be HHMM")
+        return value
+
+
+class ModelPrediction(BaseModel):
+    probability: float = Field(ge=0.0, le=1.0)
+    predicted_delayed: bool
+    threshold: float = Field(ge=0.0, le=1.0)
+    model_version: str
+    prediction_time: str = "T-24h"
+    caveat: str