Stage 3: rewrite as Lakeflow Spark Declarative Pipelines

[p17b]

Third of three stages, stacked on medallion_whl.

Replaces the ingest / transform / aggregate jobs with a single pipeline that
declares three datasets. Execution order is derived from table references in the
code; the runtime owns checkpoints, triggers, output modes, and the writes.

The transformation logic is unchanged. Both branches import the same
transform_silver and build_gold from the wheel, covered by the same 43 unit
tests. Only orchestration differs — which is what makes the two comparable, and
what the wheel from stage 2 bought us.

What disappears

Concern	Job-based	This branch
Execution order	Three jobs run in sequence	Derived from table references
Checkpoints	Explicit path per stream	Managed by the pipeline
Triggers / output modes	Per write	Implicit in the dataset type
Writes	saveAsTable + overwriteSchema	The decorator's return value
Quality reporting	A job asserting counts afterwards	Per-run metrics in the event log

Three job YAMLs and three notebooks deleted; three transformation files and one
pipeline YAML added.

Dataset types

Bronze and silver are streaming tables — append-only, incremental.

Gold is a materialized view, deliberately. Streaming tables never revisit
rows they have already emitted, so a daily aggregate built as one would go stale
the moment late or corrected data reached silver. It reads silver with
spark.read, not readStream: a streaming read would impose watermark and state
constraints for no benefit, since the anomaly baseline spans the whole history
rather than a window.

Expectations

Every one is expect_all — warn and keep, never _or_drop or _or_fail. That
is the declarative form of a decision made on stage 1: invalid readings are
flagged, not discarded. Dropping them hides sensor faults from the people who
fix them; failing the update would take the pipeline down over a few bad rows in
an otherwise good delivery.

Silver's conditions reference the boolean columns the transform already produces
rather than restating the rules in SQL — one source of truth, so the rule cannot
drift between column and expectation.

The metrics reproduce the fixture manifest independently:

Expectation	Failed	Manifest expects
turbine_id_in_expected_group	27	27
power_output_present_and_non_negative	9	9
wind_speed_present_and_non_negative	10	10
wind_direction_within_compass_range	8	8
source_file_maps_to_known_group	20	20

Summing to the manifest's 54 invalid readings. Two mechanisms, completely
different routes, same numbers. Gold's three expectations pass 465/465.

Results

Identical to the job branches: silver 11,169 / 54 invalid / 20 UNKNOWN / 465
turbine-days / 20 anomalies. The stage 1 validation job passes unchanged
against pipeline-produced tables — a fair cross-check rather than a rewritten
test.

Migration constraint

The first run failed:

Could not materialize `...`.`turbine_raw` because a MANAGED table already
exists with that name.

A pipeline cannot adopt an existing managed table. The job branches had
written plain Delta tables at those names, and they had to be dropped so the
pipeline could own them. Harmless for regenerable fixtures; on a real migration
it means a cutover plan — write to new names and swap, or accept a rebuild
window.

Trade-offs

For: far less orchestration code; dependency graph derived rather than
maintained; quality is a queryable per-run artifact; table types force an
explicit answer to "recompute or append?".

Against: the pipeline owns its tables, so migration is a cutover; debugging
moves from reading a notebook top-to-bottom to reading a graph plus an event
log; and it is Databricks-specific in a way that plain PySpark plus a scheduler
is not.

Neither is strictly better. Declarative wins when the shape is stable and
quality reporting matters; the job version is easier to reason about while the
logic is still moving, and it ports.

See docs/declarative_pipeline.md for the full write-up.