EXpiper

A lightweight, cloud-native pipeline for running ML/AI experiments on GCP — built with Vertex AI and Kubeflow Pipelines.

Architecture

_{Source: system-design.drawio — open in draw.io or the VS Code extension to edit.}

The pipeline ingests data from Cloud Storage, BigQuery, and external APIs, processes it through Kubeflow Pipeline components (validation, feature engineering, training, evaluation), logs everything to Vertex AI Experiments, and promotes passing models through a quality gate into the Model Registry for serving. A closed-loop feedback path re-triggers training when model monitoring detects drift.

Project Structure

expiper/
├── README.md
├── system-design.drawio         # Architecture diagram (open in draw.io)
├── data/                        # Local data / sample datasets
└── infra/
    └── terraform/               # Infrastructure as Code
        ├── main.tf              # Root — API enablement + module wiring
        ├── variables.tf         # Input variables
        ├── outputs.tf           # Provisioned resource references
        ├── versions.tf          # Provider & backend config
        ├── terraform.tfvars.example
        └── modules/
            ├── networking/      # VPC, subnet, Cloud NAT, firewall
            ├── storage/         # GCS buckets (raw, artifacts, pipeline root), BigQuery
            ├── iam/             # Service accounts + least-privilege bindings
            ├── gke/             # GKE Autopilot cluster (Kubeflow host)
            ├── vertex_ai/       # Metadata Store, Tensorboard, Endpoint
            ├── monitoring/      # Alert policies (latency, errors, crashloops, failures)
            └── cicd/            # Artifact Registry, Cloud Build triggers

Prerequisites

• Terraform >= 1.5
• gcloud CLI authenticated (gcloud auth application-default login)
• A GCP project with billing enabled
• A GCS bucket for Terraform remote state

Getting Started

1. Configure variables

cd infra/terraform
cp terraform.tfvars.example terraform.tfvars

Edit terraform.tfvars:

Variable	Description	Required
project_id	GCP project ID	yes
region	GCP region (default: us-central1)	no
environment	dev, staging, or prod	no
github_repo	GitHub repo in owner/repo format (enables Cloud Build triggers)	no
notification_email	Email for monitoring alerts	no

2. Initialise and apply

# Point at your state bucket
terraform init -backend-config="bucket=YOUR-TF-STATE-BUCKET" \
               -backend-config="prefix=expiper/state"

# Review the plan
terraform plan

# Apply
terraform apply

3. Install Kubeflow Pipelines on the GKE cluster

Terraform provisions the GKE Autopilot cluster; Kubeflow Pipelines is installed separately:

# Get cluster credentials
gcloud container clusters get-credentials $(terraform output -raw gke_cluster_name) \
  --region $(terraform output -raw region 2>/dev/null || echo "us-central1")

# Deploy KFP (adjust version as needed)
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=2.3.0"
kubectl wait crd/applications.app.k8s.io --for condition=established
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=2.3.0"

4. Run a pipeline

from google.cloud import aiplatform
from kfp.v2 import compiler
from kfp.v2.google.client import AIPlatformClient

aiplatform.init(
    project="my-gcp-project",
    location="us-central1",
    experiment="my-first-experiment",
)

# Compile your pipeline
compiler.Compiler().compile(
    pipeline_func=my_pipeline,
    package_path="pipeline.json",
)

# Submit to Vertex AI via Kubeflow
client = AIPlatformClient(project_id="my-gcp-project", region="us-central1")
client.create_run_from_job_spec("pipeline.json")

Terraform Modules

Module	Key Resources	Maps to Diagram
networking	VPC, subnet (pod/service secondary ranges), Cloud Router, Cloud NAT, firewall rules	Underlying network for all services
storage	3 GCS buckets (raw-data, artifacts, pipeline-root), BigQuery features dataset	Data Sources lane
iam	3 service accounts (pipeline, training, GKE node), bucket + project IAM, SA impersonation	Cross-cutting security layer
gke	GKE Autopilot cluster with private nodes and Workload Identity	Kubeflow Pipelines lane
vertex_ai	Metadata Store, Tensorboard, online prediction Endpoint	Vertex AI lane
monitoring	Email notification channel, 4 alert policies (endpoint latency p99, error rate, pod crashloops, pipeline failure)	Monitoring & CI/CD lane
cicd	Artifact Registry (Docker), 2 Cloud Build triggers (pipeline compilation, training image build)	Monitoring & CI/CD lane

Key Outputs

After terraform apply, these outputs are available:

terraform output gke_cluster_name         # GKE cluster to host Kubeflow
terraform output raw_data_bucket          # gs://expiper-dev-raw-data
terraform output artifacts_bucket         # gs://expiper-dev-artifacts
terraform output pipeline_root_bucket     # gs://expiper-dev-pipeline-root
terraform output bigquery_dataset         # expiper_dev_features
terraform output vertex_endpoint_id       # Vertex AI serving endpoint
terraform output artifact_registry_repo   # Container image registry
terraform output pipeline_service_account # SA for pipeline execution

Environments

Resources are prefixed with expiper-{environment} so dev, staging, and prod can coexist in the same project (or be split across projects):

• dev — force_destroy enabled on buckets, 90-day lifecycle cleanup, 30-day pipeline root TTL
• staging — same as dev (safe to tear down)
• prod — force_destroy disabled, no automatic object deletion

License

MIT