From 8ddc9caac9350749e7838f8f81e46442a5eb0820 Mon Sep 17 00:00:00 2001 From: Subash Pandey Date: Wed, 22 Jul 2026 19:06:17 +0545 Subject: [PATCH 1/2] Update Terraform and documentation for hobby environment - Revised the README to clarify the process for accessing the VPS and managing the firewall settings, emphasizing the use of `admin_cidrs` for SSH and kubectl access. - Introduced a new script, `fetch-hobby-kubeconfig.sh`, to automate the retrieval and configuration of the kubeconfig file after deploying the k3s cluster. - Updated Terraform configurations to replace `ssh_source_cidrs` with `admin_cidrs`, enhancing security by restricting access to specified IPs. - Enhanced cloud-init template to automatically set the TLS SAN to the VPS public IP, ensuring secure communication with the Kubernetes API. - Added output hints in Terraform to guide users on fetching the kubeconfig after deployment. These changes improve the security and usability of the hobby environment setup. --- README.md | 24 ++++---- gitops/README.md | 12 +++- .../environments/hobby/cloud-init.yaml.tpl | 20 ++++--- .../terraform/environments/hobby/main.tf | 44 +++++++------- .../terraform/environments/hobby/outputs.tf | 9 ++- .../hobby/terraform.tfvars.example | 16 +++-- .../terraform/environments/hobby/variables.tf | 5 ++ .../terraform/modules/firewall/main.tf | 51 +++++++++------- .../terraform/modules/firewall/variables.tf | 6 +- scripts/fetch-hobby-kubeconfig.sh | 58 +++++++++++++++++++ 10 files changed, 167 insertions(+), 78 deletions(-) create mode 100644 scripts/fetch-hobby-kubeconfig.sh diff --git a/README.md b/README.md index be9f707..bb8d8ca 100644 --- a/README.md +++ b/README.md @@ -91,11 +91,11 @@ terraform destroy ### Opening again (same machine, VPS still exists) 1. Hetzner console → power **ON** if you powered off. -2. Confirm your public IP still matches the firewall allow-list in `main.tf` (`ssh_source_cidrs`). If your ISP changed it, update the CIDR and `terraform apply` before SSH will work. +2. Confirm your public IP still matches `admin_cidrs` in `terraform.tfvars`. If your ISP changed it, update and `terraform apply` before SSH/kubectl will work (`curl -4 ifconfig.me`). 3. Point kubectl at the hobby cluster and verify: ```bash -export KUBECONFIG=~/.kube/hobby.yaml +./scripts/fetch-hobby-kubeconfig.sh # or: export KUBECONFIG=~/.kube/hobby.yaml kubectl get nodes kubectl -n argocd get pods kubectl -n argocd port-forward svc/argocd-server 8080:443 @@ -112,13 +112,8 @@ You need: this repo, your SSH **private** key (same key Terraform registered), ` git clone https://github.com/notsubash/cloud-native-AI-platform.git cd cloud-native-AI-platform -# Laptop only — do not run scp while SSH'd into the VPS -mkdir -p ~/.kube -scp -i ~/.ssh/id_ed25519 root@:/etc/rancher/k3s/k3s.yaml ~/.kube/hobby.yaml -# Edit hobby.yaml: replace 127.0.0.1 with - -export KUBECONFIG=~/.kube/hobby.yaml -kubectl get nodes +./scripts/fetch-hobby-kubeconfig.sh +# needs terraform state (or set IP manually — see script / terraform output) # Argo UI kubectl -n argocd port-forward svc/argocd-server 8080:443 @@ -130,23 +125,26 @@ If you lack Terraform state on the new machine, manage the existing server from ```bash cd infrastructure/terraform/environments/hobby -cp terraform.tfvars.example terraform.tfvars # set ssh_public_key_path +cp terraform.tfvars.example terraform.tfvars +# set ssh_public_key_path + admin_cidrs = ["YOUR.IP/32"] (curl -4 ifconfig.me) export HCLOUD_TOKEN=... -# Update ssh_source_cidrs in main.tf to YOUR current public IP/32 terraform init && terraform plan && terraform apply -# Wait ~2–3 min for cloud-init/k3s, then copy kubeconfig (see A) +# from repo root — waits for k3s, writes ~/.kube/hobby.yaml with public IP +./scripts/fetch-hobby-kubeconfig.sh + # Install Argo CD (server-side apply), create ghcr-pull secret, apply gitops/applications/api.yaml # Full sequence: gitops/README.md ``` +Firewall allows **SSH (22)** and **kubectl (6443)** only from `admin_cidrs`. No SSH tunnel needed when your IP matches. ### Cost hygiene checklist - [ ] Before leaving for the day: power off **or** destroy (know which you chose). - [ ] If paused > 7 days: destroy, don’t leave an idle ON server. - [ ] After destroy: confirm Hetzner console shows **no** `cnai-hobby` server. - [ ] Never commit `HCLOUD_TOKEN`, `terraform.tfvars`, `*.tfstate`, or the GHCR PAT. -- [ ] Firewall is IP-locked — a new network/café IP blocks SSH until you update `ssh_source_cidrs`. +- [ ] Firewall is IP-locked — a new network/café IP blocks SSH/kubectl until you update `admin_cidrs` in `terraform.tfvars` and apply. ## Local development (Compose) diff --git a/gitops/README.md b/gitops/README.md index de8e2ce..265cf44 100644 --- a/gitops/README.md +++ b/gitops/README.md @@ -18,14 +18,19 @@ GitHub (branch in Application) → Argo CD → Helm (helm/api + values-hobby.yam ## Prerequisites (cluster already up) -- Hobby VPS with k3s (Terraform + cloud-init) -- `KUBECONFIG` pointing at the VPS (e.g. `~/.kube/hobby.yaml` with `127.0.0.1` replaced by the public IP) +- Hobby VPS with k3s (Terraform + cloud-init; API cert includes public IP via `--tls-san`) +- Firewall allows your IP on **22** and **6443** (`admin_cidrs` in `terraform.tfvars`) +- Laptop kubeconfig from `./scripts/fetch-hobby-kubeconfig.sh` → `export KUBECONFIG=~/.kube/hobby.yaml` - Argo CD installed in namespace `argocd` - Namespace `ai-platform` and docker-registry secret `ghcr-pull` (GitHub PAT with `read:packages` — not an image tag) ## Bootstrap Argo CD (once per cluster) ```bash +# From repo root, after terraform apply: +./scripts/fetch-hobby-kubeconfig.sh +export KUBECONFIG=~/.kube/hobby.yaml + kubectl create namespace argocd kubectl apply --server-side -n argocd \ -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml @@ -83,7 +88,8 @@ Automated sync / self-heal can be enabled later in `api.yaml` (`syncPolicy.autom | `values-hobby.yaml: no such file` | File not on the tracked revision / not committed | | Sync OK but `ImagePullBackOff` | Bad `ghcr-pull` secret (need PAT with `read:packages`) | | Application Healthy but empty namespace | Manual sync not run yet | -| SSH to VPS hangs | Your public IP changed; update `ssh_source_cidrs` in Terraform | +| SSH to VPS hangs / kubectl :6443 times out | Your public IP changed; update `admin_cidrs` in `terraform.tfvars` + `terraform apply` | +| Host key verification failed after recreate | Normal after destroy/recreate; `./scripts/fetch-hobby-kubeconfig.sh` clears the old key | ## Cost note diff --git a/infrastructure/terraform/environments/hobby/cloud-init.yaml.tpl b/infrastructure/terraform/environments/hobby/cloud-init.yaml.tpl index a40a9c7..9307de5 100644 --- a/infrastructure/terraform/environments/hobby/cloud-init.yaml.tpl +++ b/infrastructure/terraform/environments/hobby/cloud-init.yaml.tpl @@ -1,11 +1,10 @@ #cloud-config # ============================================================================= -# Minimal cloud-init — install single-node k3s -# ============================================================================= -# After terraform apply: -# ssh root@ -# cat /etc/rancher/k3s/k3s.yaml # this is your kubeconfig -# On your laptop: replace 127.0.0.1 with the public IP. +# First-boot: install single-node k3s with TLS SAN = this VPS public IP +# so kubectl from your laptop (https://:6443) trusts the API cert. +# +# Public IP comes from Hetzner link-local metadata (no Terraform cycle needed). +# After apply: ./scripts/fetch-hobby-kubeconfig.sh # ============================================================================= package_update: true @@ -14,6 +13,9 @@ packages: - curl runcmd: - # Install k3s as a single-node cluster (default Traefik + servicelb). - # DISABLE traefik later if you prefer nginx — for Phase 6, leave defaults. - - curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644 \ No newline at end of file + - | + set -e + PUBLIC_IP=$(curl -fsSL http://169.254.169.254/hetzner/v1/metadata/public-ipv4) + curl -sfL https://get.k3s.io | sh -s - \ + --write-kubeconfig-mode 644 \ + --tls-san "$PUBLIC_IP" diff --git a/infrastructure/terraform/environments/hobby/main.tf b/infrastructure/terraform/environments/hobby/main.tf index 9388a81..aea9f30 100644 --- a/infrastructure/terraform/environments/hobby/main.tf +++ b/infrastructure/terraform/environments/hobby/main.tf @@ -1,38 +1,38 @@ # ============================================================================= -# templatefile() reads the .tpl and substitutes ${...} variables. -# Here we keep the template simple (no vars) — empty map {}. +# Hobby env: one VPS + firewall. cloud-init installs k3s with --tls-san so +# kubectl can talk to the public IP on :6443 (firewall-locked to admin_cidrs). # -# MONEY MOMENT: the next `terraform apply` creates a billable Hetzner server. -# ========================================================================== +# MONEY: terraform apply starts Hetzner billing; destroy stops it. +# ============================================================================= locals { - ssh_public_key = trimspace(file(pathexpand(var.ssh_public_key_path))) + ssh_public_key = trimspace(file(pathexpand(var.ssh_public_key_path))) } module "firewall" { - source = "../../modules/firewall" + source = "../../modules/firewall" - name = "${var.server_name}-fw" - ssh_source_cidrs = ["103.129.135.175/32"] + name = "${var.server_name}-fw" + ssh_source_cidrs = var.admin_cidrs } module "server" { - source = "../../modules/server" + source = "../../modules/server" - name = var.server_name - server_type = var.server_type - location = var.location - ssh_public_key = local.ssh_public_key - firewall_id = module.firewall.id + name = var.server_name + server_type = var.server_type + location = var.location + ssh_public_key = local.ssh_public_key + firewall_id = module.firewall.id - user_data = templatefile("${path.module}/cloud-init.yaml.tpl", {}) + user_data = templatefile("${path.module}/cloud-init.yaml.tpl", {}) } # module "dns" { -# source = "../../modules/dns" -# count = var.enable_dns ? 1 : 0 -# -# zone_name = var.domain -# record_name = var.subdomain -# ipv4_address = module.server.ipv4_address -#} \ No newline at end of file +# source = "../../modules/dns" +# count = var.enable_dns ? 1 : 0 +# +# zone_name = var.domain +# record_name = var.subdomain +# ipv4_address = module.server.ipv4_address +# } diff --git a/infrastructure/terraform/environments/hobby/outputs.tf b/infrastructure/terraform/environments/hobby/outputs.tf index 8cb59b9..81f732f 100644 --- a/infrastructure/terraform/environments/hobby/outputs.tf +++ b/infrastructure/terraform/environments/hobby/outputs.tf @@ -1,5 +1,5 @@ output "public_ip" { - description = "VPS IPv4 — use for SSH and later DNS" + description = "VPS IPv4 — SSH and kubectl API" value = module.server.ipv4_address } @@ -8,7 +8,12 @@ output "ssh_command" { value = "ssh root@${module.server.ipv4_address}" } +output "kubeconfig_hint" { + description = "After apply + ~2 min for cloud-init, run this from repo root" + value = "./scripts/fetch-hobby-kubeconfig.sh" +} + output "dns_name" { description = "FQDN if DNS enabled" value = null -} \ No newline at end of file +} diff --git a/infrastructure/terraform/environments/hobby/terraform.tfvars.example b/infrastructure/terraform/environments/hobby/terraform.tfvars.example index 62c7423..130b4a6 100644 --- a/infrastructure/terraform/environments/hobby/terraform.tfvars.example +++ b/infrastructure/terraform/environments/hobby/terraform.tfvars.example @@ -1,9 +1,17 @@ # Copy to terraform.tfvars (gitignored) and fill in. +# +# Your current public IP (must match the network you kubectl/SSH from): +# curl -4 ifconfig.me +# If your ISP changes IP, update admin_cidrs and terraform apply. server_name = "cnai-hobby" -server_type = "cx22" +server_type = "cx23" location = "nbg1" ssh_public_key_path = "~/.ssh/id_ed25519.pub" -enable_dns = false -# domain = "example.com" -# subdomain = "lab" \ No newline at end of file + +# SSH + Kubernetes API (6443) — lock to your IP only +admin_cidrs = ["203.0.113.10/32"] + +enable_dns = false +# domain = "example.com" +# subdomain = "lab" diff --git a/infrastructure/terraform/environments/hobby/variables.tf b/infrastructure/terraform/environments/hobby/variables.tf index b3a338c..cd4368b 100644 --- a/infrastructure/terraform/environments/hobby/variables.tf +++ b/infrastructure/terraform/environments/hobby/variables.tf @@ -23,6 +23,11 @@ variable "ssh_public_key_path" { description = "Path to our public key file (contents get uploaded to Hetzner)" } +variable "admin_cidrs" { + type = list(string) + description = "Your public IP(s) as /32 — SSH (22) and kubectl API (6443). Find IP: curl -4 ifconfig.me" +} + variable "enable_dns" { type = bool description = "Wire Cloudflare DNS? Keep false until later" diff --git a/infrastructure/terraform/modules/firewall/main.tf b/infrastructure/terraform/modules/firewall/main.tf index aed2925..5e46859 100644 --- a/infrastructure/terraform/modules/firewall/main.tf +++ b/infrastructure/terraform/modules/firewall/main.tf @@ -1,30 +1,37 @@ # Why: Hetzner firewall attaches to the server; rules are declarative. +# admin_cidrs (ssh_source_cidrs): home IP/32 for SSH + kubectl API — not the world. resource "hcloud_firewall" "this" { - name = var.name + name = var.name - # SSH - rule { - direction = "in" - protocol = "tcp" - port = "22" - source_ips = var.ssh_source_cidrs - } + # SSH + rule { + direction = "in" + protocol = "tcp" + port = "22" + source_ips = var.ssh_source_cidrs + } - # HTTP / HTTPS (ingress later) - rule { - direction = "in" - protocol = "tcp" - port = "80" - source_ips = ["0.0.0.0/0", "::/0"] - } + # Kubernetes API (kubectl from laptop — same CIDRs as SSH) + rule { + direction = "in" + protocol = "tcp" + port = "6443" + source_ips = var.ssh_source_cidrs + } - rule { - direction = "in" - protocol = "tcp" - port = "443" - source_ips = ["0.0.0.0/0", "::/0"] - } + # HTTP / HTTPS (ingress later) + rule { + direction = "in" + protocol = "tcp" + port = "80" + source_ips = ["0.0.0.0/0", "::/0"] + } - # TODO (optional): allow ICMP for ping diagnostics + rule { + direction = "in" + protocol = "tcp" + port = "443" + source_ips = ["0.0.0.0/0", "::/0"] + } } \ No newline at end of file diff --git a/infrastructure/terraform/modules/firewall/variables.tf b/infrastructure/terraform/modules/firewall/variables.tf index bfc4ba7..814809c 100644 --- a/infrastructure/terraform/modules/firewall/variables.tf +++ b/infrastructure/terraform/modules/firewall/variables.tf @@ -3,7 +3,7 @@ variable "name" { } variable "ssh_source_cidrs" { - type = list(string) - description = "Who may SSH. Our home IP/32" - default = ["0.0.0.0/0"] # TODO: tighten to our IP/32 before real apply + type = list(string) + description = "Who may SSH and reach the K8s API (6443). Use your home IP/32." + # No wide-open default — hobby env must pass admin_cidrs explicitly. } \ No newline at end of file diff --git a/scripts/fetch-hobby-kubeconfig.sh b/scripts/fetch-hobby-kubeconfig.sh new file mode 100644 index 0000000..c8c800a --- /dev/null +++ b/scripts/fetch-hobby-kubeconfig.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# ============================================================================= +# After terraform apply: pull k3s kubeconfig and point it at the VPS public IP. +# Usage (from repo root): +# ./scripts/fetch-hobby-kubeconfig.sh +# +# Requires: terraform state in hobby env, SSH key that matches the server. +# Windows tip: if ssh-keygen -R fails with "Permission denied", we rewrite +# known_hosts with grep instead. +# ============================================================================= +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +TF_DIR="$ROOT/infrastructure/terraform/environments/hobby" +KUBE_OUT="${KUBECONFIG_OUT:-$HOME/.kube/hobby.yaml}" + +IP="$(terraform -chdir="$TF_DIR" output -raw public_ip)" +echo "VPS IP: $IP" + +# Drop stale host key (common after destroy/recreate on same IP) +KNOWN_HOSTS="${HOME}/.ssh/known_hosts" +if [[ -f "$KNOWN_HOSTS" ]]; then + if grep -q "$IP" "$KNOWN_HOSTS" 2>/dev/null; then + grep -v "$IP" "$KNOWN_HOSTS" > "${KNOWN_HOSTS}.tmp" && mv "${KNOWN_HOSTS}.tmp" "$KNOWN_HOSTS" + echo "Removed old known_hosts entry for $IP" + fi +fi + +mkdir -p "$(dirname "$KUBE_OUT")" + +# Wait until k3s API answers on the node (cloud-init may still be running) +echo "Waiting for SSH + k3s (up to ~3 min)..." +for i in $(seq 1 36); do + if ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=5 \ + "root@${IP}" "kubectl get nodes >/dev/null 2>&1"; then + break + fi + if [[ "$i" -eq 36 ]]; then + echo "Timed out waiting for k3s. SSH in and check: journalctl -u k3s -e" >&2 + exit 1 + fi + sleep 5 +done + +scp -o StrictHostKeyChecking=accept-new \ + "root@${IP}:/etc/rancher/k3s/k3s.yaml" "$KUBE_OUT" + +# k3s writes 127.0.0.1; replace with public IP (firewall allows your admin_cidrs) +# portable sed: write to temp then mv (works on macOS + Git Bash) +tmp="$(mktemp)" +sed "s#https://127.0.0.1:6443#https://${IP}:6443#g" "$KUBE_OUT" > "$tmp" +mv "$tmp" "$KUBE_OUT" + +export KUBECONFIG="$KUBE_OUT" +echo "" +echo "Wrote $KUBE_OUT" +echo "Run: export KUBECONFIG=$KUBE_OUT" +kubectl get nodes From 0b8f0a0dc5721f4bb0c0afc9531748ff664ce3c4 Mon Sep 17 00:00:00 2001 From: Subash Pandey Date: Thu, 23 Jul 2026 10:43:08 +0545 Subject: [PATCH 2/2] Enhance observability and metrics for API - Introduced middleware to attach X-Request-ID for log correlation and record RED metrics (request counts and duration) for all HTTP requests. - Updated logging format to include request_id for better traceability. - Added a debug endpoint to simulate errors for observability drills. - Configured Prometheus ServiceMonitor for scraping metrics from the API. - Updated Helm values to enable metrics scraping and added necessary configurations for Prometheus and Loki. These changes improve the observability and monitoring capabilities of the API, facilitating better performance tracking and error diagnosis. --- apps/api/app/main.py | 82 ++++++++++++++++++-- helm/api/templates/servicemonitor.yaml | 31 ++++++++ helm/api/values.yaml | 5 ++ monitoring/README.md | 0 monitoring/alerts/api-rules.yaml | 40 ++++++++++ monitoring/dashboards/api-red.json | 0 monitoring/kube-prometheus-stack-values.yaml | 77 ++++++++++++++++++ monitoring/loki-values.yaml | 53 +++++++++++++ 8 files changed, 283 insertions(+), 5 deletions(-) create mode 100644 helm/api/templates/servicemonitor.yaml create mode 100644 monitoring/README.md create mode 100644 monitoring/alerts/api-rules.yaml create mode 100644 monitoring/dashboards/api-red.json create mode 100644 monitoring/kube-prometheus-stack-values.yaml create mode 100644 monitoring/loki-values.yaml diff --git a/apps/api/app/main.py b/apps/api/app/main.py index e07522c..1ab611e 100644 --- a/apps/api/app/main.py +++ b/apps/api/app/main.py @@ -1,25 +1,88 @@ from __future__ import annotations +import uuid import json import logging from contextlib import asynccontextmanager from typing import Annotated - +import time import httpx import psycopg import redis -from fastapi import Depends, FastAPI, HTTPException, Response -from prometheus_client import CONTENT_TYPE_LATEST, Counter, generate_latest +from fastapi import Depends, FastAPI, HTTPException, Response, Request +from prometheus_client import CONTENT_TYPE_LATEST, Counter, generate_latest, Histogram from pydantic import BaseModel, Field - +from starlette.middleware.base import BaseHTTPMiddleware from app.config import Settings, get_settings from app.llm import summarize -logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s") +logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s request_id=%(request_id)s %(message)s") log = logging.getLogger("api") SUMMARIZE_REQUESTS = Counter("summarize_requests_total", "Summarize endpoint calls", ["status"]) +# ---- RED metrics for ALL HTTP requests ------------------------------------- +HTTP_REQUESTS = Counter( + "http_requests_total", + "HTTP requests", + ["method", "path", "status"], +) + +# Buckets in seconds — tune for your SLOs. These are fine for a learning API. +HTTP_DURATION = Histogram( + "http_request_duration_seconds", + "HTTP request latency", + ["method", "path"], + buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0), +) + +class RequestIdFilter(logging.Filter): + """Ensure every log record has request_id so the format string never KeyErrors.""" + def filter(self, record: logging.LogRecord) -> bool: + if not hasattr(record, "request_id"): + record.request_id = "-" + return True +logging.getLogger().addFilter(RequestIdFilter()) +class ObservabilityMiddleware(BaseHTTPMiddleware): + """ + One middleware, two teaching jobs: + 1) Attach X-Request-ID (client-provided or generated) for log correlation + 2) Record RED metrics for every request + Later (OTel): the same place is where a trace span often starts. + """ + async def dispatch(self, request: Request, call_next): + # Prefer inbound header so a client / gateway can propagate the id + request_id = request.headers.get("x-request-id") or str(uuid.uuid4()) + request.state.request_id = request_id + # Skip scraping /metrics itself from RED (optional; avoids noise) + path = request.url.path + track = path != "/metrics" + start = time.perf_counter() + status_code = 500 + try: + response = await call_next(request) + status_code = response.status_code + response.headers["X-Request-ID"] = request_id + return response + finally: + if track: + elapsed = time.perf_counter() - start + # Use path as-is for this tiny API; normalize if you add path params + HTTP_REQUESTS.labels( + method=request.method, + path=path, + status=str(status_code), + ).inc() + HTTP_DURATION.labels(method=request.method, path=path).observe(elapsed) + # Structured-ish log line — Loki can filter on request_id= + log.info( + "request method=%s path=%s status=%s duration_ms=%.1f", + request.method, + path, + status_code, + elapsed * 1000.0, + extra={"request_id": request_id}, + ) @asynccontextmanager async def lifespan(app: FastAPI): @@ -30,6 +93,7 @@ async def lifespan(app: FastAPI): app = FastAPI(title="cloud-native-ai-api", version="0.1.0", lifespan=lifespan) +app.add_middleware(ObservabilityMiddleware) SettingsDep = Annotated[Settings, Depends(get_settings)] @@ -99,3 +163,11 @@ def summarize_endpoint(body: SummarizeIn, settings: SettingsDep): except Exception: SUMMARIZE_REQUESTS.labels(status="error").inc() raise + +# --------------------------------------------------------------------------- +# TEACHING ONLY — remove or gate behind env before "real" use. +# Hit this to practice: metrics spike → Loki shows request_id → you diagnose. +# --------------------------------------------------------------------------- +@app.get("/debug/boom") +def boom(): + raise HTTPException(status_code=500, detail="forced failure for observability drill") \ No newline at end of file diff --git a/helm/api/templates/servicemonitor.yaml b/helm/api/templates/servicemonitor.yaml new file mode 100644 index 0000000..cae28a0 --- /dev/null +++ b/helm/api/templates/servicemonitor.yaml @@ -0,0 +1,31 @@ +{{- /* + ServiceMonitor is a Prometheus Operator CRD. + When present, the Operator configures Prometheus to scrape this Service. + + Without this (or pod annotations), Prometheus has no idea your /metrics exists. +*/ -}} +{{- if .Values.metrics.serviceMonitor.enabled }} +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: {{ include "api.fullname" . }} + labels: + {{- include "api.labels" . | nindent 4 }} + # Optional: match whatever label your Prometheus selects on + release: mon +spec: + # Which Services to scrape (label selector) + selector: + matchLabels: + {{- include "api.selectorLabels" . | nindent 6 }} + + # Namespace of the Service (usually same as the chart release) + namespaceSelector: + matchNames: + - {{ .Release.Namespace }} + + endpoints: + - port: http # must match Service port *name* in service.yaml + path: /metrics # your FastAPI route + interval: 30s +{{- end }} \ No newline at end of file diff --git a/helm/api/values.yaml b/helm/api/values.yaml index c20be60..b993a18 100644 --- a/helm/api/values.yaml +++ b/helm/api/values.yaml @@ -58,4 +58,9 @@ probes: initialDelaySeconds: 5 periodSeconds: 5 +# Prometheus scrape via Operator +metrics: + serviceMonitor: + enabled: true + # Namespace is usually set by helm -n, not hardcoded int emplates \ No newline at end of file diff --git a/monitoring/README.md b/monitoring/README.md new file mode 100644 index 0000000..e69de29 diff --git a/monitoring/alerts/api-rules.yaml b/monitoring/alerts/api-rules.yaml new file mode 100644 index 0000000..624c750 --- /dev/null +++ b/monitoring/alerts/api-rules.yaml @@ -0,0 +1,40 @@ +# ============================================================================= +# PrometheusRule — high-level alert for the learning lab +# ============================================================================= +# Apply after kube-prometheus-stack is installed (CRDs must exist): +# kubectl apply -f monitoring/alerts/api-rules.yaml +# +# Or fold into GitOps later as its own Application / Helm extraObjects. +# +# Flow: +# Prometheus evaluates expr every evaluation interval +# → if true for `for:` duration, alert becomes FIRING +# → Alertmanager routes to a receiver (webhook / null for lab) +# ============================================================================= +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: api-health + namespace: monitoring + labels: + # Must match Prometheus ruleSelector; with our values (NilUsesHelmValues=false) + # any label usually works. Some installs expect release: mon + release: mon +spec: + groups: + - name: api.rules + rules: + - alert: ApiHighErrorRate + # More than ~5% 5xx over 5m window, sustained for 2m + expr: | + ( + sum(rate(http_requests_total{namespace="ai-platform",status=~"5.."}[5m])) + / + clamp_min(sum(rate(http_requests_total{namespace="ai-platform"}[5m])), 0.001) + ) > 0.05 + for: 2m + labels: + severity: warning + annotations: + summary: "API 5xx rate is high" + description: "Error ratio > 5% for 2m. Check Grafana RED + Loki for request_id." \ No newline at end of file diff --git a/monitoring/dashboards/api-red.json b/monitoring/dashboards/api-red.json new file mode 100644 index 0000000..e69de29 diff --git a/monitoring/kube-prometheus-stack-values.yaml b/monitoring/kube-prometheus-stack-values.yaml new file mode 100644 index 0000000..52b1d63 --- /dev/null +++ b/monitoring/kube-prometheus-stack-values.yaml @@ -0,0 +1,77 @@ +# ============================================================================= +# kube-prometheus-stack — HOBBY / small VPS profile +# ============================================================================= +# Install (from laptop, kubeconfig pointing at hobby k3s): +# +# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +# helm repo update +# kubectl create namespace monitoring +# helm upgrade --install mon prometheus-community/kube-prometheus-stack \ +# -n monitoring -f monitoring/kube-prometheus-stack-values.yaml +# +# UI access while learning (no Ingress yet — Phase 10): +# kubectl -n monitoring port-forward svc/mon-grafana 3000:80 +# # admin password: kubectl get secret -n monitoring mon-grafana \ +# # -o jsonpath='{.data.admin-password}' | base64 -d; echo +# +# WHY these knobs: one node, 4–8 GB RAM. Observability must not starve the API. +# ============================================================================= + +# Grafana: keep defaults; change password in secret later (Phase 9) +grafana: + # Single replica is enough for a lab + replicas: 1 + # Persistence is nice but optional on a throwaway VPS; emptyDir loses dashboards on restart + persistence: + enabled: false + # Don't install a zoo of plugins — RAM + image size + plugins: [] + +prometheus: + prometheusSpec: + # How long to keep samples. Shorter = less disk + less memory pressure. + retention: 3d + # Scrape every 30s is fine for learning (15s is noisier / heavier) + scrapeInterval: 30s + # Cap resources so Prometheus can't eat the node + resources: + requests: + cpu: 100m + memory: 512Mi + limits: + memory: 768Mi + # Storage: emptyDir or tiny PVC. On single-node k3s, local-path PVC is fine. + storageSpec: {} + # Enable ServiceMonitor discovery in all namespaces (so ai-platform API is scraped) + serviceMonitorSelectorNilUsesHelmValues: false + podMonitorSelectorNilUsesHelmValues: false + ruleSelectorNilUsesHelmValues: false + +alertmanager: + enabled: true + alertmanagerSpec: + replicas: 1 + resources: + requests: + memory: 64Mi + limits: + memory: 128Mi + +# Node exporter + kube-state-metrics are useful and relatively cheap — keep them. +# If you OOM later, you can disable optional components, not these first. + +# Example: turn off unused bits if chart version exposes them +# (names vary slightly by chart version — check `helm show values`) +prometheus-node-exporter: + resources: + requests: + memory: 32Mi + limits: + memory: 64Mi + +kube-state-metrics: + resources: + requests: + memory: 64Mi + limits: + memory: 128Mi \ No newline at end of file diff --git a/monitoring/loki-values.yaml b/monitoring/loki-values.yaml new file mode 100644 index 0000000..bdb0cec --- /dev/null +++ b/monitoring/loki-values.yaml @@ -0,0 +1,53 @@ +# ============================================================================= +# Grafana Loki — single-binary / monomorphic mode for tiny clusters +# ============================================================================= +# Chart: grafana/loki (check current chart docs; modes change over versions) +# +# helm repo add grafana https://grafana.github.io/helm-charts +# helm upgrade --install loki grafana/loki -n monitoring -f monitoring/loki-values.yaml +# +# Teaching idea: Loki indexes LABELS, not full-text like ES. +# Good labels: namespace, app, pod. Bad labels: request_id (cardinality!). +# Put request_id IN the log line body; filter with |= "request_id=abc". +# ============================================================================= + +deploymentMode: SingleBinary + +loki: + auth_enabled: false + commonConfig: + replication_factor: 1 + storage: + type: filesystem + schemaConfig: + configs: + - from: "2024-01-01" + store: tsdb + object_store: filesystem + schema: v13 + index: + prefix: index_ + period: 24h + limits_config: + retention_period: 72h # 3 days — matches cost tactics in PLAN.md + +singleBinary: + replicas: 1 + resources: + requests: + memory: 256Mi + limits: + memory: 512Mi + +# Disable distributed components if the chart enables them by default +backend: + replicas: 0 +read: + replicas: 0 +write: + replicas: 0 + +chunksCache: + enabled: false +resultsCache: + enabled: false \ No newline at end of file