Learn/cloud and gitops

README.md

@@ -1,6 +1,6 @@
 # Cloud Native AI Platform
 
-A cost-bounded platform for a minimal AI summarization API, built with production-oriented patterns: multi-stage containers, Kubernetes probes, Helm packaging, Terraform foundations, and GitHub Actions publishing immutable images to GHCR.
+A cost-bounded platform for a minimal AI summarization API, built with production-oriented patterns: multi-stage containers, Kubernetes probes, Helm packaging, Terraform foundations, GitHub Actions publishing immutable images to GHCR, and Argo CD GitOps on a Hetzner VPS.
 
 Monthly spend is capped at **$15**. See [docs/cost-budget.md](docs/cost-budget.md).
 
@@ -10,45 +10,143 @@ Monthly spend is capped at **$15**. See [docs/cost-budget.md](docs/cost-budget.m
 |-------|--------|-----------|
 | API + local dev | Done | FastAPI (`/health`, `/ready`, `/metrics`, `POST /v1/summarize`), Compose, golden-path tests |
 | Container image | Done | Multi-stage Dockerfile, non-root runtime |
-| Terraform (hobby) | Scaffolded | Hetzner modules + hobby env; not applied until a VPS is needed |
+| Terraform (hobby) | Done | Hetzner CX22 VPS (`cnai-hobby`) + firewall; k3s via cloud-init |
 | Kubernetes (local) | Done | `kubernetes/base`, liveness/readiness probes. [Runbook](docs/runbooks/app-wont-start.md) |
 | Helm | Done | `helm/api` + Bitnami Postgres/Redis. [Runbook](docs/runbooks/helm.md) |
 | CI / registry | Done | GitHub Actions: test → build → push to GHCR on `main` |
-| GitOps / cloud deploy | Next | Argo CD + VPS |
+| GitOps / cloud deploy | Done | Argo CD on k3s; Application watches `helm/api` + `values-hobby.yaml` |
 
-Operational notes: [docs/lessons-learned.md](docs/lessons-learned.md).
+Operational notes: [docs/lessons-learned.md](docs/lessons-learned.md). GitOps details: [gitops/README.md](gitops/README.md).
 
 ## Architecture
 
 ```
 Client → FastAPI (apps/api) → PostgreSQL
                           └→ Redis
+
+GitHub (main) → GHCR image
+             └→ Argo CD (k3s on Hetzner) → Helm release in ai-platform
 ```
 
 The API exposes standard health and metrics endpoints. LLM calls go through a single `summarize()` abstraction: `stub` in tests/CI, `deepseek` via OpenAI-compatible HTTP when configured. Liveness (`/health`) stays cheap; readiness (`/ready`) gates traffic until Postgres and Redis are reachable.
 
+On the hobby VPS, desired state lives in Git. Argo CD renders the Helm chart and applies it in-cluster — you do not `helm upgrade` from the laptop for cloud deploys.
+
+## Live stack (hobby)
+
+Hetzner VPS `cnai-hobby` (Nuremberg) — ~$6.49/mo while it exists:
+
+![Hetzner hobby VPS running](assets/server_running.png)
+
+Argo CD managing the `api` Application (Healthy, path `helm/api`, namespace `ai-platform`):
+
+![Argo CD Applications dashboard](assets/argo.png)
+
 ## Repository layout
 
 ```
 apps/api/              FastAPI service, Dockerfile, tests
 kubernetes/base/       Raw Kustomize manifests
-helm/api/              Application Helm chart
+helm/api/              Application Helm chart (values-local + values-hobby)
+gitops/                Argo CD Application manifests
 infrastructure/        Terraform modules + hobby environment
 .github/workflows/     CI pipeline
 docs/                  Runbooks, architecture, cost budget
+assets/                Screenshots referenced from this README
 ```
 
 ## Deployment options
 
-Pick **one** local stack. Do not run Compose and a cluster side by side.
-
 | Goal | Path |
 |------|------|
-| Fastest dev loop | Docker Compose (below) |
+| Fastest local loop | Docker Compose (below) |
 | Learn raw K8s objects | `kubectl apply -k kubernetes/base` |
-| Day-to-day cluster work | Helm + Bitnami. [Runbook](docs/runbooks/helm.md) |
+| Day-to-day local cluster | Helm + Bitnami. [Runbook](docs/runbooks/helm.md) |
+| Cloud (hobby VPS) | Terraform → k3s → Argo CD. [gitops/README.md](gitops/README.md) |
+
+Do not run Compose and a local cluster side by side. Cloud work uses the VPS kubeconfig (`~/.kube/hobby.yaml`), not Docker Desktop.
+
+## Pause, resume, and cost control
+
+The VPS is the only recurring bill (~$6–7/mo). GHCR, GitHub Actions free tier, Argo CD, and k3s are $0. Keep the monthly total under **$15**.
 
-Cloud infrastructure (`infrastructure/terraform/environments/hobby`) stays offline until a VPS is required.
+### Closing for the day (pick one)
+
+| Intent | Action | Still billed? |
+|--------|--------|----------------|
+| Short break (hours / overnight) | Hetzner console → power **OFF** on `cnai-hobby` | **Yes** — disk/server reservation still charges |
+| Pause multi-day / keep spend flat | `terraform destroy` in `infrastructure/terraform/environments/hobby` | **No** — this is the real off switch |
+| Pause > 7 days | Always `terraform destroy` (see [docs/cost-budget.md](docs/cost-budget.md)) | No |
+
+Powering off is convenient but **does not stop the meter**. Deleting the server (Terraform destroy or Hetzner Delete) does.
+
+Before destroy: nothing unique should live only on the box — Git + GHCR are source of truth. After destroy, note spend in [docs/cost-budget.md](docs/cost-budget.md).
+
+```bash
+cd infrastructure/terraform/environments/hobby
+export HCLOUD_TOKEN=...   # Hetzner API token
+terraform destroy
+```
+
+### Opening again (same machine, VPS still exists)
+
+1. Hetzner console → power **ON** if you powered off.
+2. Confirm your public IP still matches the firewall allow-list in `main.tf` (`ssh_source_cidrs`). If your ISP changed it, update the CIDR and `terraform apply` before SSH will work.
+3. Point kubectl at the hobby cluster and verify:
+
+```bash
+export KUBECONFIG=~/.kube/hobby.yaml
+kubectl get nodes
+kubectl -n argocd get pods
+kubectl -n argocd port-forward svc/argocd-server 8080:443
+# UI: https://localhost:8080
+```
+
+### Fresh device → back to the current point
+
+You need: this repo, your SSH **private** key (same key Terraform registered), `HCLOUD_TOKEN`, a GitHub PAT with `read:packages`, and either (A) the existing VPS still running or (B) a willingness to recreate it.
+
+**A — VPS still running (cheaper resume)**
+
+```bash
+git clone https://github.com/notsubash/cloud-native-AI-platform.git
+cd cloud-native-AI-platform
+
+# Laptop only — do not run scp while SSH'd into the VPS
+mkdir -p ~/.kube
+scp -i ~/.ssh/id_ed25519 root@<VPS_IP>:/etc/rancher/k3s/k3s.yaml ~/.kube/hobby.yaml
+# Edit hobby.yaml: replace 127.0.0.1 with <VPS_IP>
+
+export KUBECONFIG=~/.kube/hobby.yaml
+kubectl get nodes
+
+# Argo UI
+kubectl -n argocd port-forward svc/argocd-server 8080:443
+```
+
+If you lack Terraform state on the new machine, manage the existing server from the Hetzner console (or copy `*.tfstate` from the old laptop). Do not `terraform apply` blindly — it may try to create a second billable server.
+
+**B — Recreate from zero (after destroy, or no state)**
+
+```bash
+cd infrastructure/terraform/environments/hobby
+cp terraform.tfvars.example terraform.tfvars   # set ssh_public_key_path
+export HCLOUD_TOKEN=...
+# Update ssh_source_cidrs in main.tf to YOUR current public IP/32
+terraform init && terraform plan && terraform apply
+
+# Wait ~2–3 min for cloud-init/k3s, then copy kubeconfig (see A)
+# Install Argo CD (server-side apply), create ghcr-pull secret, apply gitops/applications/api.yaml
+# Full sequence: gitops/README.md
+```
+
+### Cost hygiene checklist
+
+- [ ] Before leaving for the day: power off **or** destroy (know which you chose).
+- [ ] If paused > 7 days: destroy, don’t leave an idle ON server.
+- [ ] After destroy: confirm Hetzner console shows **no** `cnai-hobby` server.
+- [ ] Never commit `HCLOUD_TOKEN`, `terraform.tfvars`, `*.tfstate`, or the GHCR PAT.
+- [ ] Firewall is IP-locked — a new network/café IP blocks SSH until you update `ssh_source_cidrs`.
 
 ## Local development (Compose)
 
@@ -86,6 +184,12 @@ helm history api -n ai-platform
 
 Full sequence, DNS notes, and tear-down: [docs/runbooks/helm.md](docs/runbooks/helm.md).
 
+## GitOps (hobby cloud)
+
+Desired state: [gitops/applications/api.yaml](gitops/applications/api.yaml) → chart `helm/api` with [helm/api/values-hobby.yaml](helm/api/values-hobby.yaml) (GHCR image + `ghcr-pull` secret).
+
+App changes go through Git + Argo Sync — not `helm upgrade` on the laptop. Bootstrap, sync, and drift notes: [gitops/README.md](gitops/README.md).
+
 ## CI / container registry
 
 [`.github/workflows/ci.yml`](.github/workflows/ci.yml) runs on pull requests and pushes to `main`:
@@ -105,7 +209,7 @@ gh auth token | docker login ghcr.io -u <owner> --password-stdin
 docker pull ghcr.io/<owner>/cloud-native-ai-api:sha-<commit>
 ```
 
-PR builds validate the Dockerfile without publishing. Deploy wiring to GHCR images is handled by GitOps, not in CI.
+PR builds validate the Dockerfile without publishing. Cloud deploys consume GHCR via Argo + `values-hobby.yaml`.
 
 ## Image build
 

docs/lessons-learned.md

@@ -50,13 +50,13 @@ Do not run `make up` (Compose) alongside the K8s stack. Access the API with `kub
 
 Owned the API as `helm/api` (Chart.yaml, values, templates, helpers, NOTES) and left Postgres/Redis to Bitnami — own app charts, consume mature data-store charts.
 
-`values-local.yaml` is the local knob file: `pullPolicy: Never`, stub LLM, tiny resources, and Bitnami Service DNS (`postgres-postgresql`, `redis-master`). Defaults in `values.yaml` stay closer to Phase 3 names; local overrides win with `-f`.
+`values-local.yaml` is the local knob file: `pullPolicy: Never`, stub LLM, tiny resources, and Bitnami Service DNS (`postgres-postgresql`, `redis-master`). Defaults in `values.yaml` stay closer to the raw `kubernetes/base` names; local overrides win with `-f`.
 
 Proved the exit checklist: `helm upgrade ... --set image.tag=local-v2` moved the pod image; `helm rollback api 1` restored `cloud-native-ai-api:local` and wrote a new history revision (rollback is a new revision, not a rewind of the list).
 
 Service DNS still bites after switching charts: Bitnami release `postgres` → Service `postgres-postgresql`, Redis standalone → `redis-master`. Wrong host → `/ready` fails even when pods look fine.
 
-Do not run Phase 3 raw `api`/`postgres`/`redis` next to Helm/Bitnami in the same namespace — two Deployments fight for the same mental model. Pick one path: today that path is Bitnami + `helm/api`.
+Do not run raw `api`/`postgres`/`redis` manifests next to Helm/Bitnami in the same namespace — two Deployments fight for the same mental model. Pick one path: today that path is Bitnami + `helm/api` locally, and Argo + `values-hobby.yaml` on the VPS.
 
 `helm template` before `install` caught path typos (`./help/...`); `helm lint` + rendered YAML beat debugging CrashLoops from bad templates.
 
@@ -72,4 +72,22 @@ GHCR auth is `GITHUB_TOKEN` + `packages: write` on the build job only — no PAT
 
 Buildx + `cache-from` / `cache-to type=gha,mode=max` stores builder layers in GitHub Actions cache — first run cold (~0% cached), later runs reuse pip/install layers. The `.dockerbuild` artifact is a Buildx record, not a published package.
 
-CI produces images; it does not deploy. Helm values still point at local tags for laptop work — wiring `image.repository` to `ghcr.io/<owner>/cloud-native-ai-api` and `pullPolicy: IfNotPresent` is Phase 6 (GitOps on the VPS), not something to bolt onto Actions with `kubectl apply`.
+CI produces images; it does not deploy. Laptop Helm keeps local tags; cloud wiring (`ghcr.io/...`, `pullPolicy: IfNotPresent`, `imagePullSecrets`) lives in `values-hobby.yaml` and is applied by Argo — not bolted onto Actions with `kubectl apply`.
+
+## Hobby cloud & GitOps (Hetzner + k3s + Argo CD)
+
+Applied Terraform for real: Hetzner CX23 (`cnai-hobby`), firewall, cloud-init → single-node k3s. Billing starts at `apply`; the only reliable off switch is `terraform destroy` (console power-off still charges for the reserved server).
+
+Cloudflare provider still configures even when DNS is disabled — empty `provider "cloudflare" {}` demands a token and can pull a breaking provider major. For hobby without DNS: leave Cloudflare out of the apply path entirely (comment provider + DNS module) until you need records.
+
+Copied kubeconfig from the **laptop** with `scp root@<ip>:... ~/.kube/hobby.yaml`, then replaced `127.0.0.1` with the public IP. Running `scp` while already SSH’d into the VPS targets the server itself and fails (key-only auth, wrong destination).
+
+Argo install: prefer `kubectl apply --server-side` — client-side apply blows the last-applied annotation size limit on large CRDs (ApplicationSet). Core `Application` still works; finish install with server-side / `--force-conflicts` if needed.
+
+GitOps only sees what GitHub has. Local branch + uncommitted `values-hobby.yaml` → Argo errors like “unable to resolve revision” or missing values file. Commit, push, then Sync.
+
+`ghcr-pull` must be a docker-registry Secret whose password is a GitHub PAT (`read:packages`). An image tag like `sha-...` is not a password — that mistake yields `ImagePullBackOff` after a “healthy” Application.
+
+Kept sync **manual** at first: applying the Application CR registers desired state; pods appear after Sync. Automate prune/selfHeal later once the loop feels boring.
+
+Firewall `ssh_source_cidrs` is a single home IP `/32`. Café/VPN IP changes look like “SSH hang” — update Terraform and re-apply before debugging k3s.

gitops/README.md

@@ -0,0 +1,90 @@
+# GitOps (Argo CD)
+
+Desired state for the hobby cluster lives in Git. Argo CD watches this repo, renders the Helm chart, and applies it into `ai-platform`. You do not `helm upgrade` from your laptop for cloud deploys.
+
+```
+GitHub (branch in Application) → Argo CD → Helm (helm/api + values-hobby.yaml) → k3s
+```
+
+## Layout
+
+| Path | Role |
+|------|------|
+| `gitops/applications/api.yaml` | Argo `Application` CR — source repo/path, destination namespace, sync policy |
+| `helm/api/` | Chart Argo renders |
+| `helm/api/values-hobby.yaml` | Cloud values: GHCR image, `IfNotPresent`, `imagePullSecrets: ghcr-pull` |
+
+`root-app.yaml` is reserved for an optional app-of-apps bootstrap later; the working entry point today is `applications/api.yaml`.
+
+## Prerequisites (cluster already up)
+
+- Hobby VPS with k3s (Terraform + cloud-init)
+- `KUBECONFIG` pointing at the VPS (e.g. `~/.kube/hobby.yaml` with `127.0.0.1` replaced by the public IP)
+- Argo CD installed in namespace `argocd`
+- Namespace `ai-platform` and docker-registry secret `ghcr-pull` (GitHub PAT with `read:packages` — not an image tag)
+
+## Bootstrap Argo CD (once per cluster)
+
+```bash
+kubectl create namespace argocd
+kubectl apply --server-side -n argocd \
+  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
+# If objects already exist and conflict:
+# kubectl apply --server-side --force-conflicts -n argocd -f <same URL>
+
+kubectl -n argocd get pods   # wait until Running
+kubectl -n argocd get secret argocd-initial-admin-secret \
+  -o jsonpath='{.data.password}' | base64 -d; echo
+
+kubectl -n argocd port-forward svc/argocd-server 8080:443
+# UI: https://localhost:8080  (user: admin)
+```
+
+Use **server-side** apply: client-side `kubectl apply` can fail on large Argo CRDs (annotation size limit).
+
+## Register the Application
+
+Commit and **push** the chart path and values Argo should see. Argo clones GitHub — uncommitted or local-only branches are invisible.
+
+```bash
+# Secret once per namespace (PAT, not an image digest)
+kubectl create namespace ai-platform   # if missing
+kubectl -n ai-platform create secret docker-registry ghcr-pull \
+  --docker-server=ghcr.io \
+  --docker-username=<github-user> \
+  --docker-password='<PAT with read:packages>' \
+  --docker-email=<email>
+
+kubectl apply -f gitops/applications/api.yaml
+kubectl -n argocd get application api
+```
+
+Sync is **manual** by default (safer while learning). In the UI: open `api` → **SYNC**, or:
+
+```bash
+# after installing the argocd CLI, or use the UI Sync button
+argocd app sync api
+```
+
+Automated sync / self-heal can be enabled later in `api.yaml` (`syncPolicy.automated`).
+
+## Day-to-day loop
+
+1. Change chart or `values-hobby.yaml` (prefer pinning `image.tag` to `sha-<short>` from CI).
+2. Commit + push to the revision in `api.yaml` (`targetRevision`).
+3. Refresh / Sync in Argo.
+4. Check: `kubectl -n ai-platform get pods` and port-forward `svc/api` if needed.
+
+## Common failures
+
+| Symptom | Likely cause |
+|---------|----------------|
+| `unable to resolve '<branch>' to a commit SHA` | Branch not pushed to GitHub |
+| `values-hobby.yaml: no such file` | File not on the tracked revision / not committed |
+| Sync OK but `ImagePullBackOff` | Bad `ghcr-pull` secret (need PAT with `read:packages`) |
+| Application Healthy but empty namespace | Manual sync not run yet |
+| SSH to VPS hangs | Your public IP changed; update `ssh_source_cidrs` in Terraform |
+
+## Cost note
+
+Argo and k3s are free. The Hetzner VPS is the bill. Power-off still charges; `terraform destroy` stops billing. See the pause/resume section in the root [README.md](../README.md).

gitops/applications/api.yaml

@@ -0,0 +1,68 @@
+# =============================================================================
+# Argo CD Application — "desired state" for our API Helm chart
+# =============================================================================
+# This is NOT a Kubernetes Deployment. It is an Argo CRD that says:
+#   "Watch this Git path, render Helm, apply into that namespace."
+#
+# After you commit this file, you either:
+#   A) kubectl apply -f gitops/applications/api.yaml   (bootstrap once)
+#   B) or create it via Argo UI pointing at the same repo/path
+#
+# From then on, CHANGES TO THE APP go through Git + Argo sync —
+# not through helm upgrade on your laptop.
+# =============================================================================
+
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: api
+  namespace: argocd # Applications live in Argo's namespace by default
+  # finalizer: ensures resources are deleted when the Application is deleted
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io 
+spec:
+  project: default
+
+  # -------------------------------------------------------------------------
+  # SOURCE — where desired state lives
+  # -------------------------------------------------------------------------
+  source:
+    # Your GitHub repo (HTTPS). Private repo → register credentials in Argo.
+    repoURL: https://github.com/notsubash/cloud-native-AI-platform.git
+    targetRevision: main       # branch/tag/commit Argo tracks
+    path: helm/api                # chart directory
+
+    # Tell Argo this path is a Helm chart
+    helm:
+      valueFiles:
+        - values-hobby.yaml       # cloud values (GHCR image + pull secret)
+      # Optional: override tag without editing values file
+      # parameters:
+      #   - name: image.tag
+      #     value: sha-abc1234
+
+
+  # -------------------------------------------------------------------------
+  # DESTINATION — which cluster/namespace to apply into
+  # -------------------------------------------------------------------------
+  destination:
+    server: https://kubernetes.default.svc   # "in-cluster" (Argo talks to local API)
+    namespace: ai-platform
+
+  # Create the namespace if missing
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true
+
+    # -----------------------------------------------------------------------
+    # Start MANUAL (safer for learning). Click Sync in UI or:
+    #   argocd app sync api
+    #
+    # Later, enable automated:
+    # automated:
+    #   prune: true      # delete resources removed from Git
+    #   selfHeal: true   # revert manual kubectl edits (drift drill)
+    # -----------------------------------------------------------------------
+    # automated:
+    #   prune: true
+    #   selfHeal: true