Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ jobs:
stack:
- providers/upcloud-rke2
- providers/upcloud-uks
- providers/aws-rke2
- providers/aws-eks
- providers/exoscale-rke2
- providers/exoscale-sks
- providers/hetzner-rke2
Expand Down
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,19 +20,23 @@ Kubernetes flavour are **swappable options**, not baked-in assumptions.
├── providers/ # one self-contained stack (root module) per option
│ ├── upcloud-rke2/ # self-managed RKE2 on UpCloud VMs — TESTED ✅
│ ├── upcloud-uks/ # UpCloud Managed Kubernetes (UKS) — TESTED
│ ├── aws-rke2/ # self-managed RKE2 on AWS EC2 — UNTESTED ⚠️
│ ├── aws-eks/ # Amazon EKS (managed) — TESTED ⚠️
│ ├── exoscale-rke2/ # self-managed RKE2 on Exoscale VMs — UNTESTED ⚠️
│ ├── exoscale-sks/ # Exoscale SKS (managed) — UNTESTED ⚠️
│ ├── hetzner-rke2/ # self-managed RKE2 on Hetzner VMs — UNTESTED ⚠️
│ ├── hetzner-k8s/ # Hetzner has no first-party managed K8s — STUB / N/A
│ ├── lima-rke2/ # local Lima VMs (macOS/Linux), free — TESTED ✅ (macOS)
│ └── openshift/ # OpenShift — STUB
├── scripts/ # shared: prepare-mip-infra-fork.sh (works on any stack)
│ └── aws-iam/ # least-privilege deploy policies + identity generator
└── docs/
├── providers.md # the support matrix + what "tested" means
├── architecture.md # provider-agnostic architecture + the mip-infra contract
├── cost-comparison.md # all providers × options, with totals
├── firewall.md # the cloud-firewall model (UpCloud specifics noted)
├── cost-comparison.md # all providers × options, with totals, grid CO₂ and egress
├── firewall.md # the cloud-firewall model (stateless vs stateful noted)
├── access.md # consoles, dashboards, credentials
├── aws-credentials.md # exactly which AWS rights the stacks need, and why
└── why-self-managed.md # self-managed RKE2 vs managed Kubernetes, provider-agnostic
```

Expand Down Expand Up @@ -82,6 +86,10 @@ Provider and module versions, the RKE2 version, Helm chart versions, container
images and GitHub Actions are all kept current by **Renovate**
([renovate.json](renovate.json)) — see [docs/maintenance.md](docs/maintenance.md).

## License
## License & acknowledgements

[Apache License 2.0](LICENSE).
Apache License 2.0 — see [Apache License 2.0](LICENSE).

This project has received funding from the European Union’s Horizon 2020 Framework Partnership Agreement No. 650003 (Human Brain Project), and from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No. 101147319 (EBRAINS 2.0). This work was also supported by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract No. 23.00638 (EBRAINS 2.0).

The authors acknowledge the support of Amazon Web Services through the AWS Cloud Credit for Research program. Specifically, Amazon EC2 and Amazon EKS were used to develop this test environment's Amazon Terraform provider and advance further the testing of this repository. Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of Amazon Web Services.
5 changes: 4 additions & 1 deletion docs/access.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ mentioned are gitignored.
| **MIP frontend** (federation-A portal) | `https://federation-a.<W0_IP>.sslip.io` | EBRAINS Keycloak login — **dummy creds in dev, login fails by design** until real client credentials are provisioned via `scripts/gen_secrets.sh` |
| **Data Catalogue** | `https://datacatalogue.<W0_IP>.sslip.io` | public pages work; authenticated features need the same Keycloak fix |
| **UpCloud Hub** (billing, servers, console) | `https://hub.upcloud.com` | your UpCloud account; API token via `UPCLOUD_TOKEN` env for tofu/upctl |
| **AWS Console** (billing, EC2/EKS, serial console) | `https://console.aws.amazon.com` | your AWS account; credentials via `AWS_PROFILE` (or access keys) for tofu and the `aws` CLI |

## Dashboards without ingress (kubectl port-forward, on purpose)

Expand All @@ -28,9 +29,11 @@ mentioned are gitignored.
|---|---|
| **Kubernetes (RKE2)** | `export KUBECONFIG=<repo>/kubeconfig` — cluster-admin client cert, API `https://<CP_IP>:6443`. Refetch anytime: `SSH_KEY=~/.ssh/mip-dev-cluster ./scripts/fetch-kubeconfig.sh` |
| **Argo CD CLI** | `argocd login argocd.<CP_IP>.sslip.io --grpc-web --username admin --password "$(cat argocd-admin-password)"` |
| **Node SSH** | `ssh -i ~/.ssh/mip-dev-cluster root@<node-public-ip>` (all four nodes; IPs in `tofu output`) |
| **Node SSH** | `ssh -i ~/.ssh/mip-dev-cluster root@<node-public-ip>` (all four nodes; IPs in `tofu output`). **On the AWS stacks the login user is `ubuntu`** (Canonical AMIs have no root login) or `ec2-user` on the EKS node group's AL2023 image — so `ssh ubuntu@<eip>` then `sudo` |
| **Fork git server** | push/clone `ssh://git@<CP_IP>/home/git/mip-infra.git` with the same key; Argo CD pulls `ssh://git@10.100.1.10/...` with `~/.ssh/argocd-deploy-key` |
| **UpCloud API** | `UPCLOUD_TOKEN` env (never in files) for `tofu` and `upctl` |
| **AWS API** | `AWS_PROFILE` (or `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`) in the env, never in files, for `tofu` and the `aws` CLI. Which rights that principal needs, and a generator for a least-privilege one: [aws-credentials.md](aws-credentials.md) |
| **Kubernetes on `aws-eks`** | `export KUBECONFIG=<stack>/kubeconfig`, or `aws eks update-kubeconfig --name mip-eks --region eu-north-1`. Auth is the `aws eks get-token` exec plugin — no long-lived token is stored |

## In-cluster secrets (owner: `scripts/gen_secrets.sh`; read with kubectl)

Expand Down
40 changes: 31 additions & 9 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,16 +59,38 @@ reads them from `STACK_DIR` to generate the fork diff.

## Per-provider divergences (self-managed)

| Knob | UpCloud | Exoscale | Hetzner | Lima (local) |
|---|---|---|---|---|
| `private_iface` (Canal/flannel bind) | `eth2` | `eth1` (verify) | `enp7s0`/`ens10` (verify) | `eth0` (the user-v2 net replaces Lima's SLIRP NIC) |
| public-IP first-boot lookup | UpCloud metadata JSON | Exoscale metadata | Hetzner metadata | `echo 127.0.0.1` (loopback forwards) |
| firewall/security groups | `upcloud_firewall_rules` | `exoscale_security_group_rule` | `hcloud_firewall` | none (nothing leaves the host) |
| private-net MTU | 1500 | 1500 | **1450** (set CNI MTU) | 1500 (usernet) |
| storage for Longhorn | dedicated block volume | block storage volume | `hcloud_volume` or Longhorn-on-disk | root disk, or `limactl disk` (`data_disk_gib`) |

These are exactly the values the untested stubs are most likely to need
| Knob | UpCloud | AWS (EC2) | Exoscale | Hetzner | Lima (local) |
|---|---|---|---|---|---|
| `private_iface` (Canal/flannel bind) | `eth2` | `ens5` (verify: single ENI, also the default route) | `eth1` (verify) | `enp7s0`/`ens10` (verify) | `eth0` (the user-v2 net replaces Lima's SLIRP NIC) |
| public-IP first-boot lookup | UpCloud metadata JSON | IMDSv2 — token `PUT` then `meta-data/public-ipv4` | Exoscale metadata | Hetzner metadata | `echo 127.0.0.1` (loopback forwards) |
| firewall/security groups | `upcloud_firewall_rules` (**stateless**) | `aws_vpc_security_group_*_rule` (**stateful**, per role) | `exoscale_security_group_rule` | `hcloud_firewall` | none (nothing leaves the host) |
| private-net MTU | 1500 | 9001 (jumbo, intra-VPC; flannel auto-detects) | 1500 | **1450** (set CNI MTU) | 1500 (usernet) |
| storage for Longhorn | dedicated block volume | `aws_ebs_volume`, presented as `/dev/nvme1n1` (Nitro renames it) | block storage volume | `hcloud_volume` or Longhorn-on-disk | root disk, or `limactl disk` (`data_disk_gib`) |
| stable public-IP anchor | node public IPs (survive stop/start) | Elastic IPs (survive an EC2 stop) | node public IPs | node public IPs | host loopback |
| guest agent | `qemu-guest-agent` | none — Nitro has no equivalent (`guest_agent_package = ""`) | `qemu-guest-agent` | `qemu-guest-agent` | `qemu-guest-agent` |
| node naming | from the server resource | set via cloud-init + RKE2 `node-name` (EC2 derives hostnames from the private IP) | from the instance | from the server | from the Lima instance |

These are exactly the values the untested stacks are most likely to need
corrected on first real `apply`. The Lima stack additionally differs in *how*
VMs come up: node IPs are DHCP leases unknown at render time, so the module
gets a `__NODE_IP__` sentinel (resolved in-guest) and agents join the CP via
its stable `lima-<name>.internal` DNS name (`cp_join_host`).

## Managed stacks: how the traffic path differs

Managed clusters have no control-plane node to hang Traefik off and, on EKS, no
node whose public IP survives being recycled. `aws-eks` therefore replaces the
two node-IP anchors with **two Network Load Balancers, each holding an Elastic
IP** — the same two-anchor split, one address per entry point:

```
argocd.<ARGOCD_EIP>.sslip.io ─► NLB :80/:443 ─► node :30080/:30443 Traefik (DaemonSet, class `traefik`)
*.<APPS_EIP>.sslip.io ───────► NLB :80/:443 ─► node :80/:443 haproxy-edge (hostNetwork)
└─► haproxy-public (via cluster DNS) ─► apps
```

The one structural difference from the self-managed path: haproxy-edge resolves
`haproxy-public` by its **Service DNS name** rather than a MetalLB VIP, so the
app path holds regardless of which controller ends up owning that Service on a
managed cluster. `upcloud-uks` and `exoscale-sks` leave both entry points to the
cloud's own LB integration and document the gap instead.
Loading
Loading