This is my home cluster, built on k8s, backed by FluxCD, Terraform, Ansible, and GitOps/DevOps principles. This is a place for me to learn, and share everything I've learned along the way. Feel free to ask questions, submit improvements, and learn from all the hard work I've put into this!
There's an excellent template over at k8s-at-home/template-cluster-k3 that I based my cluster on if you feel like following along :)
My cluster is k8s provisioned overtop bare-metal Talos Linux, which is a bare-metal secure Kubernetes distro managed with IaC and the Talos CLI tool. This is a semi hyper-converged cluster, workloads and block storage are sharing the same available resources on my nodes while I have a separate server for (NFS) file storage.
- actions-runner-controller: self-hosted Github runners
- cilium: internal Kubernetes networking plugin
- cert-manager: creates SSL certificates for services in my cluster
- external-dns: automatically syncs DNS records from my cluster ingresses to a DNS provider
- external-secrets: managed Kubernetes secrets using 1Password Connect.
- ingress-nginx: ingress controller for Kubernetes using NGINX as a reverse proxy and load balancer
- rook: distributed block storage for persistent storage
- sops: managed secrets for Kubernetes, Ansible, and Terraform which are committed to Git
- tf-controller: additional Flux component used to run Terraform from within a Kubernetes cluster.
- volsync: backup and recovery of persistent volume claims
Flux watches the clusters in my kubernetes folder (see Directories below) and makes the changes to my clusters based on the state of my Git repository.
The way Flux works for me here is it will recursively search the kubernetes/${cluster}/apps folder until it finds the most top level kustomization.yaml per directory and then apply all the resources listed in it. That aforementioned kustomization.yaml will generally only have a namespace resource and one or many Flux kustomizations (ks.yaml). Under the control of those Flux kustomizations there will be a HelmRelease or other resources related to the application which will be applied.
Renovate watches my entire repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged Flux applies the changes to my cluster.
The Git repository contains the following directories under cluster and are ordered below by how Flux will apply them.
- flux: directory is the entrypoint to Flux.
- apps: directory (depends on core) is where your common applications (grouped by namespace) could be placed, Flux will prune resources here if they are not tracked by Git anymore.
- bootstrap: directory contains bootstrap procedures for the cluster(s).
- components: directory contains re-useable components that can be used in the apps directory.
📁 bootstrap # bootstrap procedures
📁 kubernetes
├── 📁 apps # applications
├── 📁 flux # core flux configuration
└── 📁 components # re-useable components for gatus, volsync, other odds and ends
📁 talos # talos IaC
├── 📁 clusterconfig # where the config gets stores locally once generated for each host
└── 📁 patches # patches for the config, things like my nvidia GPU patch| Name | CIDR |
|---|---|
| Kubernetes Nodes | 10.1.0.0/24 |
| KubeVIP | 10.1.0.117 |
| Kubernetes gateway | 10.1.0.20 |
| Kubernetes external ingress | 10.1.0.21 |
| Kubernetes internal ingress | 10.1.0.22 |
| Kubernetes cluster CIDR | 10.42.0.0/16 |
| Kubernetes services CIDR | 10.43.0.0/16 |
While most of my infrastructure and workloads are self-hosted I do rely upon the cloud for certain key parts of my setup. This saves me from having to worry about two things. (1) Dealing with chicken/egg scenarios and (2) services I critically need whether my cluster is online or not.
The alternative solution to these two problems would be to host a Kubernetes cluster in the cloud and deploy applications like HCVault, Vaultwarden, and Gatus. However, maintaining another cluster and monitoring another group of workloads is a lot more time and effort than I am willing to put in.
| Service | Use | Cost |
|---|---|---|
| 1Password | Secrets with External Secrets | ~$65/yr |
| Cloudflare | Domain and DNS | Free |
| Backblaze B2 | Object storage for backups | ~$360/yr |
| Newshosting | Usenet access | ~$100/yr |
| GitHub | Hosting this repository and continuous integration/deployments | ~$180/yr |
| NextDNS | My router DNS server which includes AdBlocking | ~$20/yr |
| Pushover | Kubernetes Alerts and application notifications | $5 OTP |
| Total: ~$60/mo |
I utilize rook-ceph for my persistent volume claims (PVCs) and have a backup and recovery solution in place using volsync, which backs up to an S3 bucket hosted by Backblaze B2.
Over WAN, I have port forwarded ports 80 and 443 to the load balancer IP of my ingress controller that's running in my Kubernetes cluster.
Cloudflare works as a proxy to hide my homes WAN IP and also as a firewall. When not on my home network, all the traffic coming into my ingress controller on port 80 and 443 comes from Cloudflare.
🔸 Cloudflare is also configured to GeoIP block all countries except a few I have whitelisted
Handled by UniFi OS on my UniFi Dream Machine Pro.
external-dns is deployed in my cluster and configure to sync DNS records to Cloudflare. The only ingress this external-dns instance looks at to gather DNS records to put in Cloudflare are ones that have an ingress class name of external and contain an ingress annotation external-dns.alpha.kubernetes.io/target.
My home IP can change at any given time and in order to keep my WAN IP address up to date on Cloudflare I have deployed a CronJob in my cluster. This periodically checks and updates the A record ipv4.domain.tld. I followed the example of onedr0p/home-ops as well as docs online. I'll probably write a blog post about it! external-dns is pure magic!
Right now, I'm running my NAS off an old Dell PowerEdge R510 12 bay server. It's a little power hungry! I run TrueNAS, and have a RAIDz10 setup.
Currently, my raw storage capacity is about 56TB. However, I only get about half that since I'm doing RAIDz10. I have a striped set of 6 mirrored pairs. Effectively I can lose up to half my drives, but if I lose both drives in any pair, I'm having a bad day.
To offset this, I back up the NAS once a week to Backblaze B2 object storage, using a data protection setting in TrueNAS SCALE.
I'm slowly replacing all the 2TB drives, which came with my NAS, as they fail or I catch a sale on nicer WD Easy Store drives that I shuck. These are usually white label WD Red drives, or sometimes even better, HGST drives. My plan is to slowly upgrade everything to 12TB drives.
My NAS can be accessed via NFS/SMB/FTP over my home network.
| Device | Count | OS Disk Size | Data Disk Size | Ram | Operating System | Purpose |
|---|---|---|---|---|---|---|
| Poweredge R720 | 1 | 512GB SSD | 1x1TB SSD | 128GB | Talos Linux v1.10 | Kubernetes Server |
| Intel NUC8i7HVK1 | 1 | 512GB NVMe | 1x1TB SSD | 64GB | Talos Linux v1.10 | Kubernetes Server |
| UCS C220 M3 | 1 | 512GB SSD | 1x1TB SSD | 96GB | Talos Linux v1.10 | Kubernetes Server |
| Minisforum UM790 Pro | 1 | 512GB NVMe | 1x2TB NVMe | 32GB | Talos Linux v1.10 | Kubernetes Worker |
| PowerEdge R510 | 1 | 256GB SSD | 2x16TB, 4x12TB, 2x8TB, 4x2TB | 16GB | TrueNAS SCALE 23.10 | Network Attached Storage (NAS) |
| Unifi Dream Machine Pro | 1 | - | - | - | UniFi OS | Network Gateway |
| Unifi Switch 24 PoE Pro | 1 | - | - | - | UniFi OS | Network Switch |
| Mikrotik CRS-317-1G-16S+ | 1 | - | - | - | RouterOS 7 | 10Gb Network Switch/Router |
Thanks to all the people who donate their time to the Kubernetes @Home community. A lot of inspiration for my cluster came from the people that have shared their clusters over at awesome-home-kubernetes.
See commit history
See LICENSE