Note: This was my cluster up until early 2026. I've since moved to another setup (I've outgrown the pi's compute), but this repo will stay as is, because it's kinda cool and unique.

lampone

My self hosted cloud, available at cocointhe.cloud.

Table of Contents

• Hardware
  • Bill of materials
  • 3D printed parts
• Software
  • Requirements
  • OS
  • Creating the cluster
  • Deploying the stack
• Other bits and pieces
  • Public facing services through a VPS
  • SOPS setup
  • OIDC-based ssh access w/ opkssh
  • Staging env
  • Backup strategy
    • restic setup
    • garage config
    • velero restore process
  • Traefik WAF + geoblocking
  • Hyperconverged storage w/ Longhorn

Hardware

This is what the cluster looked like (early 2026):

Previous 'versions':

Late 2025

Early 2025

Bill of materials

What it's made of:

• 3 raspberry pi 4 (8Go)
• 1 gigabit ethernet 5 ports switch
• 1 1To lexar ES3 usb SSD
• 3 32Go Sandisk USB3 flash drive
• 1 80mm fan
• 3 very short cat6 ethernet cables
• a 3d printed rack
• some m3 threaded inserts and screws

3D printed parts

The rack is a remix of this one. I've included the stls here that I remixed/designed, aka the vented sleds for the PI4 and the SSD, and the side fan mount.

Software

Here is a top view diagram of the main components:

The Kubernetes cluster is deployed with k3s.

The cluster state is handled by fluxcd, based on what's in this repo.

In k8s/ there are 3 main folders:

• flux, the entrypoint used by the flux controller for synchronizing the cluster. Main 'apps' are declared here. The interval for the source GitRepo is set to 1m, so changes will be picked up within a minute or so.

• infra represents what's needed for the cluster to function:

  • a nfs-server + csi-nfs-driver for handling most persistent volumes (using the 1To SSD on the master)
  • longhorn for high-performance, hyperconverged storage (using the flash drive on each node)
  • an IngressController with Traefik, one private (listens on local lan), one "public",
  • cert-manager for certificates management of my domain,
  • kube-vip for managing the cluster's VIP.
  • rathole for exposing part of my services to the outside world (see here),
  • tailscale-operator for accessing my private services from wherever (using a subnet route) and for my cluster services to access my offsite backup server
  • system-upgrade-controller for managing k8s upgrades directly in the cluster using CRDs.
  • a renovate cronjob to create PR for components updates (w/ auto merging when it's a patch level update and other rules) #- velero + garage for daily, local backups of the cluster (if an app fails and borks its files).
  • a restic cronjob that create a remote backup of the whole nfs dir (in case the server catches on fire) for DR situation,
  • kyverno for enforcing policies in the cluster,
  • goldilocks for automatic adjustments of limits/requests,
  • a vcluster where I do all my pre production testing, see here.
  • falco for eBPF based behavior detection
  • owasp/modsecurity-csr as a 'WAF' for traefik

• apps, the actual services running on the cluster:

  • adguard for DNS/DHCP
  • gitea for local git and CI/CD
  • paperless-ngx for my important files
  • immich for photos backups and sync
  • vaultwarden as my passwords manager
  • filebrowser for file sharing
  • glance as my internet homepage
  • kromgo + shields.io for exposing badge-style cluster stats publicly
  • pocketid as an OIDC provider
  • atuin for my centralized shell history
  • uptime kuma as a simple availability dashboard
  • n8n for basic automation workflows
  • bytestash for remembering short code/iaac snippets
  • grafana + prometheus + loki for monitoring the cluster
  • grist for a modern take on spreadsheets
  • jellyfin & friends (explo, podfetch,..)
  • and some other stuff like a blog , static sites, etc..

I try to adhere to gitops/automation principles. Some things aren't automated but it's mainly toil (one-time-things during setup etc..). 99% of the infrastructure should be deployable by following these instructions (assuming data and encryption keys are known).

Requirements

• ansible: infrastructure automation
• flux: cluster state mgmt
• sops + age: encryption
• git: change management
• gitleaks: secret detection as a pre-commit hook
• vcluster-cli: connect/pause/resume/reset the vcluster

# install everything needed
brew install git ansible fluxcd/tap/flux sops age gitleaks opkssh loft-sh/tap/vcluster 

# tell git where to find its hooks
git config core.hooksPath .githooks

OS

The 3 rasps are running standard Raspberry Pi OS Lite 64b. From there unnecessary packages are removed (dphys-swapfile, avahi-daemon, modemmanager and some others).

Creating the cluster

The bootstrapping is done using ansible. The playbook will simply install k3s on all the nodes.

It is assumed that a ssh key auth is configured on the nodes (ssh-copy-id ), with passwordless sudo (<user> ALL=(ALL) NOPASSWD: ALL in visudo). Otherwise install sshpass and add -k to the ansible command.

cd ansible
ansible-galaxy collection install ansible.posix
ansible-playbook -i inventory.yaml -l lampone cluster-install.yaml

Deploying the stack

1. Get a github token and set an env var:

   export GITHUB_TOKEN=xxx

2. Enter some commands

   # pre create the decryption key
   kubectl create ns flux-system
   kubectl create secret generic sops-age --namespace=flux-system --from-file=age.agekey
   
   # bootstrap flux
   flux bootstrap github \
                 --owner=k0rventen \
                 --repository=lampone \
                 --branch=main \
                 --path=./k8s/flux

3. From here, Flux will create everything that is declared in k8s/, decrypt what's secret using the private key, and keep the stack in sync.

Other bits and pieces

Public facing services through a VPS

I previously used Cloudflare Tunnels to expose some apps to WAN without opening ports on my ISP router (and also being behind CGNAT). Due to concerns regarding their ability to decrypt traffic at the edge (as they provision their own cert for the domain), I switched to rathole + a VPS. I'm currently renting one at ByteHosting in Germany. The 'server' side on the VPS forwards all the traffic from port 80/443 to the client running in my cluster, which then forwards everything to my dedicated traefik instance, which handles SSL and routing. A traefik is in front of rathole to properly pass the client's IP in the headers down the chain.

Here is the compose for the VPS:

services:
  traefik:
    image: traefik
    restart: always
    ports:
    - 80:80
    - 443:443
    volumes:
      - ./traefik.yaml:/etc/traefik/traefik.yml
  rathole:
    image: rapiz1/rathole:v0.5.0
    restart: always
    ports:
      - 2333:2333
    volumes:
      - ./server.toml:/app/config.toml
    command: --server /app/config.toml

the traefik.yaml looks like this:

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: "websecure"
          scheme: "https"
  websecure:
    address: ":443"

providers:
  file:
    directory: /etc/traefik/

tcp:
  routers:
    to-k8s-tcp:
      entryPoints:
        - websecure
      rule: "HostSNI(`*`)"
      tls:
        passthrough: true
      service: rathole-proxy-tcp

  services:
    rathole-proxy-tcp:
      loadBalancer:
        serversTransport: tcp-transport
        servers:
          - address: "rathole:443"

  serversTransports:
    tcp-transport:
      proxyProtocol:
        version: 2

And an example server config file:

[server]
bind_addr = "0.0.0.0:2333" # `2333` specifies the port that rathole listens for clients
default_token = "verylongtoken"

[server.services.traefik_https]
bind_addr = "0.0.0.0:443"

SOPS setup

This assume you have the decryption key age.agekey, and the env var configured:

SOPS_AGE_KEY_FILE=age.agekey

If you want to encrypt an already created file (eg a k8s Secret spec):

sops encrypt -i <file.yaml>

If you want to edit inline a encrypted file (eg modify a value in a encrypted Secret/Configmap) using $EDITOR:

sops <file.yaml>

OIDC-based ssh access w/ opkssh

Whenever possible, authentification is managed through my OIDC provider (pocketID). That's true for most of the services in the cluster, but also for accessing infrastructure-level stuff like my servers, using opkssh.

1. On the OIDC provider, create a new app w/ a Public Client ID (no client secret).

2. On the client, install opkssh and in .opk/config.yaml, add the provider w/ the public client id as per the doc.

3. On the servers, install using the script:

   wget -qO- "https://raw.githubusercontent.com/openpubkey/opkssh/main/scripts/install-linux.sh" | sudo bash

   Add a provider in /etc/opk/providers, then add a user to the policy with sudo opkssh add local_user oidc_email https://oidc-provider. This means that the user with oidc_email will be able to log in as local_user.

4. On the client, do a opkssh login then ssh should be seamless.

Staging env

My staging environment is managed through vcluster. A virtual cluster is deployed inside my actual cluster, and has access to the ingressclass and storageclass of the underlying cluster. The flux controller can deploy resources in the vcluster by specifying the kubeconfig to use if necessary (see the staging/deploy.yaml file).

This allows a isolated cluster for testing things like new versions of various software, deploying new CRDs or testing RBAC rules or NetPols, and other cluster-wide changes without impacting the production environment.

Quick commands:

# connect and switch kubeconfig
vcluster connect vcluster

# disconnect
vcluster disconnect

# destroy the vcluster
vcluster delete vcluster

# and recreate from scratch 
flux reconcile hr -n staging vcluster

Backup strategy

I try to follow a 3-2-1 backup rule. The 'live' data is on the nfs ssd. It's backed up daily onto the same ssd (mainly for rollbacks and potential local re-deployments). For disaster-recovery situations, it's also backed up daily onto a HDD offsite, which can be accessed through my tailnet.

restic setup

The backup tool is restic. It's deployed as a cronjob in the cluster. It launches a custom script that runs the local backup as well as the remote one (which requires commands before and after to mount the external disk on the remote side.). Here are the commands used to create the restic repos before deploying the cronjob:

1. local repo

cd /nfs
restic init nfs-backups

2. remote repo

Create a mnt-backup.mount systemd service on the remote server to mount/umount the backup disk

coco@remote_server:~ $ cat /etc/systemd/system/mnt-backup.mount
[Unit]
Description=Restic Backup External Disk mount

[Mount]
What=/dev/disk/by-label/backup
Where=/mnt/backup
Type=ext4
Options=defaults

[Install]
WantedBy=multi-user.target

Init the repo from the nfs server (this assumes passwordless ssh auth):

restic init -r sftp:<remote_server_ip>:/mnt/backup/nfs-backups

garage config

To configure garage as an s3 backend for velero, the following commands shall be ran on the garage container:

k exec garage-xx -- /garage status
k exec garage-xx -- /garage layout assign -z dc1 -c 1G node_id
k exec garage-xx -- /garage layout apply --version 1
k exec garage-xx -- /garage bucket create velero
k exec garage-xx -- /garage key create velero
k exec garage-xx -- /garage bucket allow --read --write --owner velero  --key velero

The key and secret printed when running key create can be reported in velero's config.

velero restore process

To create a one shot backup of an app:

velero backup create groceries -l app=groceries -n backups

Check its status:

velero backup describe groceries -n backups --details

To restore it:

# suspend the flux resource
flux suspend hr -n cloud groceries

# scale down the deployment
k scale --replicas 0 deploy groceries -n cloud

# delete the pvc (necessary for velero to recreate it and restore its content through kopia)
k delete pvc groceries-data -n cloud

# restore
velero restore create --from-backup groceries

# wait until done
velero restore describe groceries --details

Traefik WAF + geoblocking

Since one of my traefik is exposed publicly, it's always a good idea to add basic security hygiene. This is done using 2 plugins for traefik:

• geoblock, for whitelisting countries that are allowed to connect. By default the plugins makes a request to a public API, which is not ideal. I developed/deployed a small local service, which pulls a geoDB from IPinfo once a day, and the plugin hit it, benefiting from local resilience, and faster responses.
• modsecurity forwards requests to a owasp/modsecurity container, which detects bad behavior (SQLi, paths, ..) and stops generic attack through detection rules.

Hyperconverged storage w/ Longhorn

After encountering various issues related to NFS (for example, SQLite concurrency through NFS for jellyfin), I decided to try some hyperconverged storage. Piraeus seemed like a solid option, but it seems arm64 is not yet an available platform for it.

I went for longhorn instead. Some usb flash drives are plugged into the raspberries, and mounted in /var/lib/longhorn using a systemd mount unit. Deployment is done through their helm chart (see here), and with the redundancy, the 3x32Go flash drives give me around 56Gi of schedulable storage. The storage is at least 3x faster than through NFS (measured using dd at around 130MB/s, which is the advertised speed of the drives). The resilience seems very good, as data locality is not a requirement. I can pull (and already have done so for testing) a usb key from a node, and the workloads just keep going.

Resources-wise, the whole longhorn-system uses around 1Gi of RAM (across 3 nodes, as almost all components are replicated), so 300Mi per node. CPU usage is also up, around 5 to 6%:

> k top pods --sum -n longhorn-system
NAME                                                CPU(cores)   MEMORY(bytes)
longhorn-manager-jkx8n                              23m          122Mi
...                                                 ________     ________
                                                    254m         993Mi

As I had to move some PV data from the nfs storageclass to longhorn's, I used the following:

1. Create new PV for longhorn's storageclass
2. Scale to 0 the deployment which storage will be migrated
3. Deploy a pod with mounts for both classes:

    apiVersion: v1
    kind: Pod
    metadata:
      name: jellyfin-copy
    spec:
      containers:
        - image: busybox
          name: busybox
          command:
            - sleep
            - "inf"
          volumeMounts:
            - name: jellyfin-cache
              mountPath: "/jellyfin-cache"
            - name: jellyfin-config
              mountPath: "/jellyfin-config"
            - name: jellyfin-longhorn-cache
              mountPath: "/longhorn-cache"
            - name: jellyfin-longhorn-config
              mountPath: "/longhorn-config"
   
      volumes:
        - name: jellyfin-cache
          persistentVolumeClaim:
            claimName: jellyfin-cache
   
        - name: jellyfin-config
          persistentVolumeClaim:
            claimName: jellyfin-config
   
        - name: jellyfin-longhorn-cache
          persistentVolumeClaim:
            claimName: jellyfin-longhorn-cache
   
        - name: jellyfin-longhorn-config
          persistentVolumeClaim:
            claimName: jellyfin-longhorn-config

4. In the container, copy from one folder to another:

   cp -a /jellyfin-cache/. /longhorn-cache
   ...

5. Delete the pod, update the mounts on the deployment, scale back up.