diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/README.md b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/README.md new file mode 100644 index 0000000..c123b63 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/README.md @@ -0,0 +1,429 @@ +# Findings: native Kubernetes user namespaces with rootful DinD + +Experiment run: 2026-08-06 + +Record last updated: 2026-08-07 + +## Question tested + +Can a Coder workspace run a normal **rootful** Docker daemon and BuildKit in a +native Kubernetes user-namespace Pod, without Envbox or Sysbox, while keeping +Pod UID 0 mapped to an unprivileged host UID range? + +This was not a rootless-Docker test. The ultimately successful shape was: + +```text +EKS AL2023 node + └─ containerd RuntimeClass: stock runc + cgroup_writable = true + └─ Pod: hostUsers: false, privileged: false + ├─ workspace processes in /workspace-processes cgroup + └─ rootful dockerd managing sibling /docker cgroup hierarchy + └─ ordinary Docker containers / BuildKit workers +``` + +The Pod used `capabilities.add: ["ALL"]`, `procMount: Unmasked`, an unconfined +seccomp profile, and `allowPrivilegeEscalation: true`. Those powers were inside +the Pod's user namespace: container UID 0 mapped to a nonzero host UID range. +The resulting workspace was effectively privileged over resources owned by +that user namespace, but it was neither a Kubernetes `privileged: true` +container nor privileged in the host's initial user namespace. Consequently, +`privileged: false` here must not be read as the security posture of a +conventionally restricted application Pod. + +## Environment + +- EKS Kubernetes `v1.36.2-eks-254016e` in `us-east-2`. +- Amazon Linux 2023 `m6i.large` managed node-group nodes. +- Node kernel: `6.18.38-76.139.amzn2023.x86_64`. +- Node containerd: `2.2.5+unknown`. +- EBS CSI driver, with an experiment-specific `gp3-csi` StorageClass using + `ebs.csi.aws.com` and `WaitForFirstConsumer`. +- Docker test image: `docker:27-dind`, which resolved to Docker Engine + `27.5.1`; the replay manifest now pins `docker:27.5.1-dind`. +- Final Docker data root: an EBS/ext4 PVC. + +The MNG was selected as the debugging-friendly baseline before considering +EKS Auto Mode/Bottlerocket. It permits explicit node bootstrap and containerd +configuration. + +## Baseline results on the stock runtime + +### Native user namespace with an EBS PVC: pass + +`userns-volume-probe.yaml` ran with `hostUsers: false`, bound the EBS PVC, and +wrote and read `/workspace/probe.txt` successfully: + +```text +uid=0(root) +/proc/self/uid_map: + 0 3130523648 65536 +``` + +This proved both non-host UID mapping and compatibility with the CSI-mounted +EBS/ext4 workspace volume. + +### Namespaced privileged probe: pass + +The initial capability probe used `privileged: true` and +`procMount: Unmasked`. It retained a non-host UID mapping and successfully +created a private tmpfs mount. This established that the requested kernel +operations were available inside the user namespace, but the final DinD Pod +did not need Kubernetes `privileged: true`. + +### Rootful Docker on the stock runtime: fail + +Dockerd started and initialized `overlay2` on the PVC, and image pulling +worked. Every attempt to start a child container failed with: + +```text +unable to apply cgroup configuration: +mkdir /sys/fs/cgroup/docker: permission denied +``` + +A focused probe confirmed that the stock runtime exposed no cgroup directory +writable by the Pod. Therefore `hostUsers: false` alone was insufficient for +rootful DinD. + +## Cgroup-writable RuntimeClass follow-up + +The follow-up added a second AL2023 MNG. Its containerd configuration +registered a named handler using the stock `io.containerd.runc.v2` runtime: + +```toml +[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc-cgroup-writable] + runtime_type = 'io.containerd.runc.v2' + cgroup_writable = true +``` + +`runc-cgroup-writable` is a local RuntimeClass/handler name, not a custom runc +binary. + +### Writable-cgroup probe: pass + +A non-privileged, user-namespaced Pod scheduled through this RuntimeClass and +created a child cgroup successfully: + +```text +/proc/self/uid_map: + 0 2088894464 65536 +cgroup on /sys/fs/cgroup type cgroup2 (rw,...,nsdelegate,...) +cgroup-writable-probe-ok +``` + +This fixed the original permission-denied failure. + +### First rootful-Docker attempt: partial pass + +Without preparing the cgroup topology, the following worked: + +- dockerd startup; +- `overlay2` on the EBS/ext4 PVC; +- image pulls and ordinary `docker run`; +- BuildKit `RUN` steps; +- Docker bridge networking between nested containers. + +Only a container using `--memory=64m --pids-limit=64` failed. The hierarchy +showed: + +```text +/sys/fs/cgroup: domain threaded +/sys/fs/cgroup/docker: threaded +``` + +PID 1, dockerd, and containerd occupied the delegated root while threaded +controllers were enabled. This forced a threaded topology in which Docker +could not apply the domain `memory` controller. + +### Domain-cgroup topology: full workload pass + +Before starting dockerd, the final entrypoint: + +1. created `/sys/fs/cgroup/workspace-processes`; +2. moved PID 1 and the workspace processes into it; +3. left the delegated cgroup root empty; +4. enabled `cpuset cpu io memory pids` in the root's + `cgroup.subtree_control`; +5. started dockerd in `workspace-processes`, with Docker children under the + sibling `/docker` hierarchy. + +The recorded result was: + +```text +/proc/self/uid_map: + 0 1990918144 65536 +root cgroup type: domain +root cgroup processes: +root subtree controllers: cpuset cpu io memory pids +workspace cgroup type: domain +Docker storage driver: overlay2 +BuildKit result: buildkit-ok +resource-limited container launch: pass +``` + +Image pull, ordinary nested execution, BuildKit, bridge networking, and a +nested container configured with memory and PID limits all completed. + +This topology is not unique to the native MNG experiment. Current Envbox and +Sysbox solve the same cgroup-v2 no-internal-process constraint at two levels: + +- Envbox's outer-dockerd wrapper (`cli/wrap_dockerd.sh`) creates an `/init` + leaf, moves processes out of the visible cgroup root, and enables its + controllers before starting the outer dockerd. This keeps inner-container + cgroups beneath the Envbox Pod's host cgroup tree. +- Sysbox-runc creates an `init.scope` leaf for the system container, places + its init and exec processes there, and delegates ownership of the cgroup-v2 + control files so inner systemd or Docker can create domain sub-cgroups. + +The native wrapper's `workspace-processes` leaf and sibling `/docker` +hierarchy explicitly reproduce the latter delegation pattern using stock runc +and containerd's `cgroup_writable = true` handler. The wrapper is therefore an +explicit replacement for behavior that Sysbox normally supplies invisibly, +not an unrelated workaround. + +### Docker Compose networking follow-up: pass + +A targeted Docker Compose test created a user-defined bridge network with an +`nginx:1.27-alpine` server and an `alpine:3.21` client. It demonstrated: + +- Compose network creation and attachment of both service containers; +- Docker embedded DNS and bare service-name resolution through libc; +- HTTP from the client to `http://server`; +- outbound HTTP from the nested client to the internet; +- a nested server published as `0.0.0.0:18080->80/tcp`; +- access to that published port from the workspace through + `127.0.0.1:18080`; +- access from another Kubernetes Pod through the workspace Pod IP; and +- access through a Kubernetes ClusterIP Service targeting the published port. + +The external peer check first passed on the workspace node and then passed +from the original MNG node. The cross-node probe reached both the workspace +Pod IP (`192.168.83.149:18080`) and the ClusterIP Service, demonstrating that +Docker's nested bridge/NAT and port-publishing rules interoperated with EKS +Pod routing and Service forwarding across nodes. + +One diagnostic nuance was observed. BusyBox `nslookup server` tried the +Kubernetes search domains inherited by the nested container with `ndots:5` +and returned failure, while `nslookup server.`, `getent hosts server`, and +HTTP to the bare name `server` all resolved the Compose service correctly. +This did not prevent ordinary libc-based application resolution, but clients +with unusual raw-DNS/search-list behavior may require separate validation. + +The successful +[`rootful-dind.yaml`](cgroup-writable-runtime/rootful-dind.yaml) replay +manifest now automates the Compose service-name, HTTP, outbound-network, and +workspace-loopback checks and records explicit completion artifacts. The +companion +[`compose-network-peer.yaml`](cgroup-writable-runtime/compose-network-peer.yaml) +declaratively creates the ClusterIP and headless Services and pins a restricted +peer Pod to the original MNG. The peer resolves the headless Service to the +workspace Pod IP, accesses that IP directly, and separately accesses the +ClusterIP Service. The Docker patch release and observed Alpine and Nginx +digests are pinned for repeatability. + +These manifests encode checks that passed interactively during the recorded +experiment. Their newly combined automated orchestration has not yet itself +been rerun; a future replay must still verify the completion files, peer log, +and distinct workspace/peer node placement before treating the manifests as a +fresh pass. + +## Interpretation + +Native Kubernetes user namespaces can support rootful DinD on this EKS 1.36 +AL2023 MNG without a host-privileged workspace Pod, provided that all of the +following are supplied: + +1. `hostUsers: false` and the broad in-user-namespace security context needed + by dockerd; +2. a containerd RuntimeClass with `cgroup_writable = true`; +3. a startup wrapper that constructs a valid delegated domain-cgroup + topology before starting dockerd; +4. compatible writable storage; EBS/ext4 with `overlay2` worked here. + +The stock EKS runtime remains insufficient. The positive result depends on +purpose-built node/runtime configuration and is currently demonstrated only +on a configurable managed node group. + +### AMI compatibility boundary + +The positive result was demonstrated on the AWS EKS-optimized Amazon Linux +2023 AMI. That AMI runs `nodeadm` during boot, and `nodeadm` supports merging +additional inline containerd TOML from a `NodeConfig`. The experiment used +that supported bootstrap path to register the handler; it did not modify or +rebuild the AMI itself. + +This result does not establish compatibility with every custom, certified, or +hardened AMI. A candidate AMI must preserve the `nodeadm`/`NodeConfig` +bootstrap path, permit the containerd override, provide a containerd version +that supports `cgroup_writable`, and use the matching containerd configuration +schema. In particular, containerd 1.x and 2.x use different CRI plugin paths. +AMI hardening or compliance policy may also prohibit writable delegated +cgroups even when the image can technically accept the configuration. + +Therefore the current compatibility boundary is: + +- AWS EKS-optimized AL2023 MNG: compatible and proven by this experiment; +- custom AL2023 AMI derived from it: plausible only if the required bootstrap + and runtime behavior are preserved, and must be tested; +- arbitrary certified or hardened AMI: not guaranteed and requires vendor or + compliance validation; +- EKS Auto Mode Bottlerocket: this AL2023 bootstrap mechanism is unavailable. + +### Storage compatibility boundary + +The MNG stack supports idmapped mounts; the `hostUsers: false` volume probe +successfully mounted and wrote to an EBS/ext4 PVC. Ext4 therefore provides a +proven storage path for the normal Coder shape of one workspace Pod using one +RWO persistent volume. + +NFS volumes are not supported for Kubernetes user-namespace Pods. Kubernetes +1.36 explicitly documents that the Linux NFS client does not support idmapped +mounts, which these Pods require for every filesystem used by a Pod volume. +This also excludes standard EFS CSI volumes because EFS is mounted through +NFS. See the upstream +[user-namespace filesystem requirements](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#filesystem-support). + +This is a meaningful compatibility restriction, but not a general blocker for +an EBS-backed Envbox replacement. It becomes blocking for templates that +require NFS/EFS semantics such as RWX storage, concurrently shared home +directories or datasets, or storage without EBS availability-zone affinity. + +### Namespace and nested-networking boundaries + +Kubernetes disallows combining `hostUsers: false` with `hostNetwork: true`, +`hostPID: true`, or `hostIPC: true`. This is a native user-namespace +restriction and therefore applies to the MNG design. It is unlikely to block +an ordinary Coder workspace, which normally uses Pod networking and isolated +PID and IPC namespaces, but it excludes specialized workspaces that require +direct host networking or host process/IPC inspection. See the upstream +[user-namespace limitations](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/#limitations). + +The claim that nested networking necessarily uses userspace NAT is not true +for the rootful-Docker design tested here. Dockerd can use Linux bridges, veth +interfaces, and kernel iptables/nftables NAT within the Pod's network +namespace using its namespaced `CAP_NET_ADMIN`; traffic then passes through +the normal Pod CNI and node/VPC networking. Userspace networking such as +`slirp4netns` is principally associated with rootless Docker. See Docker's +[packet-filtering and firewall documentation](https://docs.docker.com/engine/network/packet-filtering-firewalls/). + +The experiment and Compose follow-up proved nested-container outbound +connectivity, Compose service-name resolution through libc, published-port +reachability from the workspace, and same-node and cross-node reachability +through both the workspace Pod IP and a ClusterIP Service. Still untested are +CNI NetworkPolicy behavior, large-packet/MTU correctness, IPv6, and external +NodePort, LoadBalancer, or Ingress exposure. + +### Security comparison with Envbox/Sysbox + +This approach demonstrated the same fundamental user-namespace property as +the Envbox inner container: workspace UID 0 maps to an unprivileged host UID, +and the user-controlled workspace does not run as a host-privileged container. +It is therefore reasonable to describe the two approaches as pursuing the +same core isolation objective. + +The complete security postures are not yet proven equivalent. The native +approach removes Envbox's privileged outer container and the Sysbox manager, +filesystem service, and custom runtime from each workspace's trusted stack. +Kubernetes also assigned a distinct high host-UID range to each tested Pod, +rather than using Envbox's fixed `100000` user-namespace offset. These may be +security advantages. + +Conversely, the successful native Pod required `ALL` capabilities inside its +user namespace, an unmasked `/proc`, an unconfined seccomp profile, and a +writable delegated cgroup hierarchy. It also lacks Sysbox-specific +virtualization and mediation of system-container behavior. Those differences +must be evaluated rather than assumed equivalent. + +More precisely, the successful workspace was effectively privileged inside +its own sandbox. It could administer the Pod's mounts, network namespace, +processes, delegated cgroups, nested containers, PVC contents, credentials, +and reachable network resources. This broad authority is expected for a +Docker-capable developer workspace, where the developer is intentionally +allowed complete control inside the workspace. The relevant security +requirement is therefore containment: that authority must not extend to the +node, other workspaces or their storage, cluster-wide credentials, or network +resources the workspace is not authorized to reach. + +It was not effectively host-privileged. `hostUsers: false` mapped UID 0 to an +unprivileged high host UID and scoped namespaced capabilities such as +`CAP_SYS_ADMIN` and `CAP_NET_ADMIN` to resources owned by the Pod's user +namespace; capabilities such as `CAP_SYS_MODULE` cannot affect the host from +that namespace. The manifest also did not automatically grant host UID 0, +host namespaces, arbitrary host mounts, or unrestricted host-device access. +Those are meaningful differences from a Kubernetes `privileged: true` +container. See the upstream documentation on +[user-namespace capability boundaries](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/) +and +[privileged-container behavior](https://kubernetes.io/docs/concepts/security/linux-kernel-security-constraints/#privileged-containers). + +The remaining risk is still material because all containers share the node's +kernel. An unconfined seccomp profile permits the full syscall surface, +unmasked `/proc` exposes interfaces normally hidden by the runtime, and `ALL` +capabilities plus `allowPrivilegeEscalation: true` remove most defense in depth +inside the namespace. A kernel or user-namespace vulnerability could cross the +intended boundary. This design therefore relies heavily on the Linux user +namespace as its primary host-security boundary: it is meaningfully safer than +host-privileged DinD, but it is not equivalent to a conventional restricted +Pod and still requires a focused security review. + +The supported conclusion is therefore that the native MNG design reproduces +Envbox's fundamental non-host-root workspace boundary and may have a smaller +trusted stack, but full security equivalence requires focused escape, +cross-workspace, `/proc`, cgroup, device, mount, networking, and kernel attack- +surface testing. + +In a separate +[EKS Auto Mode/Bottlerocket experiment](../eks-auto-mode-bottlerocket-userns-rootful-dind-experiment/findings.md), +the first `hostUsers: false` probe failed because the AWS-managed Bottlerocket +node had `user.max_user_namespaces = 0`. A privileged node-preparation +DaemonSet, ordered with a NodePool startup taint, successfully raised that +sysctl and allowed a `hostUsers: false` Pod to use an EBS PVC. The subsequent cgroup +probes nevertheless found no writable delegated hierarchy, including in the +user-namespaced privileged control. Auto Mode's supported NodeClass interface +still exposes no equivalent of the custom `cgroup_writable = true` containerd +handler used by this successful MNG experiment. + +## Decision and remaining validation + +This approach is now a technically credible Envbox/Sysbox alternative for +Coder workspaces on configurable EKS MNGs. It is not yet a production-readiness +or security-equivalence result. + +Before recommending it, test at least: + +1. actual enforcement of memory, CPU, PID, and IO limits under load, rather + than only successful creation with limits; +2. multiple concurrent workspaces on one node, including resource-exhaustion + and cross-Pod isolation attempts; +3. the Coder agent and representative workspace images, broader Compose + configurations, Testcontainers, and devcontainer workflows; +4. Pod restart, node reboot, autoscaling, eviction, PVC reattachment, and + cleanup behavior; +5. admission-policy requirements and whether dedicating/gating the custom + RuntimeClass is operationally acceptable; +6. a focused security review of `ALL` capabilities, unmasked `/proc`, + unconfined seccomp, writable cgroups, nested networking, and exposed + devices, even though these are bounded by the Pod user namespace; +7. monitor for a future supported EKS Auto Mode/Bottlerocket integration. A + privileged preparation DaemonSet overcame the tested node's initial + `user.max_user_namespaces = 0`, but writable cgroup delegation remained + unavailable and Auto Mode exposed no supported equivalent of the MNG's + custom containerd handler. + +The runtime-wide `cgroup_writable` handler also lacks the finer per-Pod policy +and cgroup-depth/descendant controls expected from a future first-class +Kubernetes writable-cgroups API. Until such an API is available and validated, +the custom handler should be limited to dedicated nodes and explicitly +authorized workloads. + +## Runbook corrections made during setup + +- The experiment node must not carry an untolerated custom `NoSchedule` taint + that prevents required EKS add-ons from scheduling. +- The test uses an explicit `gp3-csi` StorageClass rather than the legacy + in-tree `gp2` StorageClass. +- A new WFFC PVC was used for the second MNG to avoid binding the Docker test + to the first node's availability zone. +- The Docker test does not hide pipeline exit codes. +- A writable cgroup mount alone is insufficient: the workspace entrypoint + must keep processes out of the delegated root before enabling domain + controllers. diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-userns-probe.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-userns-probe.yaml new file mode 100644 index 0000000..052a294 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-userns-probe.yaml @@ -0,0 +1,37 @@ +apiVersion: v1 +kind: Pod +metadata: + name: cgroup-userns-probe + namespace: userns-rootful-dind +spec: + hostUsers: false + restartPolicy: Never + nodeSelector: + experiment.coder.com/userns-dind: "true" + containers: + - name: probe + image: public.ecr.aws/docker/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d + securityContext: + privileged: true + procMount: Unmasked + command: ["/bin/sh", "-c"] + args: + - | + set -eux + echo '--- uid map ---' + cat /proc/self/uid_map + echo '--- cgroup membership ---' + cat /proc/self/cgroup + echo '--- cgroup mounts ---' + mount | grep -E 'cgroup|/sys/fs/cgroup' || true + echo '--- cgroup root metadata ---' + ls -ld /sys/fs/cgroup + echo '--- cgroup-root write probe ---' + test_dir=/sys/fs/cgroup/userns-dind-write-probe + if mkdir "$test_dir"; then + echo writable + rmdir "$test_dir" + else + echo not-writable + fi + sleep 600 diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/cgroup-probe.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/cgroup-probe.yaml new file mode 100644 index 0000000..d6d9f49 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/cgroup-probe.yaml @@ -0,0 +1,41 @@ +apiVersion: v1 +kind: Pod +metadata: + name: cgroup-writable-probe + namespace: userns-rootful-dind +spec: + runtimeClassName: runc-cgroup-writable + hostUsers: false + restartPolicy: Never + containers: + - name: probe + image: public.ecr.aws/docker/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d + securityContext: + privileged: false + allowPrivilegeEscalation: true + procMount: Unmasked + capabilities: + add: ["ALL"] + seccompProfile: + type: Unconfined + command: ["/bin/sh", "-c"] + args: + - | + set -eux + echo '--- identity and user namespace ---' + id + cat /proc/self/uid_map + cat /proc/self/gid_map + echo '--- cgroup namespace and mount ---' + cat /proc/self/cgroup + mount | grep -E 'cgroup|/sys/fs/cgroup' + ls -ldn /sys/fs/cgroup + echo '--- available/delegated controllers ---' + cat /sys/fs/cgroup/cgroup.controllers + cat /sys/fs/cgroup/cgroup.subtree_control + echo '--- create child cgroup ---' + mkdir /sys/fs/cgroup/userns-dind-write-probe + ls -ldn /sys/fs/cgroup/userns-dind-write-probe + rmdir /sys/fs/cgroup/userns-dind-write-probe + echo cgroup-writable-probe-ok + sleep 600 diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/compose-network-peer.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/compose-network-peer.yaml new file mode 100644 index 0000000..bab8cc4 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/compose-network-peer.yaml @@ -0,0 +1,94 @@ +apiVersion: v1 +kind: Service +metadata: + name: compose-network-target + namespace: userns-rootful-dind +spec: + selector: + experiment.coder.com/compose-network-target: "true" + ports: + - name: http + port: 18080 + targetPort: 18080 +--- +# This headless Service provides a stable discovery mechanism for the +# workspace Pod IP. The peer resolves it, extracts the Pod IP, and then sends +# HTTP directly to that IP rather than through the ClusterIP Service. +apiVersion: v1 +kind: Service +metadata: + name: compose-network-target-headless + namespace: userns-rootful-dind +spec: + clusterIP: None + selector: + experiment.coder.com/compose-network-target: "true" + ports: + - name: http + port: 18080 + targetPort: 18080 +--- +apiVersion: v1 +kind: Pod +metadata: + name: compose-network-peer-probe + namespace: userns-rootful-dind +spec: + # The successful workspace is forced onto the cgroup-writable MNG by its + # RuntimeClass. Pinning this peer to the original MNG makes the two HTTP + # checks cross-node rather than merely same-node checks. + nodeSelector: + experiment.coder.com/userns-dind: "true" + automountServiceAccountToken: false + restartPolicy: Never + containers: + - name: probe + image: public.ecr.aws/docker/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + runAsNonRoot: true + runAsUser: 65534 + seccompProfile: + type: RuntimeDefault + env: + - name: PEER_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + command: ["/bin/sh", "-c"] + args: + - | + set -eux + + fetch_and_verify() { + url="$1" + output="$2" + attempts=0 + until wget -T 5 -t 1 -qO "$output" "$url"; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 120 ] || return 1 + sleep 2 + done + grep -q 'Welcome to nginx' "$output" + } + + service_url='http://compose-network-target.userns-rootful-dind.svc.cluster.local:18080/' + fetch_and_verify "$service_url" /tmp/service-body + echo "cross-node-clusterip-service-ok node=$PEER_NODE_NAME url=$service_url" + + headless_host='compose-network-target-headless.userns-rootful-dind.svc.cluster.local' + attempts=0 + pod_ip='' + until pod_ip="$(getent hosts "$headless_host" | awk 'NR == 1 { print $1 }')" && + [ -n "$pod_ip" ]; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 120 ] || exit 1 + sleep 2 + done + + pod_url="http://${pod_ip}:18080/" + fetch_and_verify "$pod_url" /tmp/pod-ip-body + echo "cross-node-direct-pod-ip-ok node=$PEER_NODE_NAME url=$pod_url" + echo compose-cross-node-network-tests-ok diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/nodegroup.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/nodegroup.yaml new file mode 100644 index 0000000..010a321 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/nodegroup.yaml @@ -0,0 +1,35 @@ +apiVersion: eksctl.io/v1alpha5 +kind: ClusterConfig + +metadata: + name: userns-rootful-dind-mng-136 + region: us-east-2 + version: "1.36" + +managedNodeGroups: + - name: userns-dind-cgroup-writable + amiFamily: AmazonLinux2023 + instanceType: m6i.large + minSize: 1 + desiredCapacity: 1 + maxSize: 1 + labels: + experiment.coder.com/userns-dind-cgroup-writable: "true" + iam: + withAddonPolicies: + ebs: true + # eksctl prepends this partial NodeConfig to the EKS-generated NodeConfig; + # nodeadm merges it with the cluster connection details and normal AL2023 + # defaults. Containerd 2.x uses the config-v3 CRI runtime plugin path. + overrideBootstrapCommand: | + apiVersion: node.eks.aws/v1alpha1 + kind: NodeConfig + spec: + containerd: + config: | + [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc-cgroup-writable] + runtime_type = 'io.containerd.runc.v2' + cgroup_writable = true + + [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc-cgroup-writable.options] + SystemdCgroup = true diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/pvc.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/pvc.yaml new file mode 100644 index 0000000..647bac8 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/pvc.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: workspace-data-cgroup-writable + namespace: userns-rootful-dind +spec: + accessModes: [ReadWriteOnce] + storageClassName: gp3-csi + resources: + requests: + storage: 30Gi diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind.yaml new file mode 100644 index 0000000..c319544 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/rootful-dind.yaml @@ -0,0 +1,203 @@ +apiVersion: v1 +kind: Pod +metadata: + name: rootful-dind-cgroup-writable + namespace: userns-rootful-dind + labels: + experiment.coder.com/compose-network-target: "true" +spec: + runtimeClassName: runc-cgroup-writable + hostUsers: false + restartPolicy: Never + containers: + - name: workspace + # The original docker:27-dind tag resolved to Docker Engine 27.5.1. + # Pin the patch release so a replay does not silently change engines. + image: docker:27.5.1-dind + securityContext: + privileged: false + allowPrivilegeEscalation: true + procMount: Unmasked + capabilities: + add: ["ALL"] + seccompProfile: + type: Unconfined + env: + - name: DOCKER_TLS_CERTDIR + value: "" + - name: DOCKER_HOST + value: unix:///run/docker.sock + command: ["/bin/sh", "-c"] + args: + - | + set -eu + mkdir -p /run /workspace/docker-data /workspace/docker-artifacts-cgroup-domain + artifacts=/workspace/docker-artifacts-cgroup-domain + rm -f "$artifacts/test-complete" + alpine_image='alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d' + nginx_image='nginx:1.27-alpine@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10' + + # Keep the delegated cgroup root free of processes so that domain + # controllers such as memory and io can be enabled for Docker's + # child-container hierarchy. All workspace processes, including + # this PID 1 shell and the dockerd it starts, live in a sibling + # cgroup instead. + mkdir /sys/fs/cgroup/workspace-processes + while read -r pid < /sys/fs/cgroup/cgroup.procs; do + [ -n "$pid" ] || break + echo "$pid" > /sys/fs/cgroup/workspace-processes/cgroup.procs + done + + for controller in cpu cpuset io memory pids; do + if grep -qw "$controller" /sys/fs/cgroup/cgroup.controllers; then + echo "+$controller" > /sys/fs/cgroup/cgroup.subtree_control + fi + done + + cat /sys/fs/cgroup/cgroup.type > "$artifacts/root-cgroup-type.txt" + cat /sys/fs/cgroup/cgroup.controllers > "$artifacts/root-cgroup-controllers.txt" + cat /sys/fs/cgroup/cgroup.subtree_control > "$artifacts/root-cgroup-subtree-control.txt" + cat /sys/fs/cgroup/cgroup.procs > "$artifacts/root-cgroup-procs.txt" + cat /sys/fs/cgroup/workspace-processes/cgroup.type > "$artifacts/workspace-cgroup-type.txt" + + id > "$artifacts/id.txt" + cat /proc/self/uid_map > "$artifacts/uid-map.txt" + cat /proc/self/gid_map > "$artifacts/gid-map.txt" + cat /proc/self/cgroup > "$artifacts/cgroup.txt" + mount | grep -E 'cgroup|/sys/fs/cgroup' > "$artifacts/cgroup-mount.txt" + mkdir /sys/fs/cgroup/dind-preflight + rmdir /sys/fs/cgroup/dind-preflight + + dockerd \ + --host=unix:///run/docker.sock \ + --data-root=/workspace/docker-data \ + --pidfile=/run/dockerd.pid \ + --exec-opt=native.cgroupdriver=cgroupfs \ + --cgroup-parent=docker \ + > "$artifacts/dockerd.log" 2>&1 & + dockerd_pid=$! + trap 'kill "$dockerd_pid" 2>/dev/null || true; wait "$dockerd_pid" 2>/dev/null || true' EXIT + + i=0 + until docker info > "$artifacts/docker-info.txt" 2>&1; do + i=$((i + 1)) + if [ "$i" -ge 90 ]; then + echo 'dockerd did not become ready' >&2 + cat "$artifacts/dockerd.log" >&2 || true + exit 1 + fi + sleep 2 + done + + docker version + docker info + docker info --format '{{.Driver}}' > "$artifacts/storage-driver.txt" + + docker pull "$alpine_image" + docker run --rm "$alpine_image" id > "$artifacts/child-id.txt" + cat "$artifacts/child-id.txt" + + mkdir -p /tmp/build-context + printf '%s\n' \ + "FROM $alpine_image" \ + 'RUN id' \ + 'CMD ["/bin/sh", "-c", "echo buildkit-ok"]' \ + > /tmp/build-context/Dockerfile + DOCKER_BUILDKIT=1 docker build -t userns-dind-smoke:3 /tmp/build-context \ + > "$artifacts/build.log" 2>&1 + cat "$artifacts/build.log" + docker run --rm userns-dind-smoke:3 > "$artifacts/build-result.txt" + cat "$artifacts/build-result.txt" + + docker rm -f httpd >/dev/null 2>&1 || true + docker run -d --name httpd "$nginx_image" + httpd_ip="$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' httpd)" + docker run --rm "$alpine_image" wget -qO- "http://${httpd_ip}" \ + > "$artifacts/network-result.txt" + head -c 100 "$artifacts/network-result.txt" + echo + docker rm -f httpd + + if docker run --rm --memory=64m --pids-limit=64 "$alpine_image" true \ + > "$artifacts/resource-limit-result.txt" 2>&1; then + echo pass > "$artifacts/resource-limit-status.txt" + else + echo fail > "$artifacts/resource-limit-status.txt" + cat "$artifacts/resource-limit-result.txt" >&2 + exit 1 + fi + + # Reproduce the Docker Compose networking checks that were + # originally run interactively against this workspace. + compose_dir=/tmp/compose-network-test + compose_file="$compose_dir/compose.yaml" + mkdir -p "$compose_dir" + printf '%s\n' \ + 'services:' \ + ' server:' \ + " image: $nginx_image" \ + ' ports:' \ + ' - "18080:80"' \ + ' client:' \ + " image: $alpine_image" \ + ' command: ["sh", "-c", "sleep 3600"]' \ + > "$compose_file" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + up -d --remove-orphans + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + ps > "$artifacts/compose-ps.txt" + docker network inspect coder-nettest_default \ + > "$artifacts/compose-network-inspect.json" + + docker compose \ + -p coder-nettest \ + -f "$compose_file" \ + exec -T client sh -ec ' + attempts=0 + until getent hosts server >/tmp/server-hosts && + wget -qO /tmp/internal.html http://server/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q "Welcome to nginx" /tmp/internal.html + echo compose-service-dns-and-http-ok + + attempts=0 + until wget -qO /tmp/outbound.html http://example.com/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -qi "Example Domain" /tmp/outbound.html + echo compose-outbound-network-ok + ' > "$artifacts/compose-inner-network-result.txt" + cat "$artifacts/compose-inner-network-result.txt" + + attempts=0 + until wget -qO "$artifacts/compose-published.html" \ + http://127.0.0.1:18080/; do + attempts=$((attempts + 1)) + [ "$attempts" -lt 60 ] || exit 1 + sleep 1 + done + grep -q 'Welcome to nginx' "$artifacts/compose-published.html" + echo workspace-loopback-published-port-ok \ + > "$artifacts/compose-published-port-status.txt" + cat "$artifacts/compose-published-port-status.txt" + + touch "$artifacts/test-complete" /tmp/rootful-dind-test-complete + echo 'Domain-cgroup rootful DinD and Compose tests complete; keeping dockerd alive.' + wait "$dockerd_pid" + volumeMounts: + - name: workspace-data + mountPath: /workspace + volumes: + - name: workspace-data + persistentVolumeClaim: + claimName: workspace-data-cgroup-writable diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runbook.md b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runbook.md new file mode 100644 index 0000000..f82ea4c --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runbook.md @@ -0,0 +1,170 @@ +# Cgroup-writable RuntimeClass follow-up + +This follow-up keeps the existing EKS 1.36 cluster and original AL2023 MNG as +the control. It adds a second AL2023 MNG whose containerd 2.x configuration +registers a stock-runc handler with `cgroup_writable = true`. + +The name `runc-cgroup-writable` is local configuration, not another runtime +binary. The handler still uses the node's stock `io.containerd.runc.v2`. + +## 1. Create the second MNG + +```bash +eksctl create nodegroup -f nodegroup.yaml + +kubectl get nodes \ + -L eks.amazonaws.com/nodegroup,experiment.coder.com/userns-dind-cgroup-writable +``` + +Do not continue until the new node is `Ready` and has the expected label. + +## 2. Register the RuntimeClass + +```bash +kubectl apply -f runtimeclass.yaml +kubectl get runtimeclass runc-cgroup-writable -o yaml +``` + +## 3. Prove cgroup delegation + +```bash +kubectl apply -f cgroup-probe.yaml +kubectl -n userns-rootful-dind wait \ + --for=condition=Ready pod/cgroup-writable-probe --timeout=15m +kubectl -n userns-rootful-dind logs cgroup-writable-probe +``` + +The required result is `cgroup-writable-probe-ok`. If `mkdir` under +`/sys/fs/cgroup` fails, stop and inspect the new node's generated containerd +configuration; do not proceed to Docker. + +## 4. Run rootful Docker + +Delete the probe, create the new WFFC PVC, and start the Docker Pod: + +```bash +kubectl -n userns-rootful-dind delete pod cgroup-writable-probe --wait=true +kubectl -n userns-rootful-dind delete pod rootful-dind-cgroup-writable \ + --ignore-not-found --wait=true +kubectl apply -f pvc.yaml +kubectl apply -f rootful-dind.yaml +kubectl -n userns-rootful-dind wait \ + --for=condition=Ready pod/rootful-dind-cgroup-writable --timeout=15m +kubectl -n userns-rootful-dind logs -f rootful-dind-cgroup-writable +``` + +Stop following logs after this completion message: + +```text +Domain-cgroup rootful DinD and Compose tests complete; keeping dockerd alive. +``` + +The Pod becoming Ready only proves that its container started; it does not +prove that the embedded Docker, BuildKit, resource-limit, and Compose tests +finished. Verify the completion artifact before interpreting the result: + +```bash +until kubectl -n userns-rootful-dind exec rootful-dind-cgroup-writable -- \ + test -f /tmp/rootful-dind-test-complete; do + sleep 2 +done +``` + +Inspect results with: + +```bash +kubectl -n userns-rootful-dind exec rootful-dind-cgroup-writable -- sh -c ' + artifacts=/workspace/docker-artifacts-cgroup-domain + cat "$artifacts/uid-map.txt" + cat "$artifacts/cgroup.txt" + cat "$artifacts/root-cgroup-type.txt" + cat "$artifacts/root-cgroup-subtree-control.txt" + cat "$artifacts/storage-driver.txt" + cat "$artifacts/build-result.txt" + cat "$artifacts/resource-limit-status.txt" + cat "$artifacts/compose-inner-network-result.txt" + cat "$artifacts/compose-published-port-status.txt" +' +``` + +The successful manifest pins Docker Engine 27.5.1 and the observed Alpine and +Nginx image digests. It leaves the Compose server running with port 18080 +published into the workspace Pod network namespace for the peer tests below. + +## 5. Reproduce same-node and cross-node networking + +The embedded checks above prove Compose service-name resolution, HTTP between +Compose services, outbound networking, and access to the published port from +the workspace itself. The companion manifest adds: + +- a ClusterIP Service selecting the workspace Pod; +- a headless Service used only to discover the workspace Pod IP; and +- a restricted peer Pod pinned to the original MNG, which accesses the Nginx + container through both the ClusterIP and the workspace Pod IP. + +Delete any previous completed peer Pod, apply the manifest, and wait for the +new peer to finish: + +```bash +kubectl -n userns-rootful-dind delete pod compose-network-peer-probe \ + --ignore-not-found --wait=true +kubectl apply -f compose-network-peer.yaml +kubectl -n userns-rootful-dind wait \ + --for=jsonpath='{.status.phase}'=Succeeded \ + pod/compose-network-peer-probe --timeout=10m +kubectl -n userns-rootful-dind logs compose-network-peer-probe +``` + +The required final line is: + +```text +compose-cross-node-network-tests-ok +``` + +Confirm that the peer and workspace actually ran on different nodes; do not +call this a cross-node pass based only on the peer log: + +```bash +workspace_node="$(kubectl -n userns-rootful-dind get pod \ + rootful-dind-cgroup-writable -o jsonpath='{.spec.nodeName}')" +peer_node="$(kubectl -n userns-rootful-dind get pod \ + compose-network-peer-probe -o jsonpath='{.spec.nodeName}')" +printf 'workspace_node=%s\npeer_node=%s\n' "$workspace_node" "$peer_node" +test "$workspace_node" != "$peer_node" + +kubectl -n userns-rootful-dind get \ + service/compose-network-target \ + service/compose-network-target-headless \ + pod/rootful-dind-cgroup-writable \ + pod/compose-network-peer-probe -o wide +``` + +## Interpretation + +- Cgroup probe fails: the node/runtime configuration did not produce a usable + delegation boundary. +- Cgroup probe passes but Docker fails: capture `dockerd.log`; the next + boundary is likely device, network, or controller delegation rather than + image storage. +- Docker run/build/network/resource-limit and embedded Compose tests pass, + but peer test fails: preserve the peer log and distinguish Service, DNS, + Pod-IP routing, node placement, and CNI policy failures. +- All embedded and cross-node tests pass: the native-userns design is + technically viable on a purpose-configured MNG, but the runtime-wide + writable-cgroup handler and broad in-user-namespace security context still + need production security and exhaustion reviews. + +## Cleanup + +The Service objects can be applied repeatedly. The completed peer Pod must be +deleted before each replay because a Pod's command is immutable. + +```bash +kubectl delete -f compose-network-peer.yaml --ignore-not-found +kubectl delete -f rootful-dind.yaml --ignore-not-found +kubectl delete -f pvc.yaml --ignore-not-found +kubectl delete -f runtimeclass.yaml --ignore-not-found +``` + +Delete the `userns-dind-cgroup-writable` node group separately when the entire +experiment is complete. diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runtimeclass.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runtimeclass.yaml new file mode 100644 index 0000000..8952c00 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cgroup-writable-runtime/runtimeclass.yaml @@ -0,0 +1,8 @@ +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: runc-cgroup-writable +handler: runc-cgroup-writable +scheduling: + nodeSelector: + experiment.coder.com/userns-dind-cgroup-writable: "true" diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cluster.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cluster.yaml new file mode 100644 index 0000000..0fb48fd --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/cluster.yaml @@ -0,0 +1,34 @@ +apiVersion: eksctl.io/v1alpha5 +kind: ClusterConfig + +metadata: + name: userns-rootful-dind-mng-136 + region: us-east-2 + version: "1.36" + +# Keep this baseline on a conventional managed node group even if a future +# eksctl release changes its default behavior. +autoModeConfig: + enabled: false + +# This is intentionally a conventional Amazon Linux managed node group, not +# an Auto Mode NodePool or Bottlerocket node. It is the debugging-friendly +# baseline for the native-Kubernetes-user-namespace experiment. +managedNodeGroups: + - name: userns-dind + amiFamily: AmazonLinux2023 + instanceType: m6i.large + minSize: 1 + desiredCapacity: 1 + maxSize: 2 + labels: + experiment.coder.com/userns-dind: "true" + # The experiment needs an EBS-backed PVC. This attaches the EBS CSI + # permissions to the test node role; the EKS addon below supplies the + # controller and node components. + iam: + withAddonPolicies: + ebs: true + +addons: + - name: aws-ebs-csi-driver diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/namespace-and-pvc.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/namespace-and-pvc.yaml new file mode 100644 index 0000000..f3f361b --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/namespace-and-pvc.yaml @@ -0,0 +1,20 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: userns-rootful-dind + labels: + pod-security.kubernetes.io/enforce: privileged + pod-security.kubernetes.io/audit: privileged + pod-security.kubernetes.io/warn: privileged +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: workspace-data + namespace: userns-rootful-dind +spec: + accessModes: [ReadWriteOnce] + storageClassName: gp3-csi + resources: + requests: + storage: 30Gi diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/privileged-userns-probe.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/privileged-userns-probe.yaml new file mode 100644 index 0000000..c5b8ebc --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/privileged-userns-probe.yaml @@ -0,0 +1,30 @@ +apiVersion: v1 +kind: Pod +metadata: + name: privileged-userns-probe + namespace: userns-rootful-dind +spec: + hostUsers: false + restartPolicy: Never + nodeSelector: + experiment.coder.com/userns-dind: "true" + containers: + - name: probe + image: public.ecr.aws/docker/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d + securityContext: + privileged: true + procMount: Unmasked + command: ["/bin/sh", "-c"] + args: + - | + set -eux + id + echo '--- uid/gid mapping ---' + cat /proc/self/uid_map + cat /proc/self/gid_map + mkdir /tmp/private-mount + mount -t tmpfs tmpfs /tmp/private-mount + mount | grep ' /tmp/private-mount ' + umount /tmp/private-mount + echo privileged-userns-private-mount-ok + sleep 600 diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/rootful-dind.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/rootful-dind.yaml new file mode 100644 index 0000000..7e40cdc --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/rootful-dind.yaml @@ -0,0 +1,100 @@ +apiVersion: v1 +kind: Pod +metadata: + name: rootful-dind + namespace: userns-rootful-dind +spec: + hostUsers: false + restartPolicy: Never + nodeSelector: + experiment.coder.com/userns-dind: "true" + containers: + - name: workspace + # The original docker:27-dind tag resolved to Docker Engine 27.5.1. + image: docker:27.5.1-dind + securityContext: + privileged: true + procMount: Unmasked + env: + - name: DOCKER_TLS_CERTDIR + value: "" + - name: DOCKER_HOST + value: unix:///run/docker.sock + command: ["/bin/sh", "-c"] + args: + - | + set -eu + mkdir -p /run /workspace/docker-data /workspace/docker-artifacts + alpine_image='alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d' + nginx_image='nginx:1.27-alpine@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10' + echo '--- outer user namespace mapping ---' | tee /workspace/docker-artifacts/uid-map.txt + cat /proc/self/uid_map | tee -a /workspace/docker-artifacts/uid-map.txt + cat /proc/self/gid_map | tee -a /workspace/docker-artifacts/uid-map.txt + + dockerd \ + --host=unix:///run/docker.sock \ + --data-root=/workspace/docker-data \ + --pidfile=/run/dockerd.pid \ + > /workspace/docker-artifacts/dockerd.log 2>&1 & + dockerd_pid=$! + trap 'kill "$dockerd_pid" 2>/dev/null || true; wait "$dockerd_pid" 2>/dev/null || true' EXIT + + i=0 + until docker info >/workspace/docker-artifacts/docker-info.txt 2>&1; do + i=$((i + 1)) + if [ "$i" -ge 90 ]; then + echo 'dockerd did not become ready' >&2 + cat /workspace/docker-artifacts/dockerd.log >&2 || true + exit 1 + fi + sleep 2 + done + + docker version | tee /workspace/docker-artifacts/docker-version.txt + docker info | tee /workspace/docker-artifacts/docker-info-full.txt + docker info --format '{{.Driver}}' | tee /workspace/docker-artifacts/storage-driver.txt + + time docker pull "$alpine_image" + docker run --rm "$alpine_image" id > /workspace/docker-artifacts/child-id.txt + cat /workspace/docker-artifacts/child-id.txt + + mkdir -p /tmp/build-context + printf '%s\n' "FROM $alpine_image" 'RUN id' 'CMD ["/bin/sh", "-c", "echo buildkit-ok"]' > /tmp/build-context/Dockerfile + DOCKER_BUILDKIT=1 docker build -t userns-dind-smoke:1 /tmp/build-context \ + > /workspace/docker-artifacts/build.log 2>&1 + cat /workspace/docker-artifacts/build.log + docker run --rm userns-dind-smoke:1 > /workspace/docker-artifacts/build-result.txt + cat /workspace/docker-artifacts/build-result.txt + + docker run -d --name httpd "$nginx_image" + httpd_ip="$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' httpd)" + docker run --rm "$alpine_image" wget -qO- "http://${httpd_ip}" \ + > /workspace/docker-artifacts/network-result.txt + head -c 100 /workspace/docker-artifacts/network-result.txt + docker rm -f httpd + + if docker run --rm --memory=64m --pids-limit=64 "$alpine_image" true \ + > /workspace/docker-artifacts/resource-limit-result.txt 2>&1; then + echo pass > /workspace/docker-artifacts/resource-limit-status.txt + else + echo fail > /workspace/docker-artifacts/resource-limit-status.txt + fi + + docker run --rm --privileged "$alpine_image" sh -c ' + mkdir /tmp/nested-private-mount && + mount -t tmpfs tmpfs /tmp/nested-private-mount && + umount /tmp/nested-private-mount && + echo nested-namespaced-privilege-ok + ' > /workspace/docker-artifacts/nested-privileged-result.txt + cat /workspace/docker-artifacts/nested-privileged-result.txt + + touch /workspace/docker-artifacts/test3-complete + echo 'Test 3 complete; keeping daemon alive for inspection.' + wait "$dockerd_pid" + volumeMounts: + - name: workspace-data + mountPath: /workspace + volumes: + - name: workspace-data + persistentVolumeClaim: + claimName: workspace-data diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/runbook.md b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/runbook.md new file mode 100644 index 0000000..9764357 --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/runbook.md @@ -0,0 +1,610 @@ +# EKS Amazon Linux managed-node-group user-namespace rootful-DinD runbook + +## Purpose + +Determine whether a Coder workspace can run a **rootful** Docker daemon and +BuildKit inside a native Kubernetes user-namespace Pod on an EKS Kubernetes +1.36 cluster using a dedicated **Amazon Linux managed node group (MNG)**, +without Envbox or Sysbox. + +This is the baseline experiment. Amazon Linux MNG nodes are deliberately used +before EKS Auto Mode/Bottlerocket: they remove Auto Mode provisioning and +Bottlerocket policy as confounding variables and offer a more practical place +to diagnose a failure. A pass here is necessary but not sufficient evidence +for a later Auto Mode/Bottlerocket compatibility test. + +The intended security model is: + +```text +EKS node (Amazon Linux MNG) + └─ Kubernetes Pod: hostUsers: false + └─ workspace / dockerd runs as UID 0 inside the Pod + └─ Docker child containers run as UID 0 inside Docker +``` + +The kubelet maps the Pod's UID 0 to an unprivileged, non-overlapping host ID +range. `privileged: true` therefore supplies capabilities inside the Pod's +user namespace, not equivalent host-root capabilities. This is deliberately +different from **rootless Docker**: the Docker daemon under test is rootful +from its own point of view. + +The experiment answers whether this can replace Envbox for ordinary Coder +workspaces that need Docker builds and test containers. It does not establish +feature parity with Envbox's system-container use cases (for example arbitrary +host integration or full VM-like `systemd` behavior). + +## Decision rules + +Record each result as **pass**, **fail**, or **blocked**. Do not silently +substitute an ordinary Pod for `hostUsers: false`, or a rootless Docker daemon +for rootful Docker; either would test a different design. + +The baseline is promising only if all of these pass on the MNG target: + +1. The EKS cluster admits a `hostUsers: false`, `privileged: true`, + `procMount: Unmasked` Pod. +2. The workspace's real persistent-volume filesystem can mount into that Pod + (user-namespace Pods require idmapped-mount support). +3. A normal rootful `dockerd` starts without Envbox/Sysbox and can run, + build, network, and persist Docker containers/images. +4. `/proc/self/uid_map` shows that container UID 0 maps to a nonzero host ID. +5. The chosen storage driver is acceptable. `overlay2` or a demonstrably + performant `fuse-overlayfs` result is a potential pass; `vfs` is only a + diagnostic fallback, not a production-quality result. + +A failure of native user namespaces, idmapped volume mounts, or admission +policy is a stop condition for this MNG baseline. A Docker failure after those +pass is useful: it identifies a narrower Docker/runtime issue to investigate. + +## Scope and non-goals + +This runbook tests one Linux/amd64 EKS cluster with a dedicated Amazon Linux +MNG and an EBS-backed workspace-like PVC. It does not test NFS; NFS is +currently incompatible with Kubernetes user-namespace Pods because the Linux +NFS client does not support idmapped mounts. + +It also does not claim that a `privileged` Pod will be accepted in every +customer environment. Even when its effective capabilities are user-namespace +scoped, `privileged: true` and `procMount: Unmasked` can still be rejected by +Pod Security Admission or an organization-specific admission policy. + +## Prerequisites + +- A disposable EKS cluster running Kubernetes **1.36**, with a dedicated + Amazon Linux 2023 MNG. Kubernetes user namespaces are GA in 1.36. Confirm + the exact EKS version and node runtime; do not infer support from the + Kubernetes API alone. +- `aws`, `kubectl`, and `eksctl` installed and authenticated. +- Permission to create/delete the test namespace, Pods, and PVC. If + creating a disposable cluster, the AWS identity also needs normal EKS/VPC/ + EC2/IAM creation and deletion permissions. +- Approval for a namespaced privileged Pod. No hostPath, host networking, + host PID, or host IPC is requested by this experiment. + +Before changing anything, record the cluster and client state: + +```bash +export AWS_REGION="us-east-2" +export CLUSTER_NAME="REPLACE_WITH_CLUSTER_NAME" +export EXPERIMENT_NS="userns-rootful-dind" + +aws sts get-caller-identity +aws eks describe-cluster \ + --region "$AWS_REGION" \ + --name "$CLUSTER_NAME" \ + --query 'cluster.{name:name,version:version,platformVersion:platformVersion,status:status}' \ + --output yaml | tee cluster-version.yaml +kubectl version -o yaml | tee kubectl-version.yaml +kubectl get nodes -o wide | tee nodes-before.txt +kubectl get storageclass | tee storageclasses.txt +kubectl get nodes -L eks.amazonaws.com/nodegroup,kubernetes.io/os,kubernetes.io/arch \ + > nodes-labelled-before.txt +``` + +Stop if the server version is below 1.36. Also stop if no EBS-like storage +class is available; test the same storage class that a Coder workspace would +use, not `emptyDir` alone. + +## Optional: create a disposable EKS cluster and Amazon Linux MNG + +Skip this section when an approved test cluster already exists. Otherwise, +save this as `cluster.yaml`, choosing a unique name and region: + +```yaml +apiVersion: eksctl.io/v1alpha5 +kind: ClusterConfig + +metadata: + name: userns-rootful-dind-mng-136 + region: us-east-2 + version: "1.36" + +# Preserve the conventional-MNG setup if a future eksctl release enables Auto +# Mode by default. +autoModeConfig: + enabled: false + +managedNodeGroups: + - name: userns-dind + amiFamily: AmazonLinux2023 + instanceType: m6i.large + minSize: 1 + desiredCapacity: 1 + maxSize: 2 + labels: + experiment.coder.com/userns-dind: "true" + iam: + withAddonPolicies: + ebs: true + +addons: + - name: aws-ebs-csi-driver +``` + +Create it and configure `kubectl`: + +```bash +eksctl create cluster -f cluster.yaml +export CLUSTER_NAME="userns-rootful-dind-mng-136" +aws eks update-kubeconfig --region "$AWS_REGION" --name "$CLUSTER_NAME" +``` + +Re-run the prerequisite collection commands after creation. + +## Create or select a dedicated Amazon Linux MNG + +The disposable-cluster configuration above already creates the target MNG. If +using an existing cluster, create an equivalent dedicated Amazon Linux 2023 +MNG with the label below, or apply it through the approved node-group +management path. Do not run this baseline on Auto Mode/Bottlerocket nodes. + +Do not taint this small, dedicated MNG: EBS CSI and other cluster add-ons need +to schedule on it. The label is enough to ensure the experiment Pods select it. + +Required node label: + +```yaml +labels: + experiment.coder.com/userns-dind: "true" +``` + +Confirm the target nodes are Amazon Linux MNG nodes and capture them: + +```bash +kubectl get nodes -l experiment.coder.com/userns-dind=true -o wide \ + | tee userns-dind-nodes.txt +kubectl get nodes -l experiment.coder.com/userns-dind=true -o yaml \ + > userns-dind-nodes.yaml +``` + +Stop if this selector yields no nodes, if the nodes are not Linux/amd64, or if +they are not members of the intended Amazon Linux MNG. Record the MNG AMI and +container-runtime versions if they are visible from the node description. + +## Create the namespace and PVC + +The namespace deliberately requests the `privileged` Pod Security Admission +profile, because the main probe requests `privileged: true` and an unmasked +proc mount. If organizational policy rejects this namespace label, record the +rejection as an admission-policy block rather than weakening the experiment. + +Prefer a direct `ebs.csi.aws.com` StorageClass rather than a legacy +`kubernetes.io/aws-ebs` class. The included `storageclass.yaml` creates a +`gp3-csi` class for this purpose. Apply it before the included +`namespace-and-pvc.yaml` manifest. + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: userns-rootful-dind + labels: + pod-security.kubernetes.io/enforce: privileged + pod-security.kubernetes.io/audit: privileged + pod-security.kubernetes.io/warn: privileged +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: workspace-data + namespace: userns-rootful-dind +spec: + accessModes: [ReadWriteOnce] + storageClassName: gp3-csi + resources: + requests: + storage: 30Gi +``` + +Apply it and record its initial state: + +```bash +kubectl apply -f storageclass.yaml +kubectl apply -f namespace-and-pvc.yaml +kubectl -n "$EXPERIMENT_NS" get namespace,pvc -o yaml > namespace-and-pvc.actual.yaml +``` + +Many EBS StorageClasses use `WaitForFirstConsumer`. For those, a newly created +PVC correctly remains `Pending` until Test 1 schedules; do **not** wait for it +to become `Bound` before creating that Pod. Test 1's successful start and PVC +write are the binding/mount proof. + +## Test 1: native user namespace plus real PVC + +This probe has no Docker daemon and no privileged security context. It +separates Kubernetes user-namespace and idmapped-volume support from the later +Docker test. Save as `userns-volume-probe.yaml`: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: userns-volume-probe + namespace: userns-rootful-dind +spec: + hostUsers: false + restartPolicy: Never + nodeSelector: + experiment.coder.com/userns-dind: "true" + containers: + - name: probe + image: public.ecr.aws/docker/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d + command: ["/bin/sh", "-c"] + args: + - | + set -eux + id + echo '--- uid/gid mapping ---' + cat /proc/self/uid_map + cat /proc/self/gid_map + echo '--- PVC write ---' + echo "$(date -u +%FT%TZ) userns-volume-probe" > /workspace/probe.txt + cat /workspace/probe.txt + sleep 600 + volumeMounts: + - name: workspace-data + mountPath: /workspace + volumes: + - name: workspace-data + persistentVolumeClaim: + claimName: workspace-data +``` + +Run it: + +```bash +kubectl apply -f userns-volume-probe.yaml +kubectl -n "$EXPERIMENT_NS" wait --for=condition=Ready pod/userns-volume-probe --timeout=15m +kubectl -n "$EXPERIMENT_NS" logs userns-volume-probe | tee test1-userns-volume.log +kubectl -n "$EXPERIMENT_NS" get pod userns-volume-probe -o yaml > test1-pod.yaml +kubectl -n "$EXPERIMENT_NS" describe pod userns-volume-probe > test1-describe.txt +export NODE_NAME="$(kubectl -n "$EXPERIMENT_NS" get pod userns-volume-probe -o jsonpath='{.spec.nodeName}')" +kubectl get node "$NODE_NAME" -o yaml > test1-node.yaml +``` + +Expected result: + +- the Pod is Ready; +- `uid_map` contains a mapping from container ID `0` to a nonzero host ID; +- the PVC write succeeds. + +If it fails with `MOUNT_ATTR_IDMAP`, the PVC filesystem or node runtime does +not support this design. If `hostUsers` is rejected or ignored, stop: the +cluster cannot test the proposed security model. + +Delete the probe before the next test so the RWO PVC can attach to the Docker +Pod: + +```bash +kubectl -n "$EXPERIMENT_NS" delete pod userns-volume-probe --wait=true +``` + +## Test 2: admission of namespaced privileged / unmasked proc + +This short probe identifies an admission-policy or runtime restriction before +Docker muddies the result. Save as `privileged-userns-probe.yaml`: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: privileged-userns-probe + namespace: userns-rootful-dind +spec: + hostUsers: false + restartPolicy: Never + nodeSelector: + experiment.coder.com/userns-dind: "true" + containers: + - name: probe + image: public.ecr.aws/docker/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d + securityContext: + privileged: true + procMount: Unmasked + command: ["/bin/sh", "-c"] + args: + - | + set -eux + id + cat /proc/self/uid_map + mkdir /tmp/private-mount + mount -t tmpfs tmpfs /tmp/private-mount + mount | grep /tmp/private-mount + umount /tmp/private-mount + sleep 600 +``` + +Apply and collect results: + +```bash +kubectl apply -f privileged-userns-probe.yaml +kubectl -n "$EXPERIMENT_NS" wait --for=condition=Ready pod/privileged-userns-probe --timeout=15m +kubectl -n "$EXPERIMENT_NS" logs privileged-userns-probe | tee test2-privileged-userns.log +kubectl -n "$EXPERIMENT_NS" describe pod privileged-userns-probe > test2-describe.txt +``` + +The private `tmpfs` mount is expected to be possible inside this namespaced +privileged Pod. It is not a host mount and does not demonstrate host access. +If the Pod is rejected, preserve the full API/server message. That is a real +operational limitation even though the user namespace would confine the +effective kernel capabilities. + +Delete it before continuing: + +```bash +kubectl -n "$EXPERIMENT_NS" delete pod privileged-userns-probe --wait=true +``` + +## Test 3: rootful Docker and BuildKit + +Save this as `rootful-dind.yaml`. The original `docker:27-dind` image resolved +to Docker Engine 27.5.1, so the replay pins `docker:27.5.1-dind`. It supplies +dockerd and the Docker client; the command starts a normal rootful daemon +manually. Do not use a `*-dind-rootless` image for this test. + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: rootful-dind + namespace: userns-rootful-dind +spec: + hostUsers: false + restartPolicy: Never + nodeSelector: + experiment.coder.com/userns-dind: "true" + containers: + - name: workspace + image: docker:27.5.1-dind + securityContext: + privileged: true + procMount: Unmasked + env: + - name: DOCKER_TLS_CERTDIR + value: "" + - name: DOCKER_HOST + value: unix:///run/docker.sock + command: ["/bin/sh", "-c"] + args: + - | + set -eu + mkdir -p /run /workspace/docker-data /workspace/docker-artifacts + alpine_image='alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d' + nginx_image='nginx:1.27-alpine@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10' + echo '--- outer user namespace mapping ---' | tee /workspace/docker-artifacts/uid-map.txt + cat /proc/self/uid_map | tee -a /workspace/docker-artifacts/uid-map.txt + cat /proc/self/gid_map | tee -a /workspace/docker-artifacts/uid-map.txt + + dockerd \ + --host=unix:///run/docker.sock \ + --data-root=/workspace/docker-data \ + --pidfile=/run/dockerd.pid \ + > /workspace/docker-artifacts/dockerd.log 2>&1 & + dockerd_pid=$! + trap 'kill "$dockerd_pid" 2>/dev/null || true; wait "$dockerd_pid" 2>/dev/null || true' EXIT + + i=0 + until docker info >/workspace/docker-artifacts/docker-info.txt 2>&1; do + i=$((i + 1)) + if [ "$i" -ge 90 ]; then + echo 'dockerd did not become ready' >&2 + cat /workspace/docker-artifacts/dockerd.log >&2 || true + exit 1 + fi + sleep 2 + done + + docker version | tee /workspace/docker-artifacts/docker-version.txt + docker info | tee /workspace/docker-artifacts/docker-info-full.txt + docker info --format '{{.Driver}}' | tee /workspace/docker-artifacts/storage-driver.txt + + time docker pull "$alpine_image" + docker run --rm "$alpine_image" id > /workspace/docker-artifacts/child-id.txt + cat /workspace/docker-artifacts/child-id.txt + + mkdir -p /tmp/build-context + printf '%s\n' "FROM $alpine_image" 'RUN id' 'CMD ["/bin/sh", "-c", "echo buildkit-ok"]' > /tmp/build-context/Dockerfile + DOCKER_BUILDKIT=1 docker build -t userns-dind-smoke:1 /tmp/build-context \ + > /workspace/docker-artifacts/build.log 2>&1 + cat /workspace/docker-artifacts/build.log + docker run --rm userns-dind-smoke:1 > /workspace/docker-artifacts/build-result.txt + cat /workspace/docker-artifacts/build-result.txt + + docker run -d --name httpd "$nginx_image" + httpd_ip="$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' httpd)" + docker run --rm "$alpine_image" wget -qO- "http://${httpd_ip}" \ + > /workspace/docker-artifacts/network-result.txt + head -c 100 /workspace/docker-artifacts/network-result.txt + docker rm -f httpd + + # This checks the ordinary Docker resource-control path. A failure is + # informative; record it rather than silently removing the limit. + if docker run --rm --memory=64m --pids-limit=64 "$alpine_image" true \ + > /workspace/docker-artifacts/resource-limit-result.txt 2>&1; then + echo pass > /workspace/docker-artifacts/resource-limit-status.txt + else + echo fail > /workspace/docker-artifacts/resource-limit-status.txt + fi + + # A nested privileged container may have namespaced capabilities. It + # must still not be interpreted as a host-privilege test. + docker run --rm --privileged "$alpine_image" sh -c ' + mkdir /tmp/nested-private-mount && + mount -t tmpfs tmpfs /tmp/nested-private-mount && + umount /tmp/nested-private-mount && + echo nested-namespaced-privilege-ok + ' > /workspace/docker-artifacts/nested-privileged-result.txt + cat /workspace/docker-artifacts/nested-privileged-result.txt + + touch /workspace/docker-artifacts/test3-complete + echo 'Test 3 complete; keeping daemon alive for inspection.' + wait "$dockerd_pid" + volumeMounts: + - name: workspace-data + mountPath: /workspace + volumes: + - name: workspace-data + persistentVolumeClaim: + claimName: workspace-data +``` + +Run and observe the startup rather than assuming the Pod will become Ready: + +```bash +kubectl apply -f rootful-dind.yaml +kubectl -n "$EXPERIMENT_NS" get pod rootful-dind -w +``` + +In another terminal, collect the outcome: + +```bash +kubectl -n "$EXPERIMENT_NS" logs -f rootful-dind +kubectl -n "$EXPERIMENT_NS" describe pod rootful-dind > test3-describe.txt +kubectl -n "$EXPERIMENT_NS" get pod rootful-dind -o yaml > test3-pod.yaml +kubectl -n "$EXPERIMENT_NS" exec rootful-dind -- sh -c ' + find /workspace/docker-artifacts -maxdepth 1 -type f -printf "%f\\n" | sort + cat /workspace/docker-artifacts/storage-driver.txt + cat /workspace/docker-artifacts/uid-map.txt +' +``` + +Expected functional results: + +- `dockerd` starts without Envbox/Sysbox; +- `docker info` reports an intentional storage driver; +- the child `alpine` container and BuildKit build succeed; +- one nested container reaches another over Docker's bridge network; +- the inner privileged-container test succeeds only within its own private + namespaces; +- the outer Pod UID map still maps ID 0 to a nonzero host ID. + +If dockerd fails, preserve `dockerd.log` from the PVC before deleting the Pod: + +```bash +kubectl -n "$EXPERIMENT_NS" exec rootful-dind -- \ + cat /workspace/docker-artifacts/dockerd.log > test3-dockerd.log || true +``` + +### Storage-driver fallback diagnostics + +Do not change the first test: its default driver tells us whether this design +works naturally. If it fails specifically on overlay storage, make two clearly +labelled diagnostic reruns: + +1. Add `--storage-driver=fuse-overlayfs` **only if** `fuse-overlayfs` exists + in the image and `/dev/fuse` is available inside the Pod. +2. Add `--storage-driver=vfs` only to prove that the remaining Docker stack + works. Treat a vfs-only result as a performance/design failure, not a final + acceptance. + +For each rerun, use a fresh PVC or remove `/workspace/docker-artifacts` and +`/workspace/docker-data` deliberately, then record the exact daemon +arguments. + +## Test 4: Docker-store persistence across a workspace restart + +The workspace PVC should preserve Docker's `/workspace/docker-data` store, +while the Pod itself is recreated. This distinguishes a usable per-workspace +Docker cache from an `emptyDir`-only demo. + +After Test 3 has created `userns-dind-smoke:1`: + +```bash +kubectl -n "$EXPERIMENT_NS" delete pod rootful-dind --wait=true +kubectl apply -f rootful-dind.yaml +kubectl -n "$EXPERIMENT_NS" wait --for=condition=Ready pod/rootful-dind --timeout=15m +until kubectl -n "$EXPERIMENT_NS" exec rootful-dind -- docker info >/dev/null 2>&1; do sleep 2; done +kubectl -n "$EXPERIMENT_NS" exec rootful-dind -- \ + docker image inspect userns-dind-smoke:1 > test4-image-inspect.json +``` + +Record whether the image is immediately available. This is only a +per-workspace cache. It does **not** give the node-wide sharing that the +Envbox image-cache proposal targets; a second workspace will have its own +Docker store unless a separate registry/cache mechanism is used. + +## Interpret results + +Use this matrix in the experiment result note: + +| Result | Interpretation | Next action | +|---|---|---| +| Test 1 fails | Native Kubernetes user namespaces or the real PVC filesystem are unavailable. | Stop this baseline design; do not debug Docker. | +| Test 2 fails | The cluster policy/runtime will not allow the required namespaced privileged shape. | Determine whether a less-privileged tailored dockerd/BuildKit configuration is possible; otherwise stop. | +| Tests 1–2 pass, dockerd fails | Platform supports the isolation model, but rootful DinD needs a storage, mount, cgroup, or networking adjustment. | Diagnose from `dockerd.log`; compare `overlay2`, `fuse-overlayfs`, then `vfs`. | +| Docker works only with vfs | Functional proof only; likely too slow/disk-heavy for Coder workspaces. | Investigate `fuse-overlayfs` or another supported storage layout. | +| Tests 1–4 pass with acceptable storage | Viable candidate for a Coder workspace backend. | Run Coder-agent and real repository/devcontainer tests next. | + +In the recorded run, Tests 1 and 2 passed, but the stock-runtime Docker Pod +could not create `/sys/fs/cgroup/docker`. Continue with +[`cgroup-writable-runtime/runbook.md`](cgroup-writable-runtime/runbook.md) to +reproduce the purpose-configured MNG follow-up and its successful +domain-cgroup topology. Do not modify this baseline control to hide that +failure. + +## Follow-up Coder validation + +Only after the base test passes, replace the synthetic workspace command with +a Coder agent and run a representative workflow: + +1. provision a workspace from the candidate template; +2. clone/open `~/coder` in VS Code; +3. reopen in its `.devcontainer` configuration if requested; +4. run the Docker-using Coder tests and a BuildKit build; +5. verify Coder resource limits, stop/start behavior, and the PVC-backed + Docker-store persistence; +6. compare cold/warm image pulls with Envbox on the same node class. + +Do not describe this as an Envbox replacement until those end-to-end tests +pass. In particular, retain Envbox for workloads requiring system-container +features that the native user-namespace model does not provide. + +## Collect artifacts and clean up + +Before cleanup, collect events and all PVC-backed logs: + +```bash +kubectl -n "$EXPERIMENT_NS" get events --sort-by=.lastTimestamp > events.txt +kubectl -n "$EXPERIMENT_NS" get all -o yaml > final-kubernetes-objects.yaml +kubectl -n "$EXPERIMENT_NS" exec rootful-dind -- \ + tar -C /workspace -czf - docker-artifacts > docker-artifacts.tgz || true +kubectl get node "$NODE_NAME" -o yaml > final-node.yaml +kubectl get nodes -L eks.amazonaws.com/nodegroup -o yaml > final-nodes.yaml +``` + +Delete the experiment resources when results are recorded: + +```bash +kubectl delete -f rootful-dind.yaml --ignore-not-found +kubectl delete namespace "$EXPERIMENT_NS" --ignore-not-found +``` + +If this runbook created a disposable cluster, delete it after the namespace +has terminated: + +```bash +eksctl delete cluster --region "$AWS_REGION" --name "$CLUSTER_NAME" +``` + +Confirm no test EBS volumes, EC2 instances, managed node groups, or cluster resources +remain. The experiment incurs AWS charges while the cluster and PVC exist. diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/storageclass.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/storageclass.yaml new file mode 100644 index 0000000..4056a2c --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/storageclass.yaml @@ -0,0 +1,13 @@ +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: gp3-csi + labels: + experiment.coder.com/userns-dind: "true" +provisioner: ebs.csi.aws.com +parameters: + type: gp3 + fsType: ext4 +reclaimPolicy: Delete +volumeBindingMode: WaitForFirstConsumer +allowVolumeExpansion: true diff --git a/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/userns-volume-probe.yaml b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/userns-volume-probe.yaml new file mode 100644 index 0000000..dd3ba7d --- /dev/null +++ b/deploy/eks-mng-amazon-linux-userns-rootful-dind-experiment/userns-volume-probe.yaml @@ -0,0 +1,33 @@ +apiVersion: v1 +kind: Pod +metadata: + name: userns-volume-probe + namespace: userns-rootful-dind +spec: + # This is the feature under test. Do not change it to true or omit it. + hostUsers: false + restartPolicy: Never + nodeSelector: + experiment.coder.com/userns-dind: "true" + containers: + - name: probe + image: public.ecr.aws/docker/library/alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d + command: ["/bin/sh", "-c"] + args: + - | + set -eux + id + echo '--- uid/gid mapping ---' + cat /proc/self/uid_map + cat /proc/self/gid_map + echo '--- PVC write ---' + echo "$(date -u +%FT%TZ) userns-volume-probe" > /workspace/probe.txt + cat /workspace/probe.txt + sleep 600 + volumeMounts: + - name: workspace-data + mountPath: /workspace + volumes: + - name: workspace-data + persistentVolumeClaim: + claimName: workspace-data