Our C++ and Go services all run on docker.io/library/ubuntu via _create_oci_image in bazel/rules/oci.bzl. Distroless would be smaller and would remove a shell and a package manager from production. The thing standing in the way is healthchecks.
Why it is blocked today
Five services check themselves by shelling out. deploy/consolidated/compose.yaml, verbatim:
test: ["CMD", "timeout", "4", "bash", "-c",
'exec 3<>/dev/tcp/127.0.0.1/8083 && printf "GET /health HTTP/1.0\r\n\r\n" >&3 && head -1 <&3 | grep -q " 200"']That is bash, timeout, head and grep — and /dev/tcp, which is a bash builtin, not a file. Distroless has none of them. Docker's HEALTHCHECK needs an executable inside the container, so the answer is a small compiled binary rather than a shell pipeline.
What to build
1. A healthcheck binary. Connects to a port, optionally issues GET /health, checks for a 200, exits 0 or 1, handles its own timeout. Statically linked so it drops into any image. Go is the path of least resistance here — the repo already cross-compiles Go for images (linux_oci_go) and a static Go binary needs nothing at runtime.
2. A base parameter on linux_amd64_oci_binary / linux_oci_go, so services move over one at a time instead of in a big bang. distroless_base is already pulled and pinned in bazel/oci.MODULE.bazel and is currently used by nothing.
3. Compose healthchecks rewritten to test: ["CMD", "/healthcheck", "-port", "8083"] per service, as each moves.
Verified, so nobody has to re-check
Against the pinned distroless_base digest already in the repo:
- ships
./etc/ssl/certs/ca-certificates.crt
- ships
ld-linux-x86-64.so.2, libc.so.6, libm.so.6 — which is exactly what one_d4_worker links against, libstdc++ being static
- ships
etc/nsswitch.conf, libnss_dns.so.2, libnss_files.so.2, libresolv.so.2, so glibc DNS resolution works — this is the usual distroless trap and distroless/base is not subject to it
Start with one_d4_worker
It is the only service with no healthcheck and no ingress, so it can move before the binary exists and prove the base out at zero cost. Every other service needs step 1 first.
Note what is not a reason to do this: the missing CA bundle that broke the C++ worker's chess.com calls is fixed separately by adding the bundle as a layer, keeping ubuntu. This issue is about image size and attack surface, not that outage.
Cost worth stating
No shell in the container means no docker exec … bash for debugging. For a worker whose only interfaces are its log and Postgres that is cheap; for a service someone pokes at during an incident it is a real loss, and worth deciding per service rather than globally.
Our C++ and Go services all run on
docker.io/library/ubuntuvia_create_oci_imageinbazel/rules/oci.bzl. Distroless would be smaller and would remove a shell and a package manager from production. The thing standing in the way is healthchecks.Why it is blocked today
Five services check themselves by shelling out.
deploy/consolidated/compose.yaml, verbatim:That is
bash,timeout,headandgrep— and/dev/tcp, which is a bash builtin, not a file. Distroless has none of them. Docker'sHEALTHCHECKneeds an executable inside the container, so the answer is a small compiled binary rather than a shell pipeline.What to build
1. A healthcheck binary. Connects to a port, optionally issues
GET /health, checks for a 200, exits 0 or 1, handles its own timeout. Statically linked so it drops into any image. Go is the path of least resistance here — the repo already cross-compiles Go for images (linux_oci_go) and a static Go binary needs nothing at runtime.2. A
baseparameter onlinux_amd64_oci_binary/linux_oci_go, so services move over one at a time instead of in a big bang.distroless_baseis already pulled and pinned inbazel/oci.MODULE.bazeland is currently used by nothing.3. Compose healthchecks rewritten to
test: ["CMD", "/healthcheck", "-port", "8083"]per service, as each moves.Verified, so nobody has to re-check
Against the pinned
distroless_basedigest already in the repo:./etc/ssl/certs/ca-certificates.crtld-linux-x86-64.so.2,libc.so.6,libm.so.6— which is exactly whatone_d4_workerlinks against, libstdc++ being staticetc/nsswitch.conf,libnss_dns.so.2,libnss_files.so.2,libresolv.so.2, so glibc DNS resolution works — this is the usual distroless trap and distroless/base is not subject to itStart with one_d4_worker
It is the only service with no healthcheck and no ingress, so it can move before the binary exists and prove the base out at zero cost. Every other service needs step 1 first.
Note what is not a reason to do this: the missing CA bundle that broke the C++ worker's chess.com calls is fixed separately by adding the bundle as a layer, keeping ubuntu. This issue is about image size and attack surface, not that outage.
Cost worth stating
No shell in the container means no
docker exec … bashfor debugging. For a worker whose only interfaces are its log and Postgres that is cheap; for a service someone pokes at during an incident it is a real loss, and worth deciding per service rather than globally.