fix(ci): the vm matrix and selinux jobs, with the real causes - #23
Merged
Conversation
… causes Issue #22 documented these with suspects; the raw log archive gave the actual answers, and both were environment changes dated to the breakage window. THE VM MATRIX: docker refuses to create from a CMD-less image 'Error response from daemon: no command specified'. The ci-kernels images are file carriers with no CMD, and the docker in runner images since 2026-08-23 refuses docker create without a command - all five matrix entries died identically at Fetch the kernel. create never executes the command, so a trailing placeholder word satisfies it. The other docker create in this file targets debian:bookworm-slim, which defines a CMD, and is untouched. SELINUX: one interface, two names, headers that know only one checkmodule: syntax error at token files_runtime_filetrans. Fedora's selinux-policy renamed files_pid_filetrans to files_runtime_filetrans with the /var/run -> /run migration, but the policy-devel headers in BOTH current CI containers define only the pid spelling, and an unknown interface is a hard parse error. m4 ifdef picks whichever the build host defines, so the module now compiles on either side of the rename. This PR's own CI is the proof: rhel.yml and ebpf.yml both trigger on the paths it touches.
…me length The first pass moved each error instead of clearing it, which is progress wearing failure's clothes, and both follow-ups are instructive. selinux: the .te compiled past line 228 and died at 268 - on OUR OWN colony_firewall_stream_connect interface, whose expansion in the .if file calls files_search_runtime. Same rename family, same fix, two more sites. The .if had never been grepped: the caller was searched, the definition was not, and m4 expansion is where the definition's body meets the compiler. vm matrix: the placeholder command got docker past 'no command specified' straight into 'Invalid container name (k)' - the same runner-image docker now rejects single-character container names (the regex in its own error message requires two). kernelsrc, three sites. Two environment breakages stacked on one command line, discovered one per run because each error masks the next.
…g for m4's sake THE VM MATRIX NEVER WORKED, AND THE GUEST FINALLY SAID WHY No filesystem could mount root, tried: vfat msdos iso9660 The pinned ci-kernels have no ext4 at all. The workflow's comment claimed 'virtio_blk + ext4 - both built in'; nobody had checked, and no run of this matrix has ever been green - issue #22's 'broken since Aug 25' timeline was this author's own unverified inference. The rootfs is now a full cpio initramfs: the kernel unpacks it into ramfs and runs /init, which requires no filesystem driver whatsoever. The earlier revision rejected 'an initramfs' believing it implied 9p/virtiofs modules - backwards: a self-contained initramfs is what removes every driver requirement. Layer three of the docker create saga stands: placeholder command, longer container name. SELINUX: THE CLEVER FIX BROKE M4 ITSELF m4:colony_firewall.te:250: non-numeric argument to builtin 'incr' Wrapping an interface call in ifdef arguments breaks refpolicy's line-counting machinery. Both CI containers define the classic pid spelling, so the .te and .if now just use it - files_pid_filetrans, files_search_pids - with a note saying why. If a future policy drops the compat spelling the error will name the line plainly, which beats clever.
…ing to headers VM: the register dump told the whole story. A docker-export rootfs has no device nodes, so /init ran with no stdout - an entire round executed invisibly - and then died at 127 on poweroff, which bookworm-slim does not ship. devtmpfs is now mounted first and the console re-exec'd before anything else prints; and exiting IS the shutdown: panic=1 reboots when PID 1 dies, -no-reboot turns that into a qemu exit, and the CFC_EXIT marker is already out by then. selinux: three rounds, three different failures, all inside interface machinery this module does not control - runtime spelling unknown, pid spelling's deprecation shim leaking file:line literals into the stream, ifdef breaking m4's line counter. The module now uses raw allow and type_transition statements against var_run_t, which is a *type*, aliased across every rename, and owes nothing to any header's health. A diagnostic grep of files.if ships with the job so a fifth round, if any, starts from ground truth instead of another guess.
…o m4
The console fix paid off immediately: the guest now narrates its own
failure. Every mount succeeds, the marker prints, init exits cleanly - and
the test binary cannot load: built on the ubuntu-noble runner against
glibc 2.39, refused by bookworm's 2.36. trixie-slim's glibc covers it; the
guest userland must be at least as new as the build host.
And the fourth selinux failure was mine, said plainly by the error: my
multi-line gen_require unbalanced m4's quoting. Round five removes m4 from
the equation entirely - a native require { type var_run_t; } block, which
is checkmodule syntax, not macro expansion. The ground-truth grep shipped
last round stays: if a sixth failure exists, it will name itself.
… raised selinux: the wreckage of rounds three through five - deprecation shims apparently leaking file:line literals, gen_require unbalancing m4 - was one bug, and it was mine: refpolicy disables the m4 comment character, so inside the backtick quotes of an interface body an apostrophe terminates the quote. My comments said the te-apostrophe-s, twice. Every exotic symptom downstream was quoting poisoned at that character. The comments are reworded, the raw statements stay - they are proven independent of header health - and a warning comment now stands where the trap is. vm: the test finally ran and failed honestly - ENETUNREACH sending to 127.0.0.1, because loopback was never up: trixie-slim ships neither ip nor ifconfig and the init line failed into its own || true. A static busybox from the runner brings lo up and owes the guest libc nothing.
The 5.10 guest finally ran the suite and failed honestly: the test asserted verified_insns is never empty, but bpf_prog_info.verified_insns only exists since kernel 5.16 - verifier_cost() even documents returning None on older kernels, two screens above the assertion that ignored it. The assertion now checks the running kernel first, so it still catches the loader silently losing the counts on kernels that do provide them. Also in that guest log, a note read '... enforcement cannot be pinned and will stop when this daemon does; it will stop when this daemon does': the inner bpffs error and its wrapper each appended the same consequence. The inner error now states the fact only; the wrapper owns the consequence for every error path, not just that one.
… overrule it On 5.10 the suite now passes inside the guest (CFC_EXIT=0), and the job still failed: under set -euo pipefail the step died on qemu's own exit code. The shutdown is a deliberate panic - init exits under panic=1 and -no-reboot turns the reset into a qemu exit - and how qemu reports that reset varies by guest kernel: the 6.x reboot path exits 0, the 5.10 one exits 1, for the same healthy shutdown. The step already refused to trust qemu on the other side (a boot failure must not read as a pass, hence the marker); now it also refuses on this side. A hang or a crashed qemu is still caught: either way the marker never got printed.
…ug, second home On 5.10 the suite passes, qemu exits 0, and the step still died with no message: the step-summary block greps guest.log for verified_insns lines, kernels before 5.16 report none, grep exits 1 on no matches, and under pipefail as the last command of the brace group that took the whole step down two lines before exit CODE. This is the identical assumption the round-7 test fix removed, living in the workflow instead of the test: no counts is what an old kernel reporting honestly looks like. Swept the workflows for the class: every other unguarded grep either always matches or is a deliberate assertion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #22, closes #12.
The raw log archive gave what the interleaved live logs hid: the vm matrix dies on
no command specified(docker in runner images since Aug 23 refusescreateon CMD-less images — ci-kernels are file carriers), and selinux dies because both CI containers' policy-devel headers definefiles_pid_filetranswhile the .te used the post-renamefiles_runtime_filetransspelling — an unknown interface is a hard parse error. One placeholder word and one m4 ifdef. This PR's own CI exercises both workflows.