hack: don't let artifact tar warnings fail otherwise-green CI runs - #4345
hack: don't let artifact tar warnings fail otherwise-green CI runs#4345pujitha24 wants to merge 1 commit into
Conversation
Motivation:
CI post-steps sometimes fail with:
tar: kcp/audit.log: file changed as we read it
tar: kcp: Cannot rmdir: Directory not empty
tar: Exiting with failure status due to previous errors
in hack/run-with-prow.sh, even though the actual test run passed. This
happens because processes started during the test (e.g. kcp servers)
can still be shutting down and writing to their log/audit files at the
moment `tar cjf artifacts.tar.bz2 --remove-files *` runs. tar then
exits non-zero (either just a warning about the changed file, or a
harder failure when it leaves a file behind and the subsequent rmdir
of its directory fails). Because the whole script runs under
`set -o errexit`, that non-zero exit aborts the script immediately,
before it ever reaches the final `exit "${EXIT_CODE}"` line that
reports the real test result - so a passing test run is reported to
CI as failed, purely due to this artifact-packaging race.
Approach:
Wrap the tar invocation the same way the script already wraps the
test command a few lines above: temporarily disable `errexit`, capture
tar's exit code, and print a warning if it's non-zero instead of
letting it abort the script. The script always proceeds to package
what it can and exit with the real test EXIT_CODE, consistent with how
the file already treats test-command failures (recorded, not used to
short-circuit the script).
Validation:
This is a CI shell script with no existing unit tests and no way to
spin up real Prow infrastructure locally, so this can't be reproduced
end-to-end in this environment. What was run and verified:
- `bash -n hack/run-with-prow.sh` - passes, no syntax errors.
- `shellcheck hack/run-with-prow.sh` - produces exactly one pre-existing
SC2035 info-level note on the (unmodified) `tar` glob line; the change
introduces zero new findings.
- Manually traced the errexit/exit-code control flow: previously, any
non-zero tar exit under `set -o errexit` aborted the script before
reaching `exit "${EXIT_CODE}"`; with this change, errexit is
suspended around the tar call, so execution always reaches
`echo 'Done compressing files.'` and then `exit "${EXIT_CODE}"`,
which carries the real test result.
Report: kcp-dev#4343
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Assisted-by: claude-sonnet-5 (via Claude Code)
kcp-ci-bot
added
release-note-none
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kcp-ci-bot
added
size/S
|
Hi @pujitha24. Thanks for your PR. I'm waiting for a kcp-dev member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Why can't we just wait for all processes to exit? |
|
Good question, so I actually went and checked. The command run-with-prow.sh runs directly ( The race comes from the shared/sharded flows ( So "wait for all processes to exit" is the right fix, but it belongs in those two |
3 participants
Summary
Prevent
hack/run-with-prow.shfrom reporting a CI job as failed when thepost-test artifact-compression step hits a non-fatal
tarerror caused bylingering test processes still flushing log/audit files.
What Type of PR Is This?
/kind bug
Related Issue(s)
Fixes #
Release Notes
Fixes #4343