Fix TOCTOU crash in rotate_log() under concurrent invocations - #136
Merged
Conversation
rotate_log() checks os.path.exists(log_path) then calls os.rename(), with no exclusion between the two. When multiple obsah invocations share the same log file (the common case), one process can rename the file away between another's exists() check and its own rename(), crashing with an unhandled FileNotFoundError. Confirmed live against the real installed package (obsah 1.10.0, not this patched fork): 150 concurrent `foremanctl auth-bundle` invocations against a shared log file crashed 8 of them this way across several rounds. After suppressing FileNotFoundError around the rename, 120 further concurrent invocations produced zero crashes. Unlike the parameters.yaml read-modify-write fixed in the previous two commits, this doesn't need mutual exclusion: it doesn't matter which process wins the rotation, only that the loser doesn't crash, so a plain contextlib.suppress is sufficient. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ehelms
reviewed
Aug 18, 2026
|
|
||
| def test_rotate_log_tolerates_concurrent_rotation(tmp_path): | ||
| # Another concurrent obsah invocation can rename the same log file away | ||
| # between our exists() check and our own rename. |
Member
There was a problem hiding this comment.
I am not seeing how to connect this comment to the test itself. Maybe drop it?
Contributor
Author
There was a problem hiding this comment.
Fair point — that comment was describing the production race, not this test. The test never starts another process; it patches exists() so the file is already gone when rename() runs. Dropped it.
The test simulates the race by stealing the file from a patched exists() check, not by starting another process, so the old comment was misleading. Co-authored-by: Cursor <cursoragent@cursor.com>
ehelms
approved these changes
Aug 19, 2026
pablomh
added a commit
to redhat-performance/satperf
that referenced
this pull request
Aug 19, 2026
…e to auth-bundle
Bundle of independent foremanctl role improvements, tracked separately
from load-balanced capsule support (not committed here - unconfirmed,
stays local until proven working):
- Split foremanctl_features into foremanctl_add_features/
foremanctl_remove_features, enabling selective feature removal (e.g.
cloud-connector) rather than only addition.
- COPR cleanup: de-nest the EL9 COPR task; EL10 now enables the
official @theforeman/{foreman,katello,plugins}-nightly-staging COPRs
via a loop instead of the old personal ekohl/foreman-nightly-staging
COPR.
- Add a step to apply satellite vendor overrides before deployment.
- Deployment robustness: replace the old ignore_errors/failed_when:
... is failed debug-task pattern with failed_when: false plus an
explicit ansible.builtin.assert on rc == 0, for both the initial
deploy and the add/remove-features deploy.
- Redact the admin password in the initial-deploy debug output - a
real secret-leak fix, since a failed task previously echoed the raw
password via Ansible's own fatal-task output.
- Rename certificate-bundle to auth-bundle, tracking foremanctl's own
CLI rename, and drop the old separate OAuth consumer key/secret file
handling now that the auth bundle covers it. The new "Generate proxy
auth bundle" task's throttle: 1 references theforeman/obsah#136 (a
rotate_log() TOCTOU race under concurrent capsule delegation) - the
same obsah concurrency class as the separately-planned parameters.yaml
locking fix.
- Make foremanctl_deployment_type a required, explicit per-playbook
variable instead of a role default: foremanctl_specific.yaml sets it
to server, and the new foremanctl_capsules_specific.yaml sets it to
proxy for capsule/smart-proxy hosts.
- Restructure proxy flavor selection into _foremanctl_proxy_flavor
(Foreman -> foreman-proxy-content, Satellite -> capsule), replacing
the static foremanctl_proxy_flavor default.
- capsule role: restructure certs-generate so Foreman/Satellite
fact-setting happens inside the certs-generate block, and gate
concurrent vs. sequential certs-generate execution on
sat_version == 'stream' vs. not, replacing ad hoc throttle: 1
"XXX: Submit PR" hacks on the installer-script and
refresh-features tasks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
rotate_log()checksos.path.exists(log_path)and then callsos.rename(log_path, backup_path), with no exclusion between the two steps. When multipleobsah-based CLIs (e.g.foremanctl) run concurrently and share the same log file — a normal situation, since log rotation happens unconditionally at the top ofmain()before any other work — one process can rename the file away in the gap between another process'sexists()check and its ownrename()call, crashing with an unhandledFileNotFoundError.Live reproduction
Confirmed against the real installed package (
obsah1.10.0, unmodified) on a live Satellite host, usingforemanctl auth-bundle(which shares one log file across all invocations): 150 concurrent invocations across several rounds crashed 8 of them with this exact traceback:After applying this fix on the same host, 120 further concurrent invocations produced zero crashes.
Why no locking is needed here
Unlike a genuine read-modify-write (e.g. merging into a shared parameters file), log rotation doesn't need mutual exclusion. Every concurrent process wants the same outcome — "the old log is archived and a fresh one starts" — and that outcome is fully satisfied once any one process's rename succeeds. There's no per-process contribution to preserve, so the fix is simply to tolerate losing the race rather than to coordinate around it.
This is a well-established idiom rather than something new to this codebase:
persist_args(), a few lines away in this same file, already usesos.makedirs(persist_dir, mode=0o770, exist_ok=True)to solve the identical TOCTOU shape for directory creation (exist_ok=Trueexists in the stdlib specifically for this). This PR applies the same idiom toos.rename().lib/ansible/plugins/connection/ssh.py:Testing
Added
test_rotate_log_tolerates_concurrent_rotation, which reconstructs the actual race (not just a raised exception):os.path.exists()is patched so that, on the one call that matters, it returnsTruebut also renames the file away as a side effect — simulating another process winning the race — so the real, unmockedos.rename()call fails naturally. Assertsrotate_log()doesn't raise, that the "stolen" file exists (confirming the simulated race actually happened), and that the original log file is gone (matching the real post-race state).Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com