Skip to content

Avoid redundant foreman restart on re-deploy - #793

Open
ehelms wants to merge 1 commit into
theforeman:masterfrom
ehelms:fix-foreman-redeploy-double-restart
Open

Avoid redundant foreman restart on re-deploy#793
ehelms wants to merge 1 commit into
theforeman:masterfrom
ehelms:fix-foreman-redeploy-double-restart

Conversation

@ehelms

@ehelms ehelms commented Aug 29, 2026

Copy link
Copy Markdown
Member

Why are you introducing these changes? (Problem description, related links)

Refs #769

On re-deploy, the "Restart foreman" step took ~120s (vs ~26s on a fresh
deploy and ~26s for a manual systemctl restart foreman).

Root cause: foreman was restarted twice, ~0.6s apart. The unconditional
state: restarted on foreman-db-migrate (the "Migrate and seed" task)
propagated a restart down to foreman.service via Requires=, starting
foreman (attempt #1). The Restart foreman handler then restarted it again
(attempt #2), sending SIGTERM to the still-booting attempt #1. Because Puma
runs as container PID 1 and has not yet installed its signal traps that early
in boot, the SIGTERM is ignored until TimeoutStopSec (default 90s) kills the
container — adding ~90s to every re-deploy. The same propagation redundantly
bounced the dynflow-sidekiq instances, and also caused foreman downtime on
otherwise no-change deploys (issue #769).

What are the changes introduced in this pull request?

  • Change Requires=foreman-db-migrate.service to Wants= on the foreman and
    dynflow-sidekiq quadlets, so restarting foreman-db-migrate no longer
    propagates an extra restart onto foreman/dynflow.
  • After=foreman-db-migrate.service is kept, so migrations still run before
    foreman on boot. Migration failure is still caught by the explicit "Migrate
    and seed" task, which fails the play before foreman is restarted. The
    Restart foreman handler is now the single, deterministic restart.

Measured impact

Restart foreman task timing on re-deploy:

Time
Re-deploy before fix ~121s (115–118s across runs)
Re-deploy after fix 27.00s
Reduction ~94s (~78% faster on this task)

27s now matches a fresh deploy (26.80s) and a manual
systemctl restart foreman (~26s), i.e. all artificial delay is gone and only
Foreman's real boot time remains. End-to-end this cuts ~94s off the re-deploy
(~380s → 286s, ~25% faster).

How to test this pull request

Steps to reproduce:

  • Deploy: ./foremanctl deploy --foreman-initial-admin-password=changeme --tuning development
  • Re-run the same deploy command (a re-deploy).
  • Observe the "Restart foreman" task timing. Before this change it was ~120s;
    after, it should be ~27s (a single clean restart).
  • Confirm foreman comes up healthy: systemctl status foreman and
    curl -k https://localhost/api/v2/ping.
  • Confirm no state 'stop-sigterm' timed out. Killing. entries appear for
    foreman.service in journalctl -u foreman.service during the re-deploy.

Checklist

  • Tests added/updated (if applicable)
  • Documentation updated (if applicable)

Refs theforeman#769

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant