Support offline restore on proxy deployments by sjha4 · Pull Request #684 · theforeman/foremanctl

sjha4 · 2026-07-20T20:58:28Z

Why are you introducing these changes? (Problem description, related links)

Support offline restore on proxies. Built on top of offline restore commit hence marking draft.

What are the changes introduced in this pull request?

Adds some proxy specific logic to offline restore to be able to run against proxy deployments.

How to test this pull request

Set up a proxy
Run foremanctl backup /tmp/backup --target-host proxy
Run foremanctl restore /tmp/backup/backup_dir --target-host proxy --force

Checklist

Tests added/updated (if applicable)
Documentation updated (if applicable)

sjha4 · 2026-07-21T18:45:38Z

In first read of the PR, this should work with #611 with this so we can claim proxy backup-restore really brings proxy to backed up state. Otherwise we'll require a complete proxy sync after restore to handle container gateway. I'll test both PRs together and see if we need any updates.

sjha4 · 2026-07-23T17:07:52Z

I did some testing of this with Cheyenne's restore PR and Ian's container_gateway PR with this. Was able to backup container_gateway DB and restore the DB and redeploy proxy with correct features, content and container_gateway. Will keep this in draft till the restore PR goes in so the commits can be cleaned up.

Backup on proxy with pulp and container_gateway

TASK [backup : Display backup completion] ********************************************************************************************************
ok: [localhost] => 
    msg: |-
        Backup completed successfully.
        Location: /tmp/backup/foreman-backup-20260723T130214
        Databases: pulp, container_gateway

Restore validate:

TASK [restore : Verify database dump files exist] ************************************************************************************************
ok: [localhost] => (item=pulp)
ok: [localhost] => (item=container_gateway)

Restore:

TASK [restore : Build database restore configuration] ********************************************************************************************
ok: [localhost] => (item=pulp)
ok: [localhost] => (item=container_gateway)

TASK [restore : Verify dump files exist] *********************************************************************************************************
ok: [localhost] => (item=pulp.dump)
ok: [localhost] => (item=container_gateway.dump)

TASK [restore : Drop existing databases] *********************************************************************************************************
changed: [localhost] => (item=pulp)
changed: [localhost] => (item=container_gateway)

TASK [restore : Create empty databases] **********************************************************************************************************
changed: [localhost] => (item=pulp)
changed: [localhost] => (item=container_gateway)

TASK [restore : Enable postgres_fdw extension for IOP databases] *********************************************************************************
skipping: [localhost] => (item=pulp) 
skipping: [localhost] => (item=container_gateway) 
skipping: [localhost]

TASK [restore : Restore databases from dump files] ***********************************************************************************************
changed: [localhost] => (item=pulp.dump → pulp)
changed: [localhost] => (item=container_gateway.dump → container_gateway)

ehelms · 2026-08-10T15:18:35Z

+  import_playbook: ../deploy-proxy/deploy-proxy.yaml
+  when:
+    - not validate | default(false)
+    - restore_flavor | default('') is match('foreman-proxy.*')


This can work for now, but this will really need to change to be feature based or to have a separate command (e.g. restore-proxy).

+1 for making this deploy and the one below feature-based instead of using flavor. Since enabled_features are informed by flavors and features I'd use them as a source of truth.

ehelms · 2026-08-10T15:20:47Z

+    - not validate | default(false)
+    - restore_flavor | default('') == 'katello'
+
+- name: Verify restore completion


Can we rely on a subset of our existing checks here instead?

We can remove the entire section post deploy cause deploy has these checks built in.

+1 to that. If there are any additional checks needed you should add them as a role to ./checks.

qcjames53

I've been working on testing this playbook all day. Backup / restore seems to work great for empty proxies but I haven't yet had a content backup go through. It's some sort of pulpcore versioning issue I'm working through; hoping to get this addressed ASAP and approve the PR.

qcjames53 · 2026-08-11T21:41:01Z

+    - not validate | default(false)
+    - restore_flavor | default('') == 'katello'
+
+- name: Verify restore completion


+1 to that. If there are any additional checks needed you should add them as a role to ./checks.

qcjames53 · 2026-08-11T21:44:41Z

+  import_playbook: ../deploy-proxy/deploy-proxy.yaml
+  when:
+    - not validate | default(false)
+    - restore_flavor | default('') is match('foreman-proxy.*')


+1 for making this deploy and the one below feature-based instead of using flavor. Since enabled_features are informed by flavors and features I'd use them as a source of truth.

qcjames53

WOW that took a long time to fully test. Two full workdays. You'll have to excuse a little weariness and mental fog in this review :)

I haven't been able to validate proxy restore works at all. I think it's my fault but I want to document what's failed for me:

I set up foreman on quadlet + proxy on another vagrant VM using the foreman-proxy feature. Content sync worked. Backup went through but restore did not function due to some obsah_state issues encountered on restore setup?
I set up foreman and the smart proxy on different RHEL 9 satlab hosts. Foreman was configured regularly and then I created an auth bundle and used ./foremanctl deploy-proxy --auth-bundle ... to register the proxy. Backup would not go through with this proxy. According to Claude it was attempting to back up the foreman and candlepin dbs in addition to pulp and they were not being found on the proxy.

Based on this I have a couple of questions:

Is restore supposed to work on a quadlet-based environment?
Should that satlab proxy backup have gone through?
I don't think --force does anything? It seems to always be enabled when restoring a proxy due to broken obsah handling on the local VM proxy I tried to restore. It might work on a remote proxy but I did not get that far.

Also a few requests:

The postgreql service shutdown is really flakey in src/roles/restore/tasks/prepare_system.yaml and src/roles/backup/tasks/main.yaml. I had to do it manually on 9/10 runs across multiple installs. I know this isn't a part of this PR but could we address both before saying backup/restore is good to go? I'd really want to see this improved.
docs/user/backup-restore.md severely needs reformatted. It is 400 lines of AI-generated filler and it is so hard to find useful info in that file. I think we could get away with like 40 lines of info in that file. Can you please give this an honest try? I'd personally do commands and common params, a BRIEF list of what is backed up, and supported configs like external DBs. We don't need to include anything about compression ratios or tell users they need to rotate the backups. Why does BACKUP_DIR, the most important param, not say anything about it being a remote directory? Please 🥺

qcjames53

Samir, thank you for meeting up yesterday morning to explain the details of proxy backup/restore with me. For future reference, the missing pieces were this:

Foremanctl backup and restore currently requires a single-environment foremanctl client like foreman-maintain. We can't support multi-host foremanctl for backup/restore until obsah can support mutli-host cleanly.
--force is single-environment only for now
Documentation is a lot of carryover from various satellite docs mixed together.

I was able to verify backup/restore of a proxy hosted on a local VM functions great provided you have two foremanctl instances configured (one for the quadlet and one for the proxy VMs). --force works great for overriding an existing install. Restore onto a blank VM also functioned provided Foreman was pre-configured to use that VM as a proxy. I imagine configuring the proxy in Foreman itself would have worked as well. Existing content (+Pulp db) restores okay and new content can sync after restore.

I didn't have much luck getting #611 to work in conjunction with my environment. Nothing related to this PR; I just couldn't get the container gateway feature enabled properly for some reason (It looks like the PR is still in flux, anyways). I'd be happy to revisit this if it's a hard requirement but I'm thinking if something is broken we can fix it down the road.

qcjames53 · 2026-08-14T15:56:08Z

I'd love if you could add some of these critical details to the backup/restore docs but I'm happy to merge as-is.

ehelms

I'll wait to merge to see @sjha4 response to the docs request.

sjha4 · 2026-08-21T16:16:53Z

Had some discussions with @qcjames53 and opened a PR here: #778 which has some documentation about using backup/restore in dev setups. The current backup-restore doc in user docs does not talk about non-localhost deployments which is correct since user installs always map quadlet and proxy to localhost. In dev setups, one has to consider some things I am trying to document here: #778

sjha4 · 2026-08-21T18:48:10Z

Thanks @qcjames53 @ehelms .. This is good to merge now.

pr-processor Bot added the Not yet reviewed label Jul 20, 2026

sjha4 mentioned this pull request Jul 23, 2026

Container Gateway support #611

Merged

2 tasks

sjha4 mentioned this pull request Aug 3, 2026

Add foremanctl restore command - Complete offline backup restore #549

Merged

sjha4 force-pushed the proxy_restore branch from e9641da to afa9e34 Compare August 10, 2026 13:12

sjha4 marked this pull request as ready for review August 10, 2026 13:13

ehelms reviewed Aug 10, 2026

View reviewed changes

pr-processor Bot removed the Not yet reviewed label Aug 10, 2026

ehelms reviewed Aug 10, 2026

View reviewed changes

qcjames53 suggested changes Aug 11, 2026

View reviewed changes

pr-processor Bot added the Waiting on contributor label Aug 11, 2026

Support offline restore on proxy

fb38f7a

sjha4 force-pushed the proxy_restore branch from afa9e34 to fb38f7a Compare August 12, 2026 15:02

pr-processor Bot added Needs re-review and removed Waiting on contributor labels Aug 12, 2026

qcjames53 suggested changes Aug 12, 2026

View reviewed changes

pr-processor Bot added Waiting on contributor and removed Needs re-review labels Aug 12, 2026

qcjames53 approved these changes Aug 14, 2026

View reviewed changes

ehelms approved these changes Aug 20, 2026

View reviewed changes

aneta-petrova mentioned this pull request Aug 21, 2026

Include Smart Proxy backup and restore in containerized theforeman/foreman-documentation#5237

Merged

9 tasks

qcjames53 approved these changes Aug 21, 2026

View reviewed changes

ehelms merged commit 53bf4ca into theforeman:master Aug 23, 2026
16 checks passed

Uh oh!

Conversation

sjha4 commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are you introducing these changes? (Problem description, related links)

What are the changes introduced in this pull request?

How to test this pull request

Checklist

Uh oh!

sjha4 commented Jul 21, 2026

Uh oh!

sjha4 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ehelms Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

qcjames53 Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

ehelms Aug 10, 2026

Choose a reason for hiding this comment

Uh oh!

sjha4 Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

qcjames53 Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

qcjames53 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

qcjames53 Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

qcjames53 Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

qcjames53 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qcjames53 left a comment

Choose a reason for hiding this comment

Uh oh!

qcjames53 commented Aug 14, 2026

Uh oh!

ehelms left a comment

Choose a reason for hiding this comment

Uh oh!

sjha4 commented Aug 21, 2026

Uh oh!

sjha4 commented Aug 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sjha4 commented Jul 20, 2026 •

edited

Loading

sjha4 commented Jul 23, 2026 •

edited

Loading

qcjames53 left a comment •

edited

Loading