Switch container networking from host to named bridge networks

development/playbooks/deploy-dev/deploy-dev.yaml

@@ -46,10 +46,25 @@
     - role: systemd_target
     - role: certificates
     - role: postgresql
+      vars:
+        postgresql_network: host
     - role: valkey
+      vars:
+        valkey_network: host
     - role: candlepin
+      vars:
+        candlepin_networks: host
+        candlepin_database_host: localhost
+        candlepin_healthcheck_host: localhost
+        candlepin_tomcat_key: "{{ localhost_key }}"
+        candlepin_tomcat_certificate: "{{ localhost_certificate }}"
     - role: httpd
     - role: pulp
+      vars:
+        pulp_networks: host
+        pulp_migration_networks: host
+        pulp_database_host: localhost
+        pulp_redis_url: "redis://localhost:6379/8"
     - role: foreman_development
       vars:
         foreman_development_oauth_consumer_key: "{{ foreman_oauth_consumer_key }}"

development/playbooks/remote-database/remote-database.yaml

@@ -18,6 +18,8 @@
     - role: pre_install
     - role: certificates
     - role: postgresql
+      vars:
+        postgresql_network: host
 
   tasks:
     - name: Fetch PostgreSQL SSL CA

docs/developer/deployment.md

@@ -441,6 +441,25 @@ The external authentication configuration is managed through `foremanctl` comman
 
 If `hammer` feature is enabled and `--external-authentication` is set to `ipa_with_api`, `hammer` will be configured to use negotiate-based authentication.
 
+## Container networking
+
+Server deployments create four named Podman bridge networks:
+
+- `foreman-db`: internal, isolated network for the internal PostgreSQL container and its clients
+- `foreman-cache`: internal, isolated network for Valkey and its clients
+- `foreman-app`: shared application network for Foreman, Candlepin, and Pulp
+- `foreman-proxy`: smart-proxy network used when the `foreman-proxy` feature is enabled
+
+Services communicate across these bridges by container DNS name instead of `localhost`. The internal database and cache containers are not exposed on host TCP ports. Foreman and Pulp continue to integrate with host `httpd` through the existing systemd socket activation and Unix-socket backends, so the bridge migration does not reintroduce loopback port publishing for those services.
+
+Candlepin is reachable from Foreman as `https://candlepin:23443/candlepin`, so the deployment issues a dedicated certificate for the `candlepin` DNS name and mounts that certificate into the Candlepin container. Foreman validates that hostname using the existing installer CA trust.
+
+The `foreman-proxy` deployment always registers the smart proxy with Foreman under its own real FQDN URL (`foreman_proxy_name`/`foreman_proxy_url`); this identity is unaffected by the bridge-network split. `--registration-url` is a separate, optional override: when set, it is written into the proxy's own `settings.d/registration.yml` as the endpoint the proxy tells newly-registering hosts to actually use (for example, a load-balancer DNS name in front of multiple proxies), so registration traffic really does go wherever it points. What it does *not* do is change how Foreman itself manages the proxy, or require the proxy's own certificate to cover that hostname: foremanctl's deployment automation neither dials that URL nor validates it against any certificate, leaving both concerns (does the URL actually work for hosts, is it certificate-valid on whatever terminates TLS there) to the operator, the same way master does today.
+
+Development deployments intentionally keep `postgresql`, `valkey`, `candlepin`, and `pulp` on `network: host`. In that workflow the Rails process runs directly on the host, so preserving `localhost` endpoints avoids a parallel set of development-only service URL rewrites.
+
+Containers inherit host `/etc/hosts` entries through Podman's `base_hosts_file` setting. This keeps host-only name mappings usable from inside the smart proxy and other containers even when bridge DNS is in use.
+
 ## Deployment architecture
 
 The primary way of deployment is to install `foremanctl` on a system and then let `foremanctl` deploy the various components on the same system.

docs/iop.md

@@ -10,7 +10,7 @@ The `iop` feature depends on `rh-cloud`, which installs the `foreman_rh_cloud` p
 
 ## Architecture
 
-IOP runs as a set of containerized services managed via podman quadlets on the `iop-core-network` (bridge, `10.130.0.0/24`). The gateway is registered as a Foreman smart proxy at `https://localhost:24443`.
+IOP runs as a set of containerized services managed via podman quadlets on the isolated `iop-core-network` (bridge, `10.130.0.0/24`). Foreman stays on the shared `foreman-app` bridge, while the gateway is dual-homed onto both networks and registered as a smart proxy at `https://iop-core-gateway:8443`. The same gateway remains exposed on host loopback at `https://localhost:24443` for host-side access.
 
 ```mermaid
 graph TB
@@ -79,10 +79,10 @@ graph TB
 | puptoo | `iop-core-puptoo` | - | Puppet/system facts processor |
 | yuptoo | `iop-core-yuptoo` | - | Yum/package data processor |
 | engine | `iop-core-engine` | - | Insights rules engine |
-| gateway | `iop-core-gateway` | 127.0.0.1:24443 | nginx proxy, smart proxy relay to Foreman |
+| gateway | `iop-core-gateway` | 8443 (internal), 127.0.0.1:24443 (host loopback) | nginx proxy, smart proxy relay to Foreman |
 | inventory | `iop-core-host-inventory`, `iop-core-host-inventory-api` | 8081 (internal) | Host inventory with MQ consumer and REST API |
 | advisor | `iop-service-advisor-backend-api`, `iop-service-advisor-backend-service` | 8000 (internal) | Advisor recommendations |
-| remediation | `iop-service-remediations-api` | 3000 (host network) | Remediation playbook generation |
+| remediation | `iop-service-remediations-api` | 3000 (internal) | Remediation playbook generation |
 | vmaas | `iop-service-vmaas-reposcan`, `iop-service-vmaas-webapp-go` | - | Vulnerability metadata and advisory sync |
 | vulnerability | 8 containers (manager, taskomatic, grouper, listener, evaluators, vmaas-sync) | 8443 (internal) | Vulnerability assessment pipeline |
 
@@ -96,7 +96,7 @@ Advisor and vulnerability frontend assets are extracted from container images an
 
 ### Databases
 
-IOP creates five PostgreSQL databases, all accessible to containers via `host.containers.internal:5432`:
+IOP creates five PostgreSQL databases. In the supported internal database mode, IoP database clients join the `foreman-db` bridge and reach PostgreSQL as `postgresql:5432`:
 
 | Database | User |
 |----------|------|
@@ -122,7 +122,7 @@ Set in the playbook vars or inventory to match your Foreman deployment:
 
 ### Certificates
 
-Gateway certificates use the default certificate paths:
+Gateway certificates continue to use the default certificate paths. The gateway server certificate is issued for both `localhost` and `iop-core-gateway` so the same endpoint works from the host and from the shared Foreman container network without introducing a separate host-side DNS mapping:
 
 - Server: `/var/lib/foremanctl/certs/certs/localhost.crt`
 - Client: `/var/lib/foremanctl/certs/certs/localhost-client.crt`

src/playbooks/deploy-proxy/deploy-proxy.yaml

@@ -25,6 +25,21 @@
         certificate_checks_certificate: "{{ server_certificate }}"
         certificate_checks_key: "{{ server_key }}"
         certificate_checks_ca: "{{ server_ca_certificate }}"
+    - role: deploy_network
+      vars:
+        deploy_network_name: foreman-db
+        deploy_network_internal: true
+        deploy_network_isolate: true
+      when:
+        - database_mode == 'internal'
+    - role: deploy_network
+      vars:
+        deploy_network_name: foreman-cache
+        deploy_network_internal: true
+        deploy_network_isolate: true
+    - role: deploy_network
+      vars:
+        deploy_network_name: foreman-app
     - role: oauth_from_bundle
     - role: postgresql
       when:

src/playbooks/deploy/deploy.yaml

@@ -26,6 +26,24 @@
         certificate_checks_certificate: "{{ server_certificate }}"
         certificate_checks_key: "{{ server_key }}"
         certificate_checks_ca: "{{ server_ca_certificate }}"
+    - role: deploy_network
+      vars:
+        deploy_network_name: foreman-db
+        deploy_network_internal: true
+        deploy_network_isolate: true
+    - role: deploy_network
+      vars:
+        deploy_network_name: foreman-cache
+        deploy_network_internal: true
+        deploy_network_isolate: true
+    - role: deploy_network
+      vars:
+        deploy_network_name: foreman-app
+    - role: deploy_network
+      vars:
+        deploy_network_name: foreman-proxy
+      when:
+        - "enabled_features | has_feature('foreman-proxy')"
     - role: postgresql
       when:
         - database_mode == 'internal'
@@ -40,6 +58,10 @@
         - "enabled_features | has_feature('iop')"
         - database_mode == 'internal'
     - role: foreman_proxy
+      vars:
+        foreman_proxy_network: foreman-proxy
+        foreman_proxy_ports:
+          - "0.0.0.0:8443:8443"
       when:
         - "enabled_features | has_feature('foreman-proxy')"
     - role: hammer

src/plugins/modules/migrate_answers.py

@@ -113,6 +113,16 @@ def flatten_nested_dict(nested_dict, parent_key=''):
     return dict(items)
 
 
+def normalize_internal_database_host(mapped_config):
+    """Rewrite installer loopback DB hosts to container DNS for internal DB deployments."""
+    if mapped_config.get('database_mode') != 'internal':
+        return
+
+    database_host = mapped_config.get('database_host')
+    if database_host in {'localhost', '127.0.0.1', '::1'}:
+        mapped_config['database_host'] = 'postgresql'
+
+
 def apply_mappings(old_config):
     """
     Transform old config to new format using mapping table.
@@ -152,6 +162,8 @@ def apply_mappings(old_config):
                     param_name = str(old_key)
                 unmappable.append(param_name)
 
+    normalize_internal_database_host(result)
+
     return {
         'mapped': result,
         'unmappable': unmappable

src/roles/backup/tasks/main.yaml

@@ -34,6 +34,14 @@
       ansible.builtin.include_tasks:
         file: preflight.yaml
 
+    - name: Select database access host for backup
+      ansible.builtin.set_fact:
+        backup_database_host: >-
+          {{
+            (backup_database_mode == 'internal')
+            | ternary((postgresql_socket_dir | default('/var/run/postgresql')), database_host)
+          }}
+
     - name: Create timestamped backup directory
       ansible.builtin.file:
         path: "{{ backup_dir_full }}"
@@ -63,7 +71,7 @@
 
     - name: Wait for PostgreSQL readiness
       ansible.builtin.command:
-        cmd: pg_isready -h {{ database_host }} -p {{ database_port }}
+        cmd: pg_isready -h {{ backup_database_host }} -p {{ database_port }}
       register: backup_pg_ready
       retries: "{{ backup_postgresql_ready_retries }}"
       delay: "{{ backup_postgresql_ready_delay }}"
@@ -77,7 +85,7 @@
         db_entry:
           name: "{{ item.name }}"
           database: "{{ item.database }}"
-          host: "{{ database_host }}"
+          host: "{{ backup_database_host }}"
           port: "{{ database_port }}"
           user: "{{ item.user }}"
           password: "{{ item.password }}"
@@ -128,6 +136,9 @@
         name: foreman.target
         state: started
 
+    - name: Wait for Foreman Proxy API to be reachable from Foreman
+      ansible.builtin.include_tasks: ../../foreman_proxy/tasks/wait_for_reachable.yaml
+
     - name: Display backup completion
       ansible.builtin.debug:
         msg: |

src/roles/candlepin/defaults/main.yml

@@ -1,6 +1,9 @@
 ---
 candlepin_ssl_port: 23443
-candlepin_hostname: localhost
+candlepin_hostname: "0.0.0.0"
+candlepin_healthcheck_host: candlepin
+candlepin_networks: >-
+  {{ ((database_mode == 'internal') | ternary(['foreman-db'], [])) + ['foreman-app'] }}
 candlepin_tls_versions:
   - "TLSv1.2"
   - "TLSv1.3"
@@ -13,7 +16,7 @@ candlepin_container_image: quay.io/foreman/candlepin
 candlepin_container_tag: "4.4.14"
 candlepin_secret_mount_opts: "mode=0440,uid=0,gid=53,type=mount"
 
-candlepin_database_host: localhost
+candlepin_database_host: postgresql
 candlepin_database_port: 5432
 candlepin_database_ssl: false
 candlepin_database_ssl_mode: disable

src/roles/candlepin/tasks/main.yml

@@ -74,7 +74,7 @@
     name: "candlepin"
     image: candlepin.image
     state: quadlet
-    network: host
+    network: "{{ candlepin_networks }}"
     hostname: "{{ ansible_facts['hostname'] }}.local"
     secrets:
       - 'candlepin-ca-cert,target=/etc/candlepin/certs/candlepin-ca.crt,{{ candlepin_secret_mount_opts }}'
@@ -97,7 +97,11 @@
         After=valkey.service postgresql.service
         [Service]
         TimeoutStartSec=300
-    healthcheck: curl --fail --insecure --noproxy localhost https://localhost:23443/candlepin/status
+    healthcheck: >-
+      curl --fail --cacert /etc/candlepin/certs/candlepin-ca.crt
+      --noproxy {{ candlepin_healthcheck_host }}
+      --resolve {{ candlepin_healthcheck_host }}:{{ candlepin_ssl_port }}:127.0.0.1
+      https://{{ candlepin_healthcheck_host }}:{{ candlepin_ssl_port }}/candlepin/status
     sdnotify: healthy
 
 - name: Run daemon reload to make Quadlet create the service files

src/roles/certificates/defaults/main.yml

@@ -12,6 +12,7 @@ certificates_output_directory_keys: "{{ certificates_output_directory }}/private
 certificates_output_directory_requests: "{{ certificates_output_directory }}/requests"
 certificates_ca_subject: 'Foreman Self-signed CA'
 certificates_server_aliases: []
+certificates_hostname_aliases: {}
 certificates_algorithm_type: RSA
 certificates_algorithm_size: 4096
 certificates_ca_validity_days: 7300

src/roles/certificates/tasks/issue.yml

@@ -1,7 +1,7 @@
 ---
 - name: Issue server certificate
   when:
-    - (certificates_source != 'custom_server') or (certificates_hostname == 'localhost')
+    - (certificates_source != 'custom_server') or (certificates_hostname in ['localhost', 'candlepin'])
   block:
     - name: 'Create server private key'
       community.crypto.openssl_privatekey:
@@ -22,8 +22,11 @@
         extended_key_usage:
           - serverAuth
       vars:
-        _certificates_extra_sans: "{{ certificates_server_aliases if certificates_hostname != 'localhost' else [] }}"
-        _certificates_desired_server_sans: "{{ ([certificates_hostname] + _certificates_extra_sans) | map('regex_replace', '^', 'DNS:') | list }}"
+        _certificates_extra_sans: >-
+          {{ (certificates_server_aliases if certificates_hostname != 'localhost' else [])
+             + ((certificates_hostname_aliases | default({})).get(certificates_hostname, [])) }}
+        _certificates_desired_server_sans: >-
+          {{ ([certificates_hostname] + _certificates_extra_sans) | unique | map('regex_replace', '^', 'DNS:') | list }}
 
     - name: 'Sign server certificate'
       community.crypto.x509_certificate:

src/roles/check_duplicate_permissions/tasks/main.yaml

@@ -5,7 +5,8 @@
     login_db: "{{ foreman_database_name }}"
     login_user: "{{ foreman_database_user }}"
     login_password: "{{ foreman_database_password }}"
-    login_host: "{{ foreman_database_host }}"
+    login_host: "{{ (database_mode == 'internal') | ternary(omit, foreman_database_host) }}"
+    login_unix_socket: "{{ (database_mode == 'internal') | ternary(postgresql_socket_dir | default('/var/run/postgresql'), omit) }}"
     query: |
       SELECT id, name
       FROM permissions p

src/roles/check_foreman_tasks/tasks/main.yaml

@@ -4,7 +4,8 @@
     login_db: "{{ foreman_database_name }}"
     login_user: "{{ foreman_database_user }}"
     login_password: "{{ foreman_database_password }}"
-    login_host: "{{ foreman_database_host }}"
+    login_host: "{{ (database_mode == 'internal') | ternary(omit, foreman_database_host) }}"
+    login_unix_socket: "{{ (database_mode == 'internal') | ternary(postgresql_socket_dir | default('/var/run/postgresql'), omit) }}"
     query: |
       SELECT count(*) AS count
         FROM foreman_tasks_tasks

src/roles/check_host_facts_count/tasks/main.yaml

@@ -4,7 +4,8 @@
     login_db: "{{ foreman_database_name }}"
     login_user: "{{ foreman_database_user }}"
     login_password: "{{ foreman_database_password }}"
-    login_host: "{{ foreman_database_host }}"
+    login_host: "{{ (database_mode == 'internal') | ternary(omit, foreman_database_host) }}"
+    login_unix_socket: "{{ (database_mode == 'internal') | ternary(postgresql_socket_dir | default('/var/run/postgresql'), omit) }}"
     query: |
       SELECT fact_values.host_id, count(fact_values.id) as count
       FROM fact_values

src/roles/deploy_network/defaults/main.yaml

@@ -0,0 +1,12 @@
+---
+deploy_network_name: foreman-network
+deploy_network_driver: bridge
+deploy_network_internal: false
+deploy_network_isolate: false
+deploy_network_ipv6: false
+
+# Optional explicit network settings. Podman auto-assigns these when omitted.
+# deploy_network_subnet: "10.89.0.0/24"
+# deploy_network_gateway: "10.89.0.1"
+# deploy_network_dns:
+#   - 8.8.8.8

src/roles/deploy_network/tasks/main.yaml

@@ -0,0 +1,3 @@
+---
+- name: Deploy network
+  ansible.builtin.include_tasks: podman.yaml