Add cloud-connector as a native foremanctl feature

[jeremylenz]

Summary

• Re-implements the upstream satellite_operations.cloud_connector role natively in foremanctl (SAT-45966 / SAT-44641)
• New cloud-connector feature gated behind --add-feature cloud-connector, with rh-cloud dependency
• Installs rhc and yggdrasil-worker-forwarder, configures the worker, starts rhcd, sets rhc_instance_id via the Foreman API, and announces to Sources
• Works with both foremanctl deploy and forge deploy-dev
• Uses OAuth for all Foreman API calls during setup, matching the pattern used by other roles (foreman_proxy, iop_core, pulp, backup)
• Creates a dedicated cloud_connector_user service user with limited dispatch_cloud_requests permission for runtime worker authentication
• Adds a reusable foremanctl_api custom module for OAuth-authenticated API calls where no FAM module exists
• Early pre-checks validate consumer certificate presence and mutual exclusion with the iop feature
• Adds Foreman CA to system trust store so the worker binary can verify Foreman's certificate
• Optional --cloud-connector-http-proxy flag for environments without direct internet access

Companion PRs:

theforeman/foreman_rh_cloud#1214 (adds the announce_to_sources API endpoint)
theforeman/yggdrasil-worker-forwarder#5 (allows specifying a certificate to use)

Test plan

• foremanctl deploy --add-feature cloud-connector completes successfully
• foremanctl features --list-enabled includes cloud-connector
• systemctl status rhcd shows active/running
• /etc/rhc/workers/foreman_rh_cloud.toml has correct content with service user credentials
• hammer settings info --name rhc_instance_id shows the consumer cert CN
• hammer user info --login cloud_connector_user shows the service user with Cloud Connector role
• Second foremanctl deploy is idempotent
• forge deploy-dev --add-feature cloud-connector completes successfully
• Enabling both cloud-connector and iop fails early with a clear error

🤖 Generated with Claude Code

[ehelms]

This is the part we still need to work out how to properly handle, that is, the authentication.

[jeremylenz]

As I see it our options are
1 - Keep things as they are here, feeding admin/changeme to rhc - Not acceptable because the admin password ends up in the worker config.toml.
2 - Create a service user with limited permissions - Doable in the existing PRs here and in foreman_rh_cloud.
3 - Create a service user + personal access token and use that, same as the previous architecture - FAM doesn't have a module for personal access tokens, so this would be a fair bit of work

I think Option 2 seems promising and am pushing that update now.

[jeremylenz]

@ehelms Updated to use OAuth, but we still need the service user because yggdrasil-worker-forwarder only supports HTTP basic auth, and doesn't support OAuth.

[jeremylenz]

Pushed updates addressing review feedback (1bfe640):

• Handler: switched from ansible.builtin.service to ansible.builtin.systemd_service since we use daemon_reload
• Workers directory: removed the task — confirmed rhc package on EL9 creates /etc/rhc/workers/
• Settings API: replaced raw uri calls with theforeman.foreman.setting module for rhc_instance_id and allow_auto_inventory_upload
• Lint fix: added noqa: no-static-secrets on the defaults password (same pattern as foreman_development)

[jeremylenz]

rebased & fixed conflicts

[qcjames53]

I'm not enough in the loop to give this a full review but had a few comments that might be helpful. Consider this 80% good points 20% ass-talking. Here's what I was able to test directly:

• The iop conflict and missing consumer cert checks both work on the positive and negative cases (used manual subscription manager commands for the latter).
• The yggdrasil-worker-forwarder check works in the negative (was unable to figure out package install on centos9).
• The cloud connector role works.

Beyond these tests, I have questions about the whole approach of cloud connector even after reading the planning docs and such.

1. How are we splitting RHEL and Centos 9 deploys? I think the differentiation needs to be clearer since (best I can tell) yggdrasil-worker-forwarder is RHEL only. If the packaged container images are centos9 and the rpms needed for cloud connector are RHEL-only, that seems like a big problem to me!
2. deploy-dev needs to work with cloud connector. At minimum a procedure but ideally out of the box. I think we really need a procedure for setting up a centos9 deploy-dev environment to build/test cloud connector. If that's not possible, then we need a good procedure for setting up a RHEL environment as deploy-dev.
3. If it's possible to pass in subscription manager credentials safely, I would greatly prefer if we could deploy an environment with cloud connector immediately instead of requiring manual registration for the quadlet VM, THEN adding the feature. I understand this won't be an issue for customers but this would be HUGE for testing purposes.

I have to gracefully bow out of reviewing this for a few reasons: capacity + artemis stuff + this baby could come any day! Sorry to not complete testing but I put in an honest effort.

[jeremylenz]

I'll respond with thoughts off the top of my head :)

1. How are we splitting RHEL and Centos 9 deploys? I think the differentiation needs to be clearer since (best I can tell) yggdrasil-worker-forwarder is RHEL only. If the packaged container images are centos9 and the rpms needed for cloud connector are RHEL-only, that seems like a big problem to me!

In order for cloud connector to work you need available on your base system:

1. ansible-runner command - Downstream, from the Satellite repo, e.g. satellite-6.19-for-rhel-9-x86_64-rpms. Upstream, from the Foreman Plugins repo, e.g. https://yum.theforeman.org/plugins/3.19/el9/x86_64/
2. yggdrasil-worker-forwarder - Downstream, from the Satellite repo. Upstream, from the Foreman Plugins repo, e.g. https://yum.theforeman.org/plugins/3.19/el9/x86_64/
3. rhc - from AppStream (CentOS or RHEL)

In the previous architecture, these weren't even listed as prerequisites, even in upstream docs. Downstream, you'd already have all of these since the Satellite system must be registered and have those repos enabled. But upstream, anything can happen. It seems this is a documentation gap. cc @Lennonka

2. deploy-dev needs to work with cloud connector. At minimum a procedure but ideally out of the box. I think we really need a procedure for setting up a centos9 deploy-dev environment to build/test cloud connector. If that's not possible, then we need a good procedure for setting up a RHEL environment as deploy-dev.

I was testing with deploy-dev on RHEL (with --target-host localhost). I think it works fine, as long as you have the repos available that I mentioned above.

[jeremylenz]

@qcjames53 Addressed 2 comments and replied to the third

[lfu]

Tested cloud connector feature in development mode. All core functionality works correctly. Found 3 necessary configuration changes for development environment.

---

🔍 Testing Methodology

Environment: Development mode using forge deploy-dev with Rails source code
Test Scope:

• Cloud connector deployment
• Service user authentication
• API endpoint functionality
• Worker configuration

🔗 Related/Dependent PRs

Foreman RH Cloud Plugin

• PR: Repurpose CloudConnectorAnnounceTask for daily Sources registration foreman_rh_cloud#1214
• Purpose: Cloud connector service user and role support
• Status: Pending
• Dependency: Required for cloud connector service user authentication functionality

Foremanctl Branch Preservation

• PR: Add --preserve-plugin-branches flag to deploy-dev #568
• Purpose: Preserve plugin branches during development deployment
• Status: Pending
• Dependency: Needed for --preserve-plugin-branches flag used in testing

Note: These PRs should be merged before or alongside this cloud connector feature PR.

---

✅ What Works

1. Cloud Connector Deployment

• ✓ Cloud connector role created successfully
• ✓ Required packages present (rhc, yggdrasil-worker-forwarder) - installed as prerequisite
• ✓ Worker config file created: /etc/rhc/workers/foreman_rh_cloud.toml
• ✓ rhcd service running and enabled

Note: The cloud connector role does not install packages - it assumes rhc and yggdrasil-worker-forwarder are already present (installed manually or by prerequisite roles).

2. Service User Authentication

Created service user cloud_connector_user with proper credentials:

User ID: 5
Role: Cloud Connector
Auth Source: Internal
Last Login: Successfully authenticated via API

3. API Endpoint Verification

Manual test of /api/v2/rh_cloud/cloud_request endpoint:

Request:

curl -u cloud_connector_user:$PASSWORD \
  -X POST \
  http://localhost:3000/api/v2/rh_cloud/cloud_request \
  -H 'Content-Type: application/json' \
  -d '{"directive":"foreman_rh_cloud", ...}'

Result:

✓ Authenticated user cloud_connector_user against INTERNAL authentication source
✓ Authorized user cloud_connector_user(cloud_connector_user)
✓ Request processed (playbook URL validation occurred as expected)

4. Configuration Validation

• ✓ rhc_instance_id setting configured: 5fc35187-ee43-48a8-b61a-01c0153d0562
• ✓ Worker URL correct: http://localhost:3000/api/v2/rh_cloud/cloud_request
• ✓ Credentials properly stored in worker config

---

🔧 Required Code Changes

Found 3 necessary changes for development environment support:

Change 1: Development URL Configuration

File: development/playbooks/deploy-dev/deploy-dev.yaml

Issue: Development uses Rails dev server on port 3000, not containerized Foreman

Fix:

vars:
  cloud_connector_url: "http://localhost:3000"
  cloud_connector_admin_user: "{{ foreman_development_admin_user }}"
  cloud_connector_admin_password: "{{ foreman_development_admin_password }}"

Why: Points worker to the correct development Rails server endpoint

---

Change 2: Disable Certificate Validation for HTTP

File: src/roles/cloud_connector/tasks/main.yaml

Issue: Development uses HTTP (no SSL), certificate validation fails

Fix:

- name: Create cloud connector service user
  theforeman.foreman.user:
    # ... existing parameters ...
    validate_certs: false  # <-- Add this

Why: HTTP connections don't have valid certificates in development

---

Change 3: Make CA Certificate Path Optional

File: src/roles/cloud_connector/tasks/main.yaml

Issue: No CA certificate exists for HTTP connections, undefined variable error

Fix:

- name: Create cloud connector service user
  theforeman.foreman.user:
    # ... existing parameters ...
    ca_path: "{{ foreman_ca_certificate | default(omit) }}"  # <-- Change this

Why: Prevents Ansible error when foreman_ca_certificate is undefined (HTTP mode)

---

⚠️ Known Limitations

Automated pytest Suite Skipped

Issue: All 6 cloud connector tests were skipped:

tests/cloud_connector_test.py::test_rhc_package_installed SKIPPED
tests/cloud_connector_test.py::test_yggdrasil_worker_forwarder_package_installed SKIPPED
tests/cloud_connector_test.py::test_workers_directory_exists SKIPPED
tests/cloud_connector_test.py::test_worker_config_exists SKIPPED
tests/cloud_connector_test.py::test_worker_script_exists SKIPPED
tests/cloud_connector_test.py::test_rhcd_service_running SKIPPED

Root Cause:

• pytest hardcoded to check ./foremanctl features (production state)
• Development uses ./forge deploy-dev (separate state)
• Feature flags from --add-feature are temporary, not persisted

Impact:

• ❌ Automated test suite cannot verify development deployments
• ✅ Manual verification confirms all test criteria pass

---

Testing completed with: Claude Code

[jeremylenz]

Re: Change 1 - cloud_connector_url defaults to foreman_url which is https://{{ fqdn }},
and httpd proxies to the Rails server. The HTTPS URL through httpd works fine in dev — we tested it. Using http://localhost:3000 directly would bypass httpd and also break the ca_path on
all the FAM module calls. The worker TOML would also have an HTTP URL, which means the runtime cloud requests would go over unencrypted HTTP.

Change 2 is not necessary unless I were to do Change 1, which I think is not needed for the reason above.

Updating now with Change 3.

[jeremylenz]

Btw, Our role does install the packages — line 2-7 of tasks/main.yaml:

  - name: Install rhc and yggdrasil-worker-forwarder
    ansible.builtin.package:
      name:
        - rhc
        - yggdrasil-worker-forwarder

[jeremylenz]

/packit copr-build

[ekohl]

Marking as request changes because I think this needs some fundamental things:

• Stop using theforeman.foreman.setting
• Resolve the registration task discussion

[jeremylenz]

Marking as request changes because I think this needs some fundamental things:

• Stop using theforeman.foreman.setting
• Resolve the registration task discussion

This no longer uses theforeman.foreman.setting so that part is resolved IMO. See comment above

Added a new commit with the purpose-built module for announce to sources. Tested again and deploy succeeds, so I'm happy with it if you are.

[jeremylenz]

rebased

[jeremylenz]

🍏

[jeremylenz]

@ekohl Are your concerns addressed for now? @ehelms Can we get this in soon? I'll be away starting 5 September so would like to get it merged before then; also having it in will help us with the DBUS rewrite of yggdrasil-worker-forwarder.

[jeremylenz]

• Stop using theforeman.foreman.setting

I realized this isn't entirely accurate - we did stop using it to set allow_auto_inventory_upload, but we still use it to set rhc_instance_id. However, rhc_instance_id isn't really a user preference; it's derived from the consumer certificate and is overwritten by foremanctl on every deploy. So I think that satisfies the "controlled by foremanctl" requirement.

[jeremylenz]

Found a way to make it even better, stand by

[jeremylenz]

@ekohl Okay, found a way to not use theforeman.foreman.setting module at all (see the latest commit). 👍