[caliper] orchestration: export: make the workspace config mandatory by kpouget · Pull Request #107 · openshift-psap/forge

kpouget · 2026-07-02T10:34:12Z

Summary by CodeRabbit

Bug Fixes
- Export runs now require a workspace to be explicitly set, preventing unexpected use of a default.
- The verbose export output now shows the actual workspace value instead of a generic fallback.
Configuration
- Added a workspace setting for the export backend so exports use the intended destination by default.

openshift-ci · 2026-07-02T10:34:17Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sjmonson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2026-07-02T10:34:44Z

Warning

Review limit reached

@kpouget, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 3 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9928a21f-53be-4fa0-b121-77ae0dda95cc

📥 Commits

Reviewing files that changed from the base of the PR and between a73ecb1 and a79e2b7.

📒 Files selected for processing (3)

projects/caliper/orchestration/export.py
projects/fournos_launcher/orchestration/submit.py
projects/rhaiis/orchestration/config.yaml

📝 Walkthrough

Walkthrough

The multi-run export path in export.py now requires a non-empty workspace value from the merged MLflow config, raising a ValueError if absent, and prints the raw workspace value in verbose output. The rhaiis config.yaml adds a workspace value for the MLflow export backend.

Changes

Mandatory MLflow Workspace

Layer / File(s)	Summary
Enforce and configure required workspace `projects/caliper/orchestration/export.py`, `projects/rhaiis/orchestration/config.yaml`	Adds a ValueError when workspace is missing in multi-run export, removes the `'(default)'` fallback in verbose output, and sets `workspace: forge-rhaiis` in the rhaiis MLflow export config.

Estimated code review effort: 1 (Trivial) | ~3 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: making the export workspace configuration mandatory.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@projects/caliper/orchestration/export.py`:
- Around line 338-340: The workspace validation in _run_multi_run_export raises
a bare ValueError before the existing CLI error handling kicks in, so replace it
with the same user-facing pattern used elsewhere in the export flow (for example
in artifacts_export_run.py): emit the message with click.echo(..., err=True) and
return a non-zero status instead of letting the exception escape. Use
_run_multi_run_export and the workspace check to locate the change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 21f0f4a6-0874-4001-8af8-077f11d397cd

📥 Commits

Reviewing files that changed from the base of the PR and between 48ac14c and a73ecb1.

📒 Files selected for processing (2)

projects/caliper/orchestration/export.py
projects/rhaiis/orchestration/config.yaml

kpouget · 2026-07-02T11:01:32Z

/test fournos skeleton collect_cluster_info
/cluster psap-mgmt
/var caliper.export.backend.mlflow.config.workspace: null

psap-forge-bot · 2026-07-02T11:03:58Z

🟢 Execution of `skeleton` collect_cluster_info 🟢

Execution Engine Configuration

forge:
  args:
  - collect_cluster_info
  configOverrides: {}
  project: skeleton

Artifact Links

Test Logs

✅ 00 Preflight `2 seconds`

✅ 01 Test `12 seconds`

Test Description

This test executes the skeleton project with the collect_cluster_info flag enabled to gather cluster metadata and infrastructure details.

🔄 02 Export-Artifacts

psap-forge-bot · 2026-07-02T11:04:14Z

🟢 Submission of skeleton collect_cluster_info succeeded after 1 minute, 47 seconds 🟢

/test fournos skeleton collect_cluster_info
/var caliper.export.backend.mlflow.config.workspace: null
/cluster psap-mgmt

psap-forge-bot · 2026-07-02T11:13:03Z

🟢 Execution of `skeleton` collect_cluster_info 🟢

Execution Engine Configuration

forge:
  args:
  - collect_cluster_info
  configOverrides: {}
  project: skeleton

Artifact Links

Test Logs

✅ 00 Preflight `0 seconds`

✅ 01 Test `12 seconds`

Test Description

This test executes the skeleton project with the collect_cluster_info preset enabled to validate cluster information collection functionality. It configures the run with 30 max requests, a rate of 1, and 128 prompt tokens, while setting the test duration to 0 seconds to focus on metadata gathering rather than performance load testing.

🔄 02 Export-Artifacts

kpouget · 2026-07-02T11:16:43Z

/test fournos skeleton collect_cluster_info
/cluster psap-mgmt
/var caliper.export.backend.mlflow.config.workspace: null

psap-forge-bot · 2026-07-02T11:18:54Z

🟢 Execution of `skeleton` collect_cluster_info 🟢

Execution Engine Configuration

forge:
  args:
  - collect_cluster_info
  configOverrides: {}
  project: skeleton

Artifact Links

Test Logs

✅ 00 Preflight `0 seconds`

✅ 01 Test `13 seconds`

Test Description

This test executes the skeleton project with the collect_cluster_info preset enabled to verify the system's ability to gather cluster metadata and status information. It isolates this collection functionality by setting skeleton.collect_cluster_info: true without applying load testing variations.

🔄 02 Export-Artifacts

psap-forge-bot · 2026-07-02T11:19:10Z

🟢 Submission of skeleton collect_cluster_info succeeded after 1 minute, 36 seconds 🟢

/test fournos skeleton collect_cluster_info
/var caliper.export.backend.mlflow.config.workspace: null
/cluster psap-mgmt

kpouget · 2026-07-02T11:19:30Z

/test fournos skeleton collect_cluster_info
/var caliper.export.backend.mlflow.config.workspace: null
/cluster psap-mgmt

psap-forge-bot · 2026-07-02T11:21:29Z

🟢 Execution of `skeleton` collect_cluster_info 🟢

Execution Engine Configuration

forge:
  args:
  - collect_cluster_info
  configOverrides: {}
  project: skeleton

Artifact Links

Test Logs

✅ 00 Preflight `0 seconds`

✅ 01 Test `13 seconds`

Test Description

This test executes the skeleton project with the collect_cluster_info preset enabled to gather cluster metrics and configuration details. It applies a low request rate and minimal load parameters to focus specifically on data collection rather than performance benchmarking.

🔄 02 Export-Artifacts

psap-forge-bot · 2026-07-02T11:21:43Z

🟢 Submission of skeleton collect_cluster_info succeeded after 1 minute, 26 seconds 🟢

/test fournos skeleton collect_cluster_info
/var caliper.export.backend.mlflow.config.workspace: null
/cluster psap-mgmt

kpouget · 2026-07-02T11:22:53Z

/test fournos skeleton collect_cluster_info
/var caliper.export.backend.mlflow.config.workspace: null
/cluster psap-mgmt

psap-forge-bot · 2026-07-02T11:24:37Z

🟢 Execution of `skeleton` collect_cluster_info 🟢

Execution Engine Configuration

forge:
  args:
  - collect_cluster_info
  configOverrides:
    caliper.export.backend.mlflow.config.workspace: {}
  project: skeleton

Artifact Links: No direct links available

psap-forge-bot · 2026-07-02T11:24:54Z

🔴 Submission of skeleton collect_cluster_info failed after 1 minute, 16 seconds 🔴

/test fournos skeleton collect_cluster_info
/var caliper.export.backend.mlflow.config.workspace: null
/cluster psap-mgmt

kpouget · 2026-07-02T11:26:34Z

/test fournos skeleton collect_cluster_info

psap-forge-bot · 2026-07-02T11:27:06Z

🔴 Submission of skeleton collect_cluster_info failed after 0 seconds 🔴

Error: ValueError: cluster.name must be configured in config.yaml - cannot submit job without target cluster

OCPCI Test results
Task logs
Execution logs
Test results not available

/test fournos skeleton collect_cluster_info

…ge in the notification

kpouget · 2026-07-02T11:30:27Z

/test fournos skeleton collect_cluster_info

psap-forge-bot · 2026-07-02T11:30:56Z

🔴 Submission of skeleton collect_cluster_info failed after 0 seconds 🔴

OCPCI Test results
Task logs
Execution logs
Test results not available

Error: ValueError: /cluster (cluster.name) must be set - cannot submit job without target cluster

/test fournos skeleton collect_cluster_info

kpouget · 2026-07-02T12:17:52Z

/test fournos skeleton collect_cluster_info
/cluster psap-mgmt

psap-forge-bot · 2026-07-02T12:19:49Z

🟢 Execution of `skeleton` collect_cluster_info 🟢

Execution Engine Configuration

forge:
  args:
  - collect_cluster_info
  configOverrides: {}
  project: skeleton

Artifact Links

Test Logs

✅ 00 Preflight `0 seconds`

✅ 01 Test `20 seconds`

Test Description

This test validates the skeleton project's cluster information collection capability by enabling the collect_cluster_info preset. It executes the collection task with a limit of 30 requests at a rate of 1, utilizing 128 prompt tokens to assess the data gathering process.

🔄 02 Export-Artifacts

psap-forge-bot · 2026-07-02T12:20:10Z

🟢 Submission of skeleton collect_cluster_info succeeded after 1 minute, 48 seconds 🟢

/test fournos skeleton collect_cluster_info
/cluster psap-mgmt

kpouget requested review from MML-coder and ashtarkb and removed request for MML-coder July 2, 2026 10:34

coderabbitai Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread projects/caliper/orchestration/export.py

kpouget added 2 commits July 2, 2026 13:01

[rhaiis] orchestration: config: define the workspace config

1bfe778

[caliper] orchestration: export: make the workspace config mandatory

9c085aa

kpouget force-pushed the rhaiis branch from a73ecb1 to 9c085aa Compare July 2, 2026 11:01

kpouget force-pushed the rhaiis branch from 63a2e8f to 9f33e95 Compare July 2, 2026 11:19

kpouget force-pushed the rhaiis branch from 9f33e95 to bcaf4b8 Compare July 2, 2026 11:21

kpouget force-pushed the rhaiis branch from bcaf4b8 to f3b9f8e Compare July 2, 2026 11:26

[fournos_launcher] orchestration: submit: include the exception messa…

a79e2b7

…ge in the notification

kpouget force-pushed the rhaiis branch from f3b9f8e to a79e2b7 Compare July 2, 2026 11:30

kpouget merged commit 0ab215a into openshift-psap:main Jul 2, 2026
6 of 7 checks passed

Uh oh!

Conversation

kpouget commented Jul 2, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

openshift-ci Bot commented Jul 2, 2026

Uh oh!

coderabbitai Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kpouget commented Jul 2, 2026

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

🟢 Execution of skeleton collect_cluster_info 🟢

✅ 00 Preflight 2 seconds

✅ 01 Test 12 seconds

Test Description

🔄 02 Export-Artifacts

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

🟢 Execution of skeleton collect_cluster_info 🟢

✅ 00 Preflight 0 seconds

✅ 01 Test 12 seconds

Test Description

🔄 02 Export-Artifacts

Uh oh!

kpouget commented Jul 2, 2026

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

🟢 Execution of skeleton collect_cluster_info 🟢

✅ 00 Preflight 0 seconds

✅ 01 Test 13 seconds

Test Description

🔄 02 Export-Artifacts

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

Uh oh!

kpouget commented Jul 2, 2026

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

🟢 Execution of skeleton collect_cluster_info 🟢

✅ 00 Preflight 0 seconds

✅ 01 Test 13 seconds

Test Description

🔄 02 Export-Artifacts

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

Uh oh!

kpouget commented Jul 2, 2026

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

🟢 Execution of skeleton collect_cluster_info 🟢

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

Uh oh!

kpouget commented Jul 2, 2026

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

Uh oh!

kpouget commented Jul 2, 2026

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

Uh oh!

kpouget commented Jul 2, 2026

Uh oh!

psap-forge-bot Bot commented Jul 2, 2026

🟢 Execution of skeleton collect_cluster_info 🟢

✅ 00 Preflight 0 seconds

✅ 01 Test 20 seconds

Test Description

kpouget commented Jul 2, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading

🟢 Execution of `skeleton` collect_cluster_info 🟢

✅ 00 Preflight `2 seconds`

✅ 01 Test `12 seconds`

🟢 Execution of `skeleton` collect_cluster_info 🟢

✅ 00 Preflight `0 seconds`

✅ 01 Test `12 seconds`

🟢 Execution of `skeleton` collect_cluster_info 🟢

✅ 00 Preflight `0 seconds`

✅ 01 Test `13 seconds`

🟢 Execution of `skeleton` collect_cluster_info 🟢

✅ 00 Preflight `0 seconds`

✅ 01 Test `13 seconds`

🟢 Execution of `skeleton` collect_cluster_info 🟢

🟢 Execution of `skeleton` collect_cluster_info 🟢

✅ 00 Preflight `0 seconds`

✅ 01 Test `20 seconds`