Skip to content

sjayaram/k8s excalibur - #5254

Draft
sjayaram-nv wants to merge 10 commits into
NVIDIA-NeMo:mainfrom
sjayaram-nv:sjayaram/k8s-excalibur
Draft

sjayaram/k8s excalibur#5254
sjayaram-nv wants to merge 10 commits into
NVIDIA-NeMo:mainfrom
sjayaram-nv:sjayaram/k8s-excalibur

Conversation

@sjayaram-nv

Copy link
Copy Markdown

This branch contains changes for adding the XCalibur executor backend to enable llm benchmarking runs in K8s.

@sjayaram-nv
sjayaram-nv requested review from a team, erhoo82 and malay-nagda as code owners August 3, 2026 05:34
@copy-pr-bot

copy-pr-bot Bot commented Aug 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@sjayaram-nv
sjayaram-nv marked this pull request as draft August 3, 2026 06:16
malay-nagda
malay-nagda previously approved these changes Aug 3, 2026
@malay-nagda
malay-nagda requested a review from yaoyu-33 August 3, 2026 07:58
@malay-nagda malay-nagda added the needs-review PR is ready for code review and waiting on a reviewer label Aug 3, 2026
Comment thread scripts/performance/setup_experiment.py Outdated
Comment thread scripts/performance/setup_experiment.py Outdated
Comment on lines +51 to +55
try:
from nemo_run.core.execution.xcalibur import XCaliburExecutor as _XCaliburExecutor
except ImportError:
_XCaliburExecutor = None # type: ignore[assignment,misc]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block appears to be unused.

Comment thread scripts/performance/utils/executors.py Outdated
Comment on lines +614 to +615
# XCalibur uses run_script.py directly; all other executors use bootstrap.py.
script_name = "run_script.py" if xcalibur_namespace else ENTRYPOINT_BOOTSTRAP

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this change? The other executors go through this ENTRYPOINT mechanism. I think doing so skips any recipe level env changes would mean jobs run on xcal and other executors are not apples to apples

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bootstrap.py is a new entrypoint; currently, the container image (nvcr.io/nvidia/nemo:26.04.01) has an older Megatron-Bridge baked into /opt/Megatron-Bridge - it only has run_script.py. This is a temp workaround; will test with nemo container v 26.06 to see if bootstrap.py is available.

Comment thread scripts/performance/setup_experiment.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants