Skip to content

feat(eks): add GB300 EFA layout and GPU-conditional worker taint - #44

Merged
mchmarny merged 2 commits into
mchmarny:mainfrom
njhensley:feat/gb300-efa-and-gpu-conditional-taint
Aug 27, 2026
Merged

feat(eks): add GB300 EFA layout and GPU-conditional worker taint#44
mchmarny merged 2 commits into
mchmarny:mainfrom
njhensley:feat/gb300-efa-and-gpu-conditional-taint

Conversation

@njhensley

Copy link
Copy Markdown
Contributor

Fixes #

Proposed Changes

  • GB300 EFA layout — derive gpu_family=gb300 from the p6e-gb300 / p6e-gb300r instance-type prefixes and emit a card-0-only interface layout (primary interface + one efa-only, both on network card 0). p6e-gb300r.36xlarge reports network cards 1–8 with MaximumNetworkInterfaces=0, so a real ASG launch rejects any interface on those cards — a single EFA on card 0 is the only valid layout. Note that run-instances --dry-run does not enforce the per-card ENI limit and returns false positives; the real ASG launch fails with ENI limits exceeded on Network Card 1. Verified end-to-end on a live p6e-gb300r.36xlarge cluster.
  • GPU-conditional worker taint — apply the default dedicated=worker-workload taint only to GPU pools (gpu_family != "na"), so non-GPU (CPU) workers[] pools come out untainted, matching the GKE and AKS providers, which taint only their GPU pools. EKS previously tainted every worker pool unconditionally, forcing CPU workloads to carry a dedicated=worker-workload toleration. An empty taint string renders as --register-with-taints= in user-data (no taints).
  • Tests + schema — add terraform test coverage for the GB300 layout/prefix derivation and the GPU/CPU taint behavior; update the accelerator schema description.

Note on the taint change: any existing non-GPU workers[] pool loses its dedicated=worker-workload taint on the next apply (launch-template change → instance refresh). GPU pools and the gb200 layout are byte-identical (no diff).

Release Note

EKS: add GB300 (p6e-gb300/p6e-gb300r) EFA support with a card-0-only single-EFA layout, and apply the default worker-workload taint only to GPU node groups so non-GPU worker pools are untainted (matching GKE/AKS).

Two independent EKS compute changes, both exercised on a live
p6e-gb300r.36xlarge (GB300 NVL72 slice) cluster.

GB300 EFA layout: derive gpu_family=gb300 from the p6e-gb300 /
p6e-gb300r instance-type prefixes and emit a card-0-only interface
layout (primary interface + one efa-only, both on network card 0).
p6e-gb300r reports network cards 1-8 with MaximumNetworkInterfaces=0,
so a real ASG launch rejects any interface on those cards. Note that
run-instances --dry-run does NOT enforce the per-card ENI limit and
returns false positives, so only the EC2 API's per-card limit plus a
real launch are trustworthy here; a single EFA on card 0 is the only
valid layout.

GPU-conditional worker taint: apply the default
dedicated=worker-workload taint only to GPU pools (gpu_family != "na").
Non-GPU (CPU) worker pools now come out untainted, matching the GKE and
AKS providers, which taint only their GPU pools. Previously EKS tainted
every workers[] pool unconditionally, forcing CPU workloads to carry a
dedicated=worker-workload toleration. An empty taint string renders as
`--register-with-taints=` in user-data (no taints).

Adds terraform test coverage for the GB300 layout, prefix derivation,
and the GPU/CPU taint behavior; updates the accelerator schema
description.

Signed-off-by: Nathan Hensley <nhensley@nvidia.com>
@mchmarny
mchmarny merged commit 95aa3e9 into mchmarny:main Aug 27, 2026
@njhensley
njhensley deleted the feat/gb300-efa-and-gpu-conditional-taint branch August 27, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants