feat(eks): add GB300 EFA layout and GPU-conditional worker taint - #44
Merged
mchmarny merged 2 commits intoAug 27, 2026
Merged
Conversation
Two independent EKS compute changes, both exercised on a live p6e-gb300r.36xlarge (GB300 NVL72 slice) cluster. GB300 EFA layout: derive gpu_family=gb300 from the p6e-gb300 / p6e-gb300r instance-type prefixes and emit a card-0-only interface layout (primary interface + one efa-only, both on network card 0). p6e-gb300r reports network cards 1-8 with MaximumNetworkInterfaces=0, so a real ASG launch rejects any interface on those cards. Note that run-instances --dry-run does NOT enforce the per-card ENI limit and returns false positives, so only the EC2 API's per-card limit plus a real launch are trustworthy here; a single EFA on card 0 is the only valid layout. GPU-conditional worker taint: apply the default dedicated=worker-workload taint only to GPU pools (gpu_family != "na"). Non-GPU (CPU) worker pools now come out untainted, matching the GKE and AKS providers, which taint only their GPU pools. Previously EKS tainted every workers[] pool unconditionally, forcing CPU workloads to carry a dedicated=worker-workload toleration. An empty taint string renders as `--register-with-taints=` in user-data (no taints). Adds terraform test coverage for the GB300 layout, prefix derivation, and the GPU/CPU taint behavior; updates the accelerator schema description. Signed-off-by: Nathan Hensley <nhensley@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #
Proposed Changes
gpu_family=gb300from thep6e-gb300/p6e-gb300rinstance-type prefixes and emit a card-0-only interface layout (primaryinterface+ oneefa-only, both on network card 0).p6e-gb300r.36xlargereports network cards 1–8 withMaximumNetworkInterfaces=0, so a real ASG launch rejects any interface on those cards — a single EFA on card 0 is the only valid layout. Note thatrun-instances --dry-rundoes not enforce the per-card ENI limit and returns false positives; the real ASG launch fails withENI limits exceeded on Network Card 1. Verified end-to-end on a livep6e-gb300r.36xlargecluster.dedicated=worker-workloadtaint only to GPU pools (gpu_family != "na"), so non-GPU (CPU)workers[]pools come out untainted, matching the GKE and AKS providers, which taint only their GPU pools. EKS previously tainted every worker pool unconditionally, forcing CPU workloads to carry adedicated=worker-workloadtoleration. An empty taint string renders as--register-with-taints=in user-data (no taints).terraform testcoverage for the GB300 layout/prefix derivation and the GPU/CPU taint behavior; update theacceleratorschema description.Note on the taint change: any existing non-GPU
workers[]pool loses itsdedicated=worker-workloadtaint on the next apply (launch-template change → instance refresh). GPU pools and thegb200layout are byte-identical (no diff).Release Note