test(otel/multi_efa_dra): add integration test for the EFA DRA path - #753
Open
samehkhalil wants to merge 1 commit into
Open
test(otel/multi_efa_dra): add integration test for the EFA DRA path#753samehkhalil wants to merge 1 commit into
samehkhalil wants to merge 1 commit into
Conversation
Mirror the device-plugin multi_efa integration test for the Dynamic Resource Allocation (DRA) path. EFA devices are exposed via dranet (driver dra.net) through a ResourceClaimTemplate instead of the vpc.amazonaws.com/efa device-plugin resource, exercising the awsdevicepodcorrelation processor's DRA correlation end to end (ResourceClaim/ResourceSlice keying via dra.net/rdmaDevice) plus the groupbyattrs/efa split before the resource-level promote. The emitted metrics are identical to the device-plugin path, so the Go assertions match the multi_efa package, including TestMultiEFAClaimedVsUnclaimedCorrelation: on a 2-EFA node, efaburn claims one device (correlated to the pod) and the other stays unclaimed (no pod), guarding the per-device correlation collapse on the DRA path. New terraform module terraform/eks/daemon/otel-multi-efa-dra installs dranet in place of the EFA device plugin and applies a ResourceClaimTemplate-based efaburn workload. Pinned to k8s 1.34 (DRA GA v1, with v1beta1 still served for the processor's informers). Requires a chart with DRA config + resource.k8s.io RBAC (helm-charts PR aws#356) until that lands on main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the issue
EFA metrics on EKS can be exposed to pods two ways: the EFA device plugin
(
vpc.amazonaws.com/efa) and Dynamic Resource Allocation (DRA), where EFAdevices are allocated via a DRA driver (dranet, driver
dra.net) andResourceClaims. The agent's OTel Container Insights pipeline supports both, andthe
awsdevicepodcorrelationprocessor has a dedicated DRA code path that watchesResourceClaims/ResourceSlicesvia the K8s API and bridges the DRA deviceidentity (a PCI name, e.g.
pci-0000-00-1e-0) to the EFA metric label (e.g.rdmap0s30) via thedra.net/rdmaDeviceResourceSlice attribute.Today there is no integration coverage for the DRA path — only the device-plugin
path is exercised. This adds an end-to-end test so the DRA path is validated on a
real cluster: EFA metrics are emitted per device and correctly correlated to the
pods that claim them.
Description of changes
Add an integration test that provisions a cluster exposing EFA via DRA and
validates per-device EFA metric correlation end to end:
test/otel/multi_efa_dra/— queries the EFA metrics in CloudWatch andasserts the DRA path produces the expected per-device series and pod
attribution: each device is a distinct series with its EFA attributes
(
aws.efa.device, ENI, port), the device a pod claims is correlated to thatpod (name/namespace/container), and devices not claimed by any pod carry no pod
attributes.
terraform/eks/daemon/otel-multi-efa-dra/— provisions the cluster,installs dranet (
eks/aws-dranet) as the DRA driver, deploys theobservability chart and agent, and runs an
efaburnworkload that requests oneEFA through a
ResourceClaimTemplate.helm_chart_repo_urlvariable so the chart can be cloned from a fork whenvalidating chart changes not yet merged upstream (defaults to upstream).
Pinned to k8s 1.34 (the rest of the suite is on 1.35): the processor's DRA
informers watch
resource.k8s.io/v1beta1, which 1.34 still serves alongside theGA
v1. The generator entry is commented with this rationale; it moves to 1.35once the processor's DRA client is bumped to
v1.Dependencies (the EKS lane is green only after these land): the chart on
mainmust render the DRA correlation config (dra_device_typeson thedra.netdriver +dra.net/rdmaDevicekeying) and grant the agent ServiceAccountget/list/watchonresource.k8s.ioresourceclaims/resourceslices, and thereleased agent image must include the DRA processor path. Committed defaults
already point at that merged end-state (public agent image, chart
main, upstreamchart URL), so no follow-up edit is needed once those merge.
License
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Tests
Ran the module end to end on a live cluster (EKS 1.34,
c6in.32xlarge, EFA viadranet). All tests passed.
Verified against ground truth:
rdmap0s30andrdmap0s31.efaburn'sResourceClaimTemplateallocatedpci-0000-00-1e-0, which maps viadra.net/rdmaDevicetordmap0s30→ correlated to theefaburnpod(namespace
default, containerefaburn).rdmap0s31was unclaimed → no pod attributes.Each EFA metric (
efa_rx_bytes,efa_tx_bytes,efa_rx_dropped,efa_rdma_read_bytes) reported per device with correct DRA-based pod correlation.Cluster torn down after the run.