Skip to content

test(otel/multi_efa_dra): add integration test for the EFA DRA path - #753

Open
samehkhalil wants to merge 1 commit into
aws:mainfrom
samehkhalil:test/multi-efa-dra-per-device-correlation
Open

test(otel/multi_efa_dra): add integration test for the EFA DRA path#753
samehkhalil wants to merge 1 commit into
aws:mainfrom
samehkhalil:test/multi-efa-dra-per-device-correlation

Conversation

@samehkhalil

Copy link
Copy Markdown

Description of the issue

EFA metrics on EKS can be exposed to pods two ways: the EFA device plugin
(vpc.amazonaws.com/efa) and Dynamic Resource Allocation (DRA), where EFA
devices are allocated via a DRA driver (dranet, driver dra.net) and
ResourceClaims. The agent's OTel Container Insights pipeline supports both, and
the awsdevicepodcorrelation processor has a dedicated DRA code path that watches
ResourceClaims/ResourceSlices via the K8s API and bridges the DRA device
identity (a PCI name, e.g. pci-0000-00-1e-0) to the EFA metric label (e.g.
rdmap0s30) via the dra.net/rdmaDevice ResourceSlice attribute.

Today there is no integration coverage for the DRA path — only the device-plugin
path is exercised. This adds an end-to-end test so the DRA path is validated on a
real cluster: EFA metrics are emitted per device and correctly correlated to the
pods that claim them.

Description of changes

Add an integration test that provisions a cluster exposing EFA via DRA and
validates per-device EFA metric correlation end to end:

  • test/otel/multi_efa_dra/ — queries the EFA metrics in CloudWatch and
    asserts the DRA path produces the expected per-device series and pod
    attribution: each device is a distinct series with its EFA attributes
    (aws.efa.device, ENI, port), the device a pod claims is correlated to that
    pod (name/namespace/container), and devices not claimed by any pod carry no pod
    attributes.
  • terraform/eks/daemon/otel-multi-efa-dra/ — provisions the cluster,
    installs dranet (eks/aws-dranet) as the DRA driver, deploys the
    observability chart and agent, and runs an efaburn workload that requests one
    EFA through a ResourceClaimTemplate.
  • Generator entry wiring the test to its terraform module.
  • helm_chart_repo_url variable so the chart can be cloned from a fork when
    validating chart changes not yet merged upstream (defaults to upstream).

Pinned to k8s 1.34 (the rest of the suite is on 1.35): the processor's DRA
informers watch resource.k8s.io/v1beta1, which 1.34 still serves alongside the
GA v1. The generator entry is commented with this rationale; it moves to 1.35
once the processor's DRA client is bumped to v1.

Dependencies (the EKS lane is green only after these land): the chart on
main must render the DRA correlation config (dra_device_types on the
dra.net driver + dra.net/rdmaDevice keying) and grant the agent ServiceAccount
get/list/watch on resource.k8s.io resourceclaims/resourceslices, and the
released agent image must include the DRA processor path. Committed defaults
already point at that merged end-state (public agent image, chart main, upstream
chart URL), so no follow-up edit is needed once those merge.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Ran the module end to end on a live cluster (EKS 1.34, c6in.32xlarge, EFA via
dranet). All tests passed.

Verified against ground truth:

  • The node exposed 2 EFA devices, rdmap0s30 and rdmap0s31.
  • efaburn's ResourceClaimTemplate allocated pci-0000-00-1e-0, which maps via
    dra.net/rdmaDevice to rdmap0s30 → correlated to the efaburn pod
    (namespace default, container efaburn).
  • rdmap0s31 was unclaimed → no pod attributes.

Each EFA metric (efa_rx_bytes, efa_tx_bytes, efa_rx_dropped,
efa_rdma_read_bytes) reported per device with correct DRA-based pod correlation.
Cluster torn down after the run.

Mirror the device-plugin multi_efa integration test for the Dynamic Resource
Allocation (DRA) path. EFA devices are exposed via dranet (driver dra.net)
through a ResourceClaimTemplate instead of the vpc.amazonaws.com/efa
device-plugin resource, exercising the awsdevicepodcorrelation processor's DRA
correlation end to end (ResourceClaim/ResourceSlice keying via
dra.net/rdmaDevice) plus the groupbyattrs/efa split before the resource-level
promote.

The emitted metrics are identical to the device-plugin path, so the Go
assertions match the multi_efa package, including
TestMultiEFAClaimedVsUnclaimedCorrelation: on a 2-EFA node, efaburn claims one
device (correlated to the pod) and the other stays unclaimed (no pod), guarding
the per-device correlation collapse on the DRA path.

New terraform module terraform/eks/daemon/otel-multi-efa-dra installs dranet in
place of the EFA device plugin and applies a ResourceClaimTemplate-based efaburn
workload. Pinned to k8s 1.34 (DRA GA v1, with v1beta1 still served for the
processor's informers). Requires a chart with DRA config + resource.k8s.io RBAC
(helm-charts PR aws#356) until that lands on main.
@samehkhalil
samehkhalil requested a review from a team as a code owner September 2, 2026 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant