[aws] Enable Identity Federation for all remaining policy templates by seanrathier · Pull Request #20527 · elastic/integrations

seanrathier · 2026-08-04T18:17:34Z

Proposed commit message

Enable Identity Federation across the aws package for Config, Inspector, all aws-cloudwatch log streams, and all aws/metrics streams.

WHAT:

config (CEL) and inspector (HTTPJSON): full auth migration from hand-rolled SigV4 signing to the input's auth.aws configuration, with use_cloud_connectors driven by the supports_identity_federation var. The hand-rolled path never set X-Amz-Security-Token, so session tokens were silently broken; auth.aws fixes that and adds Role ARN / shared-credential support.
4 aws-cloudwatch log streams (cloudwatch_logs, ec2_logs, elb_logs, lambda_logs) and 13 aws/metrics streams (awshealth, billing, cloudwatch_metrics, dynamodb, ebs, ec2_metrics, ecs_metrics, elb_metrics, lambda, rds, sns, sqs, transitgateway): add top-level external_id and use_cloud_connectors to the stream templates. These input types embed Beats' shared ConfigAWS inline, so credentials sit at the top level of the stream config rather than under auth.aws.
Remove the hide_in_var_group_options: credential_type: [identity_federation] gates from all affected policy templates (including guardduty, whose template was already federation-ready).

WHY: completes ingest-dev#8802 — every agentless-eligible stream in the aws package (except SecurityHub, tracked separately) can onboard via Identity Federation (Cloud Connectors) with zero static credentials. use_cloud_connectors is supported for all these input types by Beats' shared ConfigAWS layer (x-pack/libbeat/common/aws/credentials.go, present on the 9.4 branch), which InitializeAWSConfig handles centrally.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs. (E2E validation pending — see test plan below; this is the first end-to-end exercise of use_cloud_connectors on the aws-cloudwatch and aws/metrics input types)
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices (no dashboards added or changed)

Author's Checklist

Config and Inspector streams validated E2E via Identity Federation (no AccessDeniedException, streams HEALTHY)
One aws-cloudwatch stream and one aws/metrics stream validated E2E via the Cloud Connectors token exchange
Regression: existing GuardDuty and S3 streams on the same role continue collecting
Merge order: [Cloud Security Integrations] Add all remaining aws package permissions to the Federated Identity CFT cloudbeat#7637 must merge and publish cloudformation-federated-identity-aws-9.4.0.yml to S3 before this PR is tested or merged — the manifest's Identity Federation quick-create link now points at that template and 404s until the publish pipeline runs

How to test this PR locally

cd packages/aws
elastic-package check
elastic-package test policy   # policy rendering covers federated + legacy credential paths

For the federated path E2E:

Deploy the paired cloudbeat CFT ([Cloud Security Integrations] Add all remaining aws package permissions to the Federated Identity CFT cloudbeat#7637) to create the federated-identity IAM role
In Fleet, create an agentless AWS integration using the Identity Federation credential type
Validate Config and Inspector streams show HEALTHY — no AccessDeniedException in agent logs
Validate one aws-cloudwatch stream (e.g. cloudwatch_logs) and one aws/metrics stream (e.g. cloudwatch_metrics) collect data via the Cloud Connectors token exchange

Related issues

Relates elastic/ingest-dev#8802
Relates [Cloud Security Integrations] Add all remaining aws package permissions to the Federated Identity CFT cloudbeat#7637 (paired CFT permissions)
Relates [aws] Enable Identity Federation for SecurityHub data streams #20529 (SecurityHub streams, tracked under elastic/ingest-dev#8812)

Migrate the Inspector HTTPJSON template from hand-rolled SigV4 signing to the input's auth.aws configuration. The template previously built the Authorization header in Go template expressions — twice, because the body changes between pagination pages — and never set X-Amz-Security-Token, which silently broke temporary credentials. With auth.aws the input signs every request (including re-signed pagination pages) at the transport layer. - Add the full auth.aws block including the use_cloud_connectors hook driven by supports_identity_federation (same pattern as GuardDuty) - Remove the manual X-Amz-Date and Authorization transforms and the pagination re-signing steps - Remove the hide_in_var_group_options gate from the inspector policy template, making Identity Federation selectable on the agentless path - Fixes session-token auth and adds Role ARN / shared-credential support, none of which the hand-rolled signing handled Part of elastic/ingest-dev#8802. The paired cloudbeat PR mirrors the stream's single API call (inspector2:ListFindings) into the static fallback template. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Removes identity_federation gates from all 22 remaining gated inputs across 15 policy templates. Per-input changes: - securityhub_findings, securityhub_findings_full_posture, securityhub_insights: full auth migration from hand-rolled SigV4 to auth.aws, adding Cloud Connectors support and fixing silent session- token breakage in the hand-rolled path. - cloudwatch_logs, ec2_logs, elb_logs, lambda_logs: add external_id to aws-cloudwatch stream templates. - awshealth, billing, cloudwatch_metrics, dynamodb, ebs, ec2_metrics, ecs_metrics, elb_metrics, lambda, rds, sns, sqs, transitgateway: add external_id to aws/metrics stream templates. - guardduty httpjson: gate removed; auth.aws + use_cloud_connectors already present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-08-04T18:19:15Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Restore identity_federation gate for the securityhub policy template and revert securityhub_findings, securityhub_findings_full_posture, and securityhub_insights hbs files to hand-rolled SigV4. SecurityHub will be migrated under elastic/ingest-dev#8812. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

….0 bump Cherry-picks the Config CEL migration (hand-rolled SigV4 to auth.aws) and its test-expectation update from the closed elastic#20437, which was never actually an ancestor of this branch — config and inspector were parallel branches, so closing elastic#20437 as superseded orphaned the CEL migration while the manifest gate removal still exposed Identity Federation for Config. Collapses the stacked 7.3.0/7.4.0 changelog entries into a single 7.2.0 entry linking this PR, since the combined PR targets main (at 7.1.1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…plates Beats already supports use_cloud_connectors for these input types: the flag lives in the shared ConfigAWS struct (x-pack/libbeat/common/aws/ credentials.go, present on the 9.4 branch), which both the aws-cloudwatch input and the metricbeat aws module embed inline and pass through InitializeAWSConfig. No auth.aws block is needed — these inputs take credentials at the top level of the stream config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The Identity Federation quick-create link now targets cloudformation-federated-identity-aws-9.4.0.yml, published by cloudbeat's publish_cft.sh when elastic/cloudbeat#7637 merges. Until that publish runs, this URL 404s — cloudbeat#7637 must merge first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

olegsu · 2026-08-05T19:12:50Z

 role_arn: {{role_arn}}
 {{/if}}
+{{#if external_id}}
+external_id: {{external_id}}


We will not be using external_id with WII

We will change this in a new PR to address WII, unless WII is worked out before this is merged

olegsu · 2026-08-05T19:14:14Z

 # newer versions go on top
+- version: "7.2.0"
+  changes:
+    - description: Enable Identity Federation for Config, Inspector, aws-cloudwatch log streams, and all aws/metrics streams. Migrates Config (CEL) and Inspector (HTTPJSON) from hand-rolled SigV4 signing to the input's auth.aws configuration, adding Cloud Connectors support and fixing silent session-token breakage. Adds external_id and use_cloud_connectors to aws-cloudwatch and aws/metrics stream templates.


Remove the external_id reference

#20527 (comment)

…ted_error.txt Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

elastic-vault-github-plugin-prod · 2026-08-06T14:20:01Z

✅ All changelog entries have the correct PR link.

elastic-vault-github-plugin-prod · 2026-08-06T16:35:34Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

infra-vault-gh-plugin-prod · 2026-08-06T16:35:39Z

💚 Build Succeeded

Buildkite Build
Commit: 244e817

History

💚 Build #47185 succeeded db79590
💔 Build #47179 failed a08fad7
💔 Build #47176 failed ed0b537
💔 Build #47175 failed 3dee35e

seanrathier and others added 4 commits July 31, 2026 16:53

Set real PR link in changelog

15228e1

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge branch 'main' into aws/inspector-identity-federation

aa2d13b

seanrathier mentioned this pull request Aug 4, 2026

[Cloud Security Integrations] Add all remaining aws package permissions to the Federated Identity CFT elastic/cloudbeat#7637

Open

Fix changelog PR link for 7.4.0 (20526 → 20527)

ed0b537

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

This was referenced Aug 4, 2026

[aws] Enable Identity Federation for the Config data stream #20437

Closed

[aws] Enable Identity Federation for the Amazon Inspector data stream #20439

Closed

seanrathier added Integration:aws AWS Team:Service-Integrations Label for the Observability Service Integrations team Team:Security-Cloud Services Security Data Experience - Cloud Services team [elastic/cloud-services] labels Aug 4, 2026

seanrathier added a commit to seanrathier/integrations that referenced this pull request Aug 4, 2026

Fix version to 7.5.0 (follows 7.4.0 from elastic#20527)

3579193

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

seanrathier added a commit to seanrathier/integrations that referenced this pull request Aug 4, 2026

Fix version to 7.3.0 (follows consolidated 7.2.0 from elastic#20527)

0d97cd3

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

seanrathier mentioned this pull request Aug 4, 2026

[aws] Enable Identity Federation for SecurityHub data streams #20529

Draft

8 tasks

seanrathier added the enhancement New feature or request label Aug 4, 2026

seanrathier marked this pull request as ready for review August 5, 2026 13:17

seanrathier requested review from a team as code owners August 5, 2026 13:17

olegsu reviewed Aug 5, 2026

View reviewed changes

efd6 reviewed Aug 5, 2026

View reviewed changes

Comment thread packages/aws/data_stream/config/_dev/test/scripts/unauthenticated_error.txt Outdated

Update packages/aws/data_stream/config/_dev/test/scripts/unauthentica…

244e817

…ted_error.txt Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Navigation Menu

Uh oh!

[aws] Enable Identity Federation for all remaining policy templates - #20527

[aws] Enable Identity Federation for all remaining policy templates#20527
seanrathier wants to merge 10 commits into
elastic:mainfrom
seanrathier:aws/remaining-identity-federation

seanrathier commented Aug 4, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

olegsu Aug 5, 2026

Uh oh!

seanrathier Aug 6, 2026

Uh oh!

olegsu Aug 5, 2026

Uh oh!

seanrathier Aug 6, 2026

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 6, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 6, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

seanrathier commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Uh oh!

github-actions Bot commented Aug 4, 2026

✅ Elastic Docs Style Checker (Vale)

Uh oh!

olegsu Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

seanrathier Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

olegsu Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

seanrathier Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 6, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 6, 2026

🚀 Benchmarks report

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 6, 2026

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

seanrathier commented Aug 4, 2026 •

edited

Loading