Add weekly DRA scheduler to prevent snapshot expiry

[gurevichdmitry]

Summary

• Adds cloudbeat-dra-scheduler Buildkite pipeline (weekly Monday 06:00 UTC) that derives maintained branches from .mergify.yml backport destinations + main and triggers the existing cloudbeat DRA pipeline for each.
• Keeps snapshot manifests fresh within the 15-day GCS retention window without per-release catalog-info schedule edits (new minors appear when version-bump adds Mergify rules).
• Documents branch SoT, EOL/add ops, and verification in .buildkite/README.md.

Fixes https://github.com/elastic/security-team/issues/18691

Test plan

• Local dry-run: SKIP_UPLOAD=true SKIP_REMOTE_CHECK=true .buildkite/scripts/dra-scheduler.sh emits triggers for main, 8.19, 9.3, 9.4, 9.5
• Local dry-run with origin check: same branch set (all branches exist on origin)
• After merge: confirm RRE created https://buildkite.com/elastic/cloudbeat-dra-scheduler and the weekly schedule
• Manually run scheduler once; confirm child cloudbeat DRA builds for all five branches (esp. 9.3)
• Confirm GCS latest/<branch>.json freshness and cloudbeat on dra-info

Made with Cursor

[mergify]

This pull request does not have a backport label. Could you fix it @gurevichdmitry? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[Copilot]

Pull request overview

Adds a weekly Buildkite “scheduler” pipeline to keep Cloudbeat DRA snapshot manifests fresh by periodically triggering the existing cloudbeat DRA pipeline for all maintained branches derived from .mergify.yml backport destinations plus main.

Changes:

• Registers a new Backstage/RRE Buildkite pipeline resource for cloudbeat-dra-scheduler with a weekly Monday 06:00 UTC schedule.
• Adds a new Buildkite pipeline definition and scheduler script that derives maintained branches and uploads trigger steps to Buildkite.
• Documents scheduler behavior, branch source-of-truth, and operational maintenance in .buildkite/README.md.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
catalog-info.yaml	Adds Backstage/RRE resource for the new weekly scheduler pipeline and its schedule.
.buildkite/dra-scheduler-pipeline.yml	Defines the scheduler pipeline step that runs the branch-derivation + trigger upload script.
.buildkite/scripts/dra-scheduler.sh	Implements deriving branches from .mergify.yml, optional origin validation, and Buildkite trigger-step generation/upload.
.buildkite/README.md	Documents the new weekly refresh mechanism and how to maintain/verify it.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

.buildkite/scripts/dra-scheduler.sh:137

• The EXIT trap currently only deletes the steps file; if install_yq uses a temp dir, it should also be cleaned up to avoid accumulating mktemp -d directories across runs.

trap 'rm -f "${STEPS_FILE}"' EXIT

.buildkite/scripts/dra-scheduler.sh:53

• EXCLUDE_BRANCHES is documented as CSV, but is_excluded does an exact string compare without trimming whitespace. Values like EXCLUDE_BRANCHES="9.3, 9.4" won’t exclude 9.4 due to the leading space.

    local IFS=','
    # shellcheck disable=SC2086
    for excl in ${EXCLUDE_BRANCHES}; do
        if [[ "${branch}" == "${excl}" ]]; then
            return 0

.buildkite/scripts/dra-scheduler.sh:27

• install_yq creates a temp directory via mktemp -d but never cleans it up, which will leak directories on every run (especially noticeable for local dry-runs). Capturing the temp dir path in a global var allows the existing EXIT trap to remove it without changing the rest of the function’s logic.

    echo "--- Installing yq ${YQ_VERSION}"
    local yq_dir yq_bin yq_asset
    yq_dir="$(mktemp -d)"
    yq_bin="${yq_dir}/yq"