Skip to content

test(bdd): propagate trace context in watch probes #1333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mikeyrcamp wants to merge 3 commits into
base: test/bdd-1292-secure-multiregion
Choose a base branch
from
Open

test(bdd): propagate trace context in watch probes #1333

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eec48b8
test(bdd): cover secure multi-region registration
sbaum1994 Aug 28, 2026
9beff1a
test(bdd): fix secure multi-region coverage (#1322)
mikeyrcamp Aug 28, 2026
737c5f8
test(bdd): propagate trace context in watch probes
mikeyrcamp Aug 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
166 changes: 166 additions & 0 deletions tests/bdd/features/multi-cluster-helmfile-llm-registration-multiregion.feature
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
@ncp-local @multi-cluster @helmfile @pki @llm-registration @multi-region
Feature: Register an LLM worker securely with routers in two local regions
As a self-managed NVCF operator,
I want recursive router discovery to retain explicit HTTPS transport,
so that one worker can register with routable Deployment and StatefulSet routers across regions.

Rule: A secure remote Watch URI expands the registered router topology

Background:
Given these environment variables are set:
| name |
| NGC_API_KEY |
| NVCF_CLI |
| REPO_ROOT |
| SAMPLE_NGC_ORG |
| SAMPLE_NGC_TEAM |
And I prepare Helmfile environment "local-bdd-registration-multiregion" for stack "self-managed" from fixture "tests/bdd/fixtures/self-managed-local-bdd-multi.yaml" with values:
| global.imagePullSecrets[0].name | nvcr-pull-secret |
| global.helm.sources.repository | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM} |
| global.image.repository | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM} |
| global.workerEndpoints.llmRequestRouterAddress | https://llm-request-router.nvcf.svc.cluster.local:50071 |
| addons.llm.requestRouter.workload.kind | Deployment |
| addons.llm.requestRouter.discovery.remoteWatchUrls[0] | https://region-b-watch.nvcf.svc.cluster.local:50071 |
| addons.llm.requestRouter.grpcTls.dnsNames[1] | region-b-watch.nvcf.svc.cluster.local |
| addons.llm.requestRouter.backendRouter.pylonGrpcDialAddress | https://llm-request-router.nvcf.svc.cluster.local:50071 |
| addons.llm.pki.dnsNames[2] | region-b-watch.nvcf.svc.cluster.local |
| addons.llm.pki.dnsNames[3] | *.llm-request-router-region-b-headless.nvcf.svc.cluster.local |
| observability.profile | disabled |
And I prepare Helmfile environment "local-bdd-registration-multiregion" for stack "nvcf-compute-plane" from fixture "tests/bdd/fixtures/nvcf-compute-plane-local-bdd-multi.yaml" with values:
| global.imagePullSecrets[0].name | nvcr-pull-secret |
| global.helm.sources.repository | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM} |
| global.image.repository | ${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM} |
| observability.profile | disabled |
And I prepare self-managed secrets file "deploy/stacks/self-managed/secrets/local-bdd-registration-multiregion-secrets.yaml" from template "deploy/stacks/self-managed/secrets/secrets.yaml.template" using the current NGC registry credential
When I run command "k3d cluster get ncp-local"
Then the command exit code should be 1
And multi-cluster ncp-local compute clusters are running:
| ncp-local-compute-1 |
And command has succeeded:
"""
kubectl config use-context k3d-ncp-local-cp
"""
And the "nvcr-pull-secret" image pull secret exists in namespaces:
| cassandra-system |
| nats-system |
| nvcf |
| api-keys |
| ess |
| sis |
| vault-system |
| nvca-operator |
| cert-manager |

@llm-registration-multiregion-install
Scenario: Operator installs two secure regions with distinct router workload identities
When I run command "make -C deploy/stacks/self-managed template HELMFILE_ENV=local-bdd-registration-multiregion"
Then the command exit code should be 0
And the rendered manifests in "deploy/stacks/self-managed/out" should contain:
| text |
| kind: Deployment |
| --remote-stargate-url=https://region-b-watch.nvcf.svc.cluster.local:50071 |

When I run command "make -C deploy/stacks/self-managed install HELMFILE_ENV=local-bdd-registration-multiregion"
Then the command exit code should be 0
When I run command "kubectl --context k3d-ncp-local-cp wait certificate llm-request-router-grpc-tls -n envoy-gateway-system --for=condition=Ready --timeout=5m"
Then the command exit code should be 0
When I run command "kubectl --context k3d-ncp-local-cp get certificate llm-request-router-grpc-tls -n envoy-gateway-system -o jsonpath={.spec.dnsNames}"
Then the command exit code should be 0
And the command output should contain "region-b-watch.nvcf.svc.cluster.local"
When I run command "kubectl --context k3d-ncp-local-cp rollout status deployment/llm-request-router -n nvcf --timeout=10m"
Then the command exit code should be 0

When I run command "tests/bdd/scripts/install-llm-region-b.sh"
Then the command exit code should be 0

# The initial region advertises an explicit HTTPS recursive seed while
# retaining every concrete Deployment pod identity.
When I run command:
"""
/bin/bash -c 'set -eu; output=$(grpcurl -max-time 3 -cacert <(kubectl --context k3d-ncp-local-cp get secret stargate-quic-tls -n nvcf -o jsonpath="{.data.ca\.crt}" | base64 -d) -authority llm-request-router.nvcf.svc.cluster.local -H "traceparent: 00-00000000000000000000000000001310-0000000000001310-01" -import-path src/libraries/rust/stargate/crates/proto/proto -proto stargate.proto 127.0.0.1:50071 stargate.StargateControlPlane/WatchStargates 2>&1 || true); pods=$(kubectl --context k3d-ncp-local-cp get pods -n nvcf -l app.kubernetes.io/instance=llm-request-router,app.kubernetes.io/name=llm-request-router -o jsonpath="{range .items[*]}{.metadata.name}{\"\\n\"}{end}"); count=0; while IFS= read -r pod; do [ -z "$pod" ] && continue; printf "%s" "$output" | grep -Fq "$pod"; count=$((count + 1)); done <<<"$pods"; [ "$count" -eq 3 ]; printf "%s" "$output" | grep -Fq "https://region-b-watch.nvcf.svc.cluster.local:50071"; printf "region-a-deployment=%s remote-watch=https\n" "$count"'
"""
Then the command exit code should be 0
And the command output should contain "region-a-deployment=3 remote-watch=https"

# The remote HTTPS authority resolves to two stable StatefulSet router
# identities and never relies on a dashed-IP SRV alias.
When I run command:
"""
/bin/bash -c 'set -eu; output=$(grpcurl -max-time 3 -cacert <(kubectl --context k3d-ncp-local-cp get secret stargate-quic-tls -n nvcf -o jsonpath="{.data.ca\.crt}" | base64 -d) -authority region-b-watch.nvcf.svc.cluster.local -H "traceparent: 00-00000000000000000000000000001322-0000000000001322-01" -import-path src/libraries/rust/stargate/crates/proto/proto -proto stargate.proto 127.0.0.1:50071 stargate.StargateControlPlane/WatchStargates 2>&1 || true); identities=$(printf "%s\n" "$output" | grep -Eo "llm-request-router-region-b-[0-9]+" | sort -u || true); expected=$(printf "llm-request-router-region-b-0\nllm-request-router-region-b-1\n"); [ "$identities" = "$expected" ]; count=$(printf "%s\n" "$identities" | grep -c .); [ "$count" -eq 2 ]; ! printf "%s" "$output" | grep -Eq "([0-9]{1,3}-){3}[0-9]{1,3}\."; printf "region-b-statefulset=%s tls=https\n" "$count"'
"""
Then the command exit code should be 0
And the command output should contain "region-b-statefulset=2 tls=https"

When I run command:
"""
${NVCF_CLI} --config ${REPO_ROOT}/tests/bdd/fixtures/nvcf-cli-local.yaml self-hosted --control-plane-stack deploy/stacks/self-managed --env local-bdd-registration-multiregion --control-plane-context k3d-ncp-local-cp --compute-plane-context k3d-ncp-local-compute-1 control-plane profile export --cluster-name ncp-local-cp
"""
Then the command exit code should be 0
And file "deploy/stacks/self-managed/out/control-plane-profile.yaml" should exist
And yaml file "deploy/stacks/self-managed/out/control-plane-profile.yaml" should have non-empty keys:
| key |
| managementTls.caBundlePem |
| transportTls.trustBundleFingerprint |
| transportTls.trustBundlePem |

And command has succeeded:
"""
/bin/sh -c '${NVCF_CLI} --config ${REPO_ROOT}/tests/bdd/fixtures/nvcf-cli-local.yaml init >/dev/null'
"""
When I run command "kubectl config use-context k3d-ncp-local-compute-1"
Then the command exit code should be 0
When I run command:
"""
make -C deploy/stacks/nvcf-compute-plane register-cluster CLUSTER_NAME=ncp-local-compute-1 CONTROL_PLANE_PROFILE=${REPO_ROOT}/deploy/stacks/self-managed/out/control-plane-profile.yaml COMPUTE_KUBE_CONTEXT=k3d-ncp-local-compute-1 NVCF_CLI=${NVCF_CLI} NVCF_CLI_CONFIG=${REPO_ROOT}/tests/bdd/fixtures/nvcf-cli-local.yaml
"""
Then the command exit code should be 0
And the "nvcr-pull-secret" image pull secret exists in namespaces:
| nvca-operator |
When I run command:
"""
make -C deploy/stacks/nvcf-compute-plane install CLUSTER_NAME=ncp-local-compute-1 HELMFILE_ENV=local-bdd-registration-multiregion COMPUTE_KUBE_CONTEXT=k3d-ncp-local-compute-1 NVCF_CLI=${NVCF_CLI}
"""
Then the command exit code should be 0
Then NVCFBackend "ncp-local-compute-1" in namespace "nvca-operator" using context "k3d-ncp-local-compute-1" should report agent status "healthy" within "10m"

@llm-registration-multiregion-runtime
Scenario: Pylon recursively registers with both regions and serves an authenticated request
Given I use NVCF CLI config "${REPO_ROOT}/tests/bdd/fixtures/nvcf-cli-local.yaml"
When I successfully create function "bdd-registration-multiregion" from image "nvcr.io/${SAMPLE_NGC_ORG}/${SAMPLE_NGC_TEAM}/nvcf-openai-compatible-sample:local" with CLI options:
| option | value |
| --function-type | LLM |
| --inference-url | /v1/chat/completions |
| --inference-port | 8000 |
| --health-uri | /health |
| --health-port | 8000 |
| --health-timeout | PT30S |
| --llm-model | name=openai-compatible-sample,uris=/v1/chat/completions\|/v1/embeddings,routingMethod=round_robin |
And I successfully deploy the function selected by NVCF CLI with options:
| option | value |
| --gpu | H100 |
| --instance-type | NCP.GPU.H100_1x |
| --backend | ncp-local-compute-1 |
| --regions | us-west-1 |
| --min-instances | 1 |
| --max-instances | 1 |
| --timeout | 900 |
And I successfully generate a function API key with CLI options:
| option | value |
| --description | bdd-registration-multiregion |
| --scopes | invoke_function,list_functions,queue_details,list_functions_details |

When I run command:
"""
/bin/sh -c 'set -eu; for attempt in $(seq 1 120); do row=$(kubectl --context k3d-ncp-local-compute-1 get pods -A -o json | jq -r "[.items[] | select(any(.spec.containers[]?; .name == \"llm-worker\")) | [.metadata.namespace,.metadata.name] | @tsv] | first // empty"); if [ -n "$row" ]; then ns=$(printf "%s" "$row" | cut -f1); pod=$(printf "%s" "$row" | cut -f2); metrics=$(kubectl --context k3d-ncp-local-compute-1 get --raw "/api/v1/namespaces/$ns/pods/$pod:9089/proxy/metrics" 2>/dev/null || true); registration=$(printf "%s\n" "$metrics" | grep -c "^pylon_registration_stream_connected.* 1$" || true); reverse=$(printf "%s\n" "$metrics" | grep -c "^pylon_reverse_tunnel_connected.* 1$" || true); if [ "$registration" -eq 5 ] && [ "$reverse" -ge 3 ]; then printf "registration=%s reverse=%s regions=2\n" "$registration" "$reverse"; exit 0; fi; fi; sleep 5; done; exit 1'
"""
Then the command exit code should be 0
And the command output should contain "registration=5"
And the command output should contain "regions=2"

When I successfully invoke model "openai-compatible-sample" at "/v1/chat/completions" with timeout "120" seconds:
"""
{"messages":[{"role":"user","content":"bdd-registration-multiregion"}]}
"""
Then the command output should contain "chat.completion"
And the command output should contain "fixed 128-byte response"
And I successfully undeploy the function selected by NVCF CLI
Loading