From 083041ba5936566c2ec16eb20e963760a84568f6 Mon Sep 17 00:00:00 2001 From: Ankit Anand Date: Sat, 29 Aug 2026 13:29:00 -0700 Subject: [PATCH 1/2] docs(examples): add ModelExpress Dynamo sample for self-managed compute planes Signed-off-by: Ankit Anand --- examples/README.md | 1 + .../modelexpress-dynamo-sample/README.md | 334 ++++++++++++++++++ .../modelexpress-dynamo/.helmignore | 27 ++ .../modelexpress-dynamo/Chart.yaml | 21 ++ .../templates/dynamo-graph-deployment.yaml | 113 ++++++ .../templates/hf_token_secret.yaml | 26 ++ .../modelexpress-dynamo/values.yaml | 121 +++++++ .../modelexpress-server-values.yaml | 151 ++++++++ .../override.yaml.sample | 131 +++++++ 9 files changed, 925 insertions(+) create mode 100644 examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/README.md create mode 100644 examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/.helmignore create mode 100644 examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/Chart.yaml create mode 100644 examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/templates/dynamo-graph-deployment.yaml create mode 100644 examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/templates/hf_token_secret.yaml create mode 100644 examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/values.yaml create mode 100644 examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-server-values.yaml create mode 100644 examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/override.yaml.sample diff --git a/examples/README.md b/examples/README.md index ac71154ab..fb25de8ce 100644 --- a/examples/README.md +++ b/examples/README.md @@ -20,6 +20,7 @@ Functions are long-running services that respond to HTTP or gRPC invocations. | [Multi-Node Helm Function](function-samples/helmchart-samples/multi-node-helm-function-test/) | Multi-node Helm chart for running NCCL and GPU bandwidth tests via NVCF. | | [Ray Serve Helm Chart](function-samples/helmchart-samples/ray-serve-sample/) | Helm chart that deploys a Ray Serve application as an NVCF function. | | [Dynamo Operator Sample](function-samples/helmchart-samples/dynamo-operator-sample/) | Helm chart for a vLLM disaggregated router deployed through NVCF. | +| [ModelExpress Dynamo Sample](function-samples/helmchart-samples/modelexpress-dynamo-sample/) | Dynamo vLLM function that loads model weights peer-to-peer through ModelExpress. | | [Load Tester Supreme](function-samples/load-tester-supreme/) | HTTP and gRPC echo servers designed for load and throughput testing. | ## Task Samples diff --git a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/README.md b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/README.md new file mode 100644 index 000000000..dcc5b6d75 --- /dev/null +++ b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/README.md @@ -0,0 +1,334 @@ +# ModelExpress with Dynamo on a self-managed compute plane + +This sample runs a Dynamo vLLM function whose workers load model weights through +[ModelExpress](https://github.com/ai-dynamo/modelexpress) instead of each worker +downloading the model itself. When several workers start at once, the first one +to hold the weights serves them to its peers over RDMA, so scale-out cold starts +do not repeat the same download. + +ModelExpress is optional. It is not part of the NVCF compute-plane stack, and +nothing here changes an existing installation. See +[the ModelExpress guide](../../../../docs/user/cluster-management/modelexpress.md) +for the full explanation, verification steps, and troubleshooting. + +## How this sample is split + +ModelExpress needs one cluster-wide install and one per-function chart: + +- `modelexpress-server-values.yaml` configures the upstream ModelExpress chart. + A cluster administrator installs this once. It is not part of the function. +- `modelexpress-dynamo/` is the NVCF function chart. It contains only a + `DynamoGraphDeployment` whose workers point at the server above. +- `override.yaml.sample` holds the provider-specific fabric settings. The chart + defaults are provider-neutral. + +ModelExpress is wired into the workers entirely through the +`DynamoGraphDeployment` `envs` field, at `spec.services.VllmDecodeWorker.envs`, +together with `--load-format modelexpress` on the container args. Nothing else in +the function is ModelExpress-aware, which is why `modelExpress.enabled=false` +yields the same topology on the native loader. The +[Dynamo API reference](https://github.com/ai-dynamo/dynamo/blob/v1.2.1/docs/kubernetes/api-reference.md#dynamographdeployment) +documents the field. Note that the Dynamo operator also injects +`MODEL_EXPRESS_URL` on clusters deployed with `infrastructure.modelExpressURL` +configured; values in `envs` override it, so the address set here wins. + +The server, its CRDs, and its RBAC are installed by an administrator rather than +shipped inside the function chart, because a CRD is a cluster-scoped object that +a function chart should not own, and the server is shared by every function that +uses it. The custom resources themselves are namespaced. + +## Prerequisites + +- An NVCF self-hosted cluster with the Dynamo, Grove, and KAI Scheduler add-ons. + See [Self-Managed Clusters](../../../../docs/user/cluster-management/self-managed.md), + and for local testing the + [local Dynamo Operator guide](../../../../tools/ncp-local-cluster/docs/dynamo-operator.md). +- Two or more GPU nodes. One worker cannot demonstrate peer-to-peer transfer. +- On AWS EFA, enough nodes to supply one EFA unit per worker. An EFA interface is + not shareable, so the bound is the advertised `vpc.amazonaws.com/efa` rather + than GPUs per node. The EFA-capable G-family used here exposes one interface, so + a four-GPU instance still runs one EFA worker; larger instances advertise more. +- RDMA-capable networking between those nodes. Without it ModelExpress still + works, but over a slower transport, and the cold-start benefit shrinks. +- A node container runtime that allows a large locked-memory limit. RDMA pins + memory, and containers inherit `RLIMIT_MEMLOCK` from the runtime, which + commonly defaults to 8 MiB. That is too small for UCX to register its buffers, + so the worker fails engine init with + `ibv_reg_mr(...) failed: Cannot allocate memory`. Check and fix it on each GPU + node: + + ```bash + systemctl show containerd -p LimitMEMLOCK --value # 8388608 is too small + + sudo mkdir -p /etc/systemd/system/containerd.service.d + sudo tee /etc/systemd/system/containerd.service.d/10-memlock.conf <<'EOF' + [Service] + LimitMEMLOCK=infinity + EOF + sudo systemctl daemon-reload && sudo systemctl restart containerd + ``` + + This cannot be fixed from the chart. Kubernetes has no ulimit field, and + granting `CAP_IPC_LOCK` does not help, because the worker image runs as UID + 1000 and a non-root process has an empty effective capability set. Confirm + from inside a worker with + `grep 'Max locked memory' /proc/self/limits`. +- Enough node disk for the worker image. `vllm-runtime:1.2.1` is ~13 GB + compressed, larger unpacked, and the frontend pulls it too. On an undersized + node the pull crosses the kubelet ephemeral-storage eviction threshold, which + evicts pods and cancels the extraction with `context canceled`. +- `kubectl` cluster-admin, to install the CRDs. +- A Hugging Face [token](https://huggingface.co/docs/hub/en/security-tokens), + only if you switch to a gated model. The default, + [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B), is ungated and needs + none. For a gated model, both the server and the workers need it. The server + performs the upstream download, and a worker that finds no peer source falls + back to downloading for itself, which is the normal path for the first worker + of a model. Set `hfToken` and the chart wires it into the workers. + +## Versions + +The worker image determines everything else. Both `vllm-runtime:1.2.1` and its +`1.2.1-efa-amd64` variant contain ModelExpress client 0.4.0, vLLM 0.20.1, and +NIXL 0.10.1, so the server is pinned to 0.4.0 to match the client rather than to +the chart version. The EFA variant additionally supplies AWS libfabric. + +| Component | Version | Where it is set | +| --- | --- | --- | +| Worker image | `nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1` | `modelexpress-dynamo/values.yaml` | +| AWS EFA worker image | `nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1-efa-amd64` | `override.yaml.sample` | +| ModelExpress client | 0.4.0 | bundled in the worker image | +| ModelExpress server | `nvcr.io/nvidia/ai-dynamo/modelexpress-server:0.4.0` | `modelexpress-server-values.yaml` | +| ModelExpress CRDs | 0.4.0 | applied in step 1 below | +| ModelExpress chart | 0.5.1 | install command below | +| vLLM | 0.20.1 | bundled in the worker image | +| NIXL | 0.10.1 | bundled in the worker image | + +Only one line in that table says 0.5.1, and it is the chart. You install chart +0.5.1, but the ModelExpress that runs is 0.4.0: the chart is packaging, and the +values file in this directory overrides the server image tag it would otherwise +deploy. The CRDs are applied separately from the 0.4.0 tag for the same reason. +Left unset, chart 0.5.1 would deploy server 0.3.0, which matches nothing here. + +Confirm the client version yourself before changing any of these: + +```bash +docker run --rm --entrypoint bash \ + nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 \ + -c "pip list | grep -iE 'modelexpress|^vllm|nixl'" +``` + +## Step 1: install the CRDs (administrator, once per cluster) + +The chart does not ship its CRDs. Apply them separately, from the tag that matches +the server image rather than the chart, because the schemas differ between +releases. v0.5.1 adds fields such as `sourceType` and `artifactSource` that a +0.4.0 server never writes: + +```bash +kubectl apply -f https://raw.githubusercontent.com/ai-dynamo/modelexpress/v0.4.0/examples/crds.yaml +kubectl get crd | grep modelexpress.nvidia.com +# Example output: +# modelcacheentries.modelexpress.nvidia.com +# modelmetadatas.modelexpress.nvidia.com +``` + +If you re-pin the server image, re-apply the CRDs from the matching tag. + +## Step 2: install the ModelExpress server (administrator, once per cluster) + +Install the upstream chart with the values file in this directory: + +```bash +kubectl create namespace modelexpress +helm upgrade --install modelexpress modelexpress \ + --repo https://helm.ngc.nvidia.com/nvidia/ai-dynamo \ + --version 0.5.1 \ + --namespace modelexpress \ + -f modelexpress-server-values.yaml +``` + +No registry secret is needed: the server image is publicly pullable. Add one only +if you mirror the image privately. + +The default model, `Qwen/Qwen3-0.6B`, is ungated and needs no token either. For a +gated model, add the token before installing. The values file already wires it as +an optional secret reference, so its absence is not an error: + +```bash +kubectl create secret generic hf-token-secret \ + --namespace modelexpress \ + --from-literal=HF_TOKEN="$HF_TOKEN" +``` + +Do not install the chart with its own defaults. Several of them do not work as +shipped, and `modelexpress-server-values.yaml` documents each override inline. +Two will stop the server outright: `MX_METADATA_BACKEND` is required but left +unset, and the chart's `runAsNonRoot: true` with no `runAsUser` cannot start the +chart's own image, which runs as root. A default install fails with +`container has runAsNonRoot and image will run as root`. + +Confirm the server is serving before continuing: + +```bash +kubectl get pods -n modelexpress +kubectl logs -n modelexpress deploy/modelexpress | tail -20 +``` + +## Step 3: deploy the function + +Package the chart and register pull credentials: + +```bash +helm package modelexpress-dynamo +helm push nvcf-modelexpress-dynamo-0.1.0.tgz oci:////charts +nvcf-cli registry-credential add \ + --hostname \ + --username \ + --password \ + --artifact-type HELM \ + --artifact-type CONTAINER +``` + +Create the function. Set `helmChartServiceName` to the +`DynamoGraphDeployment` name plus `-frontend`; this chart names it +`modelexpress-dynamo`, so the service is `modelexpress-dynamo-frontend`. + +```bash +cat < function-create.json +{ + "name": "modelexpress-dynamo-function", + "inferenceUrl": "/v1/chat/completions", + "inferencePort": 8000, + "helmChartServiceName": "modelexpress-dynamo-frontend", + "helmChart": "oci:////charts/nvcf-modelexpress-dynamo-0.1.0.tgz", + "healthProtocol": "HTTP", + "healthUri": "/health", + "healthPort": 8000, + "healthTimeout": "PT10S", + "healthExpectedStatusCode": 200 +} +EOF +nvcf-cli function create --input-file ./function-create.json +``` + +Save the function and version IDs, then deploy: + +```bash +cat < function-deploy.json +{ + "functionId": "", + "versionId": "", + "deploymentSpecifications": [ + { + "gpu": "", + "instanceType": "", + "backend": "nvcf-default", + "minInstances": 1, + "maxInstances": 1 + } + ] +} +EOF +nvcf-cli function deploy create --input-file ./function-deploy.json +``` + +Invoke it: + +```bash +nvcf-cli function invoke \ + --function-id --version-id \ + --request-body '{ + "model": "Qwen/Qwen3-0.6B", + "messages": [{"role": "user", "content": "What is the capital of France?"}], + "stream": false, + "max_tokens": 30 + }' +``` + +## Step 4: confirm ModelExpress is actually being used + +A worker that cannot reach ModelExpress falls back to its native loader and +still serves traffic, so a successful invocation does not prove anything. Check +the worker logs for the ModelExpress loader: + +```bash +kubectl logs -l nvidia.com/dynamo-component-type=worker --tail=200 \ + | grep -iE 'modelexpress|mx |nixl' +``` + +Two specific things to look for: + +- The loader is `modelexpress`. If `--load-format` does not name it, vLLM uses + its own loader and ModelExpress never runs, even with every environment + variable set correctly. +- The intended backend loaded. InfiniBand and RoCE use the `UCX` default. AWS + EFA uses `LIBFABRIC` with `vllm-runtime:1.2.1-efa-amd64` and + `FI_PROVIDER=efa`; require `Backend LIBFABRIC was instantiated` and + `NIXL agent ... (backend=LIBFABRIC)` in the worker log. +- A scale-out worker logged `Receiving ... tensors from source` and `Transfer + complete`, without `Loading weights from disk`. Start one seed worker and + scale from one to two; starting both simultaneously can make both miss a + Ready source and correctly fall back to disk. + +## Step 5: observe scale-out + +Cold-start behavior is the point of ModelExpress, so measure it against the same +topology with ModelExpress switched off. Nothing else in the chart changes +between the two runs: + +```bash +helm template mx-dyn modelexpress-dynamo --set modelExpress.enabled=false +``` + +Compare worker time-to-ready at increasing worker counts, and repeat each step, +because a single run does not show whether the result is consistent: + +```bash +helm upgrade ... --set vllmDecodeWorker.replicas=2 +helm upgrade ... --set vllmDecodeWorker.replicas=4 +helm upgrade ... --set vllmDecodeWorker.replicas=10 +``` + +On EFA, each step needs that many GPU nodes. Surplus replicas stay `Pending` with +`Insufficient vpc.amazonaws.com/efa`, which quietly changes what is being +measured, so scale the node pool alongside the replica count. + +Keep the model, image, node type, and cache state identical across compared +runs. A warm page cache on one run and a cold one on the other produces a +difference that has nothing to do with ModelExpress. + +## Provider-specific networking + +The chart defaults are provider-neutral. Copy `override.yaml.sample` to +`override.yaml` and apply the block matching your fabric: + +```bash +helm template mx-dyn modelexpress-dynamo -f override.yaml +``` + +AWS EFA is the validated path: use the `-efa-amd64` image, the `LIBFABRIC` +backend, `FI_PROVIDER=efa`, and the EFA extended-resource limit in +`override.yaml.sample`. The EFA security group must allow all protocols both +inbound and outbound to itself; an outbound `0.0.0.0/0` rule is not equivalent +for non-IP OS-bypass traffic. Validate with a passing TCP control, an SRD test, +and increasing EFA receive counters on both peers. InfiniBand and RoCE use the +default `UCX` backend and remain unvalidated by NVCF. + +## Cleanup + +```bash +nvcf-cli function deploy delete --function-id --version-id +nvcf-cli function delete --function-id --version-id +``` + +To remove the cluster-wide pieces as well: + +```bash +helm uninstall modelexpress --namespace modelexpress +kubectl delete -f https://raw.githubusercontent.com/ai-dynamo/modelexpress/v0.4.0/examples/crds.yaml +kubectl delete namespace modelexpress +``` + +Deleting the CRDs deletes every `ModelMetadata` and `ModelCacheEntry` in the +cluster. Do not do it while another function is still using ModelExpress. diff --git a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/.helmignore b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/.helmignore new file mode 100644 index 000000000..2a3035a2c --- /dev/null +++ b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/.helmignore @@ -0,0 +1,27 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Patterns to ignore when building packages. +.DS_Store +.git/ +.gitignore +*.swp +*.tmp +*.orig +*~ +.project +.idea/ +*.tmproj +.vscode/ diff --git a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/Chart.yaml b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/Chart.yaml new file mode 100644 index 000000000..e57bf3c42 --- /dev/null +++ b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/Chart.yaml @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: v2 +appVersion: 0.1.0 +description: Dynamo vLLM function that loads model weights through ModelExpress +name: nvcf-modelexpress-dynamo +type: application +version: 0.1.0 diff --git a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/templates/dynamo-graph-deployment.yaml b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/templates/dynamo-graph-deployment.yaml new file mode 100644 index 000000000..0f9dae6bf --- /dev/null +++ b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/templates/dynamo-graph-deployment.yaml @@ -0,0 +1,113 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: nvidia.com/v1alpha1 +kind: DynamoGraphDeployment +metadata: + name: modelexpress-dynamo +spec: + services: + Frontend: + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.labels }} + labels: + {{- toYaml . | nindent 8 }} + {{- end }} + componentType: frontend + replicas: {{ .Values.frontend.replicas | default 1 }} + {{- with .Values.frontend.resources }} + resources: + {{- . | toYaml | nindent 8 }} + {{- end }} + {{- if gt (int .Values.frontend.nodeCount) 1 }} + multinode: + nodeCount: {{ .Values.frontend.nodeCount }} + {{- end }} + extraPodSpec: + mainContainer: + image: {{ .Values.frontend.image }} + envs: + - name: DYN_ROUTER_MODE + value: kv + VllmDecodeWorker: + {{- with .Values.annotations }} + annotations: + {{- toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.labels }} + labels: + {{- toYaml . | nindent 8 }} + {{- end }} + componentType: worker + {{- with $worker := .Values.vllmDecodeWorker }} + # Do not use Helm's `default` here: zero is considered empty, which turns + # an intentional scale-to-zero into one billable GPU worker. + replicas: {{ $worker.replicas }} + {{- if gt (int $worker.nodeCount) 1 }} + multinode: + nodeCount: {{ $worker.nodeCount }} + {{- end }} + resources: + {{- (required "vllmDecodeWorker.resources must be set" $worker.resources) | toYaml | nindent 8 }} + {{- if $.Values.hfToken }} + envFromSecret: hf-token-secret + {{- end }} + {{- if $.Values.modelExpress.enabled }} + envs: + - name: MX_SERVER_ADDRESS + value: {{ required "modelExpress.serverAddress must be set when modelExpress.enabled is true" $.Values.modelExpress.serverAddress | quote }} + {{- if $.Values.modelExpress.setLegacyUrlEnv }} + - name: MODEL_EXPRESS_URL + value: {{ $.Values.modelExpress.serverAddress | quote }} + {{- end }} + {{- if $.Values.modelExpress.setVllmPlugin }} + - name: VLLM_PLUGINS + value: modelexpress + {{- end }} + {{- with $.Values.modelExpress.nixlBackend }} + - name: MX_NIXL_BACKEND + value: {{ . | quote }} + {{- end }} + {{- with $.Values.modelExpress.extraEnv }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} + {{- /* + With modelExpress.enabled=false no ModelExpress environment is emitted at + all, so the worker uses its native loader. That absence is the baseline. + */}} + extraPodSpec: + mainContainer: + image: {{ $worker.image }} + workingDir: /workspace/examples/backends/vllm + command: + - python3 + - -m + - dynamo.vllm + args: + - --model + - {{ $.Values.model | quote }} + {{- if $.Values.modelExpress.enabled }} + # Without this, vLLM uses its own loader and ModelExpress never runs, + # even with MX_SERVER_ADDRESS set. The client registers this format. + - --load-format + - {{ $.Values.modelExpress.loadFormat | default "modelexpress" }} + {{- end }} + - --tensor-parallel-size + - {{ mul (default 1 (int $worker.nodeCount)) (default 1 (int $worker.resources.limits.gpu)) | quote }} + {{- end }} diff --git a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/templates/hf_token_secret.yaml b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/templates/hf_token_secret.yaml new file mode 100644 index 000000000..5c5ffe080 --- /dev/null +++ b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/templates/hf_token_secret.yaml @@ -0,0 +1,26 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Only rendered when hfToken is set. Workers pulling weights from ModelExpress +# do not need their own token, because the server performs the upstream fetch. +{{- if .Values.hfToken }} +kind: Secret +apiVersion: v1 +metadata: + name: hf-token-secret +type: Opaque +data: + HF_TOKEN: {{ .Values.hfToken | b64enc }} +{{- end }} diff --git a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/values.yaml b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/values.yaml new file mode 100644 index 000000000..eefcd0ab6 --- /dev/null +++ b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/values.yaml @@ -0,0 +1,121 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Model served by the function. Keep this small while validating: a small model +# makes cold-start differences easy to observe and keeps a 10 worker run cheap. +model: Qwen/Qwen3-0.6B + +modelExpress: + # Set to false to run the identical topology with native model loading. Use + # this to collect the baseline that the ModelExpress comparison is measured + # against. Nothing else in this chart changes between the two runs. + enabled: true + + # Address of the ModelExpress server installed by the cluster administrator. + # Namespace and release name come from modelexpress-server-values.yaml. + serverAddress: modelexpress.modelexpress.svc.cluster.local:8001 + + # The client in vllm-runtime:1.2.1 (modelexpress 0.4.0) reads both + # MODEL_EXPRESS_URL and MX_SERVER_ADDRESS. Setting both is safe. Only drop + # this if you confirmed the newer name in the image you actually run. + setLegacyUrlEnv: true + + # vLLM 0.23 and later detect ModelExpress without help. vllm-runtime:1.2.1 + # ships vLLM 0.20.1, which does not, so the plugin must be named explicitly. + setVllmPlugin: true + + # vLLM only uses the ModelExpress loader when load-format names it. The client + # registers "modelexpress" and "mx". Anything else silently falls back to the + # native loader, which looks like success while ModelExpress does nothing. + loadFormat: modelexpress + + # NIXL transport. The client accepts UCX or LIBFABRIC only, and defaults to + # UCX. UCX is the provider-neutral default and the path for InfiniBand/RoCE. + # AWS EFA instead needs LIBFABRIC plus the -efa runtime image that supplies + # AWS libfabric. Both peers must agree. See override.yaml.sample. + # + # Known defect on this path. With modelexpress 0.4.0 in vllm-runtime:1.2.1, + # the UCX backend segfaulted the source worker while it was serving a + # transfer, inside nixlUcxSharedThread::run(). The source dies with + # EngineDeadError and the receiver reports NIXL_ERR_REMOTE_DISCONNECT. It + # reproduced over plain TCP, so it is not specific to a fabric, and it did + # not depend on UCX_RNDV_SCHEME. LIBFABRIC on EFA does not hit it. Only EFA + # was validated here, so if you run InfiniBand or RoCE on this client + # version, transfer a model between two workers before trusting the path. + nixlBackend: UCX + + # Extra environment for the worker, passed straight through. Use this for + # provider-specific transport configuration and tuning. + # + # UCX picks a transport by advertised bandwidth, so it prefers the fabric over + # TCP whenever an RDMA device is present. If that device is present but not + # functional, the transfer stalls until it times out instead of failing fast, + # and the fallback to disk then costs minutes. Confining UCX to TCP is the way + # to keep peer-to-peer working while a fabric problem is outstanding: + # + # extraEnv: + # - name: UCX_TLS + # value: tcp,self,sm + # + # Verify the fabric with ucx_perftest before trusting it. See README.md. + extraEnv: [] + + # There is deliberately no knob here for the locked-memory limit, because it + # cannot be fixed from a chart. RDMA registers pinned memory, RLIMIT_MEMLOCK + # caps how much a process may pin, and containers inherit that limit from the + # node's container runtime. When it is too low UCX cannot register buffers: + # + # ibv_reg_mr(...) failed: Cannot allocate memory : Please set max locked + # memory (ulimit -l) to 'unlimited' (current: 8192 kbytes) + # + # and the worker crashes during engine init. Kubernetes has no ulimit field, + # and adding CAP_IPC_LOCK does not help: this image runs as UID 1000, and a + # non-root process has an empty effective capability set, so the capability is + # never active. The node's runtime has to allow it. See the prerequisites in + # README.md for how to check and set this. + +frontend: + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + replicas: 1 + nodeCount: 1 + +# Worker replica count is the scale-out dimension under test. Validate at 2, 4, +# and 10 and repeat each step, because a single run does not show whether the +# result is consistent. +# +# On AWS EFA, keep replicas at or below the advertised vpc.amazonaws.com/efa +# units. An EFA interface is not shareable between Pods, so on single-interface +# instances such as the EFA-capable G-family that equals the GPU node count. +# Surplus replicas sit Pending with Insufficient vpc.amazonaws.com/efa and the +# run stops measuring what it claims. +vllmDecodeWorker: + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 + replicas: 2 + nodeCount: 1 + resources: + limits: + # Must be "gpu", not "nvidia.com/gpu". The Dynamo operator translates it. + gpu: "1" + requests: + custom: + # Raise this for larger models. + ephemeral-storage: "8Gi" + +# Token for gated model downloads on the worker side. Leave empty when the +# ModelExpress server already holds the weights. +hfToken: "" + +annotations: {} +labels: {} diff --git a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-server-values.yaml b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-server-values.yaml new file mode 100644 index 000000000..0b20a2ce8 --- /dev/null +++ b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-server-values.yaml @@ -0,0 +1,151 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Cluster-admin values for the upstream ModelExpress chart. +# +# Apply the CRDs from the v0.4.0 tag first, matching the server pinned below +# rather than the chart version. See README.md step 1. +# +# Then install the upstream chart with this file, once per cluster, before +# deploying any function that uses ModelExpress: +# +# helm upgrade --install modelexpress modelexpress \ +# --repo https://helm.ngc.nvidia.com/nvidia/ai-dynamo \ +# --version 0.5.1 \ +# --namespace modelexpress --create-namespace \ +# -f modelexpress-server-values.yaml +# +# The only 0.5.1 here is the chart. What runs is ModelExpress 0.4.0. +# +# Every value below overrides an upstream default that is unsafe or non-working +# for a self-managed NVCF compute plane. Do not drop these overrides. + +# Pin the server to the client version that ships in the worker image, not to +# the chart version. +# +# nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1 contains modelexpress 0.4.0, so +# the server is pinned to 0.4.0 to match it. Chart 0.5.1 is used because it is +# the published chart, and its image.tag default of "0.3.0" matches neither the +# chart nor the client, so it must be overridden either way. +# +# Matching client and server avoids relying on cross-version interoperability +# that has no published compatibility statement. Revisit this pin when the +# worker image ships a newer client. +image: + repository: nvcr.io/nvidia/ai-dynamo/modelexpress-server + tag: "0.4.0" + pullPolicy: IfNotPresent + +# Upstream defaults this to a secret named nvcr-secret and assumes it exists in +# the release namespace, which makes a default install fail on a cluster that has +# no such secret. +# +# The server image is publicly pullable, verified by pulling +# modelexpress-server:0.4.0 onto a node with no registry credentials at all, so +# no secret is needed for it. Set this only if you mirror the image into a +# private registry, or if you pull other gated NGC content: +# +# kubectl create secret docker-registry nvcr-secret \ +# --namespace modelexpress \ +# --docker-server=nvcr.io \ +# --docker-username='$oauthtoken' \ +# --docker-password="$NGC_API_KEY" +imagePullSecrets: [] + +serviceAccount: + create: true + # Required for MX_METADATA_BACKEND=kubernetes. Upstream ships this disabled, + # so the server cannot reach its own CRDs until you turn it on. Grants a + # namespaced Role only, not a ClusterRole. + rbac: + enabled: true + # Not set on purpose. The chart renders + # `automountServiceAccountToken: {{ .Values.serviceAccount.automount | default true }}`, + # and Helm's `default` treats `false` as empty, so the chart's own + # `automount: false` is inert and the token is always mounted. Setting it here + # would imply control that the chart does not actually offer. The token is + # required anyway for the kubernetes metadata backend. + +# The chart's own defaults cannot start the chart's own image: it sets +# runAsNonRoot: true in both securityContext blocks and specifies no runAsUser, +# while the server image runs as root. A default install fails with +# "container has runAsNonRoot and image will run as root". +# +# Rather than dropping to root, pin an explicit non-root UID. fsGroup is what +# makes the mounted cache volume writable by that UID. +podSecurityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + fsGroup: 1000 + +securityContext: + runAsNonRoot: true + runAsUser: 1000 + +env: + MODEL_EXPRESS_SERVER_PORT: "8001" + MODEL_EXPRESS_LOGGING_LEVEL: "info" + # Not the upstream default of /root, which is unwritable once the server runs + # as a non-root UID. Must match persistence.mountPath below. + MODEL_EXPRESS_CACHE_DIRECTORY: "/cache" + # Required. The server fails to start when this is unset, and upstream leaves + # it commented out, so a default install crash-loops. + # + # "kubernetes" keeps metadata in the ModelMetadata and ModelCacheEntry CRDs + # and needs no extra datastore. "redis" is also supported but requires a Redis + # you deploy and operate yourself, because the chart bundles no Redis subchart. + MX_METADATA_BACKEND: "kubernetes" + +# Single writer against a ReadWriteOnce volume. Keep replicaCount at 1 unless +# you switch to a ReadWriteMany storage class. +replicaCount: 1 + +persistence: + enabled: true + accessMode: ReadWriteOnce + # Size this to hold every model your workers request, not just one. + size: 50Gi + # Must match MODEL_EXPRESS_CACHE_DIRECTORY above. + mountPath: /cache + # Set to a class that exists in your cluster. Empty uses the cluster default. + storageClass: "" + +# Gated or private model downloads need credentials on the server, because the +# server performs the initial fetch that peers then replicate from. +# +# kubectl create secret generic hf-token-secret \ +# --namespace modelexpress --from-literal=HF_TOKEN="$HF_TOKEN" +extraEnv: + - name: HF_TOKEN + valueFrom: + secretKeyRef: + name: hf-token-secret + key: HF_TOKEN + optional: true + +service: + type: ClusterIP + port: 8001 + +resources: + limits: + cpu: "2" + memory: 4Gi + ephemeral-storage: 2Gi + requests: + cpu: 500m + memory: 1Gi + ephemeral-storage: 512Mi diff --git a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/override.yaml.sample b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/override.yaml.sample new file mode 100644 index 000000000..ba88cd43f --- /dev/null +++ b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/override.yaml.sample @@ -0,0 +1,131 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Provider-specific overrides. Copy to override.yaml and edit. +# +# helm template mx-dyn modelexpress-dynamo -f override.yaml +# +# The chart defaults are provider-neutral and carry no cloud-specific value. +# Pick the one block below that matches your fabric. + +# --------------------------------------------------------------------------- +# AWS EFA +# --------------------------------------------------------------------------- +# EFA uses NIXL's LIBFABRIC backend. Use the -efa image variant: the stock +# vllm-runtime:1.2.1 image does not ship libfabric.so.1, while +# vllm-runtime:1.2.1-efa-amd64 supplies AWS libfabric under /opt/amazon/efa. +# A validated worker logs both "Backend LIBFABRIC was instantiated" and +# "NIXL agent ... created ... (backend=LIBFABRIC)". +# +# EFA also has to be present on the node before any of this matters. An EFA +# interface can only be attached when the instance launches, so a running node +# without one cannot be upgraded in place. Confirm the node advertises the EFA +# resource before assuming the fabric is available: +# +# kubectl get nodes -o json \ +# | jq '.items[].status.allocatable | keys' | grep -i efa +# +# The security group attached to every EFA must allow all protocols both inbound +# and outbound to itself. An outbound 0.0.0.0/0 rule is not a substitute for the +# self-reference because EFA OS-bypass traffic is non-IP. Validate the fabric +# with a TCP control followed by ucx_perftest over srd, and require EFA hardware +# receive counters to increase on both peers. +# +modelExpress: + nixlBackend: LIBFABRIC + extraEnv: + - name: FI_PROVIDER + value: efa + - name: MX_ARTIFACT_READY_URL + value: http://127.0.0.1:9090/health + - name: MX_ARTIFACT_TRANSFER + value: "1" + - name: UCX_RNDV_SCHEME + value: get_zcopy + - name: UCX_RNDV_THRESH + value: "0" + - name: VLLM_RPC_TIMEOUT + value: "7200000" + +vllmDecodeWorker: + image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.2.1-efa-amd64 + resources: + limits: + gpu: "1" + custom: + # EFA must be declared under limits, not requests. It is an extended + # resource, and Kubernetes rejects a Pod that requests one without a + # matching limit: + # + # Pod "..." is invalid: spec.containers[0].resources.limits: + # Required value: Limit must be set for non overcommitable resources + # + # The Dynamo operator passes this block through verbatim, so putting it + # under requests produces an invalid Pod that never gets created. The + # PodClique reports ERR_CREATE_POD and the pods simply never appear, + # which looks like a scheduling problem rather than a spec problem. + # + # Requires the EFA device plugin on the cluster, which is what makes the + # interface schedulable at all. Peers must also land in the same subnet; + # a cluster placement group is a latency optimization, not a requirement. + # + # This also caps workers at one per node. EFA-capable G-family instances + # expose a single EFA interface and the plugin advertises one unit per + # node, so a second worker here stays Pending with "Insufficient + # vpc.amazonaws.com/efa". Scale replicas with nodes, not with GPUs per + # node. + vpc.amazonaws.com/efa: "1" + requests: + custom: + ephemeral-storage: "8Gi" + +# --------------------------------------------------------------------------- +# InfiniBand or RoCE +# --------------------------------------------------------------------------- +# Both use the UCX backend, which is already the chart default, so no +# modelExpress override is needed. What differs is the device resource the node +# advertises. Replace the resource name below with whatever your RDMA device +# plugin exposes, then apply only the vllmDecodeWorker block. +# +# Not validated by NVCF. Neither fabric exists on AWS, so the settings below are +# derived from upstream client behavior rather than from a test run. Treat them +# as a starting point and verify on your own hardware. +# +# Verify with a real transfer, not just a healthy worker. On this client version +# the UCX backend segfaulted the source worker mid-transfer in +# nixlUcxSharedThread::run(), and it reproduced over plain TCP, so the crash is +# not fabric-specific and may well appear on InfiniBand or RoCE too. The symptom +# is a source that dies with EngineDeadError while receivers report +# NIXL_ERR_REMOTE_DISCONNECT. A worker that reaches Ready proves nothing here, +# because a failed transfer falls back to downloading and still serves correctly. +# +# The device resource goes under limits, for the same reason EFA does above: an +# extended resource must be declared there or the Pod is rejected at admission. +# The name varies by plugin, so check what your nodes advertise with +# kubectl get nodes -o json | jq '.items[].status.allocatable | keys' +# +# vllmDecodeWorker: +# resources: +# limits: +# gpu: "1" +# custom: +# rdma/hca: "1" +# requests: +# custom: +# ephemeral-storage: "8Gi" +# +# On hosts with several RDMA NICs, MX_RDMA_NIC_PIN and +# MX_RDMA_NIC_PIN_MIN_RATE_GBPS control which one is selected. Leave them unset +# until you have a measured reason to change the default. From 760c7565c56bc911e025e82d4031adf16e378447 Mon Sep 17 00:00:00 2001 From: Ankit Anand Date: Sat, 29 Aug 2026 13:43:30 -0700 Subject: [PATCH 2/2] docs(examples): correct the teardown command and memlock guidance Use function deploy remove; deploy registers remove, not delete. Note that a gated model needs the worker token even when the server holds the weights, and stop asserting that CAP_IPC_LOCK cannot work for a non-root UID. Signed-off-by: Ankit Anand --- .../modelexpress-dynamo-sample/README.md | 11 ++++++----- .../modelexpress-dynamo/values.yaml | 6 ++++-- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/README.md b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/README.md index dcc5b6d75..1a58f0a6f 100644 --- a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/README.md +++ b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/README.md @@ -68,10 +68,11 @@ uses it. The custom resources themselves are namespaced. sudo systemctl daemon-reload && sudo systemctl restart containerd ``` - This cannot be fixed from the chart. Kubernetes has no ulimit field, and - granting `CAP_IPC_LOCK` does not help, because the worker image runs as UID - 1000 and a non-root process has an empty effective capability set. Confirm - from inside a worker with + Kubernetes has no ulimit field, so the chart cannot set this. Adding + `CAP_IPC_LOCK` is the usual workload-level answer and that capability is meant + to bypass `RLIMIT_MEMLOCK`, but it did not lift the limit for this image in our + testing, and whether it works at all depends on the runtime and cluster policy. + Either way, confirm from inside a worker with `grep 'Max locked memory' /proc/self/limits`. - Enough node disk for the worker image. `vllm-runtime:1.2.1` is ~13 GB compressed, larger unpacked, and the frontend pulls it too. On an undersized @@ -318,7 +319,7 @@ default `UCX` backend and remain unvalidated by NVCF. ## Cleanup ```bash -nvcf-cli function deploy delete --function-id --version-id +nvcf-cli function deploy remove --function-id --version-id nvcf-cli function delete --function-id --version-id ``` diff --git a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/values.yaml b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/values.yaml index eefcd0ab6..794049f6d 100644 --- a/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/values.yaml +++ b/examples/function-samples/helmchart-samples/modelexpress-dynamo-sample/modelexpress-dynamo/values.yaml @@ -113,8 +113,10 @@ vllmDecodeWorker: # Raise this for larger models. ephemeral-storage: "8Gi" -# Token for gated model downloads on the worker side. Leave empty when the -# ModelExpress server already holds the weights. +# Token for gated model downloads on the worker side. A worker that finds no peer +# source downloads the model itself, which is the normal path for the first +# worker, so a gated model needs this set even when the server holds the weights. +# Safe to leave empty only for ungated models such as the default Qwen/Qwen3-0.6B. hfToken: "" annotations: {}