fix: graceful UNAVAILABLE instead of NULL-deref crash on shared-memory requests (TRI-1698)

[akhilraj9]

What does the PR do?

When a frontend is started through the in-process Python bindings (tritonfrontend),
HTTPAPIServer/gRPC are constructed with a null SharedMemoryManager (TODO DLIS-7194).
An unauthenticated client could crash the whole server process with a single inference
(or shared-memory control) request referencing shared memory, because the request handlers
dereferenced shm_manager_ with no null check (CWE-476 / remote DoS).

This guards every untrusted-input-reachable shm_manager_ dereference and returns
TRITONSERVER_ERROR_UNAVAILABLE (HTTP 503 / gRPC UNAVAILABLE) instead of dereferencing.
No behavior change on the standard tritonserver binary path (manager is non-null there).

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

• fix

Related PRs:

None.

Where should the reviewer start?

• src/http_server.cc — ParseJsonTritonIO (infer input + output guards) and HandleSystemSharedMemory / HandleCudaSharedMemory
• src/grpc/infer_handler.{cc,h} — InferGRPCToInput and InferAllocatorPayload
• src/grpc/grpc_server.cc — the six CommonHandler shm control lambdas
• src/common.h — shared kSharedMemoryManagerUnavailableErrorStr
• qa/L0_python_api/test_shared_memory_frontend.py — regression test

Test plan:

• Reproduced on nvcr.io/nvidia/tritonserver:26.03-py3 (v2.67.0): server healthy → one inference with shared_memory_region → Fatal Python error: Segmentation fault, process exit 139.

• New subprocess-isolated e2e test qa/L0_python_api/test_shared_memory_frontend.py (wired into test.sh): fails on the unpatched build (detects the crash), asserts a graceful 503 + live server on the patched build.

• e2e tests added: yes.

• CI Pipeline ID: [63707556]

Caveats:

Enabling shared memory through the Python bindings remains out of scope (tracked by DLIS-7194); this PR only makes the unsupported path fail gracefully instead of crashing.

Background

PSIRT TRI-1698. Triage requested a graceful unsupported/unavailable error rather than process termination.

Related Issues:

Relates to TRI-1698

[greptile-apps]

Greptile Summary

The PR prevents null SharedMemoryManager dereferences in Python-launched HTTP and gRPC frontends.

• Returns UNAVAILABLE for shared-memory inference inputs and outputs when no manager exists.
• Applies equivalent guards to HTTP and gRPC shared-memory management endpoints.
• Adds subprocess-isolated HTTP regression coverage that verifies a 503 response and continued server health.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/http_server.cc	Guards HTTP inference input/output resolution and system/CUDA shared-memory control handlers before manager access.
src/grpc/grpc_server.cc	Guards all six system and CUDA shared-memory management RPC handlers and returns gRPC UNAVAILABLE.
src/grpc/infer_handler.cc	Rejects shared-memory gRPC inputs before dereferencing an unavailable manager.
src/grpc/infer_handler.h	Rejects shared-memory output allocation before attempting region lookup through an unavailable manager.
src/common.h	Adds the shared client-facing error message used consistently across protocol frontends.
qa/L0_python_api/test_shared_memory_frontend.py	Adds isolated HTTP regression tests for inference and control requests, including post-error health checks.
qa/L0_python_api/test.sh	Integrates the new shared-memory frontend regression module into the Python API suite.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    C[Client shared-memory request] --> F{HTTP or gRPC frontend}
    F --> M{SharedMemoryManager available?}
    M -- No --> U[Return UNAVAILABLE]
    U --> H[Frontend remains healthy]
    M -- Yes --> R[Resolve or manage shared-memory region]
    R --> I[Continue inference or control operation]

Loading

_{Reviews (4): Last reviewed commit: "fix: return UNAVAILABLE instead of crash..." | Re-trigger Greptile}

[Vinya567]

Please remove TRI numbers.

[akhilraj9]

Good catch, thank you. I removed the TRI numbers.

[yinggeh]

It's okay if it is a TODO reminder of future works. Here is not the case.

[akhilraj9]

Thanks Yingge, agreed — trimmed the comments down to one line each and removed the TRI numbers. Much cleaner now.

[whoisj]

LGTM. @yinggeh?

[Vinya567]

same. remove TRI, and keep the comments clean.

[Vinya567]

same here.

[Vinya567]

same as above