Skip to content

Avoid same-process CUDA publish synchronization - #18

Open
nvcyc wants to merge 2 commits into
mainfrom
nvcyc/fix_intra_process_event
Open

Avoid same-process CUDA publish synchronization#18
nvcyc wants to merge 2 commits into
mainfrom
nvcyc/fix_intra_process_event

Conversation

@nvcyc

@nvcyc nvcyc commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Description

This PR removes an unnecessary publish-path synchronization when CUDA-backed messages are transported between nodes in the same process through the regular DDS path.

Previously, the CUDA serialization backend attempted to use a CUDA IPC event for every descriptor. CUDA IPC event handles cannot be reopened in the process that created them, so same-process descriptor
transport called cudaEventSynchronize() before publishing. This preserved correctness but blocked the publisher until all queued CUDA work completed.

Same-process event transport

The CUDA allocation pool now owns a local CUDA event for each IPC-capable VMM block.

When serializing a descriptor for a same-process endpoint, the backend:

  • Waits for the producer’s write event on an internal CUDA stream.
  • Records the pool-owned local event after that wait.
  • Stores the local cudaEvent_t in the descriptor.
  • Returns from publication without waiting for the GPU work to finish.

The receiving backend attaches this event to the imported CUDA buffer as a non-owning write event. A subsequent ReadHandle waits for it on the consumer stream, preserving stream ordering without
blocking the publisher thread.

If a local event cannot be created, the backend retains the previous cudaEventSynchronize() behavior as a correctness-preserving fallback.

Descriptor event types

CudaBufferDescriptor now explicitly identifies how its event data should be interpreted:

  • EVENT_NONE — no producer event is attached.
  • EVENT_LOCAL — the descriptor contains a same-process cudaEvent_t.
  • EVENT_IPC — the descriptor contains an inter-process cudaIpcEventHandle_t.

This avoids relying on PID comparisons, which can be incorrect across container PID namespaces.

Inter-process transport continues to use CUDA IPC events. Imported IPC events remain receiver-owned, while same-process local events remain owned by the publisher’s allocation pool.

Regression testing

This PR adds a dedicated Fast RTPS launch test for same-process descriptor transport.

The test:

  • Loads a dedicated CUDA publisher component and subscriber into the same process.
  • Disables ROS intra-process communication so the message uses the regular DDS serialization path.
  • Queues delayed asynchronous CUDA work before each publication.
  • Publishes five CUDA-backed messages with distinct known payloads.
  • Verifies that every publish() returns before the queued CUDA work completes.
  • Checks the received backend before constructing a ReadHandle.
  • Requires every received message to remain CUDA-backed.
  • Materializes the data only for final validation and checks every payload.
  • Requires explicit backend, content, metadata, and nonblocking validation results for all five messages.

Before the fix, the regression test receives valid CUDA payloads but fails because every publication reports nonblocking=false.

After the fix, all five publications report nonblocking=true, and all received CUDA payloads validate successfully.

@nvcyc
nvcyc marked this pull request as ready for review August 28, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant