Skip to content

Move InvokerMemoryTest services to a Dockerized Rust container#428

Merged
tillrohrmann merged 1 commit into
restatedev:mainfrom
tillrohrmann:fix-invoker-memory-test
Jun 5, 2026
Merged

Move InvokerMemoryTest services to a Dockerized Rust container#428
tillrohrmann merged 1 commit into
restatedev:mainfrom
tillrohrmann:fix-invoker-memory-test

Conversation

@tillrohrmann

Copy link
Copy Markdown
Contributor

Background: InvokerMemoryTest previously bound MemoryPressureService and StatefulObject in-process in the JVM via Endpoint.bind(...), with the Restate container reaching the SDK through the testcontainers host-port relay (host.testcontainers.internal). Under sustained sequential load, that relay throttled service-runtime traffic to a fraction of normal throughput. We were not able to pinpoint the exact mechanism (single-threaded socat hop, lack of HTTP/2 flow-control awareness, and connection-tracking on the bridge network are all candidates), but the slowdown reproduces only on the relay path — sibling-container deployments on the same bridge network are unaffected. Switching the test off the relay was the most actionable mitigation.

Implementation:

  • New aggregate Rust binary at e2e-tests/services/rust/ packaged as ghcr.io/restatedev/e2e-test-services-rs. The binary reads the SERVICES env var (already injected by ServiceSpec.withServices(...)) and conditionally binds the requested services, matching the sdk-tests convention. Adding a new Rust-side e2e service is now a module + match-arm change — no new image.
  • MemoryPressureService and StatefulObject implemented under src/invoker_memory.rs with #[restate_sdk::service] / #[restate_sdk::object] and explicit #[name = "..."] mappings so the camelCase Restate names line up with the Kotlin contracts.
  • Kotlin-side contracts promoted to top-level interfaces under e2e-tests/.../contracts/, matching the sdk-tests layout.
  • infra: new RestateDeployer.Builder.withServiceDeploymentConfig(...) so a test can pin its own image without going through the global --service-container-image CLI flag.
  • InvokerMemoryTest swaps withEndpoint(Endpoint.bind(...)) for withServiceDeploymentConfig + withServiceSpec. The runtime now reaches the service over the standard bridge network at http://invoker-memory:9080/ — no SSH relay involved.

@tillrohrmann

Copy link
Copy Markdown
Contributor Author

I chose to implement the test service in Rust to go in the direction of rustify and thereby making the e2e test infrastructure more accessible for the runtime team.

@muhamadazmy muhamadazmy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes looks good to me. But I think @slinkydeveloper can judge the Kotlin changes way better than me

Comment thread e2e-tests/services/rust/src/stateful_object.rs
@tillrohrmann tillrohrmann force-pushed the fix-invoker-memory-test branch from f2ea84e to 61670a9 Compare June 5, 2026 08:18
@slinkydeveloper

slinkydeveloper commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator

let's stall on this for the time being. I prefer we continue to cover and investigate issues of an SDK our users actually use instead than rust.

Will revisit post 1.7

@tillrohrmann

tillrohrmann commented Jun 5, 2026

Copy link
Copy Markdown
Contributor Author

I would like to get this in because it causes test instability on CI and every time there is such a failure someone needs to go and take a look at it. I am happy to change it back to something else later but stalling it is not the right call at this point imo.

Unrelated side note: Whether the Java SDK is more frequently used than the Rust SDK is not so clear to me. The big benefit is that now the runtime team can work with these tests if they are not using the Java SDK.

Background: InvokerMemoryTest previously bound MemoryPressureService and
StatefulObject in-process in the JVM via Endpoint.bind(...), with the
Restate container reaching the SDK through the testcontainers
host-port relay (host.testcontainers.internal). Under sustained
sequential load, that relay throttled service-runtime traffic to a
fraction of normal throughput. We were not able to pinpoint the exact
mechanism (single-threaded socat hop, lack of HTTP/2 flow-control
awareness, and connection-tracking on the bridge network are all
candidates), but the slowdown reproduces only on the relay path —
sibling-container deployments on the same bridge network are
unaffected. Switching the test off the relay was the most actionable
mitigation.

Implementation:
- New aggregate Rust binary at e2e-tests/services/rust/ packaged as
  ghcr.io/restatedev/e2e-test-services-rs. The binary reads the
  SERVICES env var (already injected by ServiceSpec.withServices(...))
  and conditionally binds the requested services, matching the
  sdk-tests convention. Adding a new Rust-side e2e service is now a
  module + match-arm change — no new image.
- MemoryPressureService and StatefulObject implemented under
  src/invoker_memory.rs with #[restate_sdk::service] /
  #[restate_sdk::object] and explicit #[name = "..."] mappings so the
  camelCase Restate names line up with the Kotlin contracts.
- Kotlin-side contracts promoted to top-level interfaces under
  e2e-tests/.../contracts/, matching the sdk-tests layout.
- infra: new RestateDeployer.Builder.withServiceDeploymentConfig(...)
  so a test can pin its own image without going through the global
  --service-container-image CLI flag.
- InvokerMemoryTest swaps withEndpoint(Endpoint.bind(...)) for
  withServiceDeploymentConfig + withServiceSpec. The runtime now
  reaches the service over the standard bridge network at
  http://invoker-memory:9080/ — no SSH relay involved.

Image must be published manually once before this branch passes CI:
  docker buildx build --platform linux/amd64,linux/arm64 \\
    -t ghcr.io/restatedev/e2e-test-services-rs:0.1.0 \\
    --push e2e-tests/services/rust

See e2e-tests/services/rust/README.md for the full publish flow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@tillrohrmann tillrohrmann force-pushed the fix-invoker-memory-test branch from 61670a9 to 7fd4456 Compare June 5, 2026 15:33
@tillrohrmann tillrohrmann merged commit 7fd4456 into restatedev:main Jun 5, 2026
3 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 5, 2026
@tillrohrmann tillrohrmann deleted the fix-invoker-memory-test branch June 5, 2026 15:34
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants