Skip to content

feat: add distributed map operation - #1

Draft
nvasiu wants to merge 1 commit into
mainfrom
feat/map-run
Draft

feat: add distributed map operation#1
nvasiu wants to merge 1 commit into
mainfrom
feat/map-run

Conversation

@nvasiu

@nvasiu nvasiu commented Aug 20, 2026

Copy link
Copy Markdown
Owner

Summary

Adds the distributed map operation (ctx.distributed_map) to the Python SDK:
A map run processes a bounded dataset in parallel. A customer starts a map run from a durable function, naming a source to read items from, a processor function to invoke per batch, and concurrency, retry, and failure settings. The service reads items from the source, groups them into batches, invokes the processor for each batch, retries failures, tracks progress, routes successful results and failed items to destinations, and reports completion.

Changes

concurrency/models.py

  • DistributedMapSummary: what ctx.distributed_map returns, describes run's overall outcome.
  • DistributedMapResult: returned from ctx.distributed_map when inline result collection is enabled. Contains individual map run item outcomes.
  • DistributedMapResultItem and DistributedMapItemError: represent a single item's result / error

config.py

  • DistributedMapConfig: optional settings for distributed map
  • DistributedMapSource: describes where map run items come from (inline list, S3, or a custom reader)
  • DistributedMapProcessor: describes the Lambda that processes items and how outcomes are reported back
  • ProcessorRetryConfig: configures how failing items are retried
  • DistributedMapCompletionConfig: defines item failure thresholds for marking the overall map run failed
  • SuccessDestination, FailureDestination, DistributedMapDestinationConfig, DistributedMapDestination: for routing successful and failed item records to S3

context.py

  • ctx.distributed_map: the entry point a customer calls to run a distributed map

distributed_map_helpers.py

  • Authoring wrappers: let a customer write a plain function and have it work as a processor Lambda without hand-writing the item or batch protocol, including durable-execution variants and a reader
  • Currently placed in a separate top level file, can be moved elsewhere.

operation/distributed_map.py

  • The executor: drives the operation so the caller's function suspends while the run executes and resumes with the finished outcome, and surfaces a clear error if the operation itself fails

lambda_service.py

  • Carries the operation and its results to and from the backend service

state.py

  • Stores the run's outcome in the durable execution state so it persists across suspend and resume

exceptions.py

  • DistributedMapError: the error a customer catches when a run or an item fails

__init__.py

  • Makes the distributed-map types importable by customers as public API

Tests

tests/operation/distributed_map_test.py
tests/context_test.py

  • Core operation unit tests: executor, config/argument validation, wire round trips, result types

tests/e2e/distributed_map_int_test.py

  • End to end ctx.distributed_map tests, mocking backend responses: suspend / resume, collect results, throw on failure

tests/distributed_map_helpers_test.py

  • Authoring wrapper tests: checking that they process items, report failures, reject bad inputs

tests/e2e/distributed_map_helpers_int_test.py

  • End to end authoring wrapper tests.

TODO

  • When the model changes and distributed map implementation are complete in the durable service backend, these SDK changes need to be verified against them.

Future Tasks

  • Add distributed map to the local emulator (in the testing package).
    • When this is done, we can add full end to end tests using the emulator.
  • Add distributed map examples to the examples package.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@nvasiu
nvasiu deployed to ai-pr-review August 20, 2026 21:10 — with GitHub Actions Active
@nvasiu
nvasiu had a problem deploying to ai-pr-review-runtime August 20, 2026 21:10 — with GitHub Actions Failure
@nvasiu
nvasiu had a problem deploying to ai-pr-review-runtime August 20, 2026 21:10 — with GitHub Actions Failure
- Add ctx.distributed_map with inline, S3, and reader sources
- Add DistributedMapConfig, processor, completion, and destination
  config types
- Add DistributedMapResult/Summary result types and DistributedMapError
- Add function-authoring helpers for item and batch handlers
- Serialize the DISTRIBUTED_MAP operation and add its executor
@nvasiu
nvasiu deployed to ai-pr-review August 20, 2026 21:27 — with GitHub Actions Active
@nvasiu
nvasiu had a problem deploying to ai-pr-review-runtime August 20, 2026 21:27 — with GitHub Actions Failure
@nvasiu
nvasiu had a problem deploying to ai-pr-review-runtime August 20, 2026 21:27 — with GitHub Actions Failure
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant