feat: add distributed map operation - #1
Draft
nvasiu wants to merge 1 commit into
Draft
Conversation
nvasiu
had a problem deploying
to
ai-pr-review-runtime
August 20, 2026 21:10 — with
GitHub Actions
Failure
nvasiu
had a problem deploying
to
ai-pr-review-runtime
August 20, 2026 21:10 — with
GitHub Actions
Failure
- Add ctx.distributed_map with inline, S3, and reader sources - Add DistributedMapConfig, processor, completion, and destination config types - Add DistributedMapResult/Summary result types and DistributedMapError - Add function-authoring helpers for item and batch handlers - Serialize the DISTRIBUTED_MAP operation and add its executor
nvasiu
had a problem deploying
to
ai-pr-review-runtime
August 20, 2026 21:27 — with
GitHub Actions
Failure
nvasiu
had a problem deploying
to
ai-pr-review-runtime
August 20, 2026 21:27 — with
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the distributed map operation (
ctx.distributed_map) to the Python SDK:A map run processes a bounded dataset in parallel. A customer starts a map run from a durable function, naming a source to read items from, a processor function to invoke per batch, and concurrency, retry, and failure settings. The service reads items from the source, groups them into batches, invokes the processor for each batch, retries failures, tracks progress, routes successful results and failed items to destinations, and reports completion.
Changes
concurrency/models.pyDistributedMapSummary: whatctx.distributed_mapreturns, describes run's overall outcome.DistributedMapResult: returned fromctx.distributed_mapwhen inline result collection is enabled. Contains individual map run item outcomes.DistributedMapResultItemandDistributedMapItemError: represent a single item's result / errorconfig.pyDistributedMapConfig: optional settings for distributed mapDistributedMapSource: describes where map run items come from (inline list, S3, or a custom reader)DistributedMapProcessor: describes the Lambda that processes items and how outcomes are reported backProcessorRetryConfig: configures how failing items are retriedDistributedMapCompletionConfig: defines item failure thresholds for marking the overall map run failedSuccessDestination,FailureDestination,DistributedMapDestinationConfig,DistributedMapDestination: for routing successful and failed item records to S3context.pyctx.distributed_map: the entry point a customer calls to run a distributed mapdistributed_map_helpers.pyoperation/distributed_map.pylambda_service.pystate.pyexceptions.pyDistributedMapError: the error a customer catches when a run or an item fails__init__.pyTests
tests/operation/distributed_map_test.pytests/context_test.pytests/e2e/distributed_map_int_test.pyctx.distributed_maptests, mocking backend responses: suspend / resume, collect results, throw on failuretests/distributed_map_helpers_test.pytests/e2e/distributed_map_helpers_int_test.pyTODO
Future Tasks
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.