DEV Community

Cover image for Step Functions Distributed Map: Process CSV with Lambda
Sascha Mirow-Stanitzok
Sascha Mirow-Stanitzok

Posted on Originally published at sascha-ms.de

Step Functions Distributed Map: Process CSV with Lambda

With the Step Functions Distributed Map you process huge amounts of data from S3 in parallel without building your own queues, workers or progress counters. I care about this topic because I keep seeing the same improvised setup in serverless projects: one Lambda reads a large CSV file, pushes the rows to SQS, a worker handles them, and in the end nobody really knows whether every row made it.

That is exactly the gap the Distributed Map closes. You only describe which file to read, how big the chunks are and how many of them may run at once. Step Functions takes care of everything else, including a failure threshold and a result report in S3. In this article you build the whole thing once: with the Serverless Framework, TypeScript, tests and an example you can run in your own AWS account in about half an hour.

In short

For every batch of a large S3 file, the Distributed Map starts its own child workflow and thus processes hundreds of thousands of rows in parallel. The most important setting is MaxConcurrency, because it protects your database and third-party systems from your own parallelism. The most common pitfall is missing IAM permissions for starting the child workflows and for the results folder.

What you will learn in this article

  • When the regular Map is no longer enough and the Distributed Map makes sense
  • How ItemReader, ItemBatcher, MaxConcurrency and ResultWriter work together
  • A complete CSV import with the Serverless Framework and TypeScript
  • Which IAM permissions the state machine really needs
  • How to keep failures, costs and reruns under control

When is the regular Map no longer enough?

Step Functions offers the Map state in two flavors. The inline Map runs inside the execution that started it. That is convenient but comes with tight limits: the input has to fit into the execution payload (256 KiB), every step ends up in one shared execution history capped at 25,000 entries, and you cannot go beyond 40 parallel iterations.

The Distributed Map removes these limits. Every iteration is a separate child workflow execution with its own history. The data comes straight from S3, for example from a CSV, JSON, JSON Lines or Parquet file or from a list of S3 objects. Without a limit, Step Functions starts up to 10,000 child workflows at the same time.

AWS names three signals that tell you it is time to switch. I use them as a rule of thumb:

  • Your dataset is bigger than 256 KiB.
  • A single execution history would grow past 25,000 entries.
  • You need more than 40 iterations running at once.

If none of these applies, stick with the inline Map. It is easier to read, and you avoid the extra IAM permissions we will get to in a moment.

How the Step Functions Distributed Map works

Before we write code, let’s look at the terms. You will see them all the time in the console, in the docs and in error messages.

Term What it means
Parent workflow The state machine that contains the Map state. It has to be of type Standard.
Child workflow A single run for one item or one batch. It can be Standard or Express.
Map Run All child workflows of one Map invocation, with its own ARN and its own view in the console.
ItemReader Defines where the data comes from, for example s3:getObject for a CSV file.
ItemBatcher Groups several items into a batch so that not every row starts its own child workflow.
ResultWriter Writes the results of all child workflows to S3 in bulk instead of putting them into the output.

Express workflows are a cheaper, faster flavor of Step Functions for short-lived processes. They may run for at most five minutes and are billed by count and duration. That is why they almost always fit child workflows that only hand a batch over to a Lambda function.

Architecture: S3 CSV, Distributed Map with Express child workflows, Lambda, DynamoDB and ResultWriter in S3

Catalog import: the Distributed Map reads the CSV from S3 and distributes batches to up to 20 child workflows.

The diagram shows the setup we are about to build. The parent workflow reads the file, splits it into batches and starts at most 20 child workflows at once. Each one calls the same Lambda function, which validates its 500 rows and writes them to DynamoDB. At the end, the ResultWriter stores a report in S3.

🎬 Flow of a Step Functions Distributed Map – this animated diagram only runs on my site: watch it on sascha-ms.de

The example: a catalog import for a city library

Picture a city library that receives an export from its union catalog every night. The file is a CSV that uses semicolons as delimiter, the way many Excel exports do, and it contains a few hundred thousand rows with ISBN, title, author, publication year and number of copies.

Every row needs to be validated: Is the ISBN check digit correct? Is there a title? Is the year plausible? Valid rows go into a DynamoDB table, invalid ones should show up in the report so someone can fix them. A single typo in the export must never abort the whole import.

A quick note up front: if you just want to dump data from S3 into an existing table without any logic, take a look at DynamoDB Bulk Put Items. The Distributed Map shines as soon as every row needs its own code, such as a validation, an enrichment or a call to a third-party system.

Prerequisites

  • An AWS account and configured credentials for the AWS CLI
  • Node.js 22 or newer and npm
  • Serverless Framework v4 (npm i -g serverless), logged in with serverless login
  • Basic knowledge of Lambda and Step Functions, for example from the article on the saga pattern

Step by step to a Step Functions Distributed Map

Step 1: Create the project

Create an empty folder and install the dependencies. For Step Functions I use the serverless-step-functions plugin, which lets you describe the state machine directly in serverless.yml.

mkdir katalog-import && cd katalog-import
npm init -y
npm i @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb
npm i -D typescript vitest tsx @types/node @types/aws-lambda serverless-step-functions
Enter fullscreen mode Exit fullscreen mode

You will find the complete package.json and tsconfig.json in the example repository linked at the end of the article.

Step 2: Validate a row

The business validation lives in its own module without any AWS dependencies. The reason is simple: you can test it in milliseconds. Step Functions hands over every CSV row as an object whose values are all strings. The column names come from the first line of the file.

// src/validate-row.ts
// Validates a CSV row from the catalog export and turns it into a record.
// Step Functions delivers each row as an object with string values.

export interface RawRow {
  isbn?: string;
  title?: string;
  author?: string;
  year?: string;
  copies?: string;
}

export interface Book {
  isbn: string;
  title: string;
  author: string;
  year: number;
  copies: number;
}

export type ValidationResult =
  | { ok: true; book: Book }
  | { ok: false; reason: string };

// ISBN-13: strip hyphens and spaces, then verify the check digit
export function normalizeIsbn(value: string): string | null {
  const digits = value.replace(/[-\s]/g, '');
  if (!/^97[89]\d{10}$/.test(digits)) return null;

  const sum = digits
    .slice(0, 12)
    .split('')
    .reduce((acc, d, i) => acc + Number(d) * (i % 2 === 0 ? 1 : 3), 0);
  const check = (10 - (sum % 10)) % 10;
  return check === Number(digits[12]) ? digits : null;
}

export function validateRow(row: RawRow, currentYear = new Date().getFullYear()): ValidationResult {
  const isbn = normalizeIsbn(row.isbn ?? '');
  if (!isbn) return { ok: false, reason: 'invalid ISBN' };

  const title = (row.title ?? '').trim();
  if (title.length === 0) return { ok: false, reason: 'missing title' };

  const year = Number(row.year);
  if (!Number.isInteger(year) || year < 1450 || year > currentYear + 1) {
    return { ok: false, reason: 'implausible publication year' };
  }

  const copies = Number(row.copies);
  if (!Number.isInteger(copies) || copies < 0) {
    return { ok: false, reason: 'copies is not a number ≥ 0' };
  }

  return {
    ok: true,
    book: { isbn, title, author: (row.author ?? '').trim(), year, copies },
  };
}
Enter fullscreen mode Exit fullscreen mode

The return type matters here. Instead of throwing an exception, the function returns either a finished record or a reason for rejecting the row. That way the caller decides what happens to broken rows.

Step 3: The Lambda function for one batch

With ItemBatcher enabled, the Lambda function does not receive a single item but an object with an Items field. On top of that, a batch can carry fixed values in BatchInput. The function validates all rows, writes the valid ones to DynamoDB in chunks of 25 items and returns the rejected ones.

// src/import-batch.ts
// Called once per batch by a child workflow of the Distributed Map.
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { BatchWriteCommand, DynamoDBDocumentClient } from '@aws-sdk/lib-dynamodb';
import { validateRow, type Book, type RawRow } from './validate-row.js';

const TABLE_NAME = process.env.TABLE_NAME ?? '';
const doc = DynamoDBDocumentClient.from(new DynamoDBClient({}));

// This is what the input looks like when ItemBatcher is enabled
export interface BatchEvent {
  Items: RawRow[];
  BatchInput?: { source?: string };
}

export interface BatchResult {
  written: number;
  rejected: { isbn: string; reason: string }[];
}

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

// BatchWriteItem accepts at most 25 items; resend unprocessed items
async function writeChunk(books: Book[], source: string): Promise<void> {
  let pending: Record<string, unknown>[] = books.map((book) => ({
    PutRequest: { Item: { ...book, source, importedAt: new Date().toISOString() } },
  }));

  for (let attempt = 0; pending.length > 0; attempt++) {
    if (attempt >= 5) {
      // Hands the batch back to Step Functions: retry or failure threshold applies
      throw new Error(`${pending.length} items not written after 5 attempts`);
    }
    if (attempt > 0) await sleep(Math.random() * 100 * 2 ** attempt);

    const response = await doc.send(
      new BatchWriteCommand({ RequestItems: { [TABLE_NAME]: pending } }),
    );
    pending = (response.UnprocessedItems?.[TABLE_NAME] ?? []) as Record<string, unknown>[];
  }
}

export const handler = async (event: BatchEvent): Promise<BatchResult> => {
  const source = event.BatchInput?.source ?? 'unknown';
  const valid: Book[] = [];
  const rejected: BatchResult['rejected'] = [];

  for (const row of event.Items) {
    const result = validateRow(row);
    if (result.ok) valid.push(result.book);
    else rejected.push({ isbn: row.isbn ?? '', reason: result.reason });
  }

  for (let i = 0; i < valid.length; i += 25) {
    await writeChunk(valid.slice(i, i + 25), source);
  }

  return { written: valid.length, rejected };
};
Enter fullscreen mode Exit fullscreen mode

Two details matter to me here. First, BatchWriteItem may return some items as unprocessed when the table throttles. The loop resends them with a growing, randomized delay, as described in the article on exponential backoff with jitter. Second, business errors (a broken ISBN) do not cause an exception, while technical errors (DynamoDB unreachable for good) do. Only the second kind should make a child workflow fail.

Step 4: The state machine in serverless.yml

Now for the core. The definition looks longer than it is, because most of the lines are resources and permissions. Read through it calmly once; the explanation follows right after.

# serverless.yml
service: katalog-import

frameworkVersion: '4'

plugins:
  - serverless-step-functions

provider:
  name: aws
  runtime: nodejs24.x
  architecture: arm64
  region: eu-central-1
  memorySize: 512
  timeout: 60
  environment:
    TABLE_NAME: !Ref BooksTable
  iam:
    role:
      statements:
        - Effect: Allow
          Action: dynamodb:BatchWriteItem
          Resource: !GetAtt BooksTable.Arn

functions:
  importBatch:
    handler: src/import-batch.handler

stepFunctions:
  validate: true
  stateMachines:
    katalogImport:
      id: KatalogImportStateMachine
      name: ${self:service}-${sls:stage}-katalog-import
      role:
        Fn::GetAtt: [StateMachineRole, Arn]
      definition:
        Comment: Reads a catalog CSV from S3 and imports it in batches
        StartAt: KatalogImportieren
        States:
          KatalogImportieren:
            Type: Map
            Label: KatalogImport
            MaxConcurrency: 20
            ToleratedFailurePercentage: 5
            ItemReader:
              Resource: arn:aws:states:::s3:getObject
              ReaderConfig:
                InputType: CSV
                CSVHeaderLocation: FIRST_ROW
                CSVDelimiter: SEMICOLON
              Parameters:
                Bucket:
                  Ref: ImportBucket
                Key.$: $.key
            ItemBatcher:
              MaxItemsPerBatch: 500
              BatchInput:
                source: verbundkatalog
            ItemProcessor:
              ProcessorConfig:
                Mode: DISTRIBUTED
                ExecutionType: EXPRESS
              StartAt: BatchSpeichern
              States:
                BatchSpeichern:
                  Type: Task
                  Resource: arn:aws:states:::lambda:invoke
                  Parameters:
                    FunctionName:
                      Fn::GetAtt: [ImportBatchLambdaFunction, Arn]
                    Payload.$: $
                  OutputPath: $.Payload
                  Retry:
                    - ErrorEquals:
                        - Lambda.TooManyRequestsException
                        - Lambda.ServiceException
                        - Lambda.AWSLambdaException
                        - Lambda.SdkClientException
                      IntervalSeconds: 2
                      MaxAttempts: 4
                      BackoffRate: 2
                      JitterStrategy: FULL
                  End: true
            ResultWriter:
              Resource: arn:aws:states:::s3:putObject
              Parameters:
                Bucket:
                  Ref: ImportBucket
                Prefix: ergebnisse
            End: true

resources:
  Resources:
    ImportBucket:
      Type: AWS::S3::Bucket
      Properties:
        PublicAccessBlockConfiguration:
          BlockPublicAcls: true
          BlockPublicPolicy: true
          IgnorePublicAcls: true
          RestrictPublicBuckets: true
        LifecycleConfiguration:
          Rules:
            - Id: ErgebnisseAufraeumen
              Status: Enabled
              Prefix: ergebnisse/
              ExpirationInDays: 30

    BooksTable:
      Type: AWS::DynamoDB::Table
      Properties:
        BillingMode: PAY_PER_REQUEST
        AttributeDefinitions:
          - AttributeName: isbn
            AttributeType: S
        KeySchema:
          - AttributeName: isbn
            KeyType: HASH

    StateMachineRole:
      Type: AWS::IAM::Role
      Properties:
        AssumeRolePolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Principal:
                Service: states.amazonaws.com
              Action: sts:AssumeRole
        Policies:
          - PolicyName: katalog-import
            PolicyDocument:
              Version: '2012-10-17'
              Statement:
                # Start and observe the child workflows of the Distributed Map
                - Effect: Allow
                  Action: states:StartExecution
                  Resource: arn:aws:states:${aws:region}:${aws:accountId}:stateMachine:${self:service}-${sls:stage}-katalog-import
                - Effect: Allow
                  Action: states:DescribeExecution
                  Resource:
                    - arn:aws:states:${aws:region}:${aws:accountId}:execution:${self:service}-${sls:stage}-katalog-import:*
                    - arn:aws:states:${aws:region}:${aws:accountId}:execution:${self:service}-${sls:stage}-katalog-import/*
                # Restart failed runs (redrive)
                - Effect: Allow
                  Action: states:RedriveExecution
                  Resource: arn:aws:states:${aws:region}:${aws:accountId}:execution:${self:service}-${sls:stage}-katalog-import/KatalogImport:*
                # Read the CSV
                - Effect: Allow
                  Action: s3:GetObject
                  Resource: !Join ['', [!GetAtt ImportBucket.Arn, '/eingang/*']]
                # Write the results
                - Effect: Allow
                  Action:
                    - s3:PutObject
                    - s3:GetObject
                    - s3:ListMultipartUploadParts
                    - s3:AbortMultipartUpload
                  Resource: !Join ['', [!GetAtt ImportBucket.Arn, '/ergebnisse/*']]
                - Effect: Allow
                  Action: lambda:InvokeFunction
                  Resource:
                    - !GetAtt ImportBatchLambdaFunction.Arn
                    - !Join ['', [!GetAtt ImportBatchLambdaFunction.Arn, ':*']]

  Outputs:
    ImportBucketName:
      Value: !Ref ImportBucket
    StateMachineArn:
      Value: !Ref KatalogImportStateMachine
Enter fullscreen mode Exit fullscreen mode

The most important fields of the Map state in a nutshell:

  • ItemReader reads the file with s3:getObject. CSVHeaderLocation: FIRST_ROW uses the first line as column names, and CSVDelimiter: SEMICOLON makes sure the Excel export is split correctly.
  • ItemBatcher bundles 500 rows per child workflow. In addition, a batch may be at most 256 KiB. If it gets bigger, Step Functions automatically puts fewer rows into it.
  • MaxConcurrency: 20 caps the parallelism. Without this line, Step Functions starts up to 10,000 child workflows at the same time, and your table or a third-party system will feel it.
  • ToleratedFailurePercentage: 5 keeps the run going as long as at most 5% of the items fail. Above that, the Map Run stops with States.ExceedToleratedFailureThreshold.
  • ResultWriter writes all results to ergebnisse/ in the same bucket instead of cramming them into the execution output.

The retry deliberately sits in the task inside the child workflow and not on the Map state itself. According to the docs, a retry on the Map state would start a completely new Map Run and repeat all child workflows, including the successful ones.

Step 5: Understand the permissions of the state machine

This is where most first attempts fail. The state machine role needs more than permission to invoke the Lambda function:

  • states:StartExecution on its own state machine, because technically every child workflow is a new execution of that same state machine.
  • states:DescribeExecution on the executions so the parent workflow can check on its children.
  • s3:GetObject only on the input folder, and write permissions only on the results folder, including the permissions for multipart uploads.
  • states:RedriveExecution if you want to restart failed runs later.

Because the role needs to know the ARN of the state machine and the state machine needs the role, you quickly end up with a circular dependency in CloudFormation. I avoid it by giving the state machine a fixed name and building the ARN in the role as plain text. The Serverless Framework resolves the variables ${aws:region} and ${aws:accountId} during deployment.

Step 6: Test, deploy, run

Tests first, then AWS. The tests check the ISBN logic and, using a mock, also the resending of unprocessed items.

// test/validate-row.test.ts
import { describe, expect, it } from 'vitest';
import { normalizeIsbn, validateRow } from '../src/validate-row.js';

describe('normalizeIsbn', () => {
  it('accepts a valid ISBN-13 with hyphens', () => {
    expect(normalizeIsbn('978-3-16-148410-0')).toBe('9783161484100');
  });

  it('rejects a wrong check digit', () => {
    expect(normalizeIsbn('9783161484101')).toBeNull();
  });
});

describe('validateRow', () => {
  const row = { isbn: '9783161484100', title: 'Cooking for Beginners', author: 'A. Example', year: '2021', copies: '3' };

  it('turns a clean row into a record', () => {
    const result = validateRow(row, 2026);
    expect(result).toEqual({
      ok: true,
      book: { isbn: '9783161484100', title: 'Cooking for Beginners', author: 'A. Example', year: 2021, copies: 3 },
    });
  });

  it('reports a missing title', () => {
    expect(validateRow({ ...row, title: '  ' }, 2026)).toEqual({ ok: false, reason: 'missing title' });
  });

  it('reports a publication year in the future', () => {
    const result = validateRow({ ...row, year: '2031' }, 2026);
    expect(result.ok).toBe(false);
    if (!result.ok) expect(result.reason).toMatch(/publication year/);
  });

  it('reports negative copies', () => {
    expect(validateRow({ ...row, copies: '-1' }, 2026).ok).toBe(false);
  });
});
Enter fullscreen mode Exit fullscreen mode
npx tsc --noEmit
npx vitest run
Enter fullscreen mode Exit fullscreen mode

Expected result: all tests pass; the repository contains nine of them. Next, you generate a test file. The small script builds a CSV with valid ISBNs and deliberately breaks every thousandth row.

// scripts/generate-csv.ts
// Generates a test CSV with semicolons as delimiter (like many Excel exports)
import { writeFileSync } from 'node:fs';

const rows = Number(process.argv[2] ?? 20000);

function isbnWithCheckDigit(n: number): string {
  const base = `978${String(n).padStart(9, '0')}`;
  const sum = base.split('').reduce((acc, d, i) => acc + Number(d) * (i % 2 === 0 ? 1 : 3), 0);
  return base + ((10 - (sum % 10)) % 10);
}

const lines = ['isbn;title;author;year;copies'];
for (let i = 1; i <= rows; i++) {
  // Every 1000th row is broken on purpose so you can see the error handling
  const isbn = i % 1000 === 0 ? '123' : isbnWithCheckDigit(i);
  lines.push(`${isbn};Test book ${i};Author ${i % 97};${1950 + (i % 75)};${i % 5}`);
}

writeFileSync('katalog.csv', lines.join('\n') + '\n');
console.log(`wrote katalog.csv with ${rows} rows`);
Enter fullscreen mode Exit fullscreen mode
npx tsx scripts/generate-csv.ts 200000
serverless deploy

BUCKET=$(serverless info --verbose | grep ImportBucketName | awk '{print $2}')
SM_ARN=$(serverless info --verbose | grep StateMachineArn | awk '{print $2}')

aws s3 cp katalog.csv "s3://$BUCKET/eingang/katalog.csv"
aws stepfunctions start-execution \
  --state-machine-arn "$SM_ARN" \
  --input '{"key": "eingang/katalog.csv"}'
Enter fullscreen mode Exit fullscreen mode

In the Step Functions console you now see the Map Run with a progress bar below the execution: 400 child workflows for 200,000 rows, at most 20 of them at once. You can also check the status from the CLI:

aws stepfunctions list-map-runs --execution-arn "<EXECUTION_ARN>"
aws stepfunctions describe-map-run --map-run-arn "<MAP_RUN_ARN>"
Enter fullscreen mode Exit fullscreen mode

In the test data, 200 rows are invalid on purpose. They show up as rejected in the results of their batches, while the run itself still succeeds. That is exactly how it should be.

What ends up in S3

The ResultWriter stores several files under ergebnisse/<map-run-id>/. The execution itself only returns the ARN of the Map Run and the path to manifest.json.

File Content
manifest.json Metadata of the Map Run and references to the result files
SUCCEEDED_0.json Input and output of all successful child workflows, including the rejected rows per batch
FAILED_0.json All failed or aborted child workflows with their error messages
PENDING_0.json Child workflows that never started because the Map Run was aborted earlier

A single result file can grow up to 5 GB; after that, Step Functions creates SUCCEEDED_1.json and so on. For the library, a small script that pulls all rejected entries out of SUCCEEDED_0.json and prints them as a list for catalog maintenance is enough.

Common mistakes and how to avoid them

The execution fails right away with AccessDenied

Almost always, states:StartExecution on its own state machine or read access to the CSV is missing. Also check that the path in the policy matches the path in the input. eingang/* in the policy and import/katalog.csv in the input do not go together.

Bucket in another account or region

The ResultWriter only writes to buckets in the same account and region as the state machine; otherwise the run ends with States.ResultWriterFailed. The same goes for reading: listing objects only works in your own account and region.

Odd column names after an Excel export

If the first column name suddenly contains invisible characters, the export probably wrote a byte order mark at the start of the file. Save the export without a BOM, or define the column names yourself with CSVHeaderLocation: GIVEN and CSVHeaders. And if your export uses tabs or pipes: CSVDelimiter also supports TAB, PIPE and SPACE.

One broken record takes 500 rows down with it

With ItemBatcher, an exception always fails the entire batch. That is why the Lambda function catches business errors per row and returns them. Only when something technical goes wrong may the child workflow fail.

The database is on fire

Twenty parallel child workflows with 500 rows each generate a fair amount of write load. Start with a small MaxConcurrency and raise it while you watch the throttling metrics of the table. Reruns are harmless in this example because a PutItem with the same ISBN simply overwrites the same record. If your processing has side effects, read the article on idempotency in Lambda first.

Without ResultWriter the output gets too big

If you leave out the ResultWriter, Step Functions collects all results in the output of the Map state. Above 256 KiB, the execution ends with States.DataLimitExceeded. For large files, the ResultWriter is therefore always part of my setup.

Keeping an eye on costs

Costs come from three parts. The parent workflow is a Standard workflow and is billed per state transition; in US East (N. Virginia) that is USD 0.000025 per transition, with 4,000 free transitions per month. The Express child workflows are billed per request and by duration and memory. On top of that come Lambda, DynamoDB and S3.

For our example this means: 200,000 rows turn into 400 child workflows and 400 Lambda invocations. Without ItemBatcher it would be 200,000 of each. Batching is therefore the biggest cost lever, followed by a sensible memory size for the Lambda function. You will find the exact prices for your region on the official pricing page, and for Lambda the Lambda cost calculator helps.

Distributed Map, Bulk Put Items or SQS?

The Distributed Map is not always the best choice. Here is how I decide:

Situation My choice
Load a file from S3 into an existing DynamoDB table without any logic DynamoDB Bulk Put Items
Large file, every row needs its own code, the run has a start and an end Step Functions Distributed Map
Ongoing stream of individual jobs without a fixed end SQS with Lambda and partial batch response
A few hundred items that are already part of the input Inline Map

The main difference between the Distributed Map and SQS is the end. An import is a job with a clear finish: you want to know when it is done, how much worked and what did not. With SQS you have to build that bracket yourself. For an endless stream of events, on the other hand, SQS with partial batch response is the better tool.

My take

I consider the Distributed Map one of the most underrated tools in the serverless toolbox. It replaces a whole class of home-grown pipelines with a configuration you can read and review. What I like most is that the failure threshold and the result report are built in. Exactly these parts tend to be forgotten in custom solutions and only come up when someone asks at three in the morning whether the import was complete.

I see two weak spots. The IAM permissions are not intuitive, and the related error messages are often terse. On top of that, the default parallelism of 10,000 tempts you to overrun downstream systems. My advice: never leave out MaxConcurrency, not even in your first prototype.

FAQ

What is the difference between inline Map and Distributed Map?

The inline Map runs inside the execution that started it and is limited to 40 parallel iterations and 256 KiB of input. The Distributed Map starts a separate child workflow for every item or batch, reads data straight from S3 and handles up to 10,000 parallel iterations.

Can an Express state machine use a Distributed Map?

No. The parent workflow has to be a Standard workflow. The child workflows, however, can be Standard or Express.

Which file formats can the ItemReader read?

CSV and other delimited formats, JSON, JSON Lines and Parquet, plus Athena manifests, S3 inventory lists and object lists from ListObjectsV2. Files compressed with GZIP or ZSTD are supported as well.

How do I rerun only the failed items?

Redrive the parent workflow. Step Functions then restarts only the child workflows that did not succeed. The role needs states:RedriveExecution for this.

Your next step

Take a file that you already process row by row with a script or a cron job today, and rebuild the import once with the Distributed Map. Start with 1,000 rows, MaxConcurrency 5 and a ResultWriter, and then look at the files in the results folder. Which file would you migrate first? Let me know in the comments.

Continue the series: Previous part: Step Functions timeout: getting heartbeats right · Next part: Semantic cache for Bedrock with ElastiCache and Lambda · All parts of the series “Serverless patterns in practice”

All code from this article, ready to run: smirowstanitzok/sascha-ms-examples

Sources

This post was created with AI assistance.


👉 Next step: Not sure which AWS service fits your use case? My AWS service finder asks a few questions and points you to the right one.

📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

Top comments (0)