Stop Passing Big Files Through Your Workflow. Here's What to Do Instead.

You've built the workflow, tested it with a sample file, and everything works. Then someone uploads a real spreadsheet, the kind with 40,000 rows instead of 10, and suddenly your "simple" step is dragging the whole process down.

Here's what usually causes it: the workflow is carrying the file itself through every step, like it's just another piece of data. Passed as payload, dragged from step to step, worked with inline. It feels natural to build it that way. It's also the wrong move.

Here's why, and what to do instead.

Small payloads are fine. Big ones aren't.

Passing a small JSON object between steps? No problem. Passing an actual file the same way? That's where things start breaking.

You get bloated requests and responses. Slower step-to-step execution. Debugging turns into a nightmare once an output is too big to actually read. And you're burning memory on something that has nothing to do with the real work.

None of this shows up while you're testing with a 10-row spreadsheet. It shows up the day a real file hits production.

The fix: stop moving the file. Move a path instead.

Rather than pushing file contents through the workflow, download the file once into shared storage and let each step that needs it read from disk. Steps pass metadata and results to each other, never the file itself.

That's three steps, in practice.

1. Download: stream it into storage, don't load it all into memory first

import os
import requests

def main(steps, context):
    file_url = "https://testing.s3.amazonaws.com/abcd/pqr/1234/myfile.xlsx"
    file_path = "/app/files/myfiles/customFile.xlsx"

    os.makedirs(os.path.dirname(file_path), exist_ok=True)

    try:
        with requests.get(file_url, stream=True, timeout=60) as response:
            response.raise_for_status()
            with open(file_path, "wb") as f:
                for chunk in response.iter_content(chunk_size=8192):
                    if chunk:
                        f.write(chunk)

        return {"statusMessage": f"File downloaded and saved to {file_path}", "isSuccessful": True}

    except Exception as e:
        return {"statusMessage": f"Download failed: {str(e)}", "isSuccessful": False}

Streaming in chunks is the whole point here. It's what keeps a big file from spiking memory on the way in.

2. Process: read it where it lives, return a summary, not the whole file

import pandas as pd

def main(steps, context):
    file_to_process = "/app/files/myfiles/customFile.xlsx"

    try:
        df = pd.read_excel(file_to_process)
        df = df.fillna("N/A")

        numeric_cols = df.select_dtypes(include=["number"]).columns
        numeric_summary = {col: df[col].sum() for col in numeric_cols}

        return {
            "rowCount": len(df),
            "columnCount": len(df.columns),
            "columns": list(df.columns),
            "numericSummary": numeric_summary,
            "preview": df.head(5).to_dict(orient="records")
        }

    except Exception as e:
        return {"statusMessage": f"Error processing file: {str(e)}", "isSuccessful": False}

3. Delete: clean up once you're done, every single time

import os

def main(steps, context):
    file_path = "/app/files/myfiles/customFile.xlsx"

    try:
        if os.path.exists(file_path):
            os.remove(file_path)
            message = "File deleted successfully"
        else:
            message = "File does not exist"

        return {"statusMessage": message, "isSuccessful": True}

    except Exception as e:
        return {"statusMessage": str(e), "isSuccessful": False}

Skip this step and storage quietly fills up on any process that runs a few hundred times a day.

The step people actually get wrong

It's step 2. The instinct is to return the parsed data as-is. Resist it.

Figure out what the next step genuinely needs, a row count, a list of columns, a few preview rows, and return only that. If something downstream truly needs the full dataset later, it's still sitting right there on disk.

Get this one decision right and the rest of the workflow stays light. Everything after that step is working with a few KB of summary instead of megabytes of spreadsheet.

Works past Excel too

Nothing about this pattern is Excel-specific. Download → process → clean up works the same for CSVs, PDFs, generated reports, basically any bulky file your workflow has to touch. Only the processing logic in step 2 changes.

The one habit that matters

Big files stop being a headache the moment you stop treating them as data flowing through the workflow and start treating them as a resource the workflow borrows temporarily. Download it, use it, get rid of it. Nothing downstream has to carry weight it never needed.