DEV Community: Pedro Beethoven

COPY lambda will always read the data, right? Not really

Pedro Beethoven — Wed, 19 Aug 2026 20:28:02 +0000

When I started to write a lambda that issues a Redshift COPY I thought that by default the lambda would read the data it was copying, and that's not the case (even though it could be), so you could be copying duplicated data without ever noticing it, or burning retries on poisoned data. This post will try to explain the difference between control planes and data planes, and how an AWS COPY lambda that is triggered by the creation of SQS messages fits into it.

Control planes and data planes

To clarify, the term plane refers to a layer of the architecture (originally a network one, and only conceptually) where some specific kind of task is performed, and its meaning is defined by the boundary that you are drawing.
So the difference between control and data planes in general is the following:

Control plane: it's the decision maker, the one that decides where the data should go, and does the administrative work, the setup, the security policies.

Data plane: this one carries out those decisions, moves the data from source to destination, and follows the rules determined by the control plane. It's the one that actually touches the data.

AWS has a different concept, because for them control and data plane can be differentiated inside the same service. For example, Lambda's control plane is CreateFunction, and its data plane is Invoke. In my context, these concepts involve the ingestion pipeline and not the service itself. The boundary I am drawing is basically whose memory the bytes pass through. In other words, by AWS definition this lambda does call a data-plane API, redshift-data:ExecuteStatement, since Redshift is doing its primary job there and not a management call.

In the ingestion pipeline, the COPY lambda seems at the beginning to be a data plane component, but when we look closer at the implementation we notice that the data is not read at all, because the lambda only tells Redshift to move it. It hands over a pointer (the JSON manifests) and the bytes of the real data go to the Redshift cluster without even touching the lambda memory.

What the code actually does

To do a COPY operation that batch loads the data into a Redshift cluster we don't need to read the data at all, and reading it would only give the overall lambda more points of failure. Here is a simple implementation of it, with some error handling and configuration loading left out:

import json
import time

import boto3

s3 = boto3.client("s3")
redshift = boto3.client("redshift-data")

MAX_POLL_ATTEMPTS = 60
POLL_INTERVAL_SECONDS = 1


def handler(event, context):
    urls = []
    failures = []

    for record in event["Records"]:  # SQS messages announcing files, not the files
        try:
            message = json.loads(record["body"])
            urls.extend(message["urls"])
        except (json.JSONDecodeError, KeyError):
            failures.append({"itemIdentifier": record["messageId"]})

    manifest = {"entries": [{"url": url, "mandatory": True} for url in urls]}
    key = f"manifests/{time.time_ns()}.json"
    s3.put_object(Bucket=BUCKET, Key=key, Body=json.dumps(manifest))

    statement = redshift.execute_statement(
        ClusterIdentifier=CLUSTER,
        Database=DATABASE,
        Sql=f"COPY raw_events FROM 's3://{BUCKET}/{key}' IAM_ROLE '{IAM_ROLE}' FORMAT AS JSON 'auto' MANIFEST",
    )

    status = "SUBMITTED"
    for _ in range(MAX_POLL_ATTEMPTS):
        status = redshift.describe_statement(Id=statement["Id"])["Status"]
        if status not in ("SUBMITTED", "PICKED", "STARTED"):
            break
        time.sleep(POLL_INTERVAL_SECONDS)

    if status != "FINISHED":  # one COPY, so the whole batch goes back to the queue
        return {"batchItemFailures": [{"itemIdentifier": r["messageId"]} for r in event["Records"]]}

    return {"batchItemFailures": failures}

If we look at the two clients created at the top we can see the whole point of this post, because this function only talks to S3 through put_object and to Redshift through execute_statement, and there is no get_object anywhere and no third client. The urls that come inside the message are copied into the manifest as strings and are never opened, so the lambda only writes down where the data is and asks Redshift to go and get it.

As we can see we produce a single manifest based on all the records of the SQS event, but if for some reason (like that one that I will explain in the next section) any of the messages is re-enqueued to SQS we could face duplication issues, since by nature the COPY statement is not idempotent. And since the COPY is done over a manifest, when the message comes back the whole manifest is copied again, so a retry duplicates a batch and not only one record.

What could break

The code above never validates the data. It validates the message that points to it, which is not the same thing. The records are basically references to the real data, so if there are some errors in there, the COPY will fail on Redshift's side and the lambda has no way of knowing about it beforehand. This means that the COPY statement is the first schema-enforcing read, and nothing checks those bytes against the destination schema until COPY does it.

Also, without a proper handling of it in a different lambda or a proper configuration of the DLQ (dead-letter queue), the same poison message could go back to the SQS queue again and again, resulting in the same error until the message simply expires. Let's imagine a scenario where we have a visibility timeout of 15 minutes and a redrive policy with a maxReceiveCount of 5. The visibility timeout has to be that large because it needs to cover the whole invocation, including the time the lambda can spend polling, otherwise SQS would hand the same message to a second invocation while the first one is still waiting for its COPY to finish.

With those numbers the retry story goes like this. The poison message is received, the COPY fails, and 15 minutes later the message becomes visible again, and this repeats until the fifth receive, when SQS finally moves it to the DLQ, where it will wait for the retention period (up to 14 days) until somebody looks at it. Until these limits are reached, the message will return to this lambda and we will replicate the same errors.

It is also worth saying that returning batchItemFailures is what makes this bearable. With ReportBatchItemFailures enabled on the event source mapping, only the messages listed in that return go back to the queue, and not the whole batch. The interesting part is that we can only be precise about half of it, because a message that is not valid JSON or that doesn't have the urls key is something the lambda has in memory and can check, so we are able to send back exactly those ids. In the COPY context, this is a different story, since there is a single statement over a single manifest, and the smallest thing that can fail there is the whole batch, so one bad file inside it sends every message back and on the retry every good file of that batch is loaded again. In other words, the lambda can be specific about the messages that it reads, and can only be generic about the data that it never touches.

The poll loop has a similar problem. When MAX_POLL_ATTEMPTS runs out we stop watching the statement, but stopping the watch doesn't stop the COPY, and it keeps running on the cluster. The messages go back to the queue, the next invocation writes a new manifest with the same urls, and we end up with two statements loading the same files. We could call cancel_statement in that branch, but this would only help while the COPY is still running, and by the time the poll gives up it could already have loaded everything, so the duplicate would happen anyway.

Another scenario is sending the message without thinking about idempotency. Redshift doesn't enforce primary keys or unique constraints, so if the same message is delivered twice (and with a standard SQS queue, at-least-once delivery means it can be), we could have two correct messages and duplicated entries on Redshift.

Possible fixes

In order to fix the errors described above we can develop some types of solution. For example, for avoiding poisoned messages, we should have some other lambda that will validate them before they reach SQS. Another option is to validate them on the same COPY lambda, which to me is not a good solution, since it would imply downloading the real data from the source, and it would drag the lambda onto the data path and detour it from its original goal.

For the idempotency issue, again we can have another lambda before this one to be responsible for creating some idempotency keys and passing this information to this lambda, or we can also do this work here, but again it will be another case of a component with multiple responsibilities (data plane and control plane at the same time), which is not the desired behavior. For this case the best solution is probably letting a Redshift statement be responsible for guaranteeing idempotence by discarding the duplicates. The query below, for example, can run as a model in an analytics tool like dbt, and everything downstream would read that model instead of reading the raw table:

with ranked as (
    select
        event_id,
        payload,
        loaded_at,
        row_number() over (
            partition by md5(payload)
            order by loaded_at
        ) as duplicate_rank
    from raw_events
)

select
    event_id,
    payload,
    loaded_at
from ranked
where duplicate_rank = 1

The deduplication key here is the hash of the whole payload, which already carries the identifier of the event inside it, row_number() numbers the rows inside each group of identical rows, and keeping only the rank 1 collapses each group into a single row. It is important to notice that this one keeps one row per group, because the variant that filters by count(*) over (partition by ...) = 1 looks very similar but it throws the whole duplicated group away, losing the record instead of deduplicating it. The order by only decides which copy survives, and since the rows are identical it doesn't matter which one it is.

The most attentive readers have probably already noticed that the COPY itself is still not idempotent, but since we are thinking in the context of an overall implementation of the pipeline, the final goal is to guarantee that whatever reads the data downstream doesn't see duplicated rows. Since there is always the need to run the statement above we can say that this pipeline is kind of eventually idempotent, but the call of the lambda itself is not. The word eventually is important here, because the duplicated rows are still physically in the raw table, and anything that queries it before the deduplication runs will still see both copies. So the window where the data is wrong is the interval between the COPY and the next run of that model, which means that this window is not defined by the lambda but by the schedule that deduplication tool is running on. And anything that decides to read the raw table directly instead of the model is not covered by this at all.

There is also a second thing that this design does not guarantee. Since the key is a hash of the payload, two events that are really distinct but happen to be identical are indistinguishable from a redelivery, so the deduplication will drop one of them. I accepted this because in this pipeline an identical duplicate is always a retry, but this is an assumption about the data and not a property of the code, so if one day this stops being true the statement will silently discard real rows.

Conclusion

This COPY lambda was an example to me that even a simple lambda function responsible only for creating manifests and copying data can lead to wrong assumptions if you don't look into more details, and can also introduce a kind of error that a simple look into the happy path can hide. Also, the same lambda can handle the validation of malformed data or the deduplication, but this will contradict the single responsibility principle (becoming a control and data plane at the same time). The validation fits better in the function that writes the file, because that one already has the bytes in memory and checking them there doesn't cost anything extra, while doing it here would mean downloading the same data a second time only to look at it. So having a clear vision of each step of the function and a solution to each fault scenario can improve the chances of the overall implementation not becoming a complete failure.

How one Django .delete() ran a 4 GB instance out of memory | Slicing Prod Data

Pedro Beethoven — Tue, 11 Aug 2026 16:42:23 +0000

ORMs can bring us a lot of simplicity and are easier to maintain than raw SQL commands, but as with everything in life it comes with a cost, in my case the cost was memory, more precisely an OOM (Out Of Memory) error, so in order to register this for the future and also help people to avoid the same mistakes, I decided to write more about it, its causes and possible solutions.

Context

The context in which the OOM occurred was a transform step of a pipeline that has the goal of generating a slice of production data. I know that today there are some tools to implement that, but we had some specific requirements (like leaving some rows of some tables untouched), and in order to use the same models already defined in Django and use them to simplify maintenance, we decided to use a Django command.

To create the slice, we decided to choose a percentage and delete the rest, but the problem is that the first delete involves a cascade of CASCADE rules (yes) by the use of the .delete() method. This method (and other implementation mistakes) was the cause of the OOM error.

Cause

Well, saying that the cause of the error was exclusively about the method is not the whole truth, since this only appeared once I made the first QA test with production data, and this is the cause of most backend issues: Scale and Volume. Locally, I tried to make a good seed of data to test my transform step, but none of it was enough to represent the prod data (more than 150 GB). On top of that, in production the data is much more concentrated in the leaves than in the root: to give a sense of scale (numbers illustrative), the shape looks roughly like grandfather → 20k parents → 100k grandchildren. This created a perfect scenario to expose my implementation error: the wrong use of .delete().

How delete works

When you call .delete() on a queryset, the real work is handed off to a Collector instance. Its job is to work out everything that has to be deleted before deleting anything: it recursively collect()s the base model's cascade relationships, pulling the affected objects into an in-memory dict, and only then issues the actual SQL, one DELETE ... IN (...) per table. (It can skip loading a relation when it's able to "fast delete" it with a single subquery, but anything with its own cascades, signals, or children still gets pulled in.) That "collect the whole tree first, delete second" design is exactly where the memory goes. Three parts of the Collector tell the story: where the objects are kept, how the collection recurses, and when the SQL finally runs.

First, the store (all excerpts from django/db/models/deletion.py, Django 5.2, trimmed for clarity). The objects to be deleted are accumulated in a single in-memory dict:

class Collector:
    def __init__(self, using, origin=None):
        ...
        # Initially, {model: {instances}}, later values become lists.
        self.data = defaultdict(set)   # objects to be deleted are collected here, in memory

The recursion comes from the CASCADE handler: for each related object it finds, it calls collect() again, which collects their related objects, all the way down the tree:

def CASCADE(collector, field, sub_objs, using):
    collector.collect(          # recurse: collect the children of the children ...
        sub_objs,
        source=field.remote_field.model,
        source_attr=field.name,
        nullable=field.null,
        fail_on_restricted=False,
    )

    def collect(self, objs, ...):
        new_objs = self.add(objs, ...)      # stash these instances in self.data
        ...
        for related in get_candidate_relations_to_delete(model._meta):
            field = related.field
            on_delete = field.remote_field.on_delete
            ...
            for batch in batches:
                sub_objs = self.related_objects(related_model, [field], batch)  # fetch the children
                if getattr(on_delete, "lazy_sub_objs", False) or sub_objs:
                    on_delete(self, field, sub_objs, self.using)   # CASCADE -> collect() again

Only once everything is collected does the SQL run, one DELETE ... IN (...) per model, built from the primary keys of every instance sitting in self.data:

    def delete(self):
        ...
        # delete instances
        for model, instances in self.data.items():
            query = sql.DeleteQuery(model)
            pk_list = [obj.pk for obj in instances]          # every collected pk, still in memory
            count = query.delete_batch(pk_list, self.using)  # DELETE FROM ... WHERE pk IN (...)

So the combination of the collect method being called for each cascade-related object and being put in memory by the collector instance, plus the volume of data, plus the structure of the distribution of the data, rapidly uses up the maximum memory capacity of the machine instance (4 GB to be exact).

The initial solution

So the initial solution to this, instead of basically calling the queryset with the delete method, was to delete their related descendants in batches in order to keep memory usage constant, the classical idea of solving the small problems first. Here is the implementation:

The models involved form a Blog → Post → Comment (Fictional models) cascade (grandfather → parent → grandchild), where the volume lives in the leaves. We first compute the primary keys we want to drop with raw SQL (so the ids never all land in Python), and then, instead of one big queryset.delete(), we delete them a bounded batch at a time:

from django.db import transaction

# The naive version: one call for the whole slice. queryset.delete() runs
# Django's Collector, which pulls the cascade descendants (Post -> Comment -> ...)
# into memory before issuing the DELETEs. On prod-scale data this is what
# exhausts a 4 GB instance.
Post.objects.filter(id__in=drop_ids).delete()

# The batched version: delete the same rows a bounded batch at a time, so the
# Collector only ever holds one batch worth of descendants in memory.
def batched_delete(model, ids, *, size=1000):
    # Delete through _base_manager (the unfiltered manager), not .objects:
    # a custom default manager can hide rows (e.g. soft-deleted ones) that
    # the id list still targets, which would silently under-delete.
    manager = model._base_manager
    for start in range(0, len(ids), size):
        with transaction.atomic():
            manager.filter(id__in=ids[start:start + size]).delete()
    return len(ids)

batched_delete(Post, drop_ids)

And after the new implementation was merged, we tried a new round of QA, and the OOM error was fixed, so everything is good, right? Turns out, no. At this moment, even though I had some notion of the structure of the data, I wasn't expecting that these deletions using these defined batches of deletion would take more than 2 hours to (not) be processed, which caused the whole pipeline to time out and thus the need to redesign everything again to solve this new issue.

Conclusion

Even though this first try failed at the goal of solving all the problems of the pipeline and finally delivering it, it helped me to better understand how Django works in the background with its ORM. So, in the next post, I will try to explain what my tries to definitively solve this were (hopefully), as soon as possible.

How to Actually Tailor Your Resume Without Losing Your Mind

Pedro Beethoven — Fri, 06 Mar 2026 01:45:52 +0000

``Everyone says you should tailor your resume for every job application. And they're right. But nobody talks about how incredibly tedious that process actually is.

I've been through it. Multiple times. And I've watched friends go through it too. The pattern is always the same. You find a job posting, get excited, open your resume, stare at it for twenty minutes, change two bullet points, and call it done. That's not tailoring. That's just editing with extra guilt.

The problem isn't laziness. The problem is that doing it properly takes real effort, and when you're applying to dozens of jobs, that effort becomes unsustainable very fast.

So let me share what I've learned about actually tailoring resumes in a way that works without burning out.

Start with a master resume, not a blank page

This is the single most important thing I can tell you. Never start from scratch.

Build one document that has everything. Every project, every skill, every achievement, every responsibility you've ever had. Make it ugly. Make it long. Nobody will ever see this document except you.

This is your source of truth. When you need to tailor a resume for a specific position, you pull from this master document. You don't invent new things. You select the most relevant ones.

The difference is huge. Instead of thinking "what should I write," you're thinking "what should I pick." That's a much easier decision to make, especially when you're tired and just want to get the application done.

Read the job posting like a developer reads documentation

Most people skim job postings. They look at the title, glance at the requirements, and move on. That's a mistake.

Job postings are basically specifications. They tell you exactly what the company is looking for. The keywords they use, the order they list requirements, and the specific tools they mention. All of that matters.

Here's what I do. I read the posting carefully and highlighted the key terms. Not just technical skills, but also the soft skills and the type of language they use. Suppose they say "fast-paced environment," that tells you something. If they say "collaborative team," that tells you something different.

Then I look at my master resume and pick the experiences that match those terms most closely. I don't copy their words blindly, but I make sure the language in my resume resonates with what they're looking for.

This isn't gaming the system. This is communication. You're making it easy for the person reading your resume to see the connection between what they need and what you offer.

The ATS problem is real, but not in the way people think

There's a lot of fear around Applicant Tracking Systems. People think these systems are AI overlords that reject resumes based on mysterious algorithms. The reality is simpler and also more annoying.

Most ATS systems are basically keyword matchers. They parse your resume and check if certain terms appear. If the job posting asks for "Python" and your resume says "Python," that's a match. If you wrote "py" or "Python programming language" instead, it might not match.

The fix is straightforward. Use the same terminology as the job posting. If they say "React," write "React." If they say "CI/CD," write "CI/CD." Don't get creative with naming conventions.

But here's the part people miss. ATS is just the first filter. A human still reads your resume after that. So you need to optimize for both. Keywords for the machine, clear and compelling writing for the person.

Stop listing responsibilities, start showing impact

This is the most common resume mistake I see, and I made it myself for years.

"Responsible for developing backend services" tells the reader nothing useful. Every backend developer is responsible for developing backend services. That's literally the job.

What actually matters is what happened because of your work. Did response times improve? Did you reduce costs? Did you build something that other teams started using? Did you fix a bug that was causing customer complaints?

Numbers help, but they're not mandatory. "Reduced API response time by 40%" is great. But "Built an internal tool that replaced a manual process the team had been doing for two years" is also compelling, even without a percentage.

The point is to show that your work had consequences. That it mattered beyond just completing a task.

Keep the format boring

I know this sounds counterintuitive, especially with all those beautiful resume templates floating around on the internet. But fancy formatting causes real problems.

Complex layouts with columns, icons, and graphics often break when parsed by ATS systems. They also make it harder for recruiters to quickly scan your resume. And recruiters scan fast. You have maybe ten seconds of attention before they decide to keep reading or move on.

Use a clean, single-column layout. Clear section headers. Consistent formatting. Standard fonts. It's not exciting, but it works.

Save the creativity for your portfolio or your personal website. The resume is a functional document, not a design showcase.

The "one page" rule is mostly right

For most people with less than ten years of experience, one page is enough. Not because there's some sacred rule about it, but because forcing yourself to fit everything on one page makes you prioritize.

If you can't fit everything, that's the point. You shouldn't be including everything. You should be including the things that matter most for this specific application.

Two pages become acceptable when you genuinely have enough relevant experience to justify them. But "relevant" is the keyword. Ten years of experience doesn't mean ten years of content. It means you have more to choose from when tailoring.

Tailoring at scale is the real challenge

Here's the honest part. Everything I described above works great when you're applying to five jobs. It becomes exhausting when you're applying to fifty.

And most people looking for jobs are applying to far more than five positions. The math just doesn't work. If each tailored resume takes thirty minutes, and you're applying to ten jobs a week, that's five hours just on resume customization. On top of job searching, cover letters, and the actual interviews.

This is exactly why I built Alapi. Not because I think AI should write your resume for you. But because the mechanical parts of tailoring, matching keywords, adjusting emphasis, and reformatting can be automated, you can focus on the parts that actually require your brain.

You upload your resume once. Alapi collects job opportunities from multiple platforms. When you find something interesting, it generates a tailored version of your resume for that specific position. You review it, edit whatever you want, and export it.

It doesn't replace your judgment. It just removes the tedious parts.

The uncomfortable truth about job applications

Tailoring your resume matters. But it's only one piece of a much larger and often frustrating process.

The best resume in the world won't help if you're applying to jobs that aren't a good fit. And the job search process itself, with its ghosting and automated rejections and vague feedback, can be genuinely demoralizing.

What I've learned, both from my own experience and from building a tool around this problem, is that the most important thing is to reduce friction wherever you can. Make the process as efficient as possible so you can sustain it over time without burning out.

Because finding the right job isn't usually about one perfect application. It's about consistently putting yourself out there until the right match happens. And anything that makes that process less painful is worth trying.

If you want to try Alapi for tailoring your resumes automatically, check it out at alapi.app. It's free to start.