DEV Community: Lee Harding

Energy Efficient Small File Uploads to S3

Lee Harding — Mon, 04 May 2026 18:21:17 +0000

Moving files into S3 as fast as possible has been a side-quest of mine for, well, two decades (yep, S3 is more than 20 years old).

My first encounter with moving large numbers of files to S3 was from an on-premise system, working over a backlog of many thousands of files, with thousands of new ones appearing each day.

For that project I tried the obvious path (a foreach loop doing uploads one by one) and failed. I discovered success with concurrency -- running multiple uploads simultaneously from multiple workers. It worked, and a similar technique has been built into the AWS CLI (CRT and concurrent connections config setting).

Concurrency was helpful because bandwidth wasn't the limit. The files were small; at most a 1-2 MB each, and many just a few KB. The bandwidth to S3 was relatively low by today's standards, but not the bottleneck. It was the round-trip latency, connection negotiation and retransmission getting in the way.

A few years down the road, I was tasked to put many millions of small JSON files into S3 to use as static data for a web application (a "static API"). I knew upfront the API PUT request costs would be high for this project, but as a one-time expense not a big deal.

The problem I feared was that I had to run the workload at the edge (on premises, around the world) because the source data was there. Even if we could have run inside AWS on EC2 it would still be a remarkably slow and CPU intensive process (again because of connection negotiation for governance-required TLS).

I wanted to explore UDP for file transfers back then, but didn't get the chance. Recently, for the 20th anniversary of S3, I finally have.

The problems with using UDP for file transfers are three-fold:

UDP packets are small, and files can be big.
The protocol itself doesn't support reliable delivery or ordering.
There is no real standard for encrypting UDP.

When it comes to transmitting files over a network, the points above are all significant.

To transmit a file larger than a single packet (at maximum about 1500 bytes), we need to split it. Doing so means we need to have a way to know which packets are for which files (some kind of ID). That's simple enough to add, but to keep true to the stateless nature of UDP and serverless backends it needs to be added to each packet (a little redundant).

SIDE NOTE: TCP based APIs (including HTTP/1 and HTTP/2) use state attached to the session and kept on the backend to map data to files. This is a fundamental difference when it comes to designing UDP APIs versus TCP APIs.

To ensure we have the same contents as the original requires getting all the file data (reliability) and knowing what order the chunks go in.

With UDP it's easy enough to add data to each packet to ensure the payloads they contain are assembled in the right order -- a sequence number (aka. index). With that in place, it doesn't matter what order the packets arrive, we can sort them on the backend to get the contents arranged correctly.

But we also need to make sure we have all the contents of the file. Note I didn't say "we have all the packets" -- we just need the file data.

This problem is exactly what RaptorQ solves. In a nutshell, it sends the content of the file plus extra error correction data so that a few missing packets doesn't prevent the file from being reconstructed data. It also handles sequence numbers, so we have a standard to follow.

So that covers moving the file data reliably. One check mark.

What about privacy? RaptorQ doesn't include any encryption. If we used plain UDP and RaptorQ only we'd be sharing our content with the internet at large. That's a problem for many folks.

Luckily, there are some "almost standard" options.

DTLS exists, but it's not mainstream, is ungainly to implement and enforces some "features" we don't need.

A more modern approach is WireGuard which uses the Noise handshake and fixed ChaCha20 encryption. This is a lightweight and proven way of protecting data in flight and based on UDP. Most VPN product support it. Hopefully it becomes a recognized standard someday.

Putting UDP, RaptorQ and WireGuard encryption together we get a very lightweight way to send files from the edge to the cloud.

Potentially. All this was all just a thought experiment until recently.

I first prototyped this idea with a WireGuard tunnel setup with the common wg command.

The files to be sent were passed though a RaptorQ encoder library and saved to disk (each packet as a separate file). I'm lazy like that. I then sent them with cat {file} > /dev/udp/1.2.3.4/4576. Note that sending via bash like this is slow, which is good.

Don't send UDP as fast as your computer can; it isn't nice.

To keep it short, that prototype worked and I got started on a more "real" backend. Serverless, of course.

To handle the packets without a server I used a UDP Gateway Listener configured for WireGuard. Since I was using the standard wg on the client side I enabled DecapsulatedDelivery which strips the IP and UDP headers before delivering the payloads to the backend (in this case a simple Lambda).

From there the packets were collected in DDB, and once a sufficient number had been received, then the reassembly process kicked-off. That process read all the records, decoded and reassembled them into an S3 object.

Magic happened. I got files.

I got files even with significant "packet loss" (intentionally skipping random packets).

The potential was manifest, but I needed to go fast to make it interesting.

I wanted a CLI that was small, dependency-free and fast. I chose C# and .Net 10 with Ahead-of-Time compilation to build native, dependency free, single-file executables. The raptor executable weighs-in at about 3.5MB on both Linux and Windows. The codebase is the same for both platform.

Modern cross-platform development is so very nice. No #ifdef conditional compilation anywhere.

The WireGuard encryption comes courtesy of Proxylity.WireGuardClient nuget library (MIT license, repo here) that implements a simple, UdpClient-like interface. It's about 800 lines of code in total.

As I noted above, spewing packets blindly isn't cool. Some kind of congestion control (slowing down and not overburdening an already busy network) is mandatory for a tool like this -- even when the goal is speed.

My solution to congestion control (for now) is using a fixed, configurable bandwidth for the transfer (--rate-mbps). It's easy to implement (just a delay between packets), and effective. It also has the downside of underutilizing the connection, and can add to congestion on a very busy network, but in practice this is a practical approach for me that allows tinkering.

I added a couple more interesting options. The first implements different confirmation levels from "none" to "yep, the object is in S3" (see --confirm), and the second the amount of extra repair packets to send (see --overhead). Using these together allows sending the file's packets without waiting for confirmations (--confirm NONE) while being pretty-sure the file will get there (--overhead 0.4 or whatever, depending on packet loss).

So is it faster? Yes!

In the benchmarks with 1KB, 10KB and 1MB files raptor outperforms aws s3 sync by a bit. It outperforms successive aws s3 cp by 25x. Nice.

The smaller the files, the bigger the advantage. For a few small files, it's essentially instant. And, it's more flexible.

Being real, raptor loses its advantage as files get larger because of the fixed bandwidth. The --rate-mbps option can be used to saturate a link, but in the end the repair and CPU overhead of the RaptorQ encoding starts to become significant. There are ways to fix this, but given my target use case (and limited knowledge) I'm stopping here for now.

Takeaway

Sure, it's faster but the most significant benefit of raptor is that it uses 10x less CPU power. Think about that. The world runs on TLS-protected requests and each one could use 90% less electricity. Billions per second -- 200 million per second for S3 alone -- the potential electricity savings are astounding.

Why aren't we doing this more? HTTPS and TLS are so entrenched they are rarely questioned. And moving legacy systems to more modern and efficient network stacks is understandably infeasible.

But if you're working on something new, think about making that greenfield project a little more "green".

Can you make raptor faster? Clone the repo, tweak it, experiment with different backends and let me know.

Contributions welcome!

Building a UDP-based API

Lee Harding — Mon, 13 Apr 2026 21:23:20 +0000

In my last post I introduced the concept of treating UDP network packets as a serverless event source. In this post I explore building a UDP-based application programming interfaces (APIs) with a request-response model.

Background

The modern web is brim-full of HTTP APIs, and we have excellent tooling to simplify and de-risk building them.

The simplicity means we don't need to write network stacks, web servers or parsers: we get them as part of the library or platform. Writing an HTTP server is as simple as:

from http.server import HTTPServer, BaseHTTPRequestHandler

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b"OK")

HTTPServer(("", 8080), Handler).serve_forever()

If we want to get even simpler, we can use API Gateway and write a Lambda function to avoid paying for an idle container or instance to run our server. In that case the handler may be something like:

def handler(event, context):
    return {
        "statusCode": 200,
        "body": "OK"
    }

With this approach we have no costs for reserved capacity and no infrastructure to manage. This is what I mean by de-risking: We can build such APIs all day long and pay only when (if) they're actually used. Use rate limiting and WAF for abuse protection and reduce the risk even further.

What about UDP then?

A UDP API

A server for a UDP API is very similar. We're going to write it to listen for incoming packets on any port to keep with the example from the previous article.

In this case we want to support responses as an API should. We can't use EventBridge since it has a write-only API (responses relate to the success of the write; nothing custom).

So we're going to use Lambda instead and send what it returns as the response:

import socket
import struct
import boto3
import json
import base64

raw_sock = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_UDP)
lambda_client = boto3.client("lambda")
reply_socks = {}

while True:
    data, addr = raw_sock.recvfrom(65535)

    ip_header_len = (data[0] & 0x0F) * 4
    src_port, dst_port = struct.unpack('!HH', data[ip_header_len:ip_header_len + 4])
    payload = data[ip_header_len + 8:]
    src_ip = addr[0]

    response = lambda_client.invoke(
        FunctionName="my-function",
        Payload=json.dumps({
            "Port": dst_port,
            "Data": base64.b64encode(payload).decode()
        }),
    )
    reply = json.loads(response["Payload"].read())

    if reply:
        if dst_port not in reply_socks:
            s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
            s.bind(("", dst_port))
            reply_socks[dst_port] = s

        reply_socks[dst_port].sendto(reply.encode(), (src_ip, src_port))

Note that we're keeping a map of reply sockets to ensure that our replies appear to be "from" the same port the incoming packet has as the destination (the bind() call locks the socket to sending from a single port). If we didn't do this our replies would probably be ignored (or blocked by a firewall) since the client expects to have a two-way conversation with the given server port.

Now we can implement my-function however we'd like, including port-specific handling:

import json

def handler(event, context):
    if event.get("Port") == 4321:
        return "OK"

But we really want a more "API Gateway like" experience, and we can get there. The Constrained Application Protocol (CoAP) is the UDP-based cousin of HTTP. It has paths, methods, etc. and the basis for good general purpose UDP APIs.

So let's add support for CoAP to the server:

import socket
import struct
import boto3
import json
import base64
from aiocoap import Message, ACK, CONTENT

raw_sock = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_UDP)
lambda_client = boto3.client("lambda")
reply_socks = {}

ROUTES = {
    4321: {"function": "my-function",   "protocol": "raw"},
    5678: {"function": "coap-function", "protocol": "coap"},
}

def parse_coap(data):
    msg = Message.decode(data)
    return {
        "code": str(msg.code),
        "message_id": msg.mid,
        "token": base64.b64encode(msg.token).decode(),
        "path": "/" + "/".join(msg.opt.uri_path),
        "payload": base64.b64encode(msg.payload).decode(),
    }

def build_coap_ack(coap, body):
    return Message(
        mtype=ACK,
        code=CONTENT,
        mid=coap["message_id"],
        token=base64.b64decode(coap["token"]),
        payload=body.encode()
    ).encode()

while True:
    data, addr = raw_sock.recvfrom(65535)
    ip_header_len = (data[0] & 0x0F) * 4
    src_port, dst_port = struct.unpack('!HH', data[ip_header_len:ip_header_len + 4])
    payload = data[ip_header_len + 8:]
    src_ip = addr[0]

    route = ROUTES.get(dst_port)
    if not route:
        continue

    if route["protocol"] == "coap":
        coap = parse_coap(payload)
        lambda_payload = json.dumps(coap)
    else:
        lambda_payload = json.dumps({"Port": dst_port, "Data": base64.b64encode(payload).decode()})

    response = lambda_client.invoke(FunctionName=route["function"], Payload=lambda_payload)
    reply = json.loads(response["Payload"].read())

    if reply:
        if dst_port not in reply_socks:
            s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
            s.bind(("", dst_port))
            reply_socks[dst_port] = s

        packet = build_coap_ack(coap, reply) if route["protocol"] == "coap" else reply.encode()
        reply_socks[dst_port].sendto(packet, (src_ip, src_port))

In reality the configuration would be external to the server, but we're keeping things simple here. To add a new CoAP API endpoint we can just add to the configuration. To implement a CoAP handler, we build a simple Lambda that just processes the JSON object payload.

For example, here is a Lambda implementing a "time" service:

import datetime

def handler(event, context):
    if event.get("path") == "/time":
        return datetime.datetime.now().isoformat()

Adding more paths and methods to the Lambda will feel very much like building an HTTP handler.

Summary

This is the start of an interesting architecture that decouples network listening and replies from handling the packet data:

We can add new paths and methods to the CoAP Lambda without touching the server at all.
We can add new ports and custom protocols to the server in a clean, configuration driven way.
Using CoAP, we're within sight of the simplicity of the tooling for HTTP but lack the ease of configuration and features.
We're running a server, and hence haven't really de-risked as much as we'd like.

To that last point, we can't quite match API Gateway approach (there is no AWS service that supports UDP in this way, but Proxylity does).

UDP as a Serverless Event Source

Lee Harding — Fri, 10 Apr 2026 22:44:04 +0000

Are you versed in Event Driven Architecture (EDA)? This article is going to explore applying EDA at the outside edge of systems as a boundary for traditional network traffic.

What is EDA?

Setting aside hyperbole and focusing on the "what":

Event-driven architectures have three key components: event producers, event routers, and event consumers. [AWS Documentation]

But what is an "event"?

A record of a significant change in state or a notable occurrence within a system, stated in the past tense. [Wikipedia]

In EDA, things happen and information about those occurrances flows from producers to consumers. Consumers may, of course, produce events of their own creating "flows" of events around and between systems. That sounds a lot like what network equipment does, right?

So let's make that literal.

Modeling a UDP "Event"

The User Datagram Protocol is simple intentionally. It was designed in the 1980's as the "catch-all" mechanism for moving data around the internet when TCP wasn't appropriate.

A UDP packet is just a bunch of bytes following a well-defined format:

[UDP Header][UDP Payload]

Okay, it's not quite that simple because UDP is an extension of another protocol called the Internet Protocol (IP). And IP is almost always carried over ethernet, so the full content of a unit of data travelling the internet as UDP is really:

[Ethernet Header][IP Header][UDP Header][UDP Payload]

Okay, okay, it's not that simple either but I'm going to leave it there for this article. In fact, I won't be talking about the ethernet header beyond this point so we can just focus on the UDP part (and a little about IP later).

Breaking out the UDP header in more detail:

[Source Port][Destination Port][Length][Checksum][Payload]

Normally the operating system parses that header to decide which application or service should receive it. A UDP packet with a destination port of 514 is often delivered to rsyslogd, for example. If we're going to treat that UDP packet as an event inside AWS we're going to need a JSON model of it, like:

{
  "Destination: { "Port": 514 },
  "Payload": "<base64 encoded payload?"
}

Pretty minimal, but you get the idea. What happened to SourcePort and Checksum you ask? We're just ignoring them. In reality they're optional and you may get 0 values for both in some (many?) situations.

Routing and Delivery

With that model we can listen for UDP packets and fire an event to EventBridge for each:

import socket, struct, base64, json, boto3

events = boto3.client("events")
sock = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_UDP)

while True:
    pkt, _ = sock.recvfrom(65535)

    ihl = (pkt[0] & 0x0F) * 4
    dst_port = struct.unpack("!H", pkt[ihl+2:ihl+4])[0]
    payload = pkt[ihl+8:]

    events.put_events(Entries=[{
        "Source": "udp.raw",
        "DetailType": "UdpPacket",
        "Detail": json.dumps({
            "Destination": {"Port": dst_port},
            "Payload": base64.b64encode(payload).decode()
        })
    }])

And from there we can use Rules to direct the packets to consumers like Lambda, Step Function, DynamoDB or S3. The possibilities are endless.

Let's take an example. We could setup a Rule that filters everything destined for port 514:

aws events put-rule \
  --name udp-port-514 \
  --event-pattern '{
    "source": ["udp.raw"],
    "detail": {
      "Destination": {
        "Port": [514]
      }
    }
  }'

Now we can direct those packets to a Firehose for archival:

aws events put-targets \
  --rule udp-port-514 \
  --targets "Id"="firehose-target","Arn"="arn:aws:firehose:REGION:ACCOUNT_ID:deliverystream/YOUR_STREAM"

Or to CloudWatch Logs, or maybe both:

aws events put-targets \
  --rule udp-port-514 \
  --targets "Id"="cwlogs-target","Arn"="arn:aws:logs:REGION:ACCOUNT_ID:log-group:/eventbridge/udp-514"

That's the flexibility of EDA applied to network traffic. Here the use case is as simple as it gets, but that's just to pique your interest. Far more complicated scenarios benefit even more from EDA due to strong decoupling and the scalability of AWS services.

Caveats

Scalability can work against you.

Network traffic can be FAST. Really, really fast. Connecting a public-facing service directly to AWS resources like this is a terrible idea. It can lead to huge costs should someone decide (on purpose or accidentally) to DDoS you.

Strong, stateful firewalls with rate limiting and abuse prevention must be in place.

Summary

EDA is a powerful architecture pattern that applies to networking events as well as application generated events. Thinking more broadly about the "edge" of your systems opens-up great potential for simplification and, honestly, is just plain fun.