Serverless Packet Capture

A packet capture is a log of network traffic at an interface, and useful for diagnosing a wide variety of issues in distributed (internet/intranet) systems. If you you need to know who sent what, to who and when a packet capture will tell.

How Packet Captures Help

In most cases your web/API handler gets a runtime object representing each request, and for debugging those can be logged (e.g. written to CloudWatch) so no need to actually see the network packets that made up the request, right?

Maybe.
Maybe even mostly.

But for some classes of issues, "request object" may not be trustworthy. Some bugs, some performance issue, some odd behaviors defy explanation when looking only at the request in processed form.

Perhaps you have a situation where your application code adds a header to a response but the client doesn't seem to respect it:

• Is the client buggy?
• Did some layer of server code overwrite or ignore it?
• Did the client receive the response packets at all?
• Was it gone before the request arrived, perhaps removed by a proxy?
• Did the client fail to deliver it because of a full buffer?

Sometimes it's helpful to see the actual data that traversed the internet and remove those doubts. That's the role of a packet capture.

WireShark, of course, is the tool of choice for local packet captures. Run it on your machine and get instant visibility into the actual network data arriving on your network interface (i.e. your laptop or VM). It's a fabulous tool. It can even work over SSH for remote captures.

But what if you aren't working alone and need to share what you see?

Packet capture files (.pcap) can be put into Slack, emailed, etc. But it's tedious. Sharing your screen on a video call works, but leaves you clicking and scrolling by proxy.

Wouldn't it be nice to have a way to see the traffic you're interested in when it happens and share it in real-time with your collaborators?

Real-time Sharing with AppSync Events

I regularly work with teams building custom UDP APIs and engage in such real-time, collaborative investigations. On such calls I've shared .pcap files over Slack and I have to say, it hasn't been a great experience and definitely slowed things down.

To help everyone out I put together a simple web based solution with the fabulous AWS AppSync Events at its core. It allows anyone with the link to see the capture in real-time:

The architecture here is remarkably simple. Packets sent to the Listeners are queued in SQS, then processed by a Lambda function that parses the IP and UDP headers and formats the results as JSON and sends them out as events. The UI is rendered directly from a second Lambda via a function URL:

That's it. No servers required.

Creating the AppSync resources in the template is remarkably concise. We just need to create the API, a key to use in the UI and the namespace for the packet events:

  PacketCaptureApi:
    Type: AWS::AppSync::Api
    Properties:
      Name: !Sub "${AWS::StackName}-packet-capture"
      EventConfig:
        AuthProviders:
          - AuthType: API_KEY
        ConnectionAuthModes:
          - AuthType: API_KEY
        DefaultPublishAuthModes:
          - AuthType: API_KEY
        DefaultSubscribeAuthModes:
          - AuthType: API_KEY

  PacketCaptureApiKey:
    Type: AWS::AppSync::ApiKey
    Properties:
      ApiId: !GetAtt PacketCaptureApi.ApiId

  PacketsNamespace:
    Type: AWS::AppSync::ChannelNamespace
    Properties:
      ApiId: !GetAtt PacketCaptureApi.ApiId
      Name: packets

Likewise, the code for the Lambda function to send events is almost trivial once the JSON payload representing the packet is assembled:

private static async Task PublishEventsAsync(PacketEvent[] events, ILambdaContext context)
{
  // Each element of the "events" array is published as an individual channel event.
  var body = new PublishBody
  {
    Channel = APPSYNC_CHANNEL,
    Events = [.. events.Select(e => JsonSerializer.Serialize(e, JsonContext.Default.PacketEvent))]
  };

  var json = JsonSerializer.Serialize(body, JsonContext.Default.PublishBody);

  using var request = new HttpRequestMessage(HttpMethod.Post, APPSYNC_HTTP_URL)
  {
    Content = new StringContent(json, Encoding.UTF8, "application/json")
  };
  request.Headers.Add("x-api-key", APPSYNC_API_KEY);

  var response = await Http.SendAsync(request);
  if (!response.IsSuccessStatusCode)
  {
    var error = await response.Content.ReadAsStringAsync();
    context.Logger.LogError($"AppSync publish failed ({response.StatusCode}): {error}");
    response.EnsureSuccessStatusCode();
  }
}

You can find the full code in the repo over on github, and review the video walk-through to see just how quickly it can deployed, setup and used.

If your data is sensitive, the walkthrough includes setting-up a WireGuard tunnel to keep your data secure and private.

Summary

Event based systems can be amazingly simple and provide sublime capabilities on AWS. This solution composes features from AppSync, SQS, UDP Gateway and custom code on Lambda to deliver a multi-user packet capture experience that would have been difficult to replicate in the past.

What a great time to be building software!