Building a real-time sports data pipeline with AWS Fargate and AppSync

Imagine a live football match where each frame of player and ball movement must reach dashboards, analytics systems, and fans almost instantly. Every second, 25 frames describe the field’s heartbeat — multiplied by multiple matches, this becomes hundreds of small, time-critical updates per second.

Many teams rush to heavy streaming platforms early, but for small-to-medium scales that approach creates more cost and operational load than value.

This article shows a simpler and production-ready alternative: a lightweight real-time ETL built with AWS Fargate and AWS AppSync Events API — a fully managed, serverless path to distribute events in milliseconds.

Later, when data volume and replay requirements grow, this same architecture evolves naturally into a Kafka-based backbone without re-writing the entire system.

The simple flow

1. External UDP sender streams binary or JSON frames (~1 KB each).

2. AWS ECS Fargate task receives those frames, transforms them, and publishes to AppSync Events API via HTTPS.

3. AWS AppSync instantly fans out each event to connected subscribers over WebSockets.

4. Web clients or analytics tools receive updates per "channel" (for example, per match).

A channel name encodes tenancy and environment: e.g. acme/prod/match-12345. This lightweight naming pattern naturally isolates tenants and simplifies scaling.

What we should care about

1. Speed to market: This pipeline can be deployed in days, not months. No need to manage servers, brokers, or clusters.

2. Low operational cost: A single Fargate task and an AppSync Events API can handle hundreds of messages per second. The total cost for a single match will stay below $1.

3. Elastic and low risk: Because both Fargate and AppSync are serverless, scaling up or down doesn’t require re-architecture. You pay only for what you use.

4. Future-ready: When event replay or guaranteed ordering becomes important, you can add Kafka (or Amazon MSK) behind the same publish interface, without rewriting the client or subscription logic.

Engineering summary

• Throughput: 25 fps per match × 15–20 matches (~375–500 msg/s)
• Payload: ~1 KB JSON (binary base64-encoded)
• Latency: <150 ms end-to-end (p95)
• Fargate task: 0.5 vCPU / 1 GB
• Scaling: Add tasks per 500 msg/s or per tenant group
• Auth: API Key (initial) → Cognito or IAM later

Code snippet: UDP listener → AppSync publisher

The following TypeScript example runs inside a Fargate task.

import dgram from "dgram";
import fetch from "node-fetch";

const server = dgram.createSocket("udp4");
const APPSYNC_URL = process.env.APPSYNC_EVENTS_URL!;
const API_KEY = process.env.APPSYNC_API_KEY!;
const CHANNEL = process.env.CHANNEL!; // tenant/env/matchId
async function publishWithRetry(payload: any) {
  const maxRetries = 5;
  for (let i = 0; i < maxRetries; i++) {
    try {
      const res = await fetch(`${APPSYNC_URL}/event`, {
        method: "POST",
        headers: { "x-api-key": API_KEY },
        body: JSON.stringify({
          channel: CHANNEL,
          data: { ...payload, sentAt: Date.now() },
        }),
      });
      if (res.ok) return;
      throw new Error(`HTTP ${res.status}`);
    } catch (err) {
      const delay = Math.min(200 * 2 ** i, 3000);
      await new Promise((r) => setTimeout(r, delay));
    }
  }
  console.error("Publish failed after retries");
}
server.on("message", async (msg) => {
  try {
    const frame = { frame: msg.toString("base64") }; // binary encoded
    await publishWithRetry(frame);
  } catch (e) {
    console.error("Parse/publish error", e);
  }
});
process.on("SIGTERM", () => {
  console.log("Graceful shutdown");
  server.close();
});
server.bind(41234);

This task listens for UDP packets, encodes each one, and publishes with exponential backoff for resilience.

Hint: You also can use Amplify client to to communicate with AWS AppSync Events API.

Client-side subscription example (React + Amplify)

import { useEffect, useState } from "react";
import { Amplify } from "aws-amplify";
import { client } from "aws-amplify/data";

Amplify.configure({
  API: {
    Events: {
      endpoint: import.meta.env.VITE_APPSYNC_EVENTS_URL,
      region: import.meta.env.VITE_AWS_REGION,
      apiKey: import.meta.env.VITE_APPSYNC_API_KEY,
    },
  },
});
export default function MatchStream() {
  const [frames, setFrames] = useState([]);
  const channel = "acme/prod/match-12345";
  useEffect(() => {
    const sub = client.subscriptions.onMessage(channel).subscribe({
      next: (msg) => setFrames((f) => [...f, msg.data]),
    });
    return () => sub.unsubscribe();
  }, []);
  return (
    <div className="p-4">
      <h2 className="font-bold">Live Feed: {channel}</h2>
      <pre>{JSON.stringify(frames.slice(-5), null, 2)}</pre>
    </div>
  );
}

Benchmark and load validation

A simple Node.js script can generate UDP traffic and measure round-trip latency.

import dgram from "dgram";
import { performance } from "perf_hooks";

const client = dgram.createSocket("udp4");
const RATE = 25; // frames/sec
const HOST = process.env.UDP_HOST;
const PORT = process.env.UDP_PORT;
let id = 0;
setInterval(() => {
  const sentAt = performance.now();
  const msg = Buffer.from(JSON.stringify({ id, sentAt }));
  client.send(msg, PORT, HOST);
  id++;
}, 1000 / RATE);

At 25 fps × 20 matches, total ~500 msg/sec, still below AppSync’s 10 k events/sec limit. During local testing, this configuration remained under 120 ms p95 latency end-to-end.

Cost overview (per single match session)

Compute

Runtime: 90 minutes
Rate: 25 frames/sec
Subscribers: 3
Fargate task size: 1 vCPU / 2 GB RAM ($0.07 for the 90-minute session)

Operations math:

Duration: 90 minutes = 5,400 seconds
Inbound publishes: 25 fps × 5,400 s = 135,000 operations
Outbound broadcasts: Each event goes to 3 subscribers → 135,000 × 3 = 405,000 operations
WebSocket ops (connect + subscribe + pings)
Connect + subscribe overhead: ~6 ops (3 connects + 3 subscribes) — negligible
Keep-alive pings: 1 per minute per connection → 150 minutes × 3 connections = 450 ops.

Total ops ~135,000 + 405,000 + 450 + 6 = 540,456 ~0.54 million

Event API operations cost: 0.54 M × $1.00/M = $0.54

AppSync Events API pricing (eu-central-1): $1.00 per million Event API operations + $0.08 per million connection-minutes.

Note: All publishes, broadcasts, and WebSocket operations (connect, subscribe, ping) are "operations".

Connection-minutes: 3 subscribers × 90 minutes = 270 connection-minutes: 270 / 1,000,000 × $0.08 ≈ $0.0000216 (basically zero).

Per-match session cost (eu-central-1)

Fargate (1 vCPU + 2 GB, 150 min): $0.07
AppSync Events operations: $0.54
AppSync connection-minutes: $0.0000216 (rounds to $0.00)
Estimated total (excluding data transfer): ~ $0.61 per match session

Heads-up on data transfer. Internet egress is billed separately at standard EC2 data transfer rates. Include it if our subscribers aren’t inside AWS.

Capacity & latency notes

1. AppSync Events allows ~10,000 inbound events/sec and ~1,000,000 outbound/sec (soft-adjustable). Our 25–500 msg/sec range is comfortably below this.
2. Keep-alive impact: The 60-second keep-alive contributes a tiny number of billable ops compared to our event traffic.
3. Latency target: With Fargate → AppSync → WebSocket fan-out, staying under 150 ms (p95) is realistic at these rates.

TL;DR for managers

For match-time-only workloads, Fargate + AppSync Events is both feasible and cost-efficient:

• Under $1 per match (compute + Event API ops), excluding Internet egress.
• Scales to dozens of matches with linear, predictable cost.
• Zero server management — fast to deploy, easy to tear down after each match.
• Clean migration path to Kafka/MSK later for strict ordering and replay — without throwing away our WebSocket fan-out.