DEV Community

Dinesh_gowtham
Dinesh_gowtham

Posted on

How to Stream Multi-File AI Coding Assistant Suggestions with Amazon Kinesis, Explained Simply

AI coding assistants are moving beyond single‑file autocomplete to multi‑file, agentic edits. To keep your IDE responsive, you need a way to handle a flood of edit suggestions in real time. Streaming those events through Amazon Kinesis lets you decouple generation from consumption and scale safely.

Why Streaming AI Assistant Output Matters

When an assistant suggests changes across many files, each suggestion becomes a tiny event: “Add this function to utils.js”, “Rename variable in app.ts”, etc. If you fire a separate HTTP request for every event and wait for the response, the round‑trip latency adds up quickly and your editor feels sluggish.

Think of the flow like a conveyor belt in a factory. The assistant drops packages (suggestions) onto the belt, and downstream workers (your build, testing, or review pipelines) pick them up at their own pace. The belt never stops, and if it gets jammed the whole line halts. Amazon Kinesis is the digital version of that belt – it accepts a steady stream of records and guarantees they can be read later without losing order inside a single lane (called a shard).

In plain English: By treating suggestions as a stream you avoid “one‑by‑one” blocking calls and you give yourself room to grow the number of files the assistant touches.

Minimal code: putting a single suggestion onto a stream

// npm install @aws-sdk/client-kinesis
import { KinesisClient, PutRecordCommand } from "@aws-sdk/client-kinesis";

const client = new KinesisClient({ region: "us-east-1" });

async function sendSuggestion(prId: string, file: string, diff: string) {
  const payload = JSON.stringify({ prId, file, diff, timestamp: Date.now() });

  // Each record needs a partition key – we use the PR ID so all edits for the same PR land in the same shard.
  const command = new PutRecordCommand({
    StreamName: "ai-code-suggestions",
    PartitionKey: prId,
    Data: Buffer.from(payload),
  });

  await client.send(command);
}
Enter fullscreen mode Exit fullscreen mode

Why each line matters

  • KinesisClient is the object that knows how to talk to the Kinesis service.
  • PutRecordCommand describes a single write operation.
  • PartitionKey determines which shard receives the record; keeping the same PR together helps preserve order.

Setting Up a Kinesis Stream for Code Suggestions

Before you can write or read anything, you need a stream object in AWS. A stream is a container that holds a set of shards – independent lanes that each carry up to 1 MiB per second of write throughput and 2 MiB per second of read throughput. If you exceed those limits you’ll see throttling errors, a common surprise when a team suddenly scales from a few suggestions per minute to thousands.

Tip: Start with a single shard during development. When you monitor the WriteProvisionedThroughputExceeded CloudWatch metric, add more shards to spread the load.

Minimal code: creating the stream (run once)

// npm install @aws-sdk/client-kinesis
import { KinesisClient, CreateStreamCommand, DescribeStreamCommand } from "@aws-sdk/client-kinesis";

const client = new KinesisClient({ region: "us-east-1" });

async function createStream() {
  // We ask for 1 shard; each shard gives us 1 MiB/s write and 2 MiB/s read.
  const createCmd = new CreateStreamCommand({
    StreamName: "ai-code-suggestions",
    ShardCount: 1,
  });
  await client.send(createCmd);
  console.log("Stream created, waiting for ACTIVE state...");

  // Wait until the stream is ready (polling DescribeStream)
  while (true) {
    const desc = await client.send(new DescribeStreamCommand({ StreamName: "ai-code-suggestions" }));
    if (desc.StreamDescription?.StreamStatus === "ACTIVE") break;
    await new Promise(r => setTimeout(r, 1_000));
  }
  console.log("Stream is ACTIVE.");
}
Enter fullscreen mode Exit fullscreen mode

Key concepts

  • ShardCount is the number of lanes you start with.
  • DescribeStreamCommand lets you poll the current status; a stream must be ACTIVE before you can write.

Gotcha alert

Kinesis does not let you filter messages on the service side. If you need only “JavaScript file edits”, you must filter after you read the records. Also, the iterator you obtain to read a shard expires after five minutes; a consumer that pauses longer will need to request a fresh iterator or risk missing data.

Consuming the Stream in a Node.js 22 ECS Task

Your IDE extension can push suggestions, but the heavy lifting—grouping, merging, persisting—happens in a background worker. Amazon Elastic Container Service (ECS) lets you run a container on either EC2 or Fargate. The example below assumes a Fargate task running Node.js 22, which is a recent LTS runtime that supports top‑level await.

In plain English: The consumer continuously asks Kinesis for new records, processes them, and then loops back.

Minimal code: a long‑running consumer

// npm install @aws-sdk/client-kinesis @aws-sdk/client-ecs
import { KinesisClient, GetShardIteratorCommand, GetRecordsCommand } from "@aws-sdk/client-kinesis";

const client = new KinesisClient({ region: "us-east-1" });
const streamName = "ai-code-suggestions";
const shardId = "shardId-000000000000"; // With a single shard this ID is static.

async function startConsumer() {
  // Get an iterator that starts at the latest record (you could use TRIM_HORIZON to read from the beginning).
  const iteratorCmd = new GetShardIteratorCommand({
    StreamName: streamName,
    ShardId: shardId,
    ShardIteratorType: "LATEST",
  });
  let iterator = (await client.send(iteratorCmd)).ShardIterator;

  while (iterator) {
    const recordsCmd = new GetRecordsCommand({ ShardIterator: iterator, Limit: 100 });
    const response = await client.send(recordsCmd);

    // Kinesis may return an empty batch even when there is data pending – this is normal because of eventual propagation.
    if (response.Records?.length) {
      for (const rec of response.Records) {
        const payload = JSON.parse(Buffer.from(rec.Data).toString());
        // Hand off each suggestion to the aggregation layer (see next section).
        handleSuggestion(payload);
      }
    }

    // Advance iterator for the next loop; iterator expires after 5 min, so we must keep refreshing it.
    iterator = response.NextShardIterator;
    // Small pause prevents hammering the service.
    await new Promise(r => setTimeout(r, 200));
  }
}
Enter fullscreen mode Exit fullscreen mode

Why this works

  • GetShardIteratorCommand creates a cursor (the iterator) that points to a position in the shard.
  • GetRecordsCommand fetches up to Limit records starting at that cursor.
  • The loop updates the iterator each iteration; if the iterator ever becomes null you should request a new one.

Gotcha alert

When you run this task on Fargate you might think you have GPU power for model inference, but Fargate currently does not expose GPUs. If you later add on‑device model calls, the task will fail silently. Keep inference in a separate service (e.g., SageMaker) and let the ECS consumer stay lightweight.

Aggregating Multi‑File Edits and Storing Them in S3

Each suggestion arrives as an independent JSON blob. To turn a set of blobs into a single, reviewable patch you need to:

  1. Group events by pull‑request ID (prId).
  2. Order them by the timestamp each assistant generated.
  3. Concatenate the diffs into one unified patch.

The final patch is then uploaded to an Amazon S3 bucket for later retrieval or CI consumption. An S3 bucket is a simple object storage service; each object is identified by a key (like a filename).

Tip: Use a UUID or the PR number plus a timestamp in the key so you never overwrite a previous diff.

Minimal code: aggregation and S3 upload

// npm install @aws-sdk/client-s3
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const s3 = new S3Client({ region: "us-east-1" });
const bucket = "ai-code-patches";

type Suggestion = { prId: string; file: string; diff: string; timestamp: number };
const pending: Record<string, Suggestion[]> = {}; // In‑memory store keyed by PR ID.

function handleSuggestion(sug: Suggestion) {
  // 1️⃣ Collect suggestions per PR.
  if (!pending[sug.prId]) pending[sug.prId] = [];
  pending[sug.prId].push(sug);
}

// Called periodically (e.g., every 30 s) to flush completed PRs.
async function flushAggregates() {
  for (const prId of Object.keys(pending)) {
    const list = pending[prId];
    // 2️⃣ Sort by timestamp to keep edits in the order the model produced them.
    list.sort((a, b) => a.timestamp - b.timestamp);

    // 3️⃣ Build a single diff string.
    const combined = list.map(s => `--- ${s.file}\n${s.diff}`).join("\n\n");

    // 4️⃣ Store in S3.
    const key = `pr-${prId}/${Date.now()}.diff`;
    const putCmd = new PutObjectCommand({
      Bucket: bucket,
      Key: key,
      Body: combined,
      ContentType: "text/plain",
    });
    await s3.send(putCmd);
    console.log(`Saved combined diff for PR ${prId} to s3://${bucket}/${key}`);

    // Clean up after successful upload.
    delete pending[prId];
  }
}

// Example: run flush every 30 seconds.
setInterval(() => {
  flushAggregates().catch(err => console.error("Flush error:", err));
}, 30_000);
Enter fullscreen mode Exit fullscreen mode

Explanation of steps

  • pending holds suggestions temporarily; in a production system you would use a durable store (DynamoDB) to survive restarts.
  • Sorting by timestamp guarantees the order the assistant intended, but remember the shard order guarantee only applies within a single shard. If you later add more shards, you must embed a sequence number or use a deterministic ordering key.

Gotcha alert

Because Kinesis only guarantees order inside a shard, if you change the PartitionKey strategy and end up with multiple shards for the same PR you may receive edits out of order. The safest pattern is to keep a single shard per active PR, or to add a monotonically increasing sequence field generated by the assistant and sort on that field instead of the raw arrival time.

Observability: Using diagnostics_channel for Zero‑Cost Tracing

When you run many containers, understanding where latency originates (Kinesis read, aggregation, S3 upload) becomes essential. Node.js ships with the diagnostics_channel module, a built‑in, low‑overhead way to emit named events that external tools (like CloudWatch Logs Insights or a custom collector) can subscribe to.

In plain English: Think of diagnostics_channel as a hidden microphone that only turns on when something is listening, so it never slows down your code unless you need it.

Minimal code: instrumenting the consumer

import { createChannel } from "node:diagnostics_channel";

const readChannel = createChannel("kinesis:read");
const aggChannel = createChannel("suggestion:aggregate");
const s3Channel = createChannel("s3:upload");

// Wrap Kinesis read
async function readLoop() {
  // ... iterator setup omitted for brevity
  while (true) {
    const start = Date.now();
    const response = await client.send(new GetRecordsCommand({ ShardIterator: iterator, Limit: 100 }));
    readChannel.publish({ latencyMs: Date.now() - start, recordCount: response.Records?.length ?? 0 });

    // Process each record
    for (const rec of response.Records ?? []) {
      const sug = JSON.parse(Buffer.from(rec.Data).toString());
      const aggStart = Date.now();
      handleSuggestion(sug);
      aggChannel.publish({ latencyMs: Date.now() - aggStart, prId: sug.prId });
    }

    iterator = response.NextShardIterator;
    await new Promise(r => setTimeout(r, 200));
  }
}

// Wrap S3 upload
async function flushAggregates() {
  // ... loop omitted
  const uploadStart = Date.now();
  await s3.send(putCmd);
  s3Channel.publish({ latencyMs: Date.now() - uploadStart, key });
}
Enter fullscreen mode Exit fullscreen mode

What each publish does

  • readChannel.publish emits a tiny JSON object with how long the GetRecords call took and how many records came back.
  • aggChannel.publish measures the time spent handling a single suggestion.
  • s3Channel.publish reports the time for the PutObject call.

You can attach a listener in a separate monitoring container:

import { subscribe } from "node:diagnostics_channel";

subscribe("kinesis:read", (msg) => console.log("[Kinesis read]", msg));
subscribe("suggestion:aggregate", (msg) => console.log("[Aggregate]", msg));
subscribe("s3:upload", (msg) => console.log("[S3 upload]", msg));
Enter fullscreen mode Exit fullscreen mode

Because the channel only produces output when a subscriber exists, the overhead is essentially zero when you are not actively debugging.

Gotcha alert

ECS Service Connect adds an extra network hop for intra‑service traffic. If you route the diagnostic listener through Service Connect you may see a few extra milliseconds of latency that are not present in the raw SDK calls. Measure both paths if you suspect a bottleneck.

The Takeaway

Key points to remember

  • Streaming suggestions through Kinesis decouples generation from consumption, keeping the IDE snappy even under heavy multi‑file workloads.
  • A shard is a single ordered lane; use a stable PartitionKey (e.g., PR ID) to keep related edits together, otherwise you may lose order.
  • Create the stream with the right shard count, watch the 1 MiB/s write / 2 MiB/s read limits, and remember that shard iterators expire after five minutes.
  • In an ECS Fargate task, use a simple while loop with GetRecords to pull events, then group by PR ID, sort by timestamp (or a sequence number), and upload the merged diff to S3.
  • diagnostics_channel gives you free, low‑impact visibility into read, aggregation, and upload latency; pair it with a log collector to spot problems early.
  • Common pitfalls: out‑of‑order edits when you introduce multiple shards, throttling from underestimated shard capacity, and silent permission errors caused by confusing ECS task versus execution roles.

By following these steps you turn an AI‑driven code assistant from a single‑file autocomplete toy into a production‑grade, observable pipeline that can handle dozens of files per pull request without slowing down developers. Happy streaming!


Transparency notice

This article was written with the help of an AI system — Groq (GPT OSS 120B).

Published: 2026-11-16 · Primary focus: Kinesis

All code blocks are intended to be correct and runnable, but please verify them
against the official docs for the tools mentioned before using in production.

Find an error? Drop a comment — corrections are always welcome.

Top comments (0)