Chanyeong Yun

Posted on Jul 27

31.8x Speedup by Changing One File: Async Embedding Calls on AWS Bedrock

#aws #python #performance #ai

I was testing a RAG ingestion pipeline and noticed something painful: ingesting a single document took nearly 50 seconds.

Checking the metrics revealed that the CPU was doing almost nothing. The app wasn't compute-bound — it was simply sitting idle, waiting for sequential HTTP network round-trips to return.

By converting the blocking embedding module into an asynchronous workflow, processing time dropped from 49.61 seconds to 1.56 seconds with zero infrastructure changes.

Benchmark Setup

Model: Amazon Titan Text Embeddings V2 (AWS Bedrock)
Dataset: 33 text chunks from a single document
Region: us-east-1
Test: Sequential blocking requests vs. concurrent asynchronous requests

Results

Approach	Time	Difference
Sequential (`requests` blocking loop)	49.61s	Baseline
Concurrent (`aiohttp` + `asyncio.gather`)	1.56s	31.8× faster

Why the Gap Was So Massive

Sequential (Blocking I/O)

Each request must complete before the next one starts.

chunk 1 ──> request ──> ⏳ wait (~1.5s) ──> response
chunk 2 ──> request ──> ⏳ wait (~1.5s) ──> response
chunk 3 ──> request ──> ⏳ wait (~1.5s) ──> response
...
Total Time ≈ (33 chunks × ~1.5s) ≈ 49.6s

Since network latency dominates the runtime, the event loop sits completely unused.

Concurrent (Non-blocking Async)

Requests are dispatched concurrently; total time collapses down to the duration of the slowest single request.

chunk 1  ──> request ──────────────> response
chunk 2  ──> request ──────────────> response
chunk 3  ──> request ──────────────> response
...        (in flight concurrently)
Total Time ≈ slowest single request ≈ 1.56s

The Code Change

Before: Sequential Execution

import requests

def generate_embeddings(texts: list[str]) -> list[list[float]]:
    all_embeddings = []
    for text in texts:
        response = requests.post(url, headers=headers, json={"inputText": text})
        vector = response.json().get("embedding", [])
        all_embeddings.append(vector)
    return all_embeddings

After: Concurrent Execution

import asyncio
import aiohttp

async def fetch_embedding(session: aiohttp.ClientSession, text: str) -> list[float]:
    async with session.post(url, headers=headers, json={"inputText": text}) as res:
        data = await res.json()
        return data.get("embedding", [])

async def generate_embeddings_async(texts: list[str]) -> list[list[float]]:
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_embedding(session, text) for text in texts]
        return await asyncio.gather(*tasks)

Note: If you're using boto3 instead of raw HTTP requests, look at aioboto3, or offload the blocking SDK calls with asyncio.to_thread() to get similar non-blocking behaviour.

How It Scales

Chunks	Sequential (est.)	Concurrent (est.)
31	49.61s	1.56s
100	~160s (2.7 min)	~2–3s
500	~800s (13 min)	~3–5s
1,000	~1,600s (26 min)	~5–10s

The more chunks you ingest, the more the concurrent approach pays off — batch ingestion jobs that took half an hour now finish in seconds.

Key Takeaways

Check for idle waiting first. Before scaling up infrastructure or adding complex worker queues, verify whether your pipeline is simply blocked on I/O.
Mind the service quotas. Bedrock has per-region request-rate limits — as chunk counts grow into the thousands, cap concurrency (e.g. asyncio.Semaphore) before you hit them.
Low effort, high impact. Changing just one file (embedder.py) removed the embedding stage as a bottleneck in the pipeline.

DEV Community