"API Limit 21KB!" ...Then 32KB of data caused a mysterious error. The culprit? Confusing bytes with characters.

Hey, it's your resident Old Man Developer here. One evening, while checking the logs for my self-made bot as usual, I noticed a recurring, inexplicable error in a section that interacts with an external API.

The logs showed messages like Payload too large. "Ah, typical," I thought. But something felt off. The API documentation clearly stated a "21KB payload limit," and my implementation was supposed to have a logic check to verify data size before sending it.

Yet, it was still being rejected. My local check passed, but the API screamed "too big!" This contradiction baffled me for quite a while.

The Cause: len() was the Root of All Evil

Initially, I suspected API capriciousness, or perhaps a temporary network glitch. But the error consistently reproduced with specific data patterns. Okay, this was on my end.

I pulled out the problematic data and measured its size locally.

# Assuming data_string holds the problematic data
print(len(data_string))

It was clearly smaller than 21 * 1024 = 21504. Still, the API was rejecting it. I just couldn't understand why.

After about half a day of scratching my head, it suddenly hit me: the data I was handling contained a fair amount of Japanese text. "Could it be... byte count?"

To test this hypothesis, I wrote a simple piece of code.

text = "こんにちは世界"

char_count = len(text)
byte_count_utf8 = len(text.encode('utf-8'))

print(f"String: '{text}'")
print(f"Character count: {char_count}") # => 7
print(f"Byte count (UTF-8): {byte_count_utf8}") # => 21

Seeing this, everything clicked.

"こんにちは世界" is only 7 characters. But encoded in UTF-8, it becomes 21 bytes. That's a 3x difference! For English (ASCII) text, 1 character is basically 1 byte, so len() roughly matches the byte count. But when multibyte characters like Japanese come into play, the story changes entirely.

In essence, I was checking the "character count" with len(), but the API was looking at the "byte count" calculated by len(str.encode('utf-8')). The "21KB" in the documentation had, in my mind, been unconsciously translated to something like "21k characters." Damn, it was a complete assumption on my part.

When I re-measured the actual error-causing data by byte count, it easily exceeded 21KB, coming in at nearly 32KB. No wonder the API was upset!

The Fix: Switch to Byte-Based Check Logic

Once the cause was known, the fix was simple. Just change the size check before sending to the API from character-based to byte-based.

def check_api_limit(data_string, limit_in_kb):
    # Encode the string in UTF-8 and calculate byte count
    byte_size = len(data_string.encode('utf-8'))

    # Convert KB limit to bytes for comparison
    limit_in_bytes = limit_in_kb * 1024

    if byte_size > limit_in_bytes:
        # Raise an error here to prevent the API call
        raise ValueError(f"Data size ({byte_size} bytes) exceeds limit ({limit_in_bytes} bytes)")

# Calling side
try:
    check_api_limit(my_data, 21)
    # Logic to send to API
except ValueError as e:
    print(f"Error: {e}")
    # Handle splitting, error handling, etc.

I implemented a function like this and placed it right before the API call. Now, I can accurately detect size overages locally before the API even gets a chance to complain. Since deploying this fix, the mysterious error has completely disappeared.

Lessons Learned: Question Units, Measure Empirically

The lessons I learned from this failure are quite simple yet crucial:

1. Question Unit Definitions: When documentation uses ambiguous terms like "size" or "KB," I need to make it a habit to accurately confirm whether it refers to "character count" or "byte count." If it's not specified, I should have tested with minimal data to verify behavior.
2. Don't Forget the Multibyte Trap: This issue constantly arises, especially when building AI agents that handle natural language data like Japanese. Always keep in mind that character count and byte count are completely different.
3. Assumptions are the Biggest Enemy: Python's ease of use, like getting "size" with len(), ironically became the pitfall this time. Doubting my own "I know this" assumptions is a subtle but incredibly important practice.

So, that was my story of wasting half a day due to confusing "character count" and "byte count" – a slightly embarrassing but common mistake anyone can make.

Squashing these small, unassuming errors one by one is the reality of personal development, especially when you're working on side projects. I'll share again if I screw up in another interesting way.

---

X: @oji_ai_dev
note: oji_ai_dev

---

I build and run small Python systems — trading bots, RAG APIs, scheduled automation — and write up whatever breaks along the way.

If a provider-agnostic RAG Q&A API is useful to you, mine is MIT-licensed on GitHub: rag-faq-api. It runs and passes its full test suite **with no API key* (offline stub LLM + hashing embedder), swaps to Claude / Gemini / OpenAI via one env var, and ships a retrieval-quality harness (Hit@k / MRR / Recall@k) with a chunking sweep.*