DEV Community: iwan rustiawan

Tips on Handling Files in LLMs

iwan rustiawan — Fri, 17 Jul 2026 06:38:56 +0000

Tips on Handling Files in LLMs

You might want to pull information (especially from doc/text documents) out of a file and then interact with an LLM about its contents. The easiest approach is to let a multimodal LLM extract the file and feed it straight into the context for discussion.

That's not wrong — but picture the token waste: your context gets dominated by image tokens, because every page is rendered as an image that burns far more tokens than its text would. For multi-page files, the cost climbs fast.

For cases like this, it's better to handle the file inside your own backend, where you have far more control. You can extract the text separately and assemble the LLM context more deliberately — for example, only injecting the parts you actually consider necessary.

Here's a quick comparison from my own experience:

	OCR / Text-Extraction Pipeline	Direct LLM/Vision
Stability	✅ Winner. OCR is consistent — the same input tends to produce the same text. The LLM only reasons over the extracted text, so there are fewer sources of variance and the failure modes are more bounded and well understood.	⚠️ Vision extraction varies run-to-run; it can silently drop rows on dense tables or hallucinate digits.
Cost	✅ Winner at scale. Open-source engines (Tesseract/PaddleOCR) are free to self-host, or you can use a proven managed OCR with predictable per-page pricing.	❌ Image tokens dominate — every page rendered as an image costs far more tokens than its text (unless you downscale aggressively, which then sacrifices digit accuracy).
Numeric Accuracy	✅ Winner. If accuracy on numbers is the goal, you can use OCR with a custom model purpose-built for tabular digits.	❌ Weaker on dense financial tables; digit-level errors are exactly the kind of failure you can't tolerate.
Document Classification	⚠️ OCR strips out visual cues (logos, stamps, layout). But usually the text alone is enough.	✅ Winner here. Vision sees logos/layout/scanned docs/handwriting that OCR struggles with.

One important caveat: this isn't about "vision being bad." There are cases where Direct Vision is actually the better call — low-quality scans, handwriting, or highly unstructured layouts where building a clean OCR pipeline is more trouble than it's worth. The point is to match the tool to the nature of your data, not to paint with one brush.

For any system that touches numbers or finances, my rule is simple: hand numeric extraction to a layer that's deterministic and auditable — not to a model that "thinks it's probably right."

LLM #AIEngineering #SoftwareEngineering

Building Production-Grade LLM Applications

iwan rustiawan — Tue, 14 Jul 2026 16:16:25 +0000

Building Production-Grade LLM Applications
A short note, just sharing some experience that might be useful for those who are starting to explore building LLM applications. A few things I've come to understand that have become important notes for me.

1. LLMs are stateless.
Don't assume that once you've successfully integrated with an LLM vendor's API, you automatically get the same experience as interacting with ChatGPT directly. ChatGPT is a product; what we call through the API is a model — and the model has no memory at all.

You say "my name is Iwan" in turn 1? Don't expect the LLM to remember it in turn 2. What actually happens: the entire conversation history is re-sent with every request. So "memory management" isn't the model's job — it's ours :D

And this brings its own challenges, because it isn't as simple as big vs. small memory; it's about keeping accuracy, cost, and performance in balance.

2. RAG is not part of the LLM.
RAG is an architecture around the model, not a built-in capability. We first retrieve the relevant documents, then inject them into the context. The model itself knows nothing about our database.

A simple example: say we have an FAQ document we want to use as a reference for our app. We chunk that document, turn it into embeddings, and store it in a vector database. Later, when a user asks a question, we turn the question into an embedding too, use it to find the most relevant FAQ chunks — and only then inject the results into the context before sending it to the model.

3. The LLM is an orchestrator, not a calculator.
This is the one I emphasize most. Never hand calculations over to an LLM, especially anything financial.

Not because the AI isn't smart enough, but because structurally it predicts the next token probabilistically — it doesn't execute arithmetic. When we ask it to add two numbers, it isn't calculating. It's guessing the most likely-looking answer.
The solution: function calling. The model's job is to understand the user's intent and choose which function to call. The computation is handed off to a deterministic backend — one that can be tested and audited.

It's not just about accuracy, it's also about auditability. If someone asks, "why is the number this?", the answer has to be traceable to a function whose code we can actually show. Not to a model that "thinks it's probably that."

There's still a lot I haven't figured out on my end, especially around evaluation and cost. If anything here is off, or if you've had a different experience, I'd be glad to hear your input.

LLM #AIEngineering #SoftwareEngineering