What Happened
A developer moved a 35 KB pre‑prompt from the Opus model to a self‑hosted Ollama instance. The prompt was larger than Ollama’s default token limit, so the request was truncated and the model behaved unpredictably. The developer also noticed that Ollama treats system messages and prompt formatting differently, which altered output quality.
To fix this, the prompt was rewritten to fit within Ollama’s constraints, the request payload was adjusted, and a caching layer was added to prevent repeated large prompt transmissions. The workflow now runs smoothly, but the experience revealed several hidden pitfalls when moving from cloud‑based LLMs to self‑hosted solutions.
Why This Matters for Builders
- Prompt Size Constraints: Cloud providers often allow larger prompts or automatically chunk requests. Self‑hosted models like Ollama enforce stricter limits. Audit prompt lengths before deployment.
- Serialization Differences: System messages and user prompts are encoded differently. A prompt that works in Opus may need re‑formatting for Ollama, affecting token counts and model context.
- Caching Strategy: Resending a large prompt on every run increases latency and resource usage. Caching the prompt payload (e.g., with Redis or an in‑memory store) improves throughput.
- Error Handling and Fallbacks: Truncation or token limit errors can crash automation flows. Add graceful degradation—fall back to a shorter prompt or log detailed diagnostics.
- Monitoring and Metrics: Track prompt size, token usage, and response latency. This is essential for production n8n or AI‑agent pipelines.
FAQ
Q: How can I check if my prompt exceeds Ollama’s limit?
A: Use a tokenizer that matches the model (e.g., tiktoken for GPT‑style models) to count tokens before sending the request. Compare the count to the model’s documented maximum.
Q: What’s the best way to cache a large prompt in an n8n workflow?
A: Store the prompt in a Redis instance or a local file with a key tied to the workflow run. Retrieve it before the LLM node and only regenerate it if the underlying data changes.
Q: Should I split the prompt into smaller chunks?
A: If the model’s limit is tight, break the prompt into logical sections and feed them sequentially, concatenating results or using a summarization step to maintain context.
Q: How do I handle truncation errors in production?
A: Wrap the LLM call in a try‑catch block, detect token limit errors, and either truncate the prompt gracefully or trigger a fallback workflow that logs the issue and continues with a default response.
Originally published on Automations Cookbook.
Top comments (0)