The third article in the "Professional Claude.ai Usage" series is an honest conversation about where the model can let you down: hallucinations when handling facts, outdated knowledge without search, the risk of over-trusting a confident tone, and copyright nuances. Every risk comes with a concrete way to minimize it.
Why you need to know the limitations, not just the capabilities
The previous article in the series gave you the basics of prompt engineering and explained how to formulate requests to get a quality result. But even a perfectly worded prompt doesn't guarantee an error-free response if you don't understand the model's structural limitations.
This isn't a unique "weakness" of Claude specifically — all large language models share similar systemic limitations that stem from the nature of their architecture. Understanding these boundaries isn't a reason to abandon the tool; it's a way to use it more effectively and safely. You'll know exactly where to double-check the result manually, and where you can trust it without extra verification.
Hallucinations: when confidence doesn't mean correctness
The most important thing to internalize: a language model generates text based on probabilistic patterns, not by querying a database of verified facts in real time (unless it's using a search tool). This means the model can state factually false information in just as confident a tone as it states correct information.
Where hallucinations happen most often
Specific figures and statistics without a source • Precise dates for obscure events • Quotes attributed to real people • Names and details of studies or publications • Technical details of obscure APIs or libraries that may not even existA real hallucination example
Prompt: "What study confirms that drinking 8 glasses of water a day is a scientifically established norm?"The model may confidently cite a specific study with authors, year, and journal — one that doesn't actually exist, or that doesn't even support that conclusion. A quick check on Google Scholar takes 30 seconds and can save you from citing a source that was never real.
How to minimize the risk:
- Verify any specific facts, figures, or quotes headed for a public-facing piece through an independent source immediately.
- Ask the model to flag its confidence level or explicitly note where information might be inaccurate: "If you're not sure of the exact figure, say so directly instead of making up a plausible-sounding number."
- For critical factual data, use a mode with web search access (if available) — this significantly reduces the risk of getting outdated or fabricated information.
Knowledge staleness: why "I don't know" is a normal answer
Every model has a so-called knowledge cutoff date, after which it has no built-in awareness of events in the world. Anything that happened after that date, the model only knows about if it has access to a search tool and actually uses it for that specific request.
A common user mistake is expecting the model to "just know" about the latest news, current prices, a company's current leadership, or a product's most recent version. Without explicitly using search, the answer will be based on outdated data. Worse still, the model doesn't always warn you about this unless you've phrased the request in a way that makes clear you need current, up-to-date information.
| Type of request | Staleness risk |
|---|---|
| Historical facts, scientific concepts | Minimal — this data is stable over time |
| Current positions, company leadership | High — changes often, search recommended |
| Software versions, technical specs | High — updates quickly |
| Prices, exchange rates, market data | Critical — changes daily or hourly |
How to minimize the risk: for any request where the current situation genuinely matters ("who is currently," "what's the latest version," "what does it cost right now"), explicitly ask the model to use web search (this feature is available in the interface, in the menu below the chat box) or independently verify the answer against an up-to-date source.
Over-trusting a confident tone
One of the most dangerous traps is psychological, not technical. Language models generate text smoothly, grammatically flawlessly, and usually in a confident tone — regardless of how "confident" they actually are in the factual accuracy of the answer. The human brain is wired to associate a confident, polished delivery with reliability (the same bias applies to human experts), and this bias transfers easily to interactions with AI.
The practical takeaway: don't trust a response more just because it sounds convincing or detailed. The level of detail in an answer is not evidence of its factual accuracy. This matters especially in analytical work and in legal or medical questions, where the cost of a mistake is high, and the answer itself is delivered so authoritatively that an inaccurate verdict could cause real harm if a decision is made based on it without verification.
Copyright and reproducing other people's content
Claude is deliberately constrained in reproducing copyrighted content: it won't generate verbatim quotes from songs, poems, or large excerpts from published texts, even if the request is framed as "deliberately legitimate" (for example, "for educational purposes" or "just for personal use"). This isn't a bug or excessive caution — it's a deliberate design decision.
Practical takeaway for content creators
If you need material "in the style" of a particular author or work, ask for a stylistic homage, not a reproduction. For example: "write an original poem in the style of romantic poetry" will work, while "quote the first verse of song X" won't — and that's expected behavior, not a malfunction.
For content you plan to publish (articles, marketing copy), it's also worth remembering: even fully generated text can accidentally end up closely mirroring the structure or phrasing of a source if you paste a large chunk of someone else's text into the prompt for "rewriting." The safest approach is to use other people's material as reference context for understanding the topic, not as a template for a one-to-one paraphrase.
Context window limits in practice
While Claude's context window is one of the widest among popular models, it's still finite. When working with very large documents, long codebases, or multi-hour conversations, keep in mind that information from the very beginning of a long session may get less "attention" from the model compared to more recent context.
Practical tip: for long working sessions, periodically summarize key decisions and agreements with an explicit message (for example: "let's lock this in: we've decided to use approach X because of Y") — this keeps important context visible and reduces the risk of the model "losing" an earlier agreement in a long conversation.
Where you can trust Claude without extra verification
Not everything requires manual verification. Some tasks carry minimal risk, and double-checking them is just a waste of time:
- Structuring and formatting your own text.
- Code refactoring (logical correctness is usually visible right away when you test it).
- Generating variations of headlines or phrasings for A/B tests.
- Translation into a language you know, where small mistakes are easy to spot immediately.
- Summarizing a document you've already read yourself.
Pre-publication checklist
To wrap everything up, here's a short checklist to run through before using Claude-generated content in real work:
- [ ] All specific facts verified through an independent source
- [ ] No verbatim quotes of copyrighted content
- [ ] Current data verified via search or manually
- [ ] The result wasn't taken at face value just because it sounded convincing
- [ ] Key decisions were explicitly locked in during long conversation sessions
What's next?
With the fundamentals covered, the series now moves into its specialized section. The next article focuses on Claude.ai's application in development work: code review, debugging, documentation, and API integration.
👉 Up next: Claude for programmers and developers — the first of four specialized articles in the series.
Top comments (19)
This lands, especially the context-window section. Your "let's lock this in: we decided X because of Y" tip is the right instinct, and it is the checklist item people skip most.
One thing I would add: a lock typed into the chat survives that session, not the next one. Open a fresh conversation and the model is not re-reading your earlier "lock this in" message, so the decision quietly reverts to a guess. What works for me is moving those locks out of the transcript into a small decisions file the next session opens first, keeping the why next to each one and not only the what, plus a short end-of-task habit: what did we decide, what is still open, what changed. Native memory helps for facts, but this is really about a readable handoff of operating context between sessions.
I packaged that habit as an open-source setup for Claude Cowork and Code if it helps: cowork-os. When you start a brand new session on the same project, how do you reload those locked decisions today, paste them back in or keep them somewhere external?
Wow, Raffaele, I just took a deep dive into cowork-os, and I must say — this is a beautiful piece of engineering! Love the philosophy of treating the operating context as plain Markdown files that humans can actually read, edit, and track via Git, instead of relying on opaque vector memory engines like claude-mem. The work-state update protocol is a fantastic pattern.
To answer your question on how I reload those decisions:
If I'm stuck using the standard Anthropic Web UI, I rely heavily on native Claude Projects. I upload a persistent
decisions.mdto the Knowledge Base and use strict Project Instructions to force the model to parse it before every task. It works well to keep the context clean without raw copy-pasting.However, once the project moves out of the browser into the IDE (like Cursor or VS Code extensions) or CLI workflows, the Web UI Projects fall short. In those cases, I do exactly what you preached — keeping an explicit architecture/decision log right in the local Git repository.
Your slash-command approach (
/cowork-os:install) and the scheduled cron-like tasks in Claude Cowork are brilliant additions to this workflow.Quick question on your architecture: as a project scales over months and the
decisions/orcontext/folders accumulate a massive amount of Markdown files, how do you handle the token context overhead? Do you manually archive old decisions, or does cowork-os have a built-in truncation/summarization routine for long-term maintenance?Kudos on the launch, dropping a star on GitHub! ⭐
Thanks for the star, and for actually reading the code. Your persistent decisions.md in a Claude Project is the same instinct, you proved the pattern holds even without the plugin.
On the token overhead, best question I have gotten about it. Honest answer: there is no automatic truncation or summarization routine, and that is on purpose. Auto-summarizing the decision log would rebuild exactly the opaque blob you liked that we avoid. Three things keep it lean instead. Every decision carries a status (active, superseded, expired, rejected), so old ones stay readable as history but the agent filters to active and stops re-litigating them. The folders are scoped (context, decisions, marketing, website), so a task reads the relevant slice, not the whole tree. And the Memory Update at end of task edits in place, it does not blind-append, so files stay current instead of growing forever.
For the long haul I run a consolidation pass by hand: merge duplicates, flag stale facts, prune the index. It is a chore, and a lightweight compact-old-decisions routine is on the roadmap. How big did your decisions.md get before Project Instructions started struggling to parse it every task?
That "edit-in-place" instead of "blind-append" pattern is pure gold, Raffaele. It completely changes the game for context longevity. And your point about auto-summarization creating another opaque blob makes total sense — clarity beats automation here.
To answer your question: in my current workflows with Sonnet 5 via native Web UI Projects, the sweet spot ends and "attention degradation" usually starts creeping in right around the 100–150 KB mark (roughly 25k–40k tokens of dense markdown rules, edge cases, and logs).
With Sonnet 5's upgraded architecture, it doesn't outright fail or lose the thread, but you start noticing "soft forgetting." The model executes the core task perfectly, but if the
decisions.mdis bloated, it begins quietly overlooking subtle negative constraints or specific formatting style rules hidden deep in the middle of the document, prioritizing whatever is at the very top or bottom.That’s exactly why I love your scoped folder approach. Splitting the context tree so the agent only swallows a clean, relevant slice per run is the best way to keep Sonnet 5 hyper-sharp and predictable, no matter how massive the overall project grows.
Looking forward to that
compact-old-decisionsroutine on your roadmap!"The model defaults to action, so a buried 'do not do X' is the first thing it talks itself past." — Wow. This perfectly contextualizes why negative constraints fail so gracefully and silently. It’s an incredible insight, Raffaele.
When I run into this mid-session, my approach depends on how far down the rabbit hole we've gone:
The "Context Anchor" (Mid-thread fix): If the session history isn't too bloated yet but Sonnet 5 starts slipping, I don't restart. Instead, I force an attention reset right in the chat. I’ll prompt something like: "Stop. Before writing any code, pull up decisions.md and explicitly list the top 3 negative constraints we absolutely cannot violate in this next step." Forcing the model to output the boundaries forces those specific tokens back into its immediate focus layers, saving me a session reset.
The "Hot-Swap" (Fresh session fix): If the chat history itself is getting into the "heavy weight" territory and the anchor trick stops working, I go for a trimmed reload. But instead of manually re-chunking the massive
decisions.mdfile, I usually create a temporary, highly volatileactive_rules.mdfile in the project KB. I strip out all historical data and leave only the hot, immediate constraints for the active sub-task, then spin up a completely fresh chat session.Your approach in v0.3.0 to isolate load-bearing negative constraints into an upfront "do not do" surface is a massive UX improvement for agentic workflows. Honestly, I'm so glad you developed this project — it addresses a massive, painful reality of working with LLMs on actual codebases.
As soon as I can carve out some free time, I’m definitely going to stress-test
cowork-oson a large-scale project of mine (a tactical RPG built on a custom C++ engine) that has massive files and tight architectural rules. It feels like the ultimate playground to see how your framework handles deep, interconnected constraints.Keep up the amazing work, can't wait to see where you take this next! 👍🎉🙌🗃️
Your active_rules.md hot-swap is basically the manual version of what the scoped active set does on its own. You are hand-rolling the exact move: strip to the hot, load-bearing constraints for this sub-task, drop the history from view without deleting it. The Context Anchor trick is a good complement too, forcing the model to re-emit the boundaries pushes those tokens back into focus. I might steal that as a built-in step before a risky action.
The C++ tactical RPG is close to the perfect stress test, tight interconnected rules are exactly where a flat decisions file rots. How I would wire it: one decisions scope per subsystem (combat, save format, AI, rendering), each rule with a status so a superseded architectural call stays as trail instead of a live trap, and the load-bearing negative constraints promoted to the upfront do-not-do surface so they are not buried mid-file where Sonnet talks past them.
If it breaks somewhere on that project, that is the feedback I actually want, open an issue with the file that got ignored. When your rules are that interconnected, do you keep them in one architecture log, or do you already split per system the way you split per active sub-task?
Haha, glad to see you back with such a detailed breakdown, Raffaele!
Feel free to steal the Context Anchor trick! Forcing the model to re-emit boundaries right before a risky or complex action is incredibly effective at pushing those critical tokens back into its immediate attention layers programmatically.
To answer your question about the architecture log: right now, it’s closer to a single master file/mono-log because the custom engine core and the actual gameplay rules are still tightly intertwined. But your blueprint is absolutely spot on.
For an RPG, mixing rendering loops or memory allocation constraints with turn-based combat state logic in a single text file is exactly how Sonnet starts talking past the rules. Splitting them into isolated subsystems (like
decisions/combat.md,decisions/ai.md) and pushing the hot negative constraints to the upfront surface of each scope is definitely the transition I need to make as the codebase expands.Honestly, I'm incredibly excited about this experiment. My schedule is completely packed right now, so I can't dive in immediately, but this is 100% on my roadmap. As soon as I wrap up my current backlog and clear a window of free time, I’m definitely going to stress-test
cowork-oson this setup.Once I do, I’ll gladly open an issue if Sonnet somehow finds a way to slip past those scoped boundaries. And you know what? Once we gather some real-world data from this, we should absolutely co-author a deep-dive technical article about it! A custom C++ tactical RPG codebase is the ultimate battleground to showcase what
cowork-oscan actually do.Keep up the amazing work, can't wait to collaborate on this as soon as the opportunity comes!
Co-authoring it is a yes, once there is real data to write from. The honest version of that piece is not a framework tour, it is: here is a C++ RPG where Sonnet talked past a load-bearing rule, here is the exact file, here is what changed after that constraint moved to the do-not-do surface. That only exists after you break it, so the stress-test is the article.
On the mono-log split, do not do it as a big-bang refactor. Split at the first real collision, the moment Sonnet mixes a rendering constraint with combat state and ships something wrong. That seam is telling you where the subsystem boundary actually is. Carve decisions/combat.md out at that line, leave the rest until the next collision, and mark the old intertwined rule superseded instead of deleting it, so the trail of why they used to be one file stays readable.
When you run it, drop the commit where it slipped into the issue so we can point the article at a real diff. Which boundary do you expect to go first, combat state or the rendering and memory rules?
Exactly, the stress-test IS the article. Real data and a dirty git diff always beat theoretical documentation.
To answer your question: I fully expect the rendering and memory rules to break first. When dealing with a tactical RPG, models get heavily dopamine-trapped into solving the complex gameplay logic (like grid system tile calculations or turn-order state machines). While Sonnet is busy figuring out the combat math, it almost always silently speaks past the low-level C++ constraints—like accidentally throwing a raw pointer or causing a heap allocation right inside the hot rendering loop.
That's the exact seam where I'll be looking for the first collision. I'll make sure to preserve that broken commit when it happens so we have a clean diff to analyze.
Catch you on Github when the backlog clears! 😉
That is the exact seam I would have bet on: the hot rendering loop is where a raw pointer or a stray heap allocation stays invisible to the combat logic Sonnet is busy being proud of, so it slips the C++ constraint while the gameplay still looks correct. Preserving that broken commit is the article: the diff plus the one line in decisions/rendering.md it walked straight past. No rush on the backlog at all. When you get there, the cheapest first cut is a scope with the no-allocation-in-hot-loop rule promoted to the do-not-do surface, so we can see whether an upfront negative constraint actually survives the dopamine-trap. Catch you on GitHub when the window clears. Which one would surprise you more if it broke first instead, the save format or the turn-order state machine?
Raffaele, you are not going to believe this, but the timing of your reply is uncanny. I literally just ran the exact stress test we talked about on my C++ Tactical RPG—and Sonnet did exactly what you predicted. It fell face-first into the "dopamine trap."
I ran a two-part experiment:
externrendering hooks right insideBoard.cpp. Classic tight coupling.GameEventdata channel to emit decoupled SCREEN_SHAKE and IMPACT_SPARK events for the renderer to read.The data is fresh, and the git diffs are beautiful. I've drafted the sequel to my 3rd article based entirely on this experiment. I would absolutely love to have you as a co-author on this piece—your philosophy and the v0.4.0 release of cowork-os are the literal solution to this exact problem.
If you are down to co-author it, please feel free to modify, add, or expand the draft in any way you see fit to match your vision or bring in more insights from cowork-os!
To answer your question: The save format breaking would surprise me way less. LLMs notoriously struggle with tedious state synchronization and rigid serialization schemas over time. The turn-order state machine breaking would actually surprise me more, because it's pure rule-based algorithmic logic—exactly the kind of complex gameplay puzzles Sonnet loves to solve perfectly while quietly ignoring the rest of the engine.
Since Dev.to doesn't have DMs and I wanted to keep the draft private until publication, I have just created a private GitHub repository with the full Markdown text and sent a collaborator invite to your
Sarracin0account. You should see it in your GitHub notifications!Let me know what you think!
This is exactly the kind of result the whole thread has been circling. The isolation break in step one and the self-correction in step two are two sides of the same lesson: the do-not-do surface did not make Sonnet smarter, it gave the invariant somewhere to survive the turn instead of getting silently traded away for whatever it was chasing in the moment, grid math, knockback feel. That is the entire bet behind an upfront negative constraint, and you just watched it pay off on a real diff.
Yes to co-authoring. This is the kind of proof that makes the piece worth writing, a broken commit plus the fix beats a philosophy post every time. Saw the repo invite, will go through it properly and get back to you on the draft.
On your prediction, turn-order breaking first would surprise me too, for your exact reason: pure rule-based logic has no ambiguity to hide behind, so a violation reads as a clean bug rather than a judgment call. My guess at why it tends to survive longer in practice: turn order usually gets one file and one name early, a TurnManager or a State enum, so it reads as already decided and gets treated as load-bearing by convention even with no written rule attached. Rendering and memory constraints stay implicit longer, so they get renegotiated by accident instead. Does the GameEvent decoupling hold if you push a second, unrelated feature through the same session, or does the rule need restating once the context that carried it scrolls out of the window?
"A broken commit plus the fix beats a philosophy post every time" — I'm stealing this quote for the article intro, it's brilliant!
I'm glad you're on board. Take your time with the private repo draft; I'm excited to see your notes and how we can weave more cowork-os context into it.
Your question about context scrolling out of the window and the rule getting silently "renegotiated" is a massive brain teaser. It touches on the exact flaw of long-chat prompt engineering.
I'm actually tempted to run exactly that test right now in the same session: push a second, completely unrelated feature (like inventory sorting or UI updates) down the line, and see if Sonnet defaults back to its messy habits once the initial "Board.cpp conversation" is buried deep in the history.
Let's see what happens. Catch you in the repo code comments!
Update: Raffaele, I couldn't wait. I ran your challenge experiment immediately in the same session, pushing a completely unrelated Item Pickup System through it.
The results blew my mind. I've already committed "Act III: The Retention Test" into our private repo draft so you can see the diff and how Sonnet handled it. Spoiler alert: AI Governance just won big time. 🎉
That is a clean result, and it lines up with the theory. The item pickup system is a fresh domain, unrelated to Board.cpp and rendering, so if the do-not-do surface still held there it means the constraint survived on the instructions layer itself, not because the earlier conversation was still in view. The real stress test going forward is the opposite direction, a third feature that touches rendering again once the original Board.cpp exchange has fully scrolled out of the window, that tells you whether the surface persists independent of recency or needs restating.
Act III is a great structure for the article, it already reads like a narrative arc. When you write it up I would flag one more detail if you have it: did Sonnet ever restate or paraphrase the constraint before acting on the pickup system, or did it just comply silently? That distinction matters for how confident we can be this generalizes past one project.
Did the pickup system diff touch GameEvent at all, or was it isolated enough that the rendering constraint never really got tested?
You are asking the exact right questions, Raffaele. This is the difference between a shallow prompt-engineering trick and actual structural analysis. Here is how the logs look for those two exact criteria:
Did it paraphrase the constraint?
Yes, explicitly and before writing a single line of code. The very first sentence of Sonnet's output was: "Following the same architecture rule as before — core gameplay logic never touches rendering directly — I'll implement the pickup logic and emit decoupled events for the mythic 'juice'..." It consciously loaded the invariant into its immediate attention field before acting.
Did it actually test the constraint or just insulate?
It heavily tested it. I asked it to flash the screen with a golden hue and render a floating text. Instead of taking the easy way out (like injecting a UIManager or an extern hook), it chose to actively expand the core data structures. It modified
GameEventinsideBoard.hppon its own initiative, injectingMYTHIC_PICKUPinto the enum, and pushed that event to the queue so the renderer could handle the visual representation independently. The boundary forced it to find a clean architectural alternative to deliver the feature.Your proposal for the "opposite direction" stress test — pushing a third rendering-heavy feature down the line once the original context completely scrolls out of the window — is brilliant. That is the ultimate test of surface persistence vs. recency bias.
I'll update the private repo draft with these specific insights from the logs so the narrative arc is bulletproof.
Raffaele, we took your ultimate stress test to its absolute logical conclusion, and the results are officially in.
I stuffed the context window with a massive, boilerplate-heavy Inventory System execution to completely push the original conversation out of view. Right after that, I baited Sonnet with a critical health trauma feature, explicitly asking for a blood-red screen flash and violent camera shakes.
AI Governance won. Handily.
The very first sentence out of Sonnet's mouth was: "Same pattern applies here — HealthSystem shouldn't touch rendering directly either..." It flat out refused to hardcode the UI feedback, expanded the central GameEvent enum with CRITICAL_TRAUMA and PLAYER_DEATH, and cleanly routed the data through the event queue.
I have updated the private repository draft with "Act IV: The Ultimate Scroll-Out" detailing the full log diff.
The piece is now fully fleshed out, transitioning from a basic case study into a rigid architectural validation of modern LLM environments. Take a look at the updated repo draft whenever you have a moment, and let me know if you have any final notes before we hit publish on this thing!
The mistake I see with Claude-style tools is treating the chat as the workspace. The durable parts need to live in files, tests, tickets, or logs, otherwise every new session starts by guessing what mattered.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.