Setting you up to fail

Melvyn Sopacua — Sat, 11 Jul 2026 11:59:04 +0000

I saw the following in a job description:

Design agents that utilise LLMs for reasoning, document extraction, and the end-to-end automation of complex enterprise processes. You will implement AI best practices to ensure our autonomous agents act with precision, security and industrial-grade reliability.

(emphasis mine)

This is setting you up to fail and is a classical XY problem waiting to happen.

If you don't know the XY problem, it is a problem with a perceived solution that is wrong for the original problem.

LLMs are a lot of things, but in the end they are only communicators. What's happening with LLMs now reminds me of the things people do with git, just because they can, not because git is very good at the job. Like a completely new p2p internet over git, CMS systems built around git (yes, it works, but CMS's are meant for people who like WYSIWYG editors) and fun tools like taking a picture each time you do a commit.

The perception with LLMs is that they really can brew your coffee. And LLMs are a lot of things, but they are not precise, secure nor reliable. And so, because the CTO (or more likely the Marketing Manager turned 10x Vibe Coder) has already said the solution is LLMs, you are now going to see the following problems:

Precision

Well, how about counting letters in words or asking for words with 5 letters or painting any picture with text in it. It's really easy to see that LLMs are not precise. If they were precise you wouldn't need instructor, because yes, following instructions to the letter is also about precision. I think anyone working with LLMs for more than a few minutes knows that precision isn't a trait that can be attributed to LLMs.

Secure

There's a ton of anecdotal evidence that LLMs leak data and many more found and reported responsibly.

But the most important evidence is mathematical: guardrails will never be sufficient. This makes it a cat and mouse game and it won't be long till consultant agencies will step up, since they are already skilled in the tax evasion game, the stay compliant game and the aggressive accounting game.

Reliable

Related to precision and following instructions, reliability is often achieved in software by being deterministic: same input yields the same output. This isn't true for LLMs, even at temperature 0, drift occurs. This isn't a bug, it's a feature of the architecture.

Contrary to popular belief, LLMs are ill-equipped for reliable automation of repetitive tasks. There's no telling when it will do something it shouldn't, what the consequences are, if it will do it again and how likely that will be.

When routing your emails and flagging something as urgent, you may find errors not that important. You can laugh about hallucinated fake studies in reports that nobody reads thoroughly anyway.

But when your LLM based invoice reader hallucinates the wrong invoice amount and it's agent processes the payment, you may suddenly have a cash flow problem or have insulted a key supplier with a fraction of the amount due, who then delays shipping.

In Conclusion

Be wary of jobs with these kinds of descriptions. They already bought into the LLM solution and so are unlikely to listen to solutions that actually work. Such as Named Entity Recognition, tailored to invoices you see daily and not all the invoices in whole wide world and also cat pictures.

Like Sieve email filters that access your database to route supplier emails to the procurement department.

Like not giving an "Agentic" LLM an API key to modify your entire AWS environment, but use Principle of Least Privilege. Perhaps reconsider giving an LLM access to your bank account, your cloud drive or sending emails as you. The classic usability versus security tension.

All this makes your job to deliver precise, secure and reliable agentic platforms an impossible task. You will think you succeeded till that 2am phone call. Install "Prime Agentic AI security harness" by Snake Oil Guardrails Ltd and a few weeks later sue them for not stopping to send sensitive documents, because the attacker said it was a test.

You have been warned.

Prompt Engineer shouldn't be a job title.

Melvyn Sopacua — Thu, 04 Sep 2025 19:50:34 +0000

Whenever I see a post about what a Chatbot did wrong, when asked to do something, there will be at least one "prompt engineer" blaming the user. Cause that's what it is: user blaming for the tool's inadequacy.

While companies heavily invested in AI will want you to believe that AI is coming for job, the very existence of the Prompt Engineer job title proves it won't be. It pains me to say the obvious, but if AI is a success, you won't need someone to "talk to it the right way". Then you can communicate with it like a normal human being and it would do as you ask or you could replace it with a better fit. Framed like that, you can instantly see how far away from the human equivalence requirement we really are.

It's also very unsettling to me, that PM's and UX designers everywhere have simply accepted the fact that they are needed. Let me say it again: it is user blaming. The thing you punish us developers for, when we say something is "obvious so why would the user do that". The prompt engineer's sole reason of existence is that the product (what they call "AI") is not performing according to expectations and so their job is to turn that around and rewrite the poor ignorant user's question to the almighty Chatbot.

If that's not harmful enough: the "prompt engineer" encroaches on the job of the Data Scientist who should be the one fixing the problem at the right end: the "AI". Of course, this makes it harder to sell AI as a service: it's incredibly expensive and slow to have per-user trained models. It is the technically better fix, but the harder sell and surely the harder to make profitable (profit comes from scaling the same thing to millions of users). This is the part less talked about: is the SaaS model stifling AI innovation? I think GPT-5's launch made that abundantly clear: it was void of any real innovation and just rewiring (orchestration if you will) of already tried, tested and failing methods.

Yes, there are things that can be automated and on the whole scale of things, this means less jobs. CEO's of not so well-known companies, start-up founders and the like are unfortunately listening to the industry gurus, so real jobs are disappearing (at least temporarily).

It's already clear that the promised gains aren't being met and there's also a new market emerging for seasoned developers to fix "AI enhanced" codebases.