Over the last year, I've lost count of how many conversations I've seen about prompt engineering. Every week there seems to be another article explaining how a different prompt structure, a new framework, or a carefully chosen set of words can dramatically improve the quality of AI-generated responses. It's an interesting topic, and there is certainly some truth to it. A well-written prompt usually produces a better answer than a vague one. But after spending more time using AI in real DevOps workflows, I've come to believe that we're focusing on the wrong problem.
The biggest improvements I've seen, have not come from asking better questions. They've always come from giving AI a better understanding of the environment it is supposed to reason about.
That realization didn't happen overnight. Like many engineers, I initially assumed that if an AI produced an answer that wasn't particularly useful, the fault was probably mine. Maybe the prompt wasn't detailed enough. Maybe I needed to specify the expected format. Maybe I should explain the task differently. So I experimented. I rewrote prompts, added constraints, changed wording, and tried different approaches. The responses became slightly more polished, but they didn't become significantly more useful.
The turning point came when I stopped changing the prompt and started changing the information I provided.
Instead of simply asking an AI to review a Terraform change, I began explaining how the infrastructure was organized. Instead of asking it to summarize Kubernetes logs, I described the platform architecture, the deployment workflow, and the operational constraints surrounding the application. Instead of expecting it to infer how our CI/CD pipeline worked, I gave it the context that every new engineer on the team would normally receive during onboarding.
And the difference was remarkable and it permanently changed how I approach things.
The prompts themselves became almost boring. They weren't especially clever or sophisticated. In many cases they were only a sentence or two long. What changed was that the AI finally had enough background information to understand why the question mattered in the first place. Looking back, that shouldn't have been surprising.
If a new platform engineer joined the team tomorrow, I wouldn't judge their ability based on how well they answered questions during their first hour. Before expecting useful decisions, I would explain how our environments were structured, how deployments flowed through the pipeline, which services were considered critical, where the operational boundaries existed, and perhaps most importantly, why certain engineering decisions had been made in the past. None of that information would fit neatly into a ticket description, but it would heavily influence every decision that engineer made going forward.
AI is no different. We often expect it to reason about infrastructure that took years to evolve, while giving it only a handful of configuration files and a carefully worded prompt. When the answer falls short, our instinct is to keep refining the prompt instead of asking a more fundamental question: does the model actually understand the environment it's being asked to reason about? In DevOps, the answer is often no.
Infrastructure isn't defined only by Terraform modules, Kubernetes manifests, or CI/CD pipelines. It is defined by the relationships between those pieces. It includes architectural decisions that were made years ago, operational guardrails that exist because of previous incidents, naming conventions that everyone on the team follows without thinking, business priorities that influence deployment schedules, and countless small decisions that experienced engineers carry around almost instinctively. Most of that information never appears in a prompt, yet it shapes nearly every engineering decision we make.
This is why I've become increasingly interested in what I think of as context engineering. To me, it isn't about inventing another AI buzzword. It's simply the practice of making the surrounding system understandable before expecting useful answers. Good documentation becomes part of that context. Architecture diagrams become part of that context. Repository conventions, deployment workflows, incident learnings, infrastructure boundaries, and even the reasons behind previous technical decisions all become part of the environment the AI is trying to understand.
Ironically, this is exactly the kind of work that also benefits human engineers. Every team has experienced the challenge of onboarding someone into a mature platform. The difficulty isn't teaching Terraform syntax or explaining how Kubernetes works. Most experienced engineers already know those things. The difficult part is transferring all the knowledge that exists between the lines of the documentation. Why is production deployed differently from staging? Why do certain services follow different release windows? Why is one Terraform module intentionally duplicated instead of shared? Those answers usually live in conversations, design reviews, and operational experience rather than in source code.
Perhaps that's why context engineering feels much more aligned with infrastructure engineering than prompt engineering ever did. Infrastructure has always been about understanding systems rather than individual resources. AI simply exposes the same truth from a different angle. The better the system is documented and understood, the better both humans and machines are able to work within it.
I'm sure prompt engineering will continue to evolve, and it will undoubtedly remain an important skill for getting the best out of AI tools. But I suspect that, at least in DevOps, its importance is often overstated. Once the model truly understands the environment, the prompts themselves become surprisingly ordinary. They start looking less like carefully engineered instructions and more like the kind of questions one engineer naturally asks another.
And perhaps that's the biggest lesson I've taken away from using AI in infrastructure engineering. The quality of the answer depends less on how elegantly we ask the question and far more on whether we've given the system enough context to understand our world before expecting it to solve our problems.
Note: This article was written with the assistance of AI tools for structuring and drafting. The ideas, examples, and perspectives are based on real-world experience in DevOps and cloud engineering.
Originally published on Medium: https://medium.com/@yogesh.vk/why-better-prompts-wont-fix-ai-in-devops-235c0abe1814
Top comments (0)