DEV Community

Cover image for We Gave an AI Agent Production Credentials. Here’s What It Actually Did.
Daniiar Sher
Daniiar Sher

Posted on Originally published at ardor.cloud

We Gave an AI Agent Production Credentials. Here’s What It Actually Did.

We Gave an AI Agent Production Credentials. Here’s What It Actually Did.

Giving an AI agent production credentials sounds like the first paragraph of a postmortem.
We did it anyway.

GetBlock runs blockchain infrastructure across more than 1,000 servers and handles around 40 billion requests every month. Ardor was connected to its live environment through a VPN and given access to systems the team actually uses.

The obvious question is whether it broke anything.
In this case, no.

The more interesting part is that the first valuable thing it found had nothing to do with generating code.

It found $360,000 hiding in the server fleet

Someone from the GetBlock team asked Ardor a fairly routine infrastructure question.

The request was not:

Audit our entire server fleet and find hundreds of thousands of dollars in unnecessary spending.
It was much looser than that.

Ardor pulled telemetry from across the server network and compared actual RPC traffic with billing information. It found hardware that was still being paid for despite receiving little or no traffic.

The result was a decommissioning list representing roughly $360,000 in yearly costs.
Nobody had asked it to perform a cost audit. It just had enough access and context to notice that something did not add up.

That was the moment this became more interesting than another “AI writes code faster” story.
Also, Ardor did not immediately begin deleting servers. That would have been insane. Destructive actions still required someone from the team to approve them.

This wasn’t one lucky query

The server discovery is the headline number, but most of the work was less dramatic.
When several days of Mixpanel data went missing, Ardor backfilled the gap and deployed an hourly ETL job to stop it from happening again.

It connected Jira and Zendesk data so the team could classify support tickets and understand which products, protocols and locations were generating problems.

It built and deployed a complete internal application called SitemapHQ.

Someone from the content team, with no engineering background, also asked for a tool that could plan content across 11 channels while following GetBlock’s brand guidelines. Ardor built the frontend, backend, database and Kubernetes deployment.

None of these tasks make for a sexy “I built an app in 30 seconds” demo.
They are useful because they remove work that otherwise sits in a backlog waiting for somebody with the right access, context and technical knowledge.

The difficult part wasn’t the model

Giving an agent root access and hoping for the best would be stupid.

The useful part of this setup was fairly boring:

Ardor needed to know what it could read, what it could change, which actions had to stop for approval and how every execution would be logged.

That matters more than writing a clever system prompt.

An agent that can investigate a production problem is useful.

An agent that can investigate, explain what it found and propose a fix is more useful.
An agent that can silently delete infrastructure because it believes the machines are unnecessary is a future incident report.

The line between those three behaviours is mostly permissions, context and approval design.
This changed how I think about agentic software

I used to think the main value of agentic software was building applications faster.
That is still useful, but it is becoming the least interesting part.

The bigger opportunity is giving an agent enough visibility to notice the things your team does not have time to investigate.

That does not mean removing people from the process. It means deciding where the agent can work independently and exactly where a person needs to step in.

I’m curious where other developers would draw that line.
Would you let an agent:

  • Read production logs?
  • Restart a container?
  • Deploy a tested change?
  • Modify infrastructure?

- Delete a server it believes is unnecessary?

At which point would you require a human to approve the action?

I work at Ardor, so I’m obviously not a neutral observer here. The complete GetBlock case is available here if you want the uncompressed version:

https://dub.sh/01JO7nJ

Top comments (0)