Don’t Look Up: What Happens When the AI Race Becomes More Important Than the Warning?

Full Blog on : https://www.iamleopard.com/blog/don-t-look-up-what-happens-when-the-ai-race-becomes-more-important-than-the-warning

The most unsettling part of the current AI debate isn’t that someone
thinks AI could become dangerous.

It’s that some of the people building it are now asking whether we
should slow down — while governments and markets have powerful reasons
to keep accelerating.

There is a scene in Don’t Look Up that has stayed with me.

A scientist discovers that a comet is heading toward Earth.

The mathematics are straightforward.

The danger is enormous.

The solution is technically possible.

And yet, somehow, the hardest part isn’t understanding the comet.

It’s convincing everyone else that the comet matters.

The scientists try.

Politicians calculate.

Businesses see opportunities.

The media turns catastrophe into entertainment.

The public scrolls past it.

Eventually, the question stops being:

Is the comet dangerous?

It becomes:

What happens to everyone’s incentives if we admit that it is?

That is why the movie feels strangely relevant to the current AI debate.

Because in September 2026, the people building frontier AI are openly
debating whether the technology is moving faster than our ability to
evaluate and control it.

And the political response is pointing in the opposite direction:

We cannot afford to lose the AI race.

The AI industry has reached its “Don’t Look Up” moment

Anthropic CEO Dario Amodei recently published an essay arguing that the
development of frontier AI should be deliberately paced so that safety
and evaluation can keep up with capability development.

The important distinction is that this isn’t necessarily a call to stop
AI.

It is an argument about pace.

The basic idea is simple:

If capability improves faster than our ability to understand and
control the system, the gap itself becomes a risk.

Amodei’s proposal includes stronger evaluation, independent testing,
coordination among frontier AI companies, and eventually broader
international cooperation.

That would already be a significant position coming from an AI safety
researcher.

But it is more significant when the conversation is happening inside the
companies building the frontier.

Other technology leaders have also expressed support for slowing or
pacing parts of frontier development.

Then came the political counterargument.

“Whoever wins AI wins”

Donald Trump’s position is fundamentally different.

His argument is not primarily about whether AI risks exist.

It is about what happens if the United States voluntarily slows down
while another country continues accelerating.

From that perspective, AI is not merely a software technology.

It is strategic infrastructure.

It affects:

economic productivity

military capability

scientific research

industrial competitiveness

semiconductor demand

energy infrastructure

national security

geopolitical influence

And China is the obvious competitor in that calculation.

So the argument becomes:

If AI leadership matters this much, can the United States afford to
put itself at a competitive disadvantage by slowing down?

This is a much harder question than “Is AI safe?”

Because both sides can be right about different things.

The paradox of the AI race

Imagine two countries standing at a starting line.

Country A develops a more capable AI system.

Country B sees it.

Country B accelerates.

Country A responds.

Companies compete for researchers.

Investors pour money into infrastructure.

Governments provide incentives.

Data centers expand.

Models become more capable.

Agents become more autonomous.

And eventually, slowing down becomes politically difficult.

Why?

Because slowing down only works if the other players also slow down.

This creates a classic coordination problem.

If everyone agrees to move carefully, everyone might benefit.

But if you believe your competitor will continue accelerating, slowing
down can feel less like safety and more like surrender.

That is the geopolitical version of the AI race.

AI doesn’t need to be evil

This is where Don’t Look Up offers an important lesson.

The comet doesn’t hate humanity.

It doesn’t have political beliefs.

It doesn’t want to destroy civilization.

It simply follows physics.

The danger comes from the interaction between the object and the system
around it.

AI could present a similar systems problem.

We don’t necessarily need an evil machine for things to go badly.

We need only:

increasingly capable systems

poorly understood behavior

large-scale deployment

insufficient testing

competitive pressure

economic incentives

rushed decision-making

humans assuming everything will probably be fine

That combination can create serious risk without anyone explicitly
intending harm.

The part engineers should pay attention to

For developers, discussions about AI safety can sometimes sound
abstract.

“Alignment.”

“Frontier risk.”

“AI governance.”

“Existential risk.”

These phrases can feel like something that belongs in policy conferences
rather than GitHub repositories.

But the underlying problem is familiar to engineers.

We already know what happens when software is deployed faster than it
can be tested.

You get:

bugs.

You get:

security vulnerabilities.

You get:

unexpected interactions.

You get:

production incidents.

Now increase the complexity.

Give the system tools.

Give it access to APIs.

Give it memory.

Allow it to execute code.

Connect it to other agents.

Let it operate continuously.

Then tell the team:

“We’ll monitor it.”

That starts sounding much less reassuring.

Production AI is a systems problem

This is one of the biggest lessons from actually building AI-powered
products.

A production AI system isn’t just a model.

It is:

model + prompts + tools + data + permissions + APIs + users + business
logic + monitoring + failure modes.

Consider an AI agent that can read a CRM.

That’s one level of risk.

Now give it permission to modify customer records.

Different risk.

Now allow it to send emails.

Different risk again.

Now allow it to create accounts, spend money, modify production
infrastructure, and publish content without approval.

The model may be exactly the same.

But the system is radically different.

That’s why AI safety cannot be reduced to asking whether a model is
“smart” or “aligned.”

The surrounding architecture matters.

Least privilege applies to AI too

Security engineers have known this principle for decades:

Give a system only the permissions it needs.

AI agents shouldn’t be treated differently just because they communicate
through natural language.

If an agent only needs to read customer information, don’t give it write
access.

If it needs to send an email, don’t give it access to your entire
communication platform.

If an action can cause financial, legal, or irreversible consequences,
introduce an explicit confirmation layer.

The more powerful the action, the stronger the control should be.

This is not anti-AI.

It’s good engineering.

Humans still matter

There is a temptation to interpret “AI automation” as:

Remove humans from the loop.

That isn’t always the right goal.

For low-risk tasks, removing manual approval can be fantastic.

Generating a draft?

Automate it.

Summarizing documents?

Automate it.

Classifying routine data?

Automate it.

But for high-impact actions, the question should be:

What happens if the model is wrong?

If the answer is “nothing important,” automate aggressively.

If the answer is “we could lose money, expose private data, damage a
customer relationship, or take an irreversible action,” then human
oversight becomes much more valuable.

The right architecture isn’t “human everywhere.”

It is human oversight proportional to the consequences of failure.

Regulation isn’t a magic solution either

It’s tempting to turn this into a simple argument:

AI is dangerous, therefore regulate it.

But regulation can create problems of its own.

Poorly designed regulation can:

make compliance unaffordable for startups

favor incumbent companies

slow useful research

freeze outdated assumptions into law

create conflicting requirements across countries

push development into less transparent environments

And there is another difficult problem.

If one country slows down while another doesn’t, the competitive
pressure returns.

That’s why frontier AI governance is not just a national policy problem.

It is also a coordination problem.

Maybe “slow down” is the wrong phrase

When people hear:

“Slow down AI.”

they often hear:

“Stop innovation.”

But those aren’t necessarily the same thing.

A better interpretation might be:

Don’t let capability growth consistently outrun our ability to
evaluate the consequences.

Think about aviation.

We didn’t stop building faster aircraft because aircraft can crash.

Instead, society built systems around aviation:

testing

certification

inspections

redundancy

air-traffic control

pilot training

incident reporting

emergency procedures

We didn’t eliminate progress.

We built infrastructure around progress.

AI may need a similar approach.

The race may not really be USA vs China

This is perhaps the most interesting way to frame the whole debate.

We keep talking about:

USA vs China.

But another race is happening underneath it:

capability vs understanding.

Can our ability to build increasingly capable systems advance faster
than our ability to understand what those systems are doing?

That’s an engineering question.

Imagine building a massive application without reading most of the code.

Then giving it access to your production database.

Then giving it administrator privileges.

Then allowing it to modify its own behavior.

Then deploying it globally.

Then saying:

“We’ll monitor it.”

Any experienced engineer would immediately ask:

Where are the boundaries?

Where are the tests?

What happens when it fails?

Who can stop it?

Can we roll it back?

Can we explain why it did that?

Those questions become more important, not less, as AI becomes more
autonomous.

This is where the AI engineering mindset matters

At Leopard, we spend a lot of time thinking about AI not as a magical
chatbot, but as a component inside real software systems.

Our [AI & Machine Learning work] focuses on practical applications of
AI: structured context, retrieval, automation, agentic workflows, and
systems that connect models to actual business processes.

Our [Tryneth case study] is a useful example of that approach.

The important part isn’t simply that an AI model exists.

The important part is everything around it:

orchestration

APIs

data

usage tracking

authentication

permissions

business logic

monitoring

reliability

human interaction

That is where AI becomes software engineering.

And that is also where AI safety becomes practical.

What developers can do today

We don’t control national AI policy.

We don’t control frontier model roadmaps.

We don’t control geopolitical competition.

But developers do control the systems they build.

Here are a few principles worth adopting.

1. Treat model output as untrusted input

A model can be impressive and still be wrong.

Validate important outputs.

Use schemas.

Check assumptions.

Don’t let generated text automatically become trusted application state.

1. Give agents narrow permissions

Use least privilege.

Separate read and write capabilities.

Restrict tools by task.

Require explicit confirmation for dangerous actions.

1. Build observability from day one

Log:

model calls

tool calls

important decisions

failures

retries

latency

token usage

cost

user approvals

If you can’t reconstruct what happened, debugging an autonomous system
becomes extremely difficult.

1. Test the system, not just the model

A model benchmark can tell you something about the model.

It cannot tell you everything about your application.

Test the complete workflow.

Test adversarial inputs.

Test tool misuse.

Test permission boundaries.

Test failure recovery.

Test what happens when an external service is unavailable.

1. Design for graceful failure

A good AI system should have a safe state.

If the model fails, the application shouldn’t necessarily fail
catastrophically.

Fallbacks matter.

Timeouts matter.

Rate limits matter.

Human escalation matters.

Kill switches matter.

The irony of the current moment

For years, people warned:

AI might become too powerful.

Then AI became more powerful.

People said:

We need AI safety research.

Safety research expanded.

Then companies began building increasingly autonomous systems.

Now some people inside the frontier AI industry are saying:

Maybe we should pace development.

And the political counterargument is:

What if someone else gets there first?

That’s the paradox.

The more strategically valuable AI becomes, the harder it becomes to
slow down.

The harder it becomes to slow down, the more important safety becomes.

And the more important safety becomes, the more expensive it can feel to
prioritize it.

That’s the loop.

We shouldn’t look away

I don’t think AI is literally the comet from Don’t Look Up.

That analogy would be too simplistic.

AI can create enormous benefits.

It can accelerate scientific discovery.

It can improve software development.

It can automate repetitive work.

It can make sophisticated tools accessible to smaller teams.

It can help researchers and engineers solve problems that were
previously too expensive or time-consuming.

The answer isn’t to panic.

The answer is to engineer responsibly.

Testing.

Monitoring.

Red-teaming.

Independent evaluation.

Security.

Governance.

Transparency.

Human oversight.

International coordination.

And, where necessary, pacing.

The question isn’t whether we should build AI

We should.

The real question is:

Can we build increasingly powerful systems without becoming less
capable of controlling them?

That’s the conversation worth having.

Not:

“AI will destroy humanity.”

Not:

“AI will solve everything.”

But:

How do we make powerful AI useful, reliable, observable, secure, and
controllable?

That’s an engineering problem.

And unlike the comet in Don’t Look Up, we still have the opportunity
to do something about it.

So let’s not look away.

Let’s look up.

Related Leopard projects

If you want to explore the engineering side of this topic:

AI & Machine
Learning
— our approach to building practical AI-powered systems.

Tryneth — an example of
AI-agent orchestration inside a production SaaS environment.

Work & Case Studies — more examples
of software, AI, and product engineering.

Leopard — learn more about our work.

Discussion

What do you think?

Should frontier AI development be deliberately paced so that safety
and evaluation can catch up?

Or does slowing down create an unacceptable strategic disadvantage in
the global AI race?

I’d especially like to hear from developers building AI agents and
AI-powered products.

AI #ArtificialIntelligence #MachineLearning #AISafety

AIEngineering #SoftwareEngineering #AIAgents #ResponsibleAI

AIAlignment #AIRegulation #FutureOfAI #Technology