Scams are becoming more convincing. AI tools are gaining more access. A practical guide to trust, access and accountability for engineering leaders.
This is a short version of the full article on EngLead AI. Read the full story here: https://engleadai.substack.com/p/cybersecurity-in-the-ai-era
An assistant starts by helping your engineers find documentation. A few useful integrations later, it can search repositories, read incident notes and send replies to suppliers.
Each addition has a reasonable business case.
Who reviewed what those capabilities allow it to do together?
Consider a supplier request. An email describes a failing integration, names the right systems and asks for a diagnostic report. An engineer passes it to the assistant. The assistant finds relevant material and prepares a convincing reply.
The sender is an impersonator. The draft includes internal incident information that should never leave the company.
The engineer may be experienced. The assistant may have answered accurately. The failure sits in the workflow connecting a believable request, sensitive information and the ability to send it somewhere.
Join Me
Read more about EngLead AI, 🤖Newsletter, Twitter and LinkedIn for more Career, Leadership and Growth advice.
A familiar interaction can open the door
There is a documented example of recruitment being used as part of a major intrusion. In a December 2024 announcement, US and Japanese authorities described an attacker posing as a recruiter on LinkedIn. A Ginco employee received a supposed pre-employment test linked to malicious Python code and was subsequently compromised. Authorities linked the campaign to the theft of cryptocurrency worth $308 million from DMM Bitcoin in May 2024.
That announcement does not establish AI involvement. It shows how an ordinary professional interaction can become an entry point.
AI adds ways to make those interactions more convincing. The FBI advisory on generative AI fraud describes criminals using generated text, images, voices and video to support deception and reach more people.
A polished message, familiar voice or plausible professional profile therefore needs to be assessed in the context of what the person is asking you to do.
Connected assistants give a request greater reach
An AI model can propose an action. The surrounding application supplies the tools, credentials and permissions that make the action possible. Reading an incident report, preparing an email and sending it are separate capabilities, even when the interface makes them feel like one conversation.
OWASP describes excessive agency in terms of excessive functionality, permissions and autonomy. That is a useful lens for reviewing an assistant that keeps gaining integrations.
There are also different ways a harmful request can travel through the workflow. An impersonator may persuade a person to approve something. Malicious instructions in a document or webpage may redirect an assistant beyond the user's intended task. That second mechanism is prompt injection. The NCSC explains why application design must limit its consequences.
These problems can interact. They require different controls.
Three decisions deserve leadership attention
What establishes trust? Decide how consequential requests are verified, especially when they involve new recipients, access changes, software installation or sensitive information.
What authority does the workflow need? Make its ability to read, change and disclose information visible. Useful access should have a clear purpose and an accountable owner.
What happens when something looks wrong? Give the team a way to pause the workflow, understand what happened and recover. That capability needs to exist before the assistant becomes part of everyday operations.
These decisions also make adoption easier to explain. People can move faster when they understand which actions are supported, which need review and where to ask for help.
What the full article covers
The supplier assistant offers a practical starting point. It should help an engineer answer a legitimate question using appropriate information, with a clear decision before anything is sent externally.
The full article on EngLead AI is the implementation playbook for building that workflow. It covers:
- Defining the workflow before choosing controls, with a workflow register you can actually maintain
- Putting controls where trust becomes access: independent verification, separating user authority from app identity, retrieval controls, narrow tools, and approval specific enough to enforce
- Proving that the controls work, with 13 acceptance tests
- Making intervention part of the design: audit trail, detections, and a stop and recovery rehearsal
- Rolling out in stages over four weeks, and measuring the value
By the end, you will have the material for four useful deliverables: a workflow register, an enforceable permissions design, a release test suite and an operating runbook.
Read the full story here: https://engleadai.substack.com/p/cybersecurity-in-the-ai-era
Subscribe to EngLead AI for the complete implementation playbook and practical frameworks to bring into your next design review: https://engleadai.substack.com/subscribe
Writing The AI-Native Engineering Leader: on combining AI with existing technology to manage and operate teams at scale.
Subscribe at read.englead.ai for the weekly issue.
Join Me
Read more about EngLead AI, 🤖Newsletter, Twitter and LinkedIn for more Career, Leadership and Growth advice.
Top comments (8)
Cybersecurity is becoming an impossible game with AI scammers have great advantage compared to any company or ordinary peopel.
That asymmetry is the starting point of the article. An attacker only needs one convincing message to work. A company has to get verification, access and approval right every time, for every workflow. That is why I focus on where trust becomes access, not just on spotting a fake message.
Indeed. Just look at the situation with Binance where they got hacked and $6 billion of crypto were actually stolen. It's becoming almost impossible to stay ahead of the curve in the security space.
I understand that feeling, and the scale of crypto thefts is a good example of it. The DMM Bitcoin case in the article was $308 million, from a recruitment message and a malicious test. I have not verified the $6 billion Binance figure you mention, so I will not repeat it as fact. The practical point stands either way: you cannot stay ahead by trying to predict every attack. You stay ahead by building workflows where a single compromised message cannot, on its own, move money or sensitive information, and where you can stop dispatch quickly when something looks wrong.
Yeah but companies have the same tools as attackers, right?
Not really, security is extremely difficult. Ai is actually making the game even more uneaven.
Agreed, and that is the point JobClout is making. AI lowers the cost of making a convincing request, so the volume and quality of deceptive requests go up, while the defender still has to check each one. The answer in the article is not to try to out-detect every fake, but to make the workflow itself harder to misuse: verify through an independent route, separate user authority from app identity, and make approval specific enough to enforce.
In principle, yes, the same models and tools are available to both sides. In practice, companies have constraints attackers do not: approvals, audit trails, false positives that block real work, and the need to keep services running. Attackers can experiment freely and only need one success. That difference in constraints is what makes the game uneven.