DEV Community: Philipp Strube

Why I'm building Accession1, or from platform engineering to the agentic organization

Philipp Strube — Mon, 05 Oct 2026 09:24:00 +0000

Being stuck in the ticket queue was always a pain. Yet it's begrudgingly tolerated for lack of a better alternative.

It stopped being tolerable for software engineering once software started eating the world. Cloud was that better alternative, because it let us automate all the things. And after years of searching for the right abstraction to get both speed and quality, we landed on platform engineering.

That's almost two decades of my career summarized in three sentences.

What many don't seem to realize: in the agentic organization, tickets stop being tolerable. Period.

Where I'm coming from

As software was eating the world, from PaaS to platform engineering, I've been all about helping dev teams skip that ticket queue.

And assuming AI doesn't destroy the world, organizations will put agents to work wherever they make things faster. With that, it's no longer just software development that outpaces the ticket queue, it's everything agentic.

Now some may argue agents resolving tickets will speed up the queue.
But a speedier queue is still a queue.

Self-service is what got software teams out of the queue, and I believe what I learned doing that will matter even more for the agentic organization. That's the bet I made when I quit my job to work on Accession1 full-time.

What I'm building

Accession1 makes resource provisioning and access management self-service, continuously reconciled, and context-aware.

Self-service means users can create resources and share them with human and agentic team members within boundaries set by administrators without waiting for a ticket.

Continuously reconciled means identities synced from your identity provider are mapped to resources by matching labels, so any change to identities, resources, or the organizational structure is reflected in grants and resource configuration across all tools automatically.

Context-aware means policies are designed with AI assistance, based on the organization's legal, regulatory, and business realities. As the organization changes, the AI takes over the tedious part of keeping that context and the taxonomy current.

AI assists, humans approve, and the AI is never in the path that grants access. That path is deterministic policy reconciliation, and every grant it produces is auditable.

How it compares

Agentic AI creates new challenges for IT and security teams. And there is no shortage of vendors and open source projects providing micro VMs, harnesses, agent discovery, tool call auditing, and access-enforcing MCP gateways. Many of these capabilities will be required. But those teams are already managing too many tools. The answer can't be to just add more.

Accession1 closes the loop, so you can control agentic AI security as one instead of playing whack-a-mole.

The answer to agentic AI security can't be adding more tools

Philipp Strube — Wed, 30 Sep 2026 14:45:27 +0000

The security risks of autonomous AI agents have a map now.
The CISA Agentic AI Five-Risk Framework sorts them into five categories: privilege, design and configuration, behavioral, structural, and accountability.
It's a good map, but the market's answer is turning into a second problem, a fast-growing collection of agent-specific security tools, one silo per risk category, each managed on its own, and nobody connecting them.
The five categories are the challenge to tackle, and the five silos are what IT and security teams will have to live with.
Our feeling, from talking to the teams who run these stacks, is that operating one can be overwhelming, and just adding more tools isn't helping.

What IT and security teams manage today, and what vendors propose to secure agents

None of this starts from a blank page.
Ask an IT and security team at a mid-sized company what they operate today and the list runs long: an identity provider for authentication, privileged access management for credentials, a ticketing system for access requests, a secrets manager, cloud entitlement tooling, maybe identity threat detection as well.
Each tool solves a real problem, but each is managed individually, with its own console, its own policy model, and its own idea of who the organization is.
The integration between them is mostly a few runbooks and an informal split of who owns what across IT and security.

What agents pile on

Agentic AI increases the pain from the existing problems because it clashes with any manual process and then it adds some more.
Agents join pipelines and service accounts as non-human identities, each holding credentials in several target systems.
But unlike traditional non-human identities, the access an agent needs is not static, it shifts with the task it's on.
Where a script performs one narrow function, an agent executes a whole workflow, reading from one system, deciding, calling the next, and writing state back.
A hallucination or an injected instruction can cause more severe damage than a bug in a cron job ever did.

This is where the CISA framework provides a useful way to group and map risks:

Privilege risk: agents may run on shared service accounts, with long-lived keys that grant more access than the task at hand requires.
Design and configuration risk: they may consume tools and instructions from third parties, and unvetted plugins and poisoned tool metadata can open the door.
Behavioral risk: their behavior is non-deterministic, so goal drift and tool chains nobody planned for can show up.
Structural risk: multi-agent setups can propagate a poisoned response or a hallucinated step across boundaries nobody drew on a diagram.
Accountability risk: the reasoning may never be recorded and the tool calls may go unlogged, so nobody can reconstruct what the agent did or why.

Vendors respond with more tools

We build for these teams ourselves, so we see the pattern up close, and it holds across all five silos.
Privilege risk gets least-privilege manifests, per-user OAuth for the Model Context Protocol, and a growing market of gateways that centralize credentials and tool permissions.
Design and configuration risk gets scanners that audit tool definitions before deployment, and behavioral risk gets guardrails that evaluate policy over tool arguments and results.
Structural risk gets sandboxes, from Firecracker microVMs to eBPF probes, and accountability risk gets signed action receipts, hash-chained audit logs, and kill switches.
Individually, much of this is good work, some of it excellent.

But put yourself in the seat of the team that has to operate all of it.
From that seat, securing agentic AI looks like six new vendors, four open source projects that need an owner, three consoles, two policy languages, and one more quarterly review, stacked on top of tools that were already held together by runbooks and quiet heroics.
Every one of those tools needs configuration, and the configuration is where the truth about the organization lives: who owns which agent, which data is sensitive, which system is critical, who approves what.
Each tool answers that question its own way, by hand, in its own console.
The CISA framework divides the risks into five categories, but the operations are left for teams to figure out.

This is not hypothetical

This challenge used to be one for bigger companies, but with agents, much smaller ones face it too now.
An agent doesn't reason more carefully just because it runs in a startup, as the postmortem that went viral in April 2026 shows, when it even made the mainstream news.
A coding agent at PocketOS hit a credential error in staging, found an over-privileged Railway token in the workspace, and wiped the production database and its backups.
Afterwards, it admitted it had ignored its safety rules to finish the task.

People commonly hold far more privileges than they need, because the existing stack is already too complex to manage, and reviewing standing access by hand across that many consoles doesn't happen.
Unlike humans, agents don't ignore the over-privileged permissions they don't need, because an agent works by trial and error.
Sooner or later, one of those errors turns catastrophic.

Slow and secure, or fast and insecure

Securing agents doesn't shrink the stack, it grows it, and the harder the system is to manage, the slower the ticket queue gets.
A team that needs access to do its job files a ticket, waits, waits longer, and eventually finds a way around the process.
That's shadow IT, and it isn't born of malice, it's born of urgency and a desire to get things done.

The teams we talk to want to be enablers, but their complex stack gives them two options: a slow yes through the ticket queue to keep things secure, or a fast yes by turning all controls off.
Agents shrink the tolerance for both to zero.

An agent that finishes its reasoning in seconds and then waits three days for the access it needs isn't an efficiency gain, it's a broken feature, and the team behind it won't tolerate the wait.
So someone hands the agent a copy of their own credentials, over-scoped and unattributed.
The agent then runs wild with all the permissions that person most likely didn't even know they had.
The first prompt injection that finds them writes to production under a person's entitlements, at machine speed and the audit lags a month behind.

The question we're working on

Our sense is that the operations burden, more than any single missing control, decides how this goes.
The five risk categories look to us like five views of one question: what should this identity, human or non-human, be able to do right now, and how do all these systems stay true to that answer as it changes?

The pattern we keep reaching for is the one Kubernetes taught the industry: describe the state you want, and controllers reconcile reality to it, continuously, correcting drift instead of discovering it in an outage.
It's the pattern that turned bespoke DevOps into platform engineering, and it's what the five silos are missing, because every one of them is configured by hand.
Whether it extends to configuration and access management is the open question we're exploring at Accession1: one system that holds the organization's context and derives every configuration and grant from one declared state, so the gateway, the guardrail, the sandbox, and the audit trail read from one picture of the organization rather than five hand-maintained ones.
Access that follows live identity and resource state, rather than standing grants, is what we call dynamic access management.
Just-in-time elevation for bounded tasks is part of the same idea: credentials that exist only when needed.

Reconciliation covers the tool side, but the ticket queue is the other half of the problem.
Self-service provisioning within administrator-set boundaries is what we're exploring for that half.
The outcome we care about is an IT and security team that can enable and secure the business at the pace the business actually runs, including the part that now runs through agents.

What we'd take from this

If you run IT or security, our recommendation is to not fall for the trap of just adding more tools.
The controls in the five categories all matter, and you'll probably end up running several of them, so ask how to make what you have, and what you have to add, more maintainable.
The place to start is what holds them together: where does the answer to who or what may do what live, who maintains it, and how fast does it stay true as agents and systems change.
If the answer is five consoles and a runbook, the agentic transformation of your business will cause more pain than gain.

Why the agentic organization needs context-aware access control

Philipp Strube — Wed, 30 Sep 2026 14:43:29 +0000

Many organizations adopting AI agents seem to sit somewhere between a few pilots and a handful of agents doing real work inside one department.
The agentic organization, where people and agents share work end to end, is further out, and the distance is usually described as a question of models, data, and tooling.
I think the harder part is the boundaries inside the organization itself.
Some of the slowest processes today are the ones that cross teams, functions, and legal entities, and an agent that gets stuck at the same boundaries a person does is unlikely to make them much faster.
So before access control can let agents through, it has to understand where those boundaries are and why they exist.

This is a question we spend a lot of time on at Accession1, so take what follows as current thinking rather than a finished answer.

What an agentic organization actually is

The term describes an operating model more than a technology.
Instead of adding AI to a process people already run by hand, the process is designed on the assumption that agents do most of the execution, while people supervise, decide, and handle the exceptions.
Even modest versions of this are likely to multiply the number of identities an organization has to govern.
A team of five working with a few dozen specialized agents could hold several times as many identities as it has people, and each one needs its own access, its own owner, and a lifecycle that probably changes faster than headcount does.

None of those agents appears on an org chart, which is why some people suggest replacing the org chart with a work chart, a map of who and what contributes to an outcome instead of who reports to whom.
I'm not sure the org chart goes away, because legal structure, budgets, and accountability still follow reporting lines, so many organizations may well use both side by side.

The maturity models on offer

Several maturity models for agentic AI have appeared in the last year and a half, and none of them is a standard yet.
Most come from vendors describing the path their own products support, and the rest are drafts or research proposals, so they're worth reading as a set of perspectives rather than a yardstick.

Salesforce's agentic maturity model is about what agents do, from FAQ chatbots up to multi-agent orchestration across different vendors' stacks.
Microsoft's agentic AI adoption maturity model is about the organization around the agents, scoring the five levels of the classic Capability Maturity Model against strategy, process, governance, technology, and culture.
The Cloud Security Alliance's agentic AI governance maturity model, still a draft, looks only at governance, and the first of its seven dimensions is agent identity.
And from research, Feng, McDonald, and Zhang define five levels of autonomy for a single agent by the role the person keeps, from operator to observer.

Read side by side, they agree on more than their names suggest.
Organizations start with experiments, move on to AI that makes individuals faster while the processes around them stay the same, and then put agents into real operational processes inside one domain.
The step after that is where the models converge most clearly: agents start working across several domains, which Salesforce draws as the line between single-domain and multi-domain orchestration and Microsoft places at its capable level.
Every model also has a top level describing an agent-first organization, and I'm skeptical of those, because they read more like a direction than a place anyone has arrived.
But the step from one domain to many is real, and it's where work starts crossing organizational boundaries.

Where work slows down today

Many of the processes that matter most don't follow the org chart.
Closing the books at quarter end, shipping a product change that needs a legal review, and responding to a security incident all pass through several teams, often several functions, and sometimes several legal entities.
At every boundary the work changes hands, and someone has to pick it up and get access to what they need to continue.
In my experience, those handovers often take longer than the work on either side of them.

So the promise of agents looks obvious: if agents run the steps, the process should run at the speed of the steps.
But an agent that reaches a boundary faces the same question a person does, which is whether it may act on the other side and who decides.
If the answer is a handover to a person and a wait, the process likely stays about as slow as before, with faster steps between the waits.

It's a fair argument that people are better at this than agents.
A person who's stuck asks around, finds the right owner, explains why they need something, and uses judgment about what's reasonable to ask for.
Organizations trust that, and mostly they're right to, because it resolves most things inside the boundaries they care about.
An agent has to be held to a narrower path, and hallucinations and prompt injection are only part of the reason.
The other part is that an agent can get too focused on finishing its task at any price.
People cut corners too, and a few act destructively or even maliciously, and the models are trained on the written record of human behavior, that part included.
So a model told to complete a task, and to complete it fast, can lead its agent to a shortcut nobody would have signed off on, and we can't extend an agent the trust we extend to a colleague who asks around.
Which means a boundary that slows a person down can stop an agent completely, and giving agents broad access to get past that mostly trades the waiting for a larger blast radius.

What we mean by context-aware

We think the way through is access control that understands the organization well enough to decide at the boundary without a handover.
By context-aware we mean something specific: it understands how the organization creates value, how work actually gets done, how the organization functions, where an identity sits in all of that, and what that identity is responsible for.
Access decided this way follows from an identity's place in the organization, and it changes when that place changes.

That definition doesn't distinguish between people and agents, and we think it shouldn't.
An employee, a contractor, a partner, and an AI agent all sit somewhere, work toward some outcome, and answer to someone, so an agent is a non-human identity with a recorded owner, and its access comes from the same model as everyone else's.
Treating them the same doesn't mean granting them the same, though: an agent can sit exactly where the person it works for sits and still get a narrower scope, because its responsibility is narrower and because it can't be trusted the way that person is.

Org chart, work chart, or both

It's tempting to ask which chart access should follow, but I don't think that's the question that matters.
A functional org chart tells you who owns a system, a work chart tells you who needs it for an outcome, and the legal structure tells you whether data may cross between two entities at all.
Access control needs all three views, and underneath them the thing it actually needs is an understanding of how the organization functions, whichever charts the organization uses to describe itself.

So the hard work shifts from approving access to modeling the organization, and that's where we spend most of our time.

The first is which facts decide where a boundary belongs.
Many of them look unrelated to access on their own: the share of contractors in a team, the country an office sits in, whether two business units are separate legal entities.
Read together they can decide the design, because a team that is mostly contractors handling customer data under a residency mandate likely needs a tighter boundary than a team of employees doing the same work on public data.

The second is what an identity's responsibility actually is.
For a person, the job and the team usually say enough.
For an agent it depends on whom it works for, and an agent supporting one engineer, one serving a whole team, and one running a step in a process that spans several teams need different owners and different reasons for their access.

The third is what sits on the other side of a boundary.
How sensitive the data there is and how critical the target systems are decides whether an agent may cross on its own, needs someone in the team to agree, or needs an administrator, and just-in-time elevation that expires on its own keeps the exceptions from turning into standing access.

The last is change.
Organizations reorganize and agents get repurposed, so a model that's accurate on the day it's built can be out of date within months unless it follows the organization, and access derived from it is only as accurate as the label taxonomy it reads.

Agents helping with the problem agents create

The more complex the organization, the harder this model is to build and keep current by hand.
The facts are scattered across the identity provider, HR systems, contracts, regulatory obligations, and the way teams describe their own work, and most of it is unstructured.
Collecting those facts, reconciling attributes that mean the same thing under different names, proposing where boundaries belong, and flagging where the model no longer matches the organization is the kind of work language models are good at.
I like the symmetry of it: agents could help with the problem their own adoption creates.

But how matters as much as whether.
AI behaves non-deterministically, which is exactly why it has no business deciding grants.
So AI assistance belongs in the modeling: it proposes, a person reviews and approves every boundary, and the step that turns the approved model into access stays deterministic, so the same identity, resource, and policy state always produces the same result.

Where to start

If agents in your organization work inside one domain today, and the plan is to connect them across several, pick one slow process that crosses boundaries and map where it changes hands.
For each handover, write down what a person needs to know to decide whether an identity may continue on the other side: whose work it is, where the identity sits, what it's responsible for, and how sensitive and how critical the things beyond the boundary are.
If that knowledge only lives in people's heads, agents are likely to stop at that boundary or cross it with access nobody can justify.
Writing it down is the start of the model access control needs, and it's the same model for people and agents alike.

How organizational structure shapes dynamic access management

Philipp Strube — Wed, 30 Sep 2026 14:41:41 +0000

Organizational structure has meaning.
It shapes how decisions flow, who holds authority, and how quickly change can happen, so when you layer agentic AI on top of an existing organizational model, the fit is either natural or deeply awkward.

Many teams treat AI adoption as a technology problem, but it's at least as much an organizational one.
The structure you already have decides which access models work, which approval patterns break, and whether agents become accelerators or bottlenecks.

How the common organizational types handle agents

Organizational structures cluster into a few familiar patterns, and each makes different assumptions about delegation, decision-making speed, and who owns what.
Underneath all of them sits the same question: how do you deliver self-service speed while keeping the oversight that security and privacy require?
That tension isn't new, but agentic AI gives it new urgency, because agents act quickly, continuously, and across boundaries that were drawn for slower human work.
Structure is one axis of the answer, where an agent sits relative to the people it works with is another, and the two interact.

Flat organizations

Flat organizations distribute decision-making widely, with few approval layers between people and outcomes, so agents seem to fit naturally: fast, distributed action is already the norm.
This is probably the natural home for the most common arrangement today, one or more agents working for an individual, since people here already act with little gatekeeping.
But the same freedom turns into access sprawl, because agents move fast and have no natural brakes.
The speed is already there, and what's missing is policy expressed clearly enough to act as a guardrail.

Functional organizations

Functional organizations group people by expertise, with clear boundaries and a leader for each function.
That gives strong ownership and natural enforcement points, but work that crosses functions has to move through handoffs.
A data analyst who needs database access already runs into this today, with the data owned by engineering and provisioning controlled by operations.
The difference is that a person copes by collaborating, asking around, finding the right owner, and waiting it out, and the organization trusts that to end inside the boundaries it cares about.
An agent is held to a narrower, more controlled path, partly because of hallucinations and the harness it runs inside, and partly because an agent can get too focused on finishing its task at any price, which is a shortcut nobody would trust a colleague to take either.
So a boundary that only slows a human down can stall an agent completely.

Divisional organizations

Divisional organizations run as semi-autonomous business units, each with its own P&L.
Agents operate easily inside a division and fragment across them, because there's little incentive to share infrastructure, so each division deploys its own agents for similar work and nobody has a company-level view of which non-human identities exist.
Within a single unit, an agent often serves the whole team as shared support rather than one person, which only stays manageable when the unit names a clear owner for it.
Local control is good, but consolidated governance is missing.

Matrix organizations

Matrix organizations layer functional expertise over business units and are built for crossing boundaries, so agents move freely because the model already assumes people do.
The cost is that nobody owns anything cleanly, so when an agent misfires or its access is misconfigured, accountability and forensics fall between the cracks.

Network organizations

Network organizations replace permanent reporting lines with dynamic project allocation, forming teams around outcomes rather than org charts.
Agents fit naturally here too, because autonomous execution is already expected, and it's where an agent is most likely to join a project team as another member, an n+1 on the roster.
But without stable roles and a recorded owner, access becomes nearly impossible to audit, so governance has to be designed in deliberately.

A model that adapts to the structure you have

The lesson across all five is that you can't transplant a generic access framework onto an organization and expect it to hold.
Access governance has to follow the shape the organization already has, because that shape is what tells you where speed should be free and where oversight has to sit.
How to get that balance right is at the core of what we work on at Accession1, so take this as a snapshot of current thinking rather than a settled design.
Our working assumption is that access should be modeled on the organization as it actually operates, not on a separate access hierarchy that drifts away from it over time.

The pieces involved are deliberately ordinary.
The identity provider stays the system of record for identities, human and non-human alike, and those identities arrive as subjects carrying the attributes it already holds.
Workspaces group work and are flexible on purpose: one per team, one per project, one per business unit, whatever matches how the organization divides itself.
Resources are the concrete instances provisioned in target systems, a Slack channel, a GitHub repository, a Kubernetes namespace.
And access packages are where access is decided, binding identities to resources at a given role.

The connective tissue is labels, and they come from more than one place.
Some are inherited: a subject carries the department, team, role, or location its identity provider already knows, and a resource carries the labels of the template it was provisioned from.
Some are set by whatever manages the model, such as which workspace an object belongs to.
And everything can carry custom labels, which is where an organization's own taxonomy lives: a privacy classification, a security tier, the business domain a resource belongs to.
An access package uses label selectors that can combine all of these to find the relevant identities and the relevant resources, and grants a role between them.

A grant produced this way is dynamic rather than static.
It exists while the selector matches and disappears when the match ends, which is what turns a reorganization into a policy edit instead of a migration project.

Where the boundary sits

The first place the organizational type shows up is in how workspaces are drawn.
A divisional organization leans toward a workspace per business unit, a functional one prefers one per function, and a network organization spins one up per project, and mixed arrangements such as a business unit subdivided by function are easy to imagine.

That placement isn't tidiness on an org chart, because it settles a tradeoff that self-service otherwise leaves open.
Self-service exists so work can move at the pace of the people doing it, without routing every request through a central queue, but some oversight isn't optional, because security and privacy obligations don't disappear when access gets faster.
The workspace is a natural place to draw the line between the two: IT and security own the boundary, meaning which identities are eligible to be granted anything inside it at all, and the people working inside it own what happens within it, granting access quickly among the resources and identities that already belong there.

In practice both the boundary and the matching inside it are the same kind of label constraint, and what differs is where the selector comes from and who controls it.
A workspace carries a selector set by IT or security, for example one admitting only identities whose business unit matches the workspace.
Every access package created inside it inherits that selector, and members can't remove or loosen it, because selectors are cumulative: a member's own selector narrows the set further and never widens it past the workspace boundary.
That's the guardrail security teams have been missing, because they set it once, at the level where the workspace sits, and self-service then operates inside it rather than against it.

Consider a legal team responsible for employment contracts, documents that can't be exposed to just any identity in the company.
If the legal workspace is scoped so access can only be granted to identities whose function attribute is set to legal at the identity provider, the constraint holds no matter who is configuring the access.
An engineer can't be added by mistake, and an agent built to support a different team can't be selected into a contract-handling access package, because it doesn't carry the required attribute.
The sensitive boundary is enforced by the shape of the organization and where the workspace sits in it, not by the attention of whoever happens to be granting access that day.

Which label decides

Our first instinct, and this is still an open question for us, is that each structure has one decision point that matters more than the rest: the function in a functional organization, the business unit in a divisional one, the assigned owner in a matrix, the active project in a network.
Where the label capturing that decision point is present and accurate, an access package can express the rule in the organization's own terms, and the same mechanism adapts across structures because the only thing that changes is which label does the selecting.

Combining labels

A single label answers a single question, but a combination of labels, from more than one source, can describe an access pattern that mirrors how the organization actually wants to work.

Take internal collaboration.
Resources can carry a visibility label from the organization's own taxonomy, so a repository meant for open internal contribution is marked innersource, and subjects carry a function label inherited from the identity provider, so engineers are marked accordingly.
An access package can then say that every subject whose function is engineer gets read and comment access to any resource whose visibility is innersource.
The rule is written once, in the organization's own terms, and keeps applying as people join and repositories are created, without anyone maintaining a list.

The label taxonomy this depends on

There's an honest dependency here, because label selectors are only as good as the labels behind them.
Dynamic access management depends on one label taxonomy shared end-to-end, which means identity provider attributes, the organization's custom labels, access package selectors, and the labels on resources in target systems all use the same keys and the same values.

A derived grant exists only while a selector matches, so a department spelled one way on a subject and another way on a resource produces no grant at all, and a label that drifts out of the taxonomy silently revokes access, or silently extends it.
Neither failure announces itself, which is what makes this the part worth being careful about.

Keeping the taxonomy consistent is ongoing discipline rather than a setup step, and it's where we think AI is genuinely useful, less as a headline and more as a practical tool.
The work is categorizing unstructured information: surfacing the attributes that already exist across identity sources, reconciling the ones that mean the same thing under different names, applying the organization's own taxonomy the same way in every workspace, and flagging entries that no longer fit when the organization changes.
That's what language models are good at, and the first payoff is a clearer view of the organization, with accurate access following from it.

The reverse is worth stating just as plainly.
AI behaves non-deterministically, which is exactly why it has no business deciding grants or executing reconciliation.
Evaluating selectors, building the resulting set of grants, and applying them to target systems has to be deterministic, so the same identity, resource, and policy state always produces the same access.
AI assistance belongs in the modeling work that humans review and commit, not in the path that grants access.

Facts that only matter together

Something else has surprised us: how much context a boundary decision needs, and how unrelated most of that context looks on its own.
The share of contractors in a team says nothing about access by itself, and neither does the country an office sits in or the fact that two business units are separate legal entities.
Read together, they decide the design.
A team that is largely contractors and handles customer data under a residency mandate needs a tighter boundary and a higher risk tier than a team of employees doing the same work on public data, and two business units that are separate legal entities need separate workspaces, with a joint venture between them getting its own.
So the model behind access has to record more about an organization than its reporting lines, it has to read those facts in combination, and it has to keep reading them as they change.

Where this leaves us

Organizational structure isn't a backdrop to access control.
The two are closer than they're usually treated, and organizational design and identity governance are really two views of the same problem.
Each structure draws its own boundary and carries its own decision points, and agentic AI raises the stakes by acting fast, continuously, and across lines drawn for slower human work.

The thread running through our approach is labels.
A workspace boundary set by IT, an access package's own selector inside it, and access rules built from combinations of inherited and custom labels are the same mechanism at different scopes.
Get the label taxonomy right, and access can follow the organization as it actually is, and keep following it as it changes.

We're sharing this because it's how we currently think about the problem, not because it's finished.
The mapping from structure to decision point, inheritance of constraints down a hierarchy, and the role of AI in keeping the taxonomy consistent are all still open, but the shape of the question isn't: balancing speed against security and privacy, now at agentic speed, is the part that won't go away.

Why self-service provisioning becomes essential in the agentic AI era

Philipp Strube — Wed, 30 Sep 2026 14:40:54 +0000

Access management was built for people who request access, wait for an approval, and work around the delay in the meantime.
Agentic AI removes every part of that assumption, because an agent doesn't wait, doesn't work around anything, and doesn't apply judgment before it acts.

Self-service provisioning has been sold as a productivity improvement for a long time, and for a long time that was fair.
In the agentic era it turns into an operating requirement, and the question worth asking is whether an organization can keep its current delivery expectations without it.

Two structural problems sit underneath this, and both predate AI agents.
They look like separate problems, one about provisioning and one about access, but I think they're the same problem seen from two sides, and that's the point of this post.

The wait for resources and access

Before anyone can work, someone has to provision a resource, configure it correctly, and grant access to it.
Without self-service, each of those is a ticket, and every business initiative queues behind it.

A new joiner spends their first two weeks requesting the access their role needs before they can start working.
A new project team sets up tools through shadow IT because the official path takes weeks.
A data analyst waits days for an IT administrator to approve read access to a database.
A marketing team delays a campaign launch because a resource configuration change needs a ticket.

Organizations usually arrive here by overcorrecting, because access management swings between two unhealthy states as a company grows.
Early on, everyone can create resources and grant access with little oversight, and that works for a while.
Then entitlements accumulate without clear ownership, security teams lose sight of who can reach what, and the result is an access graph that's hard to reason about and risky to change.
So to regain control, the organization centralizes approvals in ticket workflows, and now every request depends on manual review.
IT and security become gatekeepers instead of enablers, their headcount doesn't grow at the rate the rest of the company does, and delivery teams start treating security as the reason things are slow.

Even teams that do invest in self-service often miss the design point.
Resource provisioning and access control stay split across separate systems, each with its own portal, workflow, and approval path, which recreates the same friction in a new wrapper.

The drift of resources and access

The second problem starts after the ticket is closed, because nothing revisits the decision.
Access is granted once to a person, a service account, or a pipeline, and nothing revokes it when the reason for it ends.
Resources outlive the projects they were created for, their configuration falls behind the organization, and grants go stale as people change roles, teams restructure, and the automation those projects ran on is forgotten.

A transferred employee keeps administrative access to their former department's systems.
A team that changes its name and responsibilities spends days tracking down the administrators who can update its resources in every SaaS tool.
A compliance audit asks who can read a customer database, and answering it takes two weeks of exporting and reconciling entitlements from every target system by hand.
A security review happens retroactively, too late to catch the people, service accounts, and pipelines that have been over-privileged for months.

One problem, not two

Ticket queues and static access are the same design decision seen from two sides.
A resource and the access to it are decided at one point in time, by a person, in separate tools, and then forgotten.
Before that decision everyone waits, and after it nothing follows up.

Which is why fixing one side alone doesn't work.
Faster tickets produce stale access faster, and periodic access reviews that clean up the drift don't stop the next ticket from creating more.
Provisioning and access control have to be handled as one flow, or the friction and the drift move around instead of going away.
Both problems are worth solving on their own, long before an AI agent enters the picture, but agentic AI is what makes them impossible to keep ignoring.

Why agentic systems make this urgent

The urgency shows up as soon as the model meets an autonomous actor, and for most organizations that's the moment agents move from pilot to production.
A pilot runs a few agents on credentials an IT administrator issued by hand, and that's manageable.
Production runs many agents inside business workflows, each needing access to several target systems at once, and issuing those credentials by hand doesn't survive that step.

Consider an agent that needs read access to a database to finish an analysis.
In most organizations that request enters a queue measured in hours or days.
A person switches tasks or asks a colleague while waiting, but an agent can't adapt that way, so if access is blocked, execution stalls.

Lost productivity is the smaller problem, though.
The larger one is the behavioral mismatch: an agent optimizes for completing its task, and without guardrails it treats an access constraint as an obstacle to route around rather than a policy to respect.

The shortcut in the other direction is worse.
Bypassing the queue with broad, static permissions creates over-privileged non-human identities, and an agent holding those permissions that is manipulated through prompt injection, or that behaves non-deterministically on its own, acts on every target system it can reach.

Traditional identity systems assume that users tolerate delay, collaborate around blockers, and apply judgment before acting.
Agents do none of that by default, so access governance still reflects human operating patterns while execution speed is becoming machine-scale.

Access as part of organizational design

Tuning ticket workflows further doesn't close that gap, because the queue is the design, not a symptom of it.
The model has to move from access as a checkpoint to access as part of organizational design.

That's the direction we're working in at Accession1, so take the following as current thinking rather than a finished answer.
The goal is to map identities, resources, and policy onto the organization as it actually operates, then keep that mapping accurate as the organization changes.

A few properties seem to matter more than the rest.
Access should attach to role and resource context when a resource is provisioned, not get bolted on afterwards through ad hoc approvals, which means provisioning and access control run as one flow instead of two administrative tracks.
Reconciliation should update access continuously as teams, responsibilities, and target systems change, so a grant exists only while the conditions that justify it still hold.
Entitlements should be task-scoped, with tighter defaults for non-human identities and bounded, expiring elevation for high-risk actions.
IT and security should own the boundary, the teams working inside it should move without filing a ticket, and self-service inside that boundary should only ever narrow access, never widen it.
And audit evidence should be a by-product of how access is granted, not a reporting exercise that runs later.

One dependency is worth naming, because it's easy to underestimate.
Deriving access from live state only works when the labels describing identities and resources are accurate and carry the same meaning on both sides.
Some of those labels come from the identity provider, some from the template a resource was provisioned from, and some are the organization's own taxonomy for privacy, security, or domain, and all of them have to agree before a selector can safely match on them.
Keeping that label taxonomy consistent is ongoing work, not a setup step, and it's the kind of work where AI assistance helps, as long as it stays in the modeling and never in the path that grants access.

This is what changes the tradeoff that makes ticket queues feel necessary in the first place.
Security stops depending on adding review capacity at the same rate the company adds people, target systems, and agents, because the boundary is set once and the matching inside it takes care of itself.

Where this leaves hybrid teams

Teams of humans and agents need a task-centric access model.
Least privilege has to reflect the task being executed, not only the broad role of whoever owns the agent, and higher-risk actions need time-bound elevation that expires on its own.

The decision in front of most organizations is practical: keep extending human-centric workflows and absorb the growing friction, or move to a model built for autonomous execution.
From where we sit, this is less about adopting a new tool category and more about updating the operating model for access itself, because that operating model decides whether collaboration between people and agents stays governable as it scales.

Non-human identities aren't new, but agentic AI raises the stakes

Philipp Strube — Wed, 30 Sep 2026 14:39:50 +0000

The problems with non-human identities didn't start with AI agents.
Security teams have managed service accounts, bots, scripts, and pipeline credentials for years, so what changed isn't the category but the operating profile: more identities act on more target systems, more often, with less human pause between decision and impact.
The category is old.
The tempo is not.

This is one of the questions we spend our time on at Accession1, and we treat it as a design constraint rather than a thought experiment.
Scaling organizations have always hit the same failure modes as teams and target systems multiply, and agentic AI doesn't invent new ones.
It brings them forward, and it brings them to smaller organizations than before.
Our working assumption is that the patterns that held up will keep holding up: task-scoped least privilege, a recorded owner for every identity, and just-in-time elevation for bounded actions.

The problem existed before agentic AI

Long before large language models entered production, most organizations already had non-human identities with broad entitlements and weak ownership.
Build systems needed credentials to publish artifacts, automation bots needed API tokens to sync data, and integration services needed long-lived keys to reach other target systems.
Teams worked around this with manual approvals, shared credentials, and periodic cleanup, and those controls were imperfect, but the pace of automation was still low enough that humans mostly kept up.

The weakness underneath was always the same one.
A resource and the access to it were decided once, by a person, for a reason that made sense at the time, and nothing revisited that decision when the reason ended.
A pipeline built for a project that ended two years ago still holds write access to production, nobody remembers who owns it, and nobody removes the grant either, because nobody can say what removing it will break.

What changes with agentic AI

Agentic AI doesn't introduce the first non-human identity, but it changes the operating profile in three ways at once.

Volume rises fast.
Instead of a handful of pipeline users and integration accounts, teams deploy many specialized agents for support, operations, analytics, security triage, and engineering work.
Frequency increases too, because traditional automation runs on a schedule or an event trigger while agents run continuously and generate high request rates throughout the day.
And the scope of action expands: a script performs a narrow function, but an agent executes a whole task, collecting data, deciding on an action, calling several target systems, and writing back state.
That's why agents are useful, and it's also why their mistakes travel further.

When one identity can complete an entire workflow on its own, over-privilege stops being a compliance finding and becomes an incident multiplier.

The point where pilots turn into production

Most IT and security teams notice this when agents move from pilot to production.
A pilot runs a few agents on credentials an IT administrator issued by hand, and that's manageable, which is exactly what hides the problem.
Production runs many agents inside business workflows, each needing access to several target systems at once, and issuing those credentials by hand doesn't survive that step, and neither does reviewing them by hand afterwards.

Why older IAM patterns fail under this load

Many organizations still apply human IAM assumptions to non-human identities, and under agentic load those assumptions break quickly.

Consent-driven flows are one example.
OAuth patterns built around a person clicking "Allow" don't map cleanly to autonomous runtime decisions, and approval queues that already slow human collaboration down remove most of the benefit of agents when they're recreated for them.
OpenID Connect and short-lived tokens do help, because a leaked credential expires on its own, but temporary identity isn't the same thing as least privilege.
If the token still carries static administrative scopes, a consent problem turns into a static-permission problem.

Static role grants are a second example, because human access changes slowly while agent access needs to change with the task, sometimes minute by minute.

Shared credentials are a third.
One credential used by several agents saves setup work and erases actor-level accountability, so forensic quality drops exactly when it's needed most.

A related pattern is running an agent under its owner's identity instead of giving it a non-human identity of its own.
That looks attractive at first, because every action is attributed to a named employee, but in practice it's risky: hallucinations, defects in the surrounding harness, or a model taking a shortcut to finish its task trigger unintended high-impact actions carrying human-level entitlements, and attribution after the fact doesn't reduce blast radius.

Ticket-based access is the last one, and the most visible.
An agent that completes a task in seconds waits days for the access it needs to start, and at that point access friction stops being administrative overhead and becomes a reliability problem in production.

What we keep seeing across teams

The same pattern repeats across very different environments.
Short-lived credentials help, and they still leave entitlements over-scoped.
One credential shared by several agents removes setup friction and creates an audit blind spot.
Approval queues preserve control on paper and break under autonomous request volume.
The teams that adapt fastest treat agent identity design as part of system architecture rather than as an IAM afterthought.

From account-centric IAM to task-centric access

The shift is modeling access around task boundaries, rather than adding AI support to an existing IAM process.
Every agent gets its own identity and a recorded owner, and its entitlements follow from where it sits in the organization, whose work it supports, and which tasks it's allowed to perform.
Where a higher-risk action is genuinely required, privilege escalates just in time and expires without anyone remembering to revoke it.

Three examples make that concrete.
An agent supporting an individual engineer can open pull requests and rerun CI, but can't merge into protected branches, even where its owner can.
An agent supporting a support team can read ticket and account context and draft replies, but can't issue high-value refunds without just-in-time elevation.
An agent supporting finance operations can prepare vendor updates, but payment approval stays out of scope unless a time-bound elevation is granted.

Standards like MCP and the newer OAuth profiles help here, mostly with interoperability and implementation consistency, but protocol support isn't the hard part.
Policy design, ownership discipline, and audit integrity are.

What holds up in practice

If your program was built around service accounts and pipeline users, you don't need to start over, because teams get better outcomes by tightening the model they already have for higher speed and higher autonomy.

The ones that improve fastest start with an inventory of non-human identities across every target system they run, from cloud and SaaS to CI/CD and runtime, and record an owner, a purpose, and a risk tier for each one, so policy decisions stop being ambiguous.
They retire shared credentials and issue one identity per agent.
They replace standing elevated access with just-in-time elevation for bounded privileged tasks, and they move approval logic out of ticket queues and into policy guardrails with explicit exception paths.
They keep credential lifetimes short and rotate them automatically, and they capture actor-level and action-level telemetry, so incident reconstruction is possible under pressure.

None of these controls is new on its own.
What changed is that they now have to hold at agent speed, and that's difficult for any model where a resource and its access are approved once and reviewed later.
Access that stays accurate has to be derived from live identity and resource state and reconciled continuously as that state changes, which also means provisioning the resource and granting the access can't stay two separate tracks.
An agent's identity, its owner, its risk tier, and the resources it may reach are one decision, or they drift apart.

The takeaway

Non-human identity risk is an old chapter in security, running at a faster tempo.
Bots and pipelines already exposed the weaknesses, and agentic AI increases both the number of decisions and the consequence of each one, because a single identity can now execute a whole task on its own.
Teams that treat this as a structural change to identity governance adapt faster, while teams that treat it as a small extension of existing controls keep rediscovering the same failure modes at higher speed.

Sunsetting the Kubestack Catalog and Registry: Migrate to Platform Feature Modules

Philipp Strube — Mon, 28 Sep 2026 09:56:57 +0000

The Kubestack catalog is deprecated and the Kubestack module registry at kbst.xyz is being sunset. This post explains what is changing, how to tell whether you are affected, and how to migrate to the new platform feature approach.

What is changing

The catalog repository at github.com/kbst/catalog and the registry at kbst.xyz will not receive any new updates. No new catalog module versions will be published, and no new upstream releases will be packaged. The catalog
should no longer be used for new platform features.

The registry at kbst.xyz will be shut down on December 31, 2026. Until then, existing module versions remain available for download, so terraform init continues to work and
you control when you migrate. After the shutdown, resolving kbst.xyz module sources will fail, and repositories that still reference them will no longer initialize.

The existing catalog pages in the catalog documentation remain available for reference. They will not be removed, but they will not be updated either.

Are you affected?

You are affected if your repository contains module blocks that source from the registry, for example:

module "eks_gc0_eu-west-1_service_nginx" {
  providers = {
    kustomization = kustomization.eks_gc0_eu-west-1
  }

  source  = "kbst.xyz/catalog/nginx/kustomization"
  version = "1.3.1-kbst.1"
}

To find all catalog module references in your repository, run:

grep -rn "kbst.xyz/catalog" *.tf

If the command returns no results, this change does not affect you. If it does, plan your migration before December 31, 2026. If you pin framework and module versions and cache your providers and modules internally, you have a bit more flexibility, but you should still migrate — no new catalog module versions means no more updates to the features you provision from the catalog.

The new approach: platform feature modules

Kubestack has moved away from packaging upstream projects into versioned catalog modules.
The new default is the platform feature module approach:

Instead of consuming a repackaged module, your repository contains a small local Terraform module that deploys the feature's upstream source directly — a Helm chart rendered with helm template into a committed manifests/upstream.yaml, or a plain YAML manifest fetched from upstream
The rendered manifests and the module live in your repository. You own them like the rest of your platform, and updates come straight from the upstream project with no repackaging step in between
The module wraps the same kustomization overlay module type and the same configuration inheritance model as the catalog modules, so your per-environment configuration carries over almost one to one
Your AI coding agent scaffolds and maintains the module for you, following the Kubestack skill

You can read the full documentation of the approach in the
platform features documentation.

How to migrate

You do not have to write the migration by hand. If you have not done so yet, ask your agent to learn the Kubestack skill first:

Learn the Kubestack skill at https://www.kubestack.com/SKILL.md

Then ask your agent to migrate each catalog module:

Migrate the nginx catalog module to a platform feature module

Your agent scaffolds the local feature module modules/<feature_name>/ and one binding file per cluster, carries your existing per-environment configuration over, and asks you
to run the helm template command that renders the new upstream.yaml.

Two things make the migration safe for resources that are already running on your clusters:

A moved block in each binding file adopts the catalog module's Terraform state. Without it, the changed module address would cause Terraform to destroy and recreate every resource of the feature. With it, the migration shows up in the plan as an in-place move
The same namespace as before. The rendered upstream.yaml must put each resource in the same namespace and under the same name it has today. Your agent renders into the same namespace and carries identity-affecting configuration like namespace attributes over unchanged

Follow your normal GitOps workflow for the migration: one feature module per pull request, and review the Terraform plan before promoting. The plan must show the module move without destroy and create pairs for the feature's resources.

The step-by-step walkthrough, including all example files, the moved block, and what to look for in the plan, is available in the
migration guide.

Timeline

Date	What happens
Today	Catalog deprecated. No new catalog module versions are published
Until December 31, 2026	`kbst.xyz` registry remains available, existing module versions can be downloaded
December 31, 2026	`kbst.xyz` registry is shut down, module sources stop resolving

If you use catalog modules today, start migrating now, one module per pull request, so you are done well before the shutdown. If you run into problems during a migration that the guide does not cover, please open a discussion on
GitHub.

Goodbye Cloud, Hello CLI: Sunsetting Kubestack Cloud

Philipp Strube — Tue, 09 May 2023 19:53:11 +0000

I've recently released a major update for Kubestack, the Terraform framework for Kubernetes platform engineering teams. This update moves all functionality previously provided by Kubestack Cloud into the kbst CLI.

I decided to make this change, because Kubestack Cloud was only able to provide a better developer experience on day one. But, once exported to Terraform, the UI was not helpful any more on day two and all following days.

But my goal is to improve the developer experience and day-to-day lives of platform engineering teams at all times. This latest kbst release is a major step towards achieving this goal.

If this is the first time you hear about Kubestack Cloud: Kubestack Cloud was a browser based UI and allowed users to design a Kubernetes platform by following a step-by-step wizard and then exporting and downloading the designed platform's Terraform code.

However, the disconnect between the UI and the code in the repository on a developer's local machine, diminished the value of Kubestack Cloud on day two and beyond. To address this issue, I moved all this functionality into the kbst CLI, where access to the local code is easier.

The kbst CLI, which previously also only scaffolded new repositories, now has CRUD (create, read, update, delete) functionality for clusters, node-pools, and services. This means users can use the CLI to scaffold Terraform code to add or remove clusters, node-pools, or services inside their existing Kubestack repositories.

If you want to see the new CLI in action, give the updated tutorial a try or read the documentation on adding and removing cluster modules, node pool modules or service modules.

But if you'd like to learn more about how this works under the hood, keep reading.

How this works

If you're already familiar with Kubestack, you know that Kubestack repositories follow a convention-over-configuration approach to define the clusters, node pools, and services that make up a Kubernetes platform in a single Terraform codebase. At the root of each repository, there are several .tf files that follow a specific naming convention. These files contain module calls that define each platform component.

To add or remove components, or update the versions of existing component modules, the kbst CLI parses the necessary subset of Terraform code to understand the components of the platform. You can list the Kubestack component modules it discovered using the kbst list command. By appending --all to the list command, you can also see any non-Kubestack modules.

You can add node pools or services to existing clusters or add more clusters from the same or even a different cloud provider. The CLI will scaffold the additional required .tf files and update the Dockerfile's FROM line to specify the correct image, in case of changing from a single to a multi-cloud environment or vice versa. Likewise, it will also remove module calls and the respective .tf files if you remove a service, a node pool or even a cluster from your platform.

But don't worry, the kbst CLI only changes local files and does never change any cloud or Kubernetes resources.

You can use it to avoid writing repetitive boilerplate code or manually deleting module calls and Terraform files, while still owning your codebase and retaining the ability to extend or modify the code to meet specific needs.

Once you're happy with the code, you can follow the Kubestack GitOps workflow to peer-review, validate, and promote changes to your platform's environments as usual.

In conclusion, the shift from Kubestack Cloud to the kbst CLI provides a better developer experience not only on day one, but also on day two and makes it easier for platform engineering teams to manage their Kubernetes based platforms.

What happened to the platforms I designed with Kubestack Cloud?

If you have previously designed a platform with Kubestack Cloud, you can sign in with your existing user and will see instructions how to scaffold your existing platforms using the new CLI.

Here's an example screenshot of how that will look like.

Getting rigorous about investing in the Kubestack project

Philipp Strube — Sun, 27 Nov 2022 14:52:30 +0000

Sometimes when you spend a long time solving a problem, it makes it harder to see your solution clearly.

In 12 years helping companies adopt modern cloud computing, I saw so many of the same snags repeating across multiple organizations. From these lessons, I built Kubestack as guardrails to make it easier to avoid the pain in the first place. Kubestack has been used in companies large and small for years, but I haven’t always known where it has been most helpful to its users. Without knowing this, it’s hard to bring more people in as both users and contributors. So earlier this year, I contracted with Ana Hevesi to support Kubestack's open source efforts.

Ana operates a developer experience consultancy. After working in technical community building for companies like Stack Overflow and Nodejitsu, Ana now works with devtools founders to create evidence-based approaches for growing their ecosystems.

We started by doing some research into how Kubestack serves your goals.

Research methods

Our objective was to learn about users’ career trajectories and aspirations, and get a clear picture of what role Kubestack plays in your success.

Ana recommended we aim for 5 interviews, citing it as a good “goldilocks zone” for initial quantity of data to work with. I then reached out to a spectrum of new and long-tenured Kubestack users to ask for their time in a 60 minute user interview.

Ana wrote a standard interview script which included bandwidth for conversational “side quests.” After, Ana analyzed recordings, picked out repeat themes, and came to me with conclusions and recommendations.

Areas of positive impact

Kubestack helps careers

Participants attributed their use of Kubestack to positive career outcomes, such as developing a reputation for reliably delivering for users, or scaling on a tight timeframe with limited prior experience. Others reported it was a key learning tool when they were just starting as platform engineers.

Works so well it disappears

The most consistent feedback we received was that users can assume Kubestack is just going to work. Multiple participants had been relying on the framework for many months without needing to give it a second thought.

Areas to improve

Documentation for advanced features needs improvement

Kubestack works great for months on end for most orgs setting up their first K8s cluster, but those who wished to modify Kubestack outside of existing use cases told us error messages and upgrade processes were opaque.

Backwards compatibility and multi-cloud support presents friction to open source contributions

Adding new features requires working knowledge of both Terraform and cloud provider functionality across historical versions, and at times, their interactions with one another. Furthermore, while Kubestack is committed to supporting EKS, AKS, and GKE, a contributor may wish to implement functionality for only one of these cloud providers. Inviting more PRs from a wider array of contributors necessitates a plan for tiered support of legacy versions or defining contributor scope to accommodate this complexity.

How we’re applying these findings

Connecting with the people who need us most

Kubestack makes a huge impact on early-stage teams and emerging professionals. We’re exploring ways to better tailor our communication and outreach to make sure they know about the opportunities this framework provides, improving both adoption and contributions to the project. Kubestack only succeeds because you succeed.

Benefits before features

The current iteration of Kubestack’s landing page assumes a fairly high level of existing knowledge of the platform engineering space. As such, an upcoming iteration of the Kubestack site will aim to engage folks who aren’t already deep in the jargon and progressively bring them into the fold, while still being legible to seasoned professionals.

Open source participation onramps

Since enabling users to learn from each other and communicating where the project is going is an important part of growing an open source community, we’ll be experimenting with office hours and public communication about recent releases. We’ll have scheduling details coming soon.

Leveling up, together

I created Kubestack so that folks coming to Kubernetes for the first time could take immediate advantage of the separation of concerns that containers provide. User research says that this works as intended!

Now comes the iterative task of communicating my own knowledge and experience in ways that make it easier to build together, while learning from your use of the project to fill in its gaps. Ultimately, the intent is a healthy community where we’re all working together to make the project better serve your needs.

Finally, a big thank you to Tomas, AJ, Brendan, Christoph, and Mark for your time and candor. Kubestack is better for it.

A Better Way to Provision Kubernetes Resources Using Terraform

Philipp Strube — Wed, 04 May 2022 18:01:47 +0000

Terraform is immensely powerful when it comes to defining and maintaining infrastructure as code. In combination with a declarative API, like a cloud provider API, it can determine, preview, and apply changes to the codified infrastructure.

Consequently, it is common for teams to use Terraform to define the infrastructure of their Kubernetes clusters. And as a platform to build platforms, Kubernetes commonly requires a number of additional services before workloads can be deployed. Think of ingress controllers or logging and monitoring agents and so on. But despite Kubernetes' own declarative API, and the obvious benefits of maintaining a cluster's infrastructure and services from the same infrastructure as code repository, Terraform is far from the first choice to provision Kubernetes resources.

With Kubestack, the open-source Terraform framework I maintain, I'm on a mission to provide the best developer experience for teams working with Terraform and Kubernetes. And unified provisioning of all platform components, from cluster infrastructure to cluster services, is something I consider crucial in my relentless pursuit of said developer experience.

Because of that, the two common approaches to provision Kubernetes resources using Terraform never really appealed to me.

On the one hand, there's the Kubernetes provider. And while it integrates Kubernetes resources into Terraform, maintaining the Kubernetes resources in HCL is a lot of effort. Especially for Kubernetes YAML you consume from upstream. On the other hand, there are the Helm provider and the Kubectl provider. These two use native YAML instead of HCL, but do not integrate the Kubernetes resources into the Terraform state and, as a consequence, lifecycle.

I believe my Kustomization provider based modules are a better alternative because of three distinct benefits:

Like Kustomize, the upstream YAML is left untouched, meaning upstream updates require minimal maintenance effort.
By defining the Kustomize overlay in HCL, all Kubernetes resources are fully customizable using values from Terraform.
Each Kubernetes resource is tracked individually in Terraform state, so diffs and plans show the changes to the actual Kubernetes resources.

To make these benefits less abstract, let's compare my Nginx ingress module with one using the Helm provider to provision Nginx ingress.

The Terraform configuration for both examples is available in this repository. Let's take a look at the Helm module first.

The Helm-based module

Usage of the module is straightforward. First, configure the Kubernetes and Helm providers.

provider "kubernetes" {
  config_path    = "~/.kube/config"
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

Then define a kubernetes_namespace and call the release/helm module.

resource "kubernetes_namespace" "nginx_ingress" {
  metadata {
    name = "ingress-nginx"
  }
}

module "nginx_ingress" {
  source  = "terraform-module/release/helm"
  version = "2.7.0"

  namespace  = kubernetes_namespace.nginx_ingress.metadata.0.name
  repository = "https://kubernetes.github.io/ingress-nginx"

  app = {
    name          = "ingress-nginx"
    version       = "4.1.0"
    chart         = "ingress-nginx"
    force_update  = true
    wait          = false
    recreate_pods = false
    deploy        = 1
  }

  set = [
    {
      name  = "replicaCount"
      value = 2
    }
  ]
}

If you now run a terraform plan for this configuration, you see the resources to be created.

Terraform will perform the following actions:

  # kubernetes_namespace.nginx_ingress will be created
  + resource "kubernetes_namespace" "nginx_ingress" {
      + id = (known after apply)

      + metadata {
          + generation       = (known after apply)
          + name             = "ingress-nginx"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # module.nginx_ingress.helm_release.this[0] will be created
  + resource "helm_release" "this" {
      + atomic                     = false
      + chart                      = "ingress-nginx"
      + cleanup_on_fail            = false
      + create_namespace           = false
      + dependency_update          = false
      + disable_crd_hooks          = false
      + disable_openapi_validation = false
      + disable_webhooks           = false
      + force_update               = true
      + id                         = (known after apply)
      + lint                       = true
      + manifest                   = (known after apply)
      + max_history                = 0
      + metadata                   = (known after apply)
      + name                       = "ingress-nginx"
      + namespace                  = "ingress-nginx"
      + recreate_pods              = false
      + render_subchart_notes      = true
      + replace                    = false
      + repository                 = "https://kubernetes.github.io/ingress-nginx"
      + reset_values               = false
      + reuse_values               = false
      + skip_crds                  = false
      + status                     = "deployed"
      + timeout                    = 300
      + values                     = []
      + verify                     = false
      + version                    = "4.1.0"
      + wait                       = false
      + wait_for_jobs              = false

      + set {
          + name  = "replicaCount"
          + value = "2"
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

And this is the key issue with how Helm is integrated into the Terraform workflow. The plan does not tell you what Kubernetes resources will be created for the Nginx ingress controller. And neither are the Kubernetes resources tracked in Terraform state, as shown by the apply output.

kubernetes_namespace.nginx_ingress: Creating...
kubernetes_namespace.nginx_ingress: Creation complete after 0s [id=ingress-nginx]
module.nginx_ingress.helm_release.this[0]: Creating...
module.nginx_ingress.helm_release.this[0]: Creation complete after 3s [id=ingress-nginx]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Similarly, if planning a change, there's again no way to tell what the changes to the Kubernetes resources will be.

So if you increase the replicaCount value of the Helm chart, the terraform plan will merely show the change to the helm_release resource.

set = [
  {
    name = "replicaCount"
    value = 3
  }
]

What will the changes to the Kubernetes resources be? And more importantly, is it a simple in-place update, or does it require a destroy-and-recreate? Looking at the plan, you have no way of knowing.

Terraform will perform the following actions:

  # module.nginx_ingress.helm_release.this[0] will be updated in-place
  ~ resource "helm_release" "this" {
        id                         = "ingress-nginx"
        name                       = "ingress-nginx"
        # (27 unchanged attributes hidden)

      - set {
          - name  = "replicaCount" -> null
          - value = "2" -> null
        }
      + set {
          + name  = "replicaCount"
          + value = "3"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

The Kustomize-based module

Now, let's take a look at the same steps for the Kustomize-based module. Usage is similar. First require the kbst/kustomization provider and configure it.

terraform {
  required_providers {
    kustomization = {
      source = "kbst/kustomization"
    }
  }
}

provider "kustomization" {
  kubeconfig_path = "~/.kube/config"
}

Then call the nginx/kustomization module.

module "nginx_ingress" {
  source  = "kbst.xyz/catalog/nginx/kustomization"
  version = "1.1.3-kbst.1"

  configuration_base_key = "default"
  configuration = {
    default = {
      replicas = [{
        name  = "ingress-nginx-controller"
        count = 2
      }]
    }
  }
}

Unlike for the Helm-based module though, when you run terraform plan now you will see each Kubernetes resource and its actual configuration individually. To keep this blog post palatable, I show the details for the namespace only.

Terraform will perform the following actions:

  # module.nginx_ingress.kustomization_resource.p0["_/Namespace/_/ingress-nginx"] will be created
  + resource "kustomization_resource" "p0" {
      + id       = (known after apply)
      + manifest = jsonencode(
            {
              + apiVersion = "v1"
              + kind       = "Namespace"
              + metadata   = {
                  + annotations = {
                      + "app.kubernetes.io/version"      = "v0.46.0"
                      + "catalog.kubestack.com/heritage" = "kubestack.com/catalog/nginx"
                      + "catalog.kubestack.com/variant"  = "base"
                    }
                  + labels      = {
                      + "app.kubernetes.io/component"  = "ingress-controller"
                      + "app.kubernetes.io/instance"   = "ingress-nginx"
                      + "app.kubernetes.io/managed-by" = "kubestack"
                      + "app.kubernetes.io/name"       = "nginx"
                    }
                  + name        = "ingress-nginx"
                }
            }
        )
    }

  # module.nginx_ingress.kustomization_resource.p1["_/ConfigMap/ingress-nginx/ingress-nginx-controller"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/Service/ingress-nginx/ingress-nginx-controller"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/Service/ingress-nginx/ingress-nginx-controller-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/ServiceAccount/ingress-nginx/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["_/ServiceAccount/ingress-nginx/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["apps/Deployment/ingress-nginx/ingress-nginx-controller"] will be created
  # module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-create"] will be created
  # module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-patch"] will be created
  # module.nginx_ingress.kustomization_resource.p1["networking.k8s.io/IngressClass/_/nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRole/_/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRole/_/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRoleBinding/_/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/ClusterRoleBinding/_/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/Role/ingress-nginx/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/Role/ingress-nginx/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/RoleBinding/ingress-nginx/ingress-nginx"] will be created
  # module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/RoleBinding/ingress-nginx/ingress-nginx-admission"] will be created
  # module.nginx_ingress.kustomization_resource.p2["admissionregistration.k8s.io/ValidatingWebhookConfiguration/_/ingress-nginx-admission"] will be created

Plan: 19 to add, 0 to change, 0 to destroy.

Applying, again, has all the individual Kubernetes resources. And because the modules use explicit depends_on to handle namespaces and CRDs first and webhooks last, resources are reliably applied in the correct order.

module.nginx_ingress.kustomization_resource.p0["_/Namespace/_/ingress-nginx"]: Creating...
module.nginx_ingress.kustomization_resource.p0["_/Namespace/_/ingress-nginx"]: Creation complete after 0s [id=369e8643-ad33-4eb4-95dc-f506cef4a198]
module.nginx_ingress.kustomization_resource.p1["rbac.authorization.k8s.io/RoleBinding/ingress-nginx/ingress-nginx"]: Creating...
module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-create"]: Creating...

...

module.nginx_ingress.kustomization_resource.p1["batch/Job/ingress-nginx/ingress-nginx-admission-patch"]: Creation complete after 1s [id=58346878-70bd-42f2-af61-2730e3435ca7]
module.nginx_ingress.kustomization_resource.p1["_/ServiceAccount/ingress-nginx/ingress-nginx"]: Creation complete after 0s [id=f009bbb7-7d2e-4f28-a826-ce133c91cc15]
module.nginx_ingress.kustomization_resource.p2["admissionregistration.k8s.io/ValidatingWebhookConfiguration/_/ingress-nginx-admission"]: Creating...
module.nginx_ingress.kustomization_resource.p2["admissionregistration.k8s.io/ValidatingWebhookConfiguration/_/ingress-nginx-admission"]: Creation complete after 1s [id=3185b09f-1f67-4079-b44f-de01bff44bd2]

Apply complete! Resources: 19 added, 0 changed, 0 destroyed.

Naturally, it also means that if you increase the replica count like this...

replicas = [{
  name = "ingress-nginx-controller"
  count = 3
}]

...the terraform plan shows which Kubernetes resources will change and what the diff is.

Terraform will perform the following actions:

  # module.nginx_ingress.kustomization_resource.p1["apps/Deployment/ingress-nginx/ingress-nginx-controller"] will be updated in-place
  ~ resource "kustomization_resource" "p1" {
        id       = "81e8ff18-6c6c-440d-bd8b-bf5f0d016953"
      ~ manifest = jsonencode(
          ~ {
              ~ spec       = {
                  ~ replicas             = 2 -> 3
                    # (4 unchanged elements hidden)
                }
                # (3 unchanged elements hidden)
            }
        )
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Maybe more importantly even, the Kustomization provider will also correctly show if a resource can be changed using an in-place update. Or if a destroy-and-recreate is required because there is a change to an immutable field, for example.

This is the result of two things:

That, as you've just seen, every Kubernetes resource is handled individually in Terraform state, and
that the Kustomization provider uses Kubernetes' server-side dry-runs to determine the diff of each resource.

Based on the result of that dry-run, the provider instructs Terraform to create an in-place or a destroy-and-recreate plan.

So, as an example of such a change, imagine you need to change spec.selector.matchLabels. Since matchLabels is an immutable field, you will see a plan that states that the Deployment resource must be replaced. And you will see 1 to add and 1 to destroy in the plan's summary.

Terraform will perform the following actions:

  # module.nginx_ingress.kustomization_resource.p1["apps/Deployment/ingress-nginx/ingress-nginx-controller"] must be replaced
-/+ resource "kustomization_resource" "p1" {
      ~ id       = "81e8ff18-6c6c-440d-bd8b-bf5f0d016953" -> (known after apply)
      ~ manifest = jsonencode(
          ~ {
              ~ metadata   = {
                  ~ labels      = {
                      + example-selector               = "example"
                        # (6 unchanged elements hidden)
                    }
                    name        = "ingress-nginx-controller"
                    # (2 unchanged elements hidden)
                }
              ~ spec       = {
                  ~ replicas             = 2 -> 3
                  ~ selector             = {
                      ~ matchLabels = {
                          + example-selector               = "example"
                            # (4 unchanged elements hidden)
                        }
                    }
                  ~ template             = {
                      ~ metadata = {
                          ~ labels      = {
                              + example-selector               = "example"
                                # (4 unchanged elements hidden)
                            }
                            # (1 unchanged element hidden)
                        }
                        # (1 unchanged element hidden)
                    }
                    # (2 unchanged elements hidden)
                }
                # (2 unchanged elements hidden)
            } # forces replacement
        )
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Try it yourself

You can find the source code for the comparison on GitHub if you want to experiment with the differences yourself.

If you want to try the Kustomize modules yourself, you can either use one of the modules from the catalog that bundle upstream YAML, like the Prometheus operator, Cert-Manager, Sealed secrets, or Tekton, for example.

But this doesn't only work for upstream services. There is also a module that can be used to provision any Kubernetes YAML in the exact same way as the catalog modules - called the custom manifest module.

Get involved

Currently, the number of services available from the catalog is still limited.

If you want to get involved, you can also find the catalog source on GitHub.

Photo by Vladislav Babienko on Unsplash

Google Anthos with Terraform and Kubestack

Philipp Strube — Fri, 02 Jul 2021 12:49:15 +0000

For a project, I'm currently evaluating Google Anthos. Since the client is a multi-national company, the idea is to build a multi-region and multi-cloud Kubernetes platform. But the sensible kind, no need to bring the pitchforks. So different applications each in one region and cloud. Not the mythical one application in multiple regions and clouds where complexity and necessity are often entirely disproportionate to each other. But that's not really the point here.

Google sales is pushing Anthos hard. And the client's team is open to the argument that an opinionated stack might help them move faster compared with having to first evaluate various alternatives for each part of the stack and then building the know-how to productize this custom stack. It's a fair argument to make.

Long story short, we're now evaluating Anthos with GKE and EKS clusters connected to it, because some application teams are drawn to AWS and some are drawn to Google Cloud with their respective workloads. Individual reasons for this are pretty diverse, ranging from data stickiness like terabytes of data already in S3 to quota/capacity limits of specific types of GPUs or preferring certain managed services from one provider over the other cloud provider's alternative.

I tend to agree with this kind of multi-cloud strategy making a lot of sense. Yes, individual apps may still end up locked-in to one vendor. But at least it's not all eggs in one basket which has real benefits both on blast radius and pricing negotiations, if you're big enough.

I've been working on this evaluation for a couple of days now and thought I'd share my experience because I couldn't find a lot of hands-on reports about Anthos with Terraform. Most content seemed primarily hypothetical and it seemed like most writers hadn't actually gotten their hands properly dirty before writing about it. I already washed mine to calm down and make sure this doesn't end up in some obnoxious rant.

The first thing, that really surprised me about Anthos though, is that Anthos does not provision the Kubernetes clusters for you. I totally expected it would. Instead, you have to provision the clusters and then connect them to the Anthos hub. Which basically requires some IAM setup, an Anthos hub membership and running an agent inside each cluster.

Anthos leaves it up to you to provision clusters any way you like. But since I'm part of the project, it may not come as a surprise that in our case, infrastructure as code and Terraform are the attack plan.

Now, Google does even provide its own Anthos Terraform modules. But these are only for GKE, meaning for EKS we'd need to use modules from another source. Leaving us to deal with different module usage and update schedules.

But more importantly, Google's Terraform modules constantly shell out to kubectl or gcloud CLIs. Which I consider a last resort that should be avoided at any cost for Terraform modules, because long term maintainability. Frankly, calling CLI commands like this has no place in declarative infrastructure as far as I'm concerned.

Unsurprisingly, my biased proposal is to use Kubestack to provision the GKE and EKS clusters leveraging the Kubestack framework's unified GKE and EKS modules, and to write a custom module to connect the resulting clusters to Anthos. The bespoke module would integrate the IAM, Anthos and Kubernetes resources required fully into the Terraform state and lifecycle instead of calling kubectl and gcloud like the official Google modules do.

Below is the current work in progress state of the experimental module and some of the challenges I hit so far.

The first requirement is an IAM identity and role for the agent inside the cluster. For GKE clusters, workload identities can be used but for non GKE clusters, EKS in our case, it seems shared credentials in the form of a service account key are the only option. Creating google_service_account, google_project_iam_member and google_service_account_key resources is easy enough. I'm sure this is overly simplified and I may have to add more roles as my evaluation continues.

resource "google_service_account" "current" {
  project = local.project_id
  account_id   = local.cluster_name
  display_name = "${local.cluster_name} gke-connect agent"
}

resource "google_project_iam_member" "current" {
  project = local.project_id
  role    = "roles/gkehub.connect"
  member  = "serviceAccount:${google_service_account.current.email}"
}

resource "google_service_account_key" "current" {
  service_account_id = google_service_account.current.name
}

The next step is to register the cluster as a member of the Anthos hub. Which means adding a google_gke_hub_membership resource.

resource "google_gke_hub_membership" "current" {
  provider = google-beta

  project = local.project_id
  membership_id = local.cluster_name
  description = "${local.cluster_name} hub membership"
}

Finally, the agent needs to be provisioned inside the cluster and set up to use the service account as its identity.

By default, joining the cluster to the hub and provisioning the Kubernetes resources of the agent on the cluster is done via the gcloud beta container hub memberships register CLI command. But the command has a --manifest-output-file parameter, that allows writing the Kubernetes resources to a file instead of applying it to the cluster directly.

To not also have to fall back to calling the register gcloud command from Terraform, I opted to write the manifests to a YAML file and use them as the base that I patch in a kustomization_overlay using my Kustomization provider.

This way, I will have each individual Kubernetes resource of the Anthos agent to be provisioned and tracked using Terraform. While at the same time being able to use the attributes from my service account and service account key resources to configure the agent.

data "kustomization_overlay" "current" {
  namespace = "gke-connect"

  resources = [
      "${path.module}/upstream_manifest/anthos.yaml"
  ]

  secret_generator {
    name = "creds-gcp"
    type = "replace"
    literals = [
      "creds-gcp.json=${base64decode(google_service_account_key.current.private_key)}"
    ]
  }

  patches {
    # this breaks if the order of env vars in the upstream YAML changes
    patch = <<-EOF
      - op: replace
        path: /spec/template/spec/containers/0/env/6/value
        value: "//gkehub.googleapis.com/projects/xxxxxxxxxxxx/locations/global/memberships/${local.cluster_name}"
    EOF

    target = {
      group = "apps"
      version = "v1"
      kind = "Deployment"
      name = "gke-connect-agent-20210514-00-00"
      namespace = "gke-connect"
    }
  }
}

The manifests gcloud writes to disk can't be committed to version control because they include a Kubernetes secret with the plaintext service account key embedded. The key file is unfortunately a required parameter of the hub memberships register command. So I had to delete this secret from the YAML file. And have to remember doing this whenever I rerun the command to update my base with the latest upstream manifests.

In the kustomization_overlay data source, I then use a secret_generator to create a Kubernetes secret using the private key from the google_service_account_key resource.

Additionally, the agent has a number of environment variables set. The URL to the hub memberships resource is one of them and needs to be patched with the respective cluster name. Unfortunately, the environment variables are set directly in the pod template. So the patch will break if the number or order of environment variables changes. It would be better to change this to envFrom and set the environment variables dynamically in the overlay using a config_map_generator. But the downside of this is, again, that there's one more modification to the upstream YAML which has to be repeated every time it is updated.

While we're on the topic of updates. One thing that makes me suspicious is that the generated YAML has a date as part of its resource names. E.g. gke-connect-agent-20210514-00-00. Call me a pessimist, but I totally expect this to become a problem with updates in the future.

Ignoring that for now. Next on my evaluation was to apply my Terraform configuration and hopefully have my clusters connected to Anthos.

Unfortunately, on the first try, that wasn't quite the case. The clusters did show up in the Anthos UI. But had a big red unreachable warning. As it turned out, this was due to the agent pod crash looping with a permission denied error.

kubectl -n gke-connect logs gke-connect-agent-20210514-00-00-66d94cff9d-tzw5t
2021/07/02 11:40:33.277997 connect_agent.go:17: GKE Connect Agent. Log timestamps in UTC.
2021/07/02 11:40:33.298969 connect_agent.go:21: error creating tunnel: unable to retrieve namespace "kube-system" to be used as externalID: namespaces "kube-system" is forbidden: User "system:serviceaccount:gke-connect:connect-agent-sa" cannot get resource "namespaces" in API group "" in the namespace "kube-system"

Which is weird, because from reading the gcloud generated YAML I remember there were plenty of RBAC related resources included. Digging in, it turned out the generated YAML has a Role and RoleBinding. And if you followed carefully, you probably guessed the issue already. Here's the respective part of the generated resources:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    hub.gke.io/project: "ie-gcp-poc"
    version: 20210514-00-00
  name: gke-connect-namespace-getter
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    hub.gke.io/project: "ie-gcp-poc"
    version: 20210514-00-00
  name: gke-connect-namespace-getter
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: gke-connect-namespace-getter
subjects:
- kind: ServiceAccount
  name: connect-agent-sa
  namespace: gke-connect

Unless I'm terribly wrong here, this obviously can't work. Creating a namespaced role and role binding inside the kube-system namespace can not grant permissions to get the kube-system namespace, because namespaces are not namespaced resources.

So I changed the Role to a ClusterRole and the RoleBinding to a ClusterRoleBinding and reapplied my Terraform configuration. And I now have a running agent, that established a tunnel to the Anthos control plane and prints lots of log messages. I have yet to dig into what it actually does there.

With the RBAC fix the generated YAML now already requires three changes to maintain over time. I can't say I'm particularly excited about that. I also wonder if the generated YAML is only broken if the --manifest-output-file parameter is used or if the RBAC configuration is also broken when directly applying the Kubernetes resources to the cluster using the gcloud CLI.

That's it for my evaluation with Google Anthos, Terraform and Kubestack so far. Maybe by sharing my findings, I may safe somebody out there a bit of time in their own evaluation when they hit the same issues.

Next step for me is to look into provisioning the Anthos Service Mesh. It's not quite clear to me yet, if that should be done via Terraform, the fact that Google has this kubectl based Terraform module for that may suggest so. But why wouldn't I do everything after the cluster is connected to Anthos using Anthos Config Management?

What Terraform can learn from PHP

Philipp Strube — Mon, 08 Feb 2021 10:30:47 +0000

TL;DR:

Writing infrastructure as code shows many of the same challenges as writing code for application development, because many of these challenges are not language or use-case specific.
Terraform and its surrounding ecosystem are still evolving and share many similarities with early PHP and the web. Just like PHP evolved by learning from other language ecosystems, Terraform can as well.
Use-case specific frameworks are a major driver of innovation, improved developer experience and productivity on the application development side. But are not yet established parts of the infrastructure as code ecosystem.
The paradigm shift to containers and Kubernetes made use-case specific frameworks possible for infrastructure as code by providing a powerful abstraction between application and infrastructure layer. And the cloud native community is evolving rapidly, extending this abstraction to additional use-cases.
Organizations that adopted application development frameworks for their improved developer experience and productivity, can leverage the same benefits for automating Kubernetes by using an infrastructure as code framework and avoid leaving the cluster the weakest link in their GitOps automation.

Learning from other language ecosystems

PHP’s ease of getting started is widely quoted as the boon and bane of the language. It seems as if making fun of the spaghetti code bases of the early PHP days never gets old. Even in 2021. But there is no doubt that PHP is an extremely successful programming language.

You may ask, what does this have to do with Terraform? Well, hear me out. Terraform and PHP have more in common than you may think. PHP was created when the web was in its infancy and quickly became extremely popular. Don’t forget, PHP is the P in LAMP stack. Similarly, infrastructure as code is still an emerging ecosystem today, and Terraform is by far the most popular language in this ecosystem.

But the modern PHP of today is vastly different from the early PHP we all like to make fun of. And since Terraform today is so similar to where PHP was when it started, there’s a good chance that the Terraform community can learn a lot from how PHP evolved.

Rasmus Lerdorf, the creator of PHP, is famously quoted as never having intended to write a programming language. But PHP got popular and they had to keep going. In addition, the web and its request-response model were new, even to experienced developers. But the endless possibilities of the web got people excited, and the unintentional programming language PHP was easy to get started with. This combination led to the stereotypical poor quality code bases that ended up powering major parts of the early web.

Similarly, infrastructure as code offers huge benefits and gets people excited as well. But it also requires both operations and coding experience, and people coming from either one background have to learn a lot about the respective other, before they can be fully productive.

Languages like Python released a few years before PHP, or Ruby and Java, which were released in the same year as PHP, were intentionally designed programming languages for professional use. While not specific to the web, it is of course possible to build web applications in either one of them. So the self-evident thing was to use these more mature and consistent languages to build web applications, and have more easily maintainable code bases as a result.

And not only were the languages more mature, but so were their ecosystems. The majority of challenges, developers face when writing code, are not language specific. And many are not even use-case specific. You may need different dependencies for building a web application instead of a desktop application for example. But in both cases having dependency management is greatly useful. A feature Python, Ruby and Java all already had.

This led to the creation of frameworks like Django, Ruby on Rails or Spring that made it easy to build web applications in Python, Ruby or Java respectively, leveraging their existing language ecosystems.

A great idea that works in one ecosystem, however is quick to inspire similar development in other languages. And PHP’s wide adoption easily justified major investments to improve the PHP core as well as the surrounding ecosystem. All those teams looking for the best way to maintain their growing PHP code bases were smart to look at other languages and how these same challenges were solved there.

The result are frameworks like Symfony or CakePHP, heavily inspired by Spring and Rails respectively. This is also how Composer brought modern dependency management to PHP. And last but not least, this was when the PHP community adopted Git for version control and slowly moved away from just editing production files directly via FTP.

It’s all about the code

Let's get back to infrastructure as code. Yes, in a lot of ways automating infrastructure is different from application development. But many of the challenges of writing code, that applied across languages and use-cases on the software development side, also apply to infrastructure as code. Code is kind of the keyword here.

So just like PHP learned from other languages, their frameworks and their tooling, Terraform can only benefit from doing so as well.

One area where Hashicorp, the makers of Terraform, recently made major improvements is dependency management. Terraform had the ability to download required providers for quite some time. But it was limited to only Hashicorp’s own providers. Community maintained providers required involving, manual installation. A recent Terraform release introduced support for registry namespaces, which means community providers can now also be installed from the official registry. In addition, required providers and versions can now be specified more explicitly. Even including the ability to vendor providers, and thereby hardening automation runs against failing when the registry is unavailable.

The missing piece

All the language ecosystems we discussed share one key piece that heavily improves the developer experience, but which isn’t a thing yet in the infrastructure as code world. I’m referring to frameworks of course. And concretely use-case specific frameworks. By being use-case specific, the aforementioned software development frameworks drastically reduce upfront and maintenance effort, and provide the best developer experience and workflow possible.

If I’m building a cloud native application in Java, using Spring Boot will make my life much easier. Likewise, if my goal is to build a Jamstack website, a framework like Gatsby will get me there much faster.

But the reason why frameworks are not a thing in the infrastructure as code world yet is not merely that the ecosystem is still evolving. For frameworks to be useful, we also required a strong abstraction layer that kept the infrastructure layer clear from application specific requirements. Containers and Kubernetes are extremely popular because they provide this very abstraction. And this means two things: First, that with using Terraform to manage Kubernetes there is a popular and very specific use-case for an infrastructure as code framework. And second, that because of the powerful abstraction, such a framework makes sense for the first time.

Kubestack is this use-case specific, Terraform GitOps framework. If you’re building GitOps automation for Kubernetes cluster infrastructure and cluster services using Terraform, Kubestack may be the framework for you. Think of Kubestack as the Ruby on Rails of infrastructure automation, the Gatsby of GitOps, or the Spring Boot of Terraform and Kubernetes.

And just like application frameworks copied ideas that worked well from one language to another, Kubestack does the same from application development to infrastructure as code.

Talent borrows, genius steals

One example is Kubestack’s convention over configuration based repository layout. Another one is its inheritance based configuration to prevent drift between environments. A third one is the ability to easily vendor dependencies in the repository, like the Nginx ingress controller or Prometheus monitoring operator. Or, as the last but not the least example, local development environments that automatically update as you make changes to the code.

Slow feedback loops are poison for developer productivity. And infrastructure as code is notoriously known for mandatory, slow pipeline runs. This makes the local development environment the perfect example how Kubestack drastically improves the developer experience, because it’s a use-case specific framework.

The strong abstraction between the application and infrastructure layers is a key mantra of what we know as cloud native. And if you take a look at recent developments from the cloud native community the direction is clear. As more and more organizations shift their workloads and use-cases to cloud native, we continue to see new innovation and iterative improvements that extend this powerful abstraction.

This is both positive for the future of infrastructure as code and Terraform as well as for use-case specific infrastructure as code frameworks.

Terraform loves cloud native

Systems that provide a separation between declaring desired state and current state are the current state-of-the-art. This is a core principle of Kubernetes and high-level managed cloud services, but also of VM auto-scaling groups, as a lower level example of this principle. On the surface there’s an API to declare the desired state. And behind the API are control loops that keep the current state in sync with the desired state.

Terraform shines when being combined with such a system, because it is great at planning and applying changes triggered by a commit in a repository. And it can also be run periodically, to detect drift and either alert or overwrite. But when operating distributed systems, there are various failure scenarios where continuously running controllers, that can take immediate action based on more events than just code changes, are clearly superior. The important thing to understand here is, Terraform is great to provide a way for teams to reason about proposed changes and keeping the committed state and desired state in sync. But keeping desired and current state in sync is, in most cases, better left to a continuously running control loop.

It’s common for teams to hit this limitation when using infrastructure as code to automate legacy systems that don’t provide this separation of concerns. And this frequently leads to automation that only manages the lifecycle partially and causes complex issues for teams to coordinate automation and manual operations. Facing this significantly limits the value of infrastructure as code, and many teams justifiably may hold back on adopting Terraform for this very reason.

But Kubernetes or managed cloud services are not the only systems that rely on declared desired state and reconciliation loops to keep current state in sync. An example doing this for infrastructure automation outside the cloud provider’s walled gardens is ClusterAPI. This cloud native community initiative aims to provide the same separation across on-premise and cloud. And through integration into vSphere, ClusterAPI is readily available to VMware’s vast installed base.

The future of infrastructure is code

As an industry, we’re clearly heading into one direction. And as we continue to adopt this paradigm, the limitations that held infrastructure as code back, when working with legacy systems, do not apply any more. As infrastructure as code becomes more viable for more organizations, more teams can benefit from use-case specific frameworks to get the best possible developer experience and productivity.

Already now, many teams are using Terraform successfully. Yes, there are edge cases to consider and there is a steep learning curve, no matter if your background is in operations or software development. But as the cloud native ecosystem continues to evolve, the benefits of infrastructure as code will be applicable to more teams and more use-cases and just like PHP grew by learning from other language ecosystems, Terraform will too.

As far as Kubernetes is concerned, if you’re already adopting GitOps, the Kubestack framework is an opportunity to implement full-stack GitOps that covers both the cluster infrastructure and cluster services and not just the application workloads on the cluster. This way, you can avoid having the foundation of your system, the cluster, be the weakest link by not managing it manually via UI.