DEV Community: Sumas Keller

The Budget Mistake Most Companies Make in Their Second Year of AI

Sumas Keller — Fri, 26 Jun 2026 16:06:04 +0000

Year one of AI deployment, budgets are straightforward. You are buying licenses, paying for implementation, running pilots. The costs are visible and the categories make sense.

Year two is where the budget mistakes happen, and they all follow the same pattern. The organization treats AI tools as a fixed line item rather than as infrastructure that requires ongoing investment to maintain its value.

Here is what that looks like in practice.

The licenses renew automatically. That part gets handled. But the work that actually keeps the AI useful, the prompt refinement, the document hygiene, the index maintenance, the workflow adjustments as the business changes, gets absorbed invisibly into whoever happens to be paying attention to the tools. Usually that is one or two people who care, doing it on the side of their actual job description, without any formal recognition that this work is happening or any protection for the time it requires.

When those people get busy with other priorities, which happens at some point to everyone, the AI tools quietly degrade. The knowledge base gets stale. The prompts stop being updated to reflect how the business has changed. The retrieval starts returning outdated information. Users start trusting the tools less. Adoption slips. By the time leadership notices, the problem has been building for months.

The budget mistake is treating the technology cost as the whole cost and ignoring the labor cost of keeping the technology valuable.

The fix is not complicated but it requires an explicit decision. Someone needs to own AI infrastructure in the same way someone owns IT infrastructure. That ownership needs a job description, not just goodwill. The time required needs to be in that person's workload, not in addition to it. And the budget conversation in year two needs to include a line item for maintenance labor, not just license renewal.

How much time is actually required depends on the scale of the deployment. For a 100-person company with a few AI tools that are genuinely embedded in workflows, I typically see meaningful AI infrastructure maintenance running about 10 to 15 hours per week across whoever is doing it. That is a real cost. In most companies I have looked at, nobody has formally accounted for it.

The organizations that sustain AI value over multiple years are the ones that recognized early that they were building infrastructure, not buying software. Infrastructure requires ongoing investment to remain useful. The ones that did not recognize this have a graveyard of AI tools that worked well in year one and quietly became unreliable in year two.

7 Decisions I Would Make Differently Before Connecting AI to Our CRM

Sumas Keller — Thu, 18 Jun 2026 10:45:24 +0000

The first time most companies connect AI to their CRM, the conversation usually revolves around productivity.

Faster account research.

Faster customer support.

Faster reporting.

Those benefits are real.

What often gets overlooked is everything happening underneath the workflow.

After speaking with teams implementing AI across CRM systems, I've noticed the same lessons appear repeatedly. If I were starting again, these are the seven areas I would examine before connecting anything.

1. Map every data category in your CRM before connecting AI.

Many organizations treat their CRM as a single system.

In reality, it often contains multiple categories of information:

• Customer contact data

• Deal notes

• Support interactions

• Contract information

• Pricing history

• Internal discussions

An AI assistant may not distinguish between these categories unless the architecture does.

What looks like one integration decision is often several access-control decisions hidden inside a single project.

2. Ask who can query which records, not just who can access the tool.

Tool access and data access are different things.

A user being allowed to open an AI assistant does not automatically mean they should be able to retrieve information from every CRM record through natural language queries.

One of the most common governance mistakes is assuming existing CRM permissions automatically extend to AI interactions.

That assumption should always be tested.

3. Think in rooms, not systems.

Most organizations structure permissions around applications.

I increasingly think they should structure permissions around contexts.

Sales teams need access to sales information.

Finance teams need access to financial information.

Legal teams need access to legal information.

The challenge begins when AI agents operate across those boundaries without clear isolation rules.

The most resilient architectures I've seen treat data domains separately and make it difficult for information to leak across operational contexts.

The question is not whether an AI agent can access information.

The question is whether it can access only the information it should.

4. Understand how AI access is logged.

Most organizations have some form of audit logging.

The important question is whether AI interactions appear inside those logs.

Can administrators see:

• Who submitted the query?

• Which records were accessed?

• What information was retrieved?

• When the retrieval occurred?

Without visibility, governance becomes significantly more difficult.

This matters even more in regulated industries where auditability is not optional.

5. Think beyond the CRM when responding to data requests.

Regulations such as GDPR create obligations around customer data visibility and management.

Once information moves beyond the source system into retrieval pipelines, indexes, search layers, and AI workflows, answering questions like:

"What data do you hold about me?"

may become more complicated than querying the CRM alone.

Organizations should understand exactly where customer information exists throughout the AI stack.

Data ownership becomes much harder when nobody knows where the data actually lives.

6. Make sure deletion workflows include the AI layer.

Deleting information from a CRM does not automatically guarantee it disappears from every AI-related component.

Depending on the architecture, information may also exist inside:

• Retrieval indexes

• Search layers

• Cached knowledge repositories

• Agent workflows

A deletion policy should cover the entire lifecycle of customer data, not just the source record.

Otherwise compliance processes may become difficult to verify.

7. Measure governance costs alongside productivity gains.

AI-assisted CRM workflows can create significant value.

Teams often report improvements in:

• Account preparation

• Customer research

• Opportunity analysis

• Internal knowledge discovery

The challenge is that governance work grows alongside those benefits.

Access control.

Auditability.

Data ownership.

Compliance reviews.

Approval workflows.

These operational costs are real and should be part of the business case from day one.

The organizations getting the most value from CRM-connected AI usually approach the project in a different order.

They treat it as a governance initiative first.

An AI initiative second.

That sequence tends to produce fewer surprises later.

One thing I've noticed is that governance becomes much easier when conversations, files, workflows, and AI agents operate within the same controlled environment rather than being scattered across multiple disconnected systems.

The more systems involved, the harder it becomes to understand:

• Who accessed what

• Where data lives

• Which permissions apply

• How actions are audited

• Whether sensitive information remains isolated

That's one reason why platforms built around data sovereignty, room-level isolation, auditability, and human-controlled workflows are attracting increasing attention from enterprise teams.

If you're exploring how modern organizations are approaching AI governance and privacy-first infrastructure, a useful starting point is:

https://privos.ai/

The conversation around AI usually starts with productivity.

The organizations seeing long-term success are increasingly starting with control.

9 Things I Got Wrong About Enterprise AI in Year One

Sumas Keller — Wed, 17 Jun 2026 14:53:48 +0000

I have been running AI initiatives for two years now. Year one was expensive in ways that did not show up on any budget line. Here is the honest version.

1. I thought adoption would follow deployment.
It does not. Deployment is a technical event. Adoption is a human event. They require completely different effort. I treated the go-live date as the finish line. It was the starting line.

2. I underestimated how much the pilot group skewed results.
The people who volunteered to test the tool were the most enthusiastic, most technically capable people we had. Their results were real and did not represent what would happen with the rest of the organization. I presented pilot results to the board as if they were predictive. They were not.

3. I did not define what success looked like before we started.
Six months in, when the CFO asked if the investment was working, I did not have a clean answer. Not because the tool was not working but because I had never made explicit what "working" meant in measurable terms. That conversation was uncomfortable and avoidable.

4. I let the vendor run the training program.
The vendor is motivated to show features. My team needed to learn workflows. These are different things. The training we got was a product tour. What we needed was workflow redesign support. I should have run the training myself with the vendor as a resource.

5. I assumed integration meant connection.
We connected the AI tool to our data sources. I called that integration. What I had was a technical connection, not an operational integration. The tool could access the data but nobody had designed when, why, or how it would actually be used in context. Real integration requires workflow design, not just API setup.

6. I ignored the manager layer.
Individual contributors either adopted or did not based on personal motivation. Managers stayed neutral because I had given them no reason to care. AI adoption without manager engagement is adoption that stalls at the enthusiast population and never reaches the mainstream.

7. I did not model the exit.
When one of our early tools lost a key enterprise feature after a pricing restructure, I discovered that we had built processes around that feature with no plan for what to do if it went away. Now I model the exit scenario before signing. What does leaving look like, what does it cost, what do we lose.

8. I treated security as a procurement checkbox.
Legal reviewed the DPA. IT checked the security certifications. Nobody mapped the actual data flows at the operational level, which documents were being sent where, under what conditions, retained for how long. That mapping happened during a compliance review sixteen months into the deployment. It should have happened on day one.

9. I confused activity with impact.
Usage metrics went up. I reported that as progress. Usage is an input metric. The output metrics, the ones that connect AI activity to business results, took much longer to define and instrument. By the time I had clean output data, I had already made three more tool decisions based on input metrics alone.

Year two has been better, largely because year one was honest enough to learn from.

The Reference Check Questions Nobody Asks AI Vendors

Sumas Keller — Tue, 16 Jun 2026 15:53:54 +0000

I have done this process wrong more times than I would like to admit.

You call the references the vendor sends you. Three customers, all happy, all articulate, all saying roughly the same things. You hang up feeling good. You sign. Six months later you are dealing with a support team that responds every 72 hours and a renewal quote that is 40% higher than year one.

The reference check told you nothing useful. Not because the customers lied. Because you asked the wrong questions.

Here is what I ask now.

"How many people at the vendor have you spoken to in the last six months?"

One contact who handles everything and is sometimes slow — that tells you something. A team of people across sales, technical support, and leadership — that tells you something different. Enterprise AI vendors with thin account teams show their limitations at exactly the moment you need them most: when something breaks at the worst possible time.

"Tell me about the last time something broke in production."

Not IF something broke. Something always breaks. I want to know what happened next. Did the vendor show up? Did they communicate clearly while the issue was live? Did they follow up after closing the ticket, or did they close it and disappear?

The answer to this question tells me more about a vendor than any product demo.

"What do you know now that you wish you had known before signing?"

This question works because it is framed as advice, not criticism. Reference customers who would never say "this product has problems" will happily answer this one honestly. Listen for anything about pricing surprises, scope limitations that only appeared after deployment, or feature gaps the sales team glossed over.

"When you renewed, did you evaluate alternatives?"

Renewal is the honest signal. A customer who renewed without looking elsewhere is genuinely satisfied. A customer who looked at three other options and came back is someone who chose this vendor over real competition. A customer who is approaching renewal and is not sure is the most valuable reference call you can make — if you can get the vendor to put them on the list, which they will not.

"How does it work for someone who joined the team after the initial rollout?"

The initial deployment had vendor support, internal champions, and everyone paying close attention. The ongoing experience for employees who joined later and had to learn the tool on their own is the real daily experience. This is where most enterprise AI tools quietly underperform their pilot results.

None of these questions are aggressive. A vendor whose customers have good experiences will answer them confidently. A vendor whose customers have mixed experiences will give you information that changes what you negotiate before signing.

Both outcomes are better than what you get from asking "do you like the product?"

How I Build AI Usage Policies People Actually Follow

Sumas Keller — Mon, 15 Jun 2026 17:42:00 +0000

Most AI usage policies fail for a simple reason.

They are written by legal teams.

Then handed to employees.

Then forgotten.

A month later, nobody remembers where the document lives.

Three months later, people are doing whatever they want anyway.

I've seen this happen multiple times.

The problem isn't that employees dislike rules.

The problem is that most AI policies are impossible to use in real work.

A policy only works if people can remember it while they're busy.

That's the standard I use.

Not whether the policy sounds impressive.

Whether a normal employee can apply it at 4:30 PM on a Friday.

The Biggest Mistake: Writing For Auditors Instead Of Employees

Many AI policies read like compliance documents.

Pages of definitions.

Pages of legal language.

Pages of edge cases.

Technically correct.

Operationally useless.

Employees don't need a 20-page document when deciding whether they can paste information into an AI tool.

They need a simple answer.

Can I do this?

Yes or no?

The more effort required to find the answer, the less likely people are to follow the policy.

Rule #1: Classify Information Before You Classify Tools

Most companies start with tools.

I start with information.

Because tools change.

Data doesn't.

I typically group information into three buckets:

Public information

Examples:

marketing content
public product documentation
published research
public website content

Internal information

Examples:

internal processes
project plans
meeting notes
operational documents

Sensitive information

Examples:

customer records
financial data
contracts
employee information
security documentation

Once employees understand these categories, tool decisions become much easier.

The question becomes:

"What information am I sharing?"

Not:

"Which AI product am I using?"

Rule #2: Make The Safe Path The Easy Path

People naturally choose the fastest option.

Good policies acknowledge this reality.

Bad policies fight it.

If employees must jump through ten steps to use an approved AI solution, many won't.

Instead, they'll find a shortcut.

That's how shadow AI starts.

I've learned that enforcement is much easier when approved tools are more convenient than unapproved tools.

Convenience is a governance tool.

Not just a product feature.

Rule #3: Focus On High-Risk Behaviors

Another mistake I see is trying to regulate everything.

That approach rarely works.

Instead, identify the behaviors that actually create risk.

For most organizations, those include:

uploading customer data
uploading contracts
sharing credentials
exposing financial information
exposing internal security information

Those deserve attention.

Whether someone uses AI to rewrite a meeting summary usually doesn't.

Good policies prioritize.

Great policies prioritize aggressively.

Rule #4: Give Employees Examples

Examples are more useful than rules.

Compare these two approaches.

Approach A:

"Do not upload confidential information."

Approach B:

"Do not upload customer contracts, financial statements, employee records, or unpublished product roadmaps."

The second version is far easier to follow.

People remember examples.

They rarely remember policy language.

Whenever possible, I replace abstract rules with concrete scenarios.

Rule #5: Assume Policies Will Be Ignored

This sounds pessimistic.

It's actually practical.

Every policy should assume occasional mistakes.

That means organizations need:

logging
monitoring
approval workflows
permission controls
audit capabilities

Policies reduce risk.

Systems enforce risk.

Both are necessary.

A document alone has never protected a company.

Rule #6: Review Policies More Often Than You Think

AI changes quickly.

What was reasonable six months ago may already be outdated.

I've seen companies create AI policies once and never revisit them.

That's risky.

A simple review cycle works much better.

Quarterly is usually enough.

The goal isn't rewriting the entire policy.

The goal is keeping it aligned with reality.

Because reality changes.

Fast.

What Good AI Governance Actually Looks Like

Good governance doesn't feel restrictive.

It feels clear.

Employees know:

what they can do
what they cannot do
which tools are approved
what data is sensitive
who to ask when unsure

When those answers are obvious, adoption becomes easier.

And risk becomes lower.

That's the outcome most organizations want.

My Take

If employees constantly ask whether they're allowed to use AI, the policy is probably too complicated.

If nobody reads the policy, it's definitely too complicated.

The best AI usage policies aren't the longest.

They're the easiest to remember.

Because a policy only creates value when people actually follow it.

And people only follow policies they understand.

Dedicated AI Team vs Embedding AI in Existing Teams: How to Make the Right Call

Sumas Keller — Fri, 12 Jun 2026 17:06:06 +0000

There is a structural decision hiding inside every enterprise AI initiative that rarely gets the explicit attention it deserves: where does AI capability live organizationally?

The two dominant models are a centralized AI team, a dedicated function that owns AI strategy, tooling, and deployment across the organization, and a distributed model where AI capability is embedded within existing business units. Each model works in specific organizational contexts and fails in others. Getting this decision wrong adds friction to AI adoption that no amount of technology investment can fix.

I have seen both approaches succeed and both approaches fail. The difference is almost always about fit with organizational structure, not about which approach is inherently superior.

The Case for a Centralized AI Team

Centralization makes sense when AI deployment requires technical depth that is scarce and expensive.

Building enterprise AI infrastructure, self-hosted inference, retrieval pipeline engineering, integration development, security architecture, requires skills that are not evenly distributed across business units. A legal team cannot maintain an embedding model. A finance team cannot debug a RAG retrieval failure. When the technical work of AI deployment is genuinely specialized, concentrating that expertise in a dedicated team prevents the alternative: every business unit attempting to hire the same scarce skills and most of them failing.

Centralization also makes sense when consistency and compliance are paramount. If every business unit deploys AI independently, you get different security configurations, different data handling practices, different prompt engineering quality, and different audit logging completeness. For organizations in regulated industries or with strong compliance requirements, the compliance overhead of governing fifteen independent AI deployments is significantly higher than governing one centralized deployment that serves fifteen use cases.

The failure mode of centralization is distance from the business. A central AI team that operates as an internal service provider, receiving requests, building solutions, delivering them, develops a different understanding of business needs than teams embedded in day-to-day operations. The quality of the AI solutions they build is limited by the quality of the requirements they receive, and requirements translated across organizational distance are reliably incomplete.

The Case for Embedding AI in Existing Teams

Distribution makes sense when the value of AI is highly context-dependent and that context lives within business units.

Customer success teams understand customer interaction patterns in ways that no centralized team can replicate from the outside. Sales teams understand deal dynamics. Engineering teams understand their own development workflows. When the AI augmentation that matters most is deeply specific to a team's domain, the people best positioned to identify, deploy, and refine it are the ones doing the work.

Embedded AI also creates faster feedback loops. When the person responsible for AI deployment is the same person using it daily, the iteration cycle from "this isn't working" to "we changed the prompt and now it works" is hours, not weeks. Centralized teams that serve many stakeholders cannot iterate at this speed.

The failure mode of distribution is fragmentation. When every team runs its own AI deployment, the organization loses visibility into what is working across the company, cannot enforce consistent data handling standards, and duplicates engineering effort. The fourth team to solve the same integration problem because nobody told them the first three had already solved it is a real and common outcome.

The Hybrid That Actually Works

Most mature enterprise AI organizations land on a hybrid: a small central platform team that owns infrastructure, security architecture, and compliance, combined with embedded practitioners in business units who own use cases and user experience.

The platform team is responsible for the things that should be consistent: the self-hosted inference infrastructure, the security sandbox, the audit logging framework, the data handling standards, the integration patterns. They build the rails.

The embedded practitioners are responsible for the things that should be context-specific: the retrieval configurations tuned to their domain's vocabulary, the prompts calibrated for their team's use cases, the workflows designed around their actual processes. They run the trains.

This model works because it allocates decisions to the people best positioned to make them. Data governance decisions belong at the center, where the compliance requirements can be seen across the full organization. Use case design decisions belong in the business units, where the operational context lives.

The failure mode of the hybrid is unclear boundaries. When it is ambiguous whether a decision belongs to the platform team or to the business unit, both parties default to inaction or conflict. The hybrid requires a clear interface: here is what the platform team owns, here is what the business unit owns, and here is how conflicts are resolved.

How to Choose

Four questions help identify which model fits your organization.

How technically scarce is AI expertise in your current organization? If you have one or two people who can build and maintain AI systems and they cannot be distributed across business units without losing effectiveness, start centralized and distribute as you hire.

How variable are your AI use cases across business units? If marketing, sales, legal, and finance all need fundamentally different AI capabilities, the embedded model delivers more use-case-specific value. If the use cases are similar enough that a single deployment serves most of them, centralization is more efficient.

How strong is your compliance requirement for consistency? Regulated industries and enterprise companies with strong data governance requirements have a higher bar for centralized oversight. The compliance cost of governing distributed deployments rises faster than the business value in these environments.

What is your current organizational change appetite? A centralized AI team is a new organizational entity that requires budget, headcount decisions, and reporting line clarity. An embedded model can start as a responsibility addition to existing roles. If the organization cannot absorb a new team right now, the embedded model with a lightweight coordination function is the pragmatic starting point.

The answer to these four questions, taken together, points to a model with more confidence than any general preference for centralization or distribution. The right structure is the one that fits the organization you actually have, deploying the technology that actually exists, toward the outcomes that actually matter.

Build vs. Buy AI Capability: The Decision Framework I Would Use Before Spending Budget

Sumas Keller — Thu, 11 Jun 2026 17:50:09 +0000

The build vs. buy question is usually asked too late.

It often appears after the company is already emotionally committed.

Engineering wants to build.

Business teams want something fast.

Finance wants cost clarity.

Security wants control.

Leadership wants progress.

Then the debate becomes political.

That is the wrong way to decide.

Build vs. buy should not be a personality contest between technical pride and commercial urgency.

It should be an operating decision.

The real question is:

Which path gives the company the capability it needs with the right balance of speed, control, cost, risk, and maintainability?

Here is the framework I would use before spending budget.

1. Start with the workflow, not the technology.

Do not begin by asking:

“Should we build our own AI system?”

Start with:

“Which business workflow are we trying to improve?”

That workflow might be:

customer support triage
sales research
compliance evidence collection
internal knowledge search
document review
reporting automation
project status summarization
onboarding assistance
contract analysis

The workflow determines the decision.

A generic workflow often favors buying.

A highly specific workflow may justify building.

A sensitive workflow may require a hybrid approach.

Without workflow clarity, build vs. buy becomes abstract and wasteful.

The company should not decide whether to build AI before it knows what work the AI is supposed to improve.

2. Score workflow specificity.

The first dimension is specificity.

Ask:

Is this workflow common across many companies?
Is the process standard or unique to us?
Are the rules simple or deeply business-specific?
Does the workflow require proprietary logic?
Would a vendor’s default workflow cover 80 percent of the need?

If the workflow is common, buying usually makes sense.

If the workflow is deeply tied to how the company operates, building may create more value.

Example:

Common workflow: summarize meeting notes.
Decision: buy.

Specific workflow: evaluate customer risk based on internal account history, legal exceptions, renewal timing, and regional compliance rules.
Decision: consider build or heavily customized platform.

The more the workflow reflects your company’s internal operating model, the more careful you should be with off-the-shelf tools.

3. Score data sensitivity.

AI decisions change when sensitive data is involved.

Ask:

Does the workflow use customer data?
Does it use employee data?
Does it touch contracts?
Does it include financial records?
Does it involve legal notes?
Does it contain confidential strategy?
Does it include regulated information?

Low-sensitivity use cases can move faster.

High-sensitivity use cases need stronger control.

This does not always mean build.

It means the buying criteria become stricter.

For sensitive workflows, a bought solution may still work if it offers strong deployment control, permissioning, audit logs, data retention controls, and clear subprocessors.

But if the vendor cannot meet the data-control requirement, buying becomes risky.

The more sensitive the data, the more control becomes part of the ROI calculation.

4. Score internal talent honestly.

This is where many companies lie to themselves.

They say:

“We can build this internally.”

Maybe.

But can they build it, secure it, maintain it, document it, monitor it, improve it, support users, handle failures, and keep up with model changes?

Building AI capability is not only model selection.

It may require:

data engineering
backend architecture
prompt and evaluation design
security engineering
access control
monitoring
user experience
MLOps
integration work
governance
ongoing support

If the internal team does not have the capacity, building may create technical debt disguised as strategic control.

The question is not:

“Can we build a prototype?”

The question is:

“Can we own this as production infrastructure?”

A prototype proves possibility.

Production proves responsibility.

Those are not the same thing.

5. Score speed-to-value.

Some AI use cases need speed.

Others justify patience.

Ask:

Is the business pain urgent?
Is there an immediate cost or capacity issue?
Is the market moving quickly?
Is the workflow blocking revenue?
Is the use case experimental?
Can we wait three to six months?
Do we need a working solution this quarter?

If the company needs value quickly, buying or using a platform may be better.

If the capability is strategically important and durable, building may be worth the wait.

But do not confuse impatience with strategy.

Sometimes teams want to build because it feels more impressive.

Sometimes teams want to buy because they want to avoid hard architecture work.

Both instincts can be wrong.

Speed matters, but speed without ownership can become expensive later.

6. Score maintenance burden.

Every AI system has a second cost after launch.

Maintenance.

This includes:

model updates
prompt updates
evaluation changes
integration failures
data-source changes
permission changes
bug fixes
user support
security patches
compliance updates
monitoring
documentation

Bought platforms absorb some of this.

Built systems push more of it onto the company.

That can be fine if the capability is strategic.

But if the use case is not core, internal maintenance can become a bad trade.

A simple rule:

Build only what you are willing to maintain.

If the company does not want to maintain it, buying is probably the honest answer.

7. Score strategic control.

Not every AI capability deserves strategic control.

Some workflows are supporting workflows.

Others are central to how the company competes.

Ask:

Does this capability create real differentiation?
Would competitors use the same vendor in the same way?
Does this workflow contain proprietary operating knowledge?
Would owning the system improve our long-term advantage?
Would vendor dependency create risk?
Would switching vendors be painful later?

If the AI capability is core to competitive advantage, building or deeper customization may be justified.

If it is a standard productivity layer, buying is likely better.

A company should not build everything.

But it should think carefully before outsourcing the parts that define how it operates.

The real build decision is not about pride. It is about what the company needs to own.

8. Use a simple decision table.

Here is the decision model I would use.

Buy when:

the workflow is common
the data sensitivity is manageable
speed matters
internal AI talent is limited
the vendor has strong controls
the workflow is not strategic differentiation
maintenance would distract the team

Build when:

the workflow is highly specific
the data is sensitive
the capability is strategic
internal talent exists
long-term control matters
vendor limitations would constrain the business
the company is prepared to maintain it

Use a hybrid approach when:

the workflow is important but not fully unique
the company needs faster deployment
sensitive data requires deployment control
the vendor can provide infrastructure but the company needs custom logic
internal teams can own configuration and governance

Hybrid is often the realistic answer.

Most companies do not need to choose between total dependency and total internal build.

They need to decide which layers to own and which layers to buy.

9. Do not forget exit cost.

Whether you build or buy, ask how you exit.

If you buy:

can you export data?
can workflows be migrated?
can prompts and agents be moved?
can logs be retained?
can the contract be ended cleanly?
can another vendor replace it later?

If you build:

can the system be handed over?
is it documented?
can another team maintain it?
can it survive staff turnover?
can components be replaced?
does the system depend on one engineer’s memory?

Exit cost is part of total cost.

If the company cannot exit cleanly, the decision carries hidden risk.

A vendor can lock you in.

An internal build can lock you in too.

The lock-in just looks different.

Final decision rule

Build vs. buy is not a moral question.

Building is not always smarter.

Buying is not always lazy.

The right decision depends on workflow specificity, data sensitivity, internal talent, speed-to-value, maintenance burden, and strategic control.

The simplest rule is this:

Buy for speed and standard capability. Build for strategic control. Use hybrid when the workflow is important but the company cannot afford to start from zero.

The worst choice is not building or buying.

The worst choice is pretending the decision is only about software cost.

It is not.

It is about what the company wants to own, what it can realistically operate, and what kind of risk it is willing to carry.

Your AI Budget Should Be Calculated Per Workflow, Not Per Seat

Sumas Keller — Wed, 10 Jun 2026 13:01:58 +0000

The per-seat licensing model made sense for software categories where every employee uses a tool in roughly the same way. Email. Office productivity. Communication platforms. Everyone uses it, everyone benefits roughly proportionally, you multiply by headcount and the math is simple.

AI tools are not like this, and pricing them like they are leads to three predictable problems.

You overpay for employees whose workflows don't benefit significantly from AI capability. You underinvest in the workflows where AI could deliver disproportionate value. And you end up with budget conversations that focus on seat count rather than on outcomes — which means the organization optimizes for adoption metrics rather than impact metrics.

The alternative is workflow-based AI budgeting. It takes more work upfront. It produces better decisions and better outcomes.

Why Per-Seat Fails for AI

Consider a 200-person company. Of those 200 people, roughly 40 are in roles where AI assistance on knowledge work — drafting, research, analysis, summarization — could save meaningful time daily. Another 60 are in operational roles where AI assistance would provide moderate benefit. The remaining 100 are in roles where AI tools would see low utilization.

Under per-seat pricing, the organization pays for 200 seats to deliver value primarily to 40 people. The 160 people with marginal benefit subsidize the 40 people with high benefit. The cost-per-unit-of-value is inflated by a factor of 4 or 5.

More problematically, the budgeting conversation that results focuses on whether to buy licenses for more or fewer people, rather than on which workflows should be supported and at what quality level. The tool becomes a headcount decision rather than a workflow decision, and the investment thesis gets lost in a conversation about utilization rates.

The Workflow-Based Framework

Workflow-based AI budgeting starts from a different question: which specific workflows in our organization would benefit most from AI augmentation, and what is the value of that augmentation?

This requires mapping workflows to value, which is more work than multiplying headcount by a seat price. It is also more honest.

The mapping process has three steps.

Step one: identify high-value AI augmentation candidates. These are workflows characterized by high volume of repeatable cognitive tasks — drafting, research, classification, summarization, extraction — performed by employees whose time is expensive. Customer-facing roles, analytical roles, and management roles with significant documentation requirements are typical high-value candidates.

Step two: estimate the value of augmentation for each candidate workflow. This requires being specific about what AI changes in the workflow: which steps get faster, by how much, and with what quality implications. Use conservative estimates. The goal is to identify workflows where the value of AI augmentation clearly exceeds the cost, not to build the most optimistic possible business case.

Step three: calculate the budget required to support those specific workflows well. This is different from calculating the cost of company-wide deployment. It is the cost of deploying the right tools, at the right quality level, for the workflows that actually justify the investment.

What This Reveals That Per-Seat Budgeting Hides

When organizations go through this exercise honestly, two things typically emerge.

The workflows with the highest AI value often require more than a standard SaaS license. Deep integration with specific data sources, custom retrieval configuration, specific security requirements, workflow-specific agent behavior — high-value AI augmentation for knowledge-intensive workflows often requires a more substantial investment than a commodity per-seat price implies. The per-seat model underinvests in the workflows that matter most.

The workflows with the lowest AI value don't need an enterprise license at all. Consumer AI tools, which cost a fraction of enterprise licensing, are adequate for incidental AI use that doesn't involve sensitive data. The per-seat enterprise model overinvests in low-value use cases.

The result of the workflow-based approach is typically a portfolio: enterprise-grade AI investment concentrated in the workflows where it matters, lower-cost solutions for peripheral use, and explicit decisions about which workflows are not worth AI investment at any price.

The Vendor Stability Factor in Workflow Investment

There is a risk dimension to workflow-based AI investment that the per-seat model partially obscures.

When you build AI augmentation deeply into a high-value workflow — when the AI is integrated with your data, your processes, your team's habits — you are creating a dependency. That dependency is worth creating when the value is real and the vendor is stable. It is a liability when the vendor's longevity is uncertain.

Before committing to deep workflow integration with any AI vendor, verify their organizational stability as part of the investment decision. Crunchbase profiles — like the one available for PrivOS at crunchbase.com/organization/privos — give useful starting context on team depth and company history, supplemented by reference checks with current customers. A vendor who appears in your workflow ROI calculation as a 3-year investment deserves 3-year vendor due diligence.

The per-seat model encourages shallow integration across many workflows, which limits this risk. The workflow model encourages deep integration in high-value workflows, which requires taking the vendor stability question seriously.

Implementing the Shift

Moving from per-seat to workflow-based AI budgeting requires two changes in how the organization thinks about technology investment.

The first is treating AI as operational investment rather than software procurement. Per-seat pricing fits the software procurement model, where you buy a capability and distribute it. Workflow-based investment fits the operational model, where you identify a value opportunity, design a solution, and invest in proportion to the expected return.

The second is accepting that not all employees need the same AI tool, or any enterprise AI tool at all. This is culturally uncomfortable in organizations that have normalized uniform technology access — where everyone gets the same software stack regardless of their workflow needs. But it is the correct framework for an investment category where the value is highly concentrated in specific use case types.

The organizations that get this right will deploy AI resources where they actually compound. The ones that default to per-seat models will spend more, measure less, and wonder why the ROI never materialized at the scale they expected.

What I'd Fix First If I Inherited a 200-Person Company's Digital Ops Tomorrow

Sumas Keller — Tue, 09 Jun 2026 10:51:47 +0000

Let me be honest about something: most digital operations transformations are done in the wrong order.

I've walked into enough companies mid-transformation to have a strong opinion about this. The instinct, almost universally, is to start with the exciting parts. New tools, new AI capabilities, new automation. The org chart gets a Chief Digital Officer. The tech vendors get invited in for demos. The roadmap gets built around capabilities.

And then, six to twelve months later, the transformation is stalling — not because the technology failed, but because the foundation wasn't there to absorb it.

If someone handed me the keys to a 200-person company's digital operations tomorrow and asked me to fix it, I would do four things first — none of which involve buying anything.

First: I'd Run a 48-Hour Audit of What Already Exists

Not a six-week consulting engagement. Forty-eight hours of focused discovery.

I want to know three things specifically: what tools are currently in use (including the ones nobody officially approved), where the single sources of truth actually live versus where they're supposed to live, and where the most painful coordination failures happen on a weekly basis.

The gap between the official tool stack and the actual tool stack is always informative. Shadow tools exist because the official tools are failing someone. Those failures are data. The tools that got adopted without permission tell you more about real operational needs than any requirements document.

I'd spend one day talking to people in sales, support, finance, and engineering — not their managers, the people doing the work — asking one question: what's the most annoying part of your workday that involves finding or sharing information? The answers cluster around three or four root causes in almost every organization.

Second: I'd Identify the Coordination Seams

Every organization has seams — points where information has to cross from one system, team, or workflow to another. These seams are where fragmentation costs concentrate.

The handoff from sales to customer success. The status update from engineering to product. The project brief from marketing to design. The budget request from a department to finance.

Each seam that requires a person to manually translate between systems — pull data from one place, format it, put it somewhere else — is a coordination cost. In a 200-person company, there are typically 10 to 15 significant seams. Each one is consuming somewhere between 2 and 15 hours per week across the people involved.

I'd map those seams on a whiteboard. Not to immediately fix them — to understand where the highest-leverage interventions are. The seam that costs the most time and creates the most errors is the right place to start. Not the most technically interesting problem. The most expensive one.

Third: I'd Talk to the People Who've Given Up

Every organization has them: people who tried to improve processes, got blocked, and stopped trying. They're sitting on a goldmine of institutional knowledge about what doesn't work and why.

These are usually not the loudest voices in any room. They've learned that raising operational problems doesn't lead to solutions. They've adapted to the dysfunction and quietly built workarounds that nobody else knows about.

I'd find three or four of them and ask: what did you try to fix, what happened, and what are you doing instead now? Their workarounds are usually better solutions than the official process, implemented at the individual level because the organizational level was too slow to move.

The answers tell you two things. What the actual failure modes are, without the political filtering that happens when problems get escalated through management layers. And whether there's organizational capacity to actually implement change, or whether the change-blocking mechanism itself needs to be addressed first.

Fourth: I'd Set One Metric That Everyone Understands

Before touching a single tool, before running a single vendor evaluation, before writing a single line of a transformation roadmap, I'd establish one operational metric that measures the thing we're actually trying to improve.

Not a laundry list of KPIs. One metric. Something like: average time from customer request to first substantive response. Or: percentage of projects delivered on the original scope without scope change meeting. Or: average time a new employee takes to reach independent operation.

The metric has to be specific, measurable from existing data, and visibly connected to business outcomes. Not a vanity metric that IT can report on — a metric that a COO would care about and that a frontline employee can understand their contribution to.

Everything that comes after — the tool decisions, the process redesign, the AI investments — gets evaluated against that metric. Not "did we deploy the technology" but "did the number move."

Without this anchor, digital transformation becomes a series of technology deployments without a coherent theory of what better looks like. With it, every decision has a clear test.

What Comes After

Once I've done those four things — usually in the first two weeks — I have a very different picture than the one I'd get from reading the existing strategic plan.

I know where the actual pain is. I know who has institutional knowledge worth preserving. I know what the political dynamics are. I know what one metric will tell me if I'm making progress.

Then, and only then, would I start evaluating what to change. Because the worst thing that can happen to a digital transformation is that it solves the wrong problems quickly. The tools are fast to deploy. The organizational dysfunction they're deployed into takes much longer to fix.

Getting the diagnosis right first is the only approach that consistently leads to transformation that actually sticks.

The AI Adoption Curve Nobody Talks About: Why Month 3 Is When Most Rollouts Stall

Sumas Keller — Mon, 08 Jun 2026 11:06:59 +0000

Enterprise AI pilots succeed. Enterprise AI deployments stall. The gap between the two is almost always a management problem, not a technology problem.

There's a pattern in enterprise AI adoption that I've watched repeat across organizations of different sizes, industries, and geographies.

Month one: high enthusiasm. The pilot group shows impressive results. Leadership is optimistic. The vendor is engaged and responsive. Productivity metrics, where they exist, show improvement.

Month three: the numbers flatten. Adoption outside the pilot group is slower than projected. Some early adopters have quietly reverted to their previous workflows. The engineering team is managing more exceptions and edge cases than expected. The business case is starting to look less certain.

Month six: the deployment is technically still active, but it's no longer receiving executive attention. It's running on momentum rather than intention. The question "is this working" is no longer being asked, partly because nobody wants to hear the answer.

This is the AI adoption stall. It's not caused by the AI. It's caused by the organizational dynamics around the AI.

Why the Pilot Always Succeeds

Enterprise AI pilots succeed for reasons that don't automatically scale.

The pilot group is selected for enthusiasm or aptitude. The people who test new tools in pilots are, disproportionately, the people who are most interested in new tools. They represent the left tail of the adoption curve, not the mean.

The pilot receives disproportionate support. Pilot participants get hands-on vendor training, quick response to issues, and organizational attention. The rest of the organization will receive a rollout package and an FAQ document.

The pilot uses representative but not fully realistic conditions. Edge cases, legacy data issues, and workflow integration problems are managed during the pilot. During broad rollout, they surface faster than the team can address them.

Pilot success answers the question "can this work in optimal conditions." It doesn't answer the question "will this work when deployed to people who didn't volunteer, with real data, and standard support."

What Actually Happens at Month Three

By month three, the deployment has moved past the self-selected early adopters into the mainstream of the organization. This group has different characteristics.

They didn't ask for this tool. It was given to them. Their motivation to invest in learning it is lower, their tolerance for friction is lower, and their tendency to revert to familiar workflows when the tool adds difficulty is higher.

They have existing workflows that work. Not perfectly, but adequately. The activation energy required to change a workflow that works is higher than the activation energy required to adopt a workflow when you have nothing.

The tool hasn't been optimized for their specific use cases. The pilot was optimized for the pilot group's workflows. The mainstream users have different tasks, different document types, different queries. The retrieval quality, prompt calibration, and workflow integration that worked for the pilot may not work as well for them.

Middle managers are not supportive, or are actively neutral. This is the most consistent factor in stalled deployments. If people managers don't actively encourage and model AI tool adoption, their teams won't adopt. If managers haven't been given a reason to care — if AI adoption isn't in their objectives, isn't discussed in their performance conversations, isn't visibly a priority to their own managers — neutrality is the rational response.

The Management Failure at the Center of Most Stalls

Most enterprise AI deployments are launched with a technology frame and fail on a management problem.

The technology questions get extensive attention: which model, what integrations, what security configuration, what prompts. The management questions get minimal attention: who is responsible for driving adoption, how are managers being equipped to support their teams, what does success look like for each team rather than for the aggregate, what happens when someone says it's not working.

The result is a deployment that is technically functional but organizationally unsupported.

Sustainable AI adoption requires middle management to become advocates, not just users. That requires three things that most deployments don't provide:

A clear narrative about why this matters for their team specifically. "We deployed this to improve productivity" is not a useful narrative for a finance manager. "We deployed this so your team can do quarterly close analysis in two hours instead of two days" is a useful narrative.

A use case that works well for their context. Generic deployments deliver generic results. Teams that see demonstrable value in their specific workflow adopt. Teams that don't see specific value don't.

Cover to require adoption. Managers need organizational permission to make AI adoption part of how their team operates, not just something available if people want it. Without explicit organizational priority, requiring adoption feels overreaching. With it, it's responsible management.

What a Deployment That Doesn't Stall Looks Like

The deployments that achieve sustained adoption — past month three, past month six, into genuine organizational capability — share a set of practices that aren't about technology.

Team-level use case definition before deployment. Before the tool goes live for any team, someone spends time with that team understanding their workflows and defining the two or three specific use cases where the tool will deliver clear value for them. The tool is deployed with those use cases activated and the team trained specifically on them, not on the general capabilities of the system.

Manager adoption tracked and managed. Manager adoption is measured separately from individual contributor adoption. Managers who are not using the tool or supporting their teams' use are identified and supported specifically. This isn't punitive — it's recognizing that manager behavior predicts team behavior.

Feedback loops that inform iteration. A formal mechanism for users to flag what's not working, with a clear commitment about who reviews that feedback and what happens as a result. Deployments that can't absorb and respond to user feedback lose user trust, which accelerates abandonment.

Explicit checkpoints at 30, 60, and 90 days. Not to declare success or failure, but to ask honestly: what's working, what's not, and what needs to change. Organizations that build checkpoints in before deployment make better decisions than organizations that assess retrospectively after the investment has been made.

The Question to Ask Before You Deploy

Before your next AI deployment, ask this: if the rollout hits resistance from middle management in month two, what is the plan?

If the answer is "we'll deal with it then," you're likely on the path to a stalled deployment.

If the answer is a specific set of interventions — manager training, use case support, adjusted timeline, escalation path — you're building an adoption strategy, not just a technology deployment.

The technology part of enterprise AI is increasingly solved. The adoption part is not. The organizations that figure out the latter will pull ahead of the ones still focused on the former.

The Metrics That Actually Tell You If Your Enterprise AI Rollout Is Working

Sumas Keller — Thu, 04 Jun 2026 15:39:01 +0000

The Metrics That Actually Tell You If Your Enterprise AI Rollout Is Working

Time saved is not a metric. It's a hypothesis. Here's how to measure what actually matters.

Enterprise AI deployments generate a lot of optimistic claims and very little rigorous measurement.

The claims are consistent: this tool saves X hours per week, this agent reduced process time by Y percent, employees are more productive. The measurement behind those claims is almost always anecdotal, self-reported, and uncorrected for the costs that show up elsewhere.

I've been through enough AI rollout reviews to know what rigorous measurement looks like — and how rare it is.

Here's the framework I'd use.

The Problem With "Time Saved" as a Metric

Time saved is the metric that appears in almost every AI ROI calculation. It's also one of the least reliable metrics available.

The problems:

Self-reported time savings are systematically overstated. When employees are asked how much time they save with a new tool, they report the most memorable time-saving moments, not the average. The cognitive work of learning the tool, correcting AI errors, and managing new workflows doesn't enter the calculation.

Saved time doesn't automatically become productive time. If an employee saves 30 minutes per day on drafting emails, that 30 minutes may be redirected to higher-value work. Or it may be redirected to checking Slack more often. The ROI calculation assumes the first; reality often delivers the second.

The denominator changes. As AI tools become standard, the baseline expectation of output increases. What saved time at tool adoption becomes the new normal productivity expectation within 12-18 months. The "savings" get absorbed into rising expectations rather than returning to the bottom line.

None of this means AI tools don't create value. They do. But measuring that value requires metrics that are observable, not self-reported, and that account for the full cost equation.

Metrics That Actually Measure AI Value

Output volume with quality gates

Instead of measuring time saved, measure output volume and apply quality gates.

For a content team using AI writing assistance: how many pieces of content are produced per week, at what quality threshold (defined by editorial review pass rates, not impressions)? Track this before deployment, during rollout, and at 90-day intervals afterward.

This measures actual output impact rather than assumed time savings.

Process cycle time — measured, not estimated

For AI agents handling defined workflows (contract review, support ticket triage, expense categorization), measure actual cycle time end-to-end. Not estimated, not self-reported — measured via system timestamps.

If AI contract review was supposed to reduce cycle time from 5 days to 2 days, pull the timestamp data. This is a binary metric: the cycle time either changed or it didn't.

Error rates and rework volume

AI tools often shift where errors occur rather than eliminating them. An AI that drafts documents quickly but introduces factual errors that require correction doesn't save time — it shifts time from drafting to reviewing and correcting.

Measure error rates and rework volume before and after deployment. For critical workflows, this metric is more important than speed.

Tool consolidation actuals

If the AI deployment was justified partly on consolidation — replacing other tools — verify that the other tools were actually deprecated and their licenses cancelled.

This sounds obvious. In practice, most AI tool adoptions layer onto existing stacks rather than replacing components of them. If you deployed an AI project management assistant and still have the same number of project management tool licenses six months later, the consolidation ROI in your business case hasn't materialized.

Support and escalation rates

For customer-facing AI applications, support escalation rate is an important quality signal. If AI-handled interactions require human escalation at 30%, the time savings from automation are partially offset by escalation handling costs.

Track escalation rates over time. A declining escalation rate indicates the AI is improving in effectiveness. A rising rate indicates quality degradation that needs attention.

The Cost Side Requires Honest Accounting

Most AI ROI calculations measure benefits against license cost. The cost side needs to include:

Correction and oversight labor. For any AI system that produces outputs requiring human review, measure the actual time spent reviewing and correcting. This cost often gets attributed to "the reviewer's normal job" and disappears from the calculation.

Prompt maintenance. For AI tools that require ongoing prompt tuning, measure the engineering time spent on prompt iteration. This cost is real and grows as the system's use cases expand.

Integration maintenance. When upstream systems change — CRM fields are renamed, data schemas update, APIs version — AI integrations require maintenance. Track this time.

False confidence cost. This is the hardest cost to measure but often the most significant: decisions made based on AI-generated content that was wrong, and the downstream impact of those decisions. This cost doesn't appear in tool analytics. It shows up in business outcomes.

Setting Up Measurement Before Deployment

The measurement infrastructure needs to be in place before the AI tool goes live, not afterward.

Before deploying any significant AI system, define:

The specific outcome the AI is intended to improve (not "productivity" — a specific, measurable outcome like "contract review cycle time" or "support ticket resolution time").

The baseline measurement for that outcome, calculated from historical data.

The measurement methodology going forward (system timestamps, not self-reports).

The evaluation timeline: 30-day, 90-day, and 12-month checkpoints with specific targets.

What constitutes a successful deployment vs. a failed one.

Organizations that define success metrics before deployment make better decisions about continuing, adjusting, or discontinuing AI tools. Organizations that measure after deployment are rationalizing investments they've already made.

What Honest Measurement Usually Finds

When enterprises run rigorous AI ROI measurement, three patterns emerge consistently.

The benefits are real but smaller than projected. Adjusted for actual adoption rates, correction labor, and realistic time reallocation, the productivity gains are usually 40-60% of initial projections. Still positive, but not transformative without appropriate scaling.

The distribution is uneven. AI tools deliver disproportionate value for specific use cases and specific user types, and minimal value for others. The aggregate average obscures both the high-value applications (which should be expanded) and the low-value applications (which should be reconsidered).

The compounding effects take longer than expected. AI tools typically deliver most of their value after the first year, as workflows are refined, prompts are optimized, and users develop more effective interaction patterns. Measuring at 90 days captures the early adoption period, not the mature deployment value.

None of these findings are reasons not to deploy AI. They're reasons to deploy thoughtfully, measure honestly, and iterate based on evidence rather than assumption.