DEV Community: Sumas Keller

Data Sovereignty and AI Adoption: What GDPR and NIS2 Actually Require

Sumas Keller — Mon, 20 Jul 2026 17:52:44 +0000

Data privacy has become the top barrier IT leaders cite when evaluating AI adoption, and for good reason. Most popular AI tools process data on external servers, outside the direct control of the company using them, which creates a genuine tension for any organization operating under GDPR, and increasingly under NIS2 for companies in critical or important sectors across the EU.

Why "the vendor says they're compliant" isn't the full answer

A common shortcut in AI vendor evaluation is checking whether a vendor states GDPR compliance in their terms of service. This is a necessary check but not a sufficient one. GDPR compliance from a vendor covers how they handle data as a processor, but it doesn't remove the company's own obligations as a data controller, including the obligation to know exactly where data is processed, how long it's retained, and what sub-processors might touch it along the way.

For AI tools specifically, this gets more complicated because many AI products route data through multiple layers, the interface, the underlying model provider, sometimes additional third-party services for search or retrieval, each of which may have its own data handling terms. Mapping this chain accurately is often harder than checking a single compliance statement.

Self-hosting as a structural answer rather than a policy answer

One way to sidestep a large share of this complexity is architectural rather than contractual: deploying AI infrastructure on servers the company itself controls, rather than relying on a third-party's infrastructure and trusting their compliance posture. This doesn't eliminate every compliance obligation, a company still needs proper access controls, audit logging, and data handling policies, but it removes the dependency on trusting an external vendor's data handling practices for the most sensitive workloads.

This is the structural approach platforms like PrivOS are built around: a fully self-hosted deployment option where data never leaves the company's own servers, with deployment ranging from private cloud to fully on-premise, air-gapped configurations for organizations in legal, financial, or other sectors where data residency requirements are strictest.

What NIS2 adds on top of GDPR

NIS2 broadens the scope of entities subject to cybersecurity obligations across the EU, and it introduces more specific requirements around incident reporting, supply chain security, and accountability at the management level. For companies newly in scope, one practical implication is that vendor risk assessment needs to extend further than it used to, evaluating not just a vendor's own security posture but the security of their sub-processors and infrastructure dependencies.

This is where self-hosted or on-premise deployment options become particularly relevant for compliance planning: they collapse a chain of third-party dependencies into a single, directly auditable environment, which materially simplifies the vendor risk assessment work NIS2 requires.

Auditability matters as much as location

Where data lives is one part of the compliance picture. The other part is whether every action taken on that data, including actions taken autonomously by an AI agent, is logged in a way that can be audited after the fact. As AI agents get more autonomy to read files, update records, or take actions inside a workspace, the ability to produce an immutable audit trail of exactly what an agent did, and when, becomes a practical requirement for both GDPR accountability principles and, in many cases, SOC2-style audits.

Platforms designed with this in mind build auditable action logs and permission boundaries directly into the architecture, rather than treating them as an afterthought, with deny-by-default permissions and mandatory human approval checkpoints for higher-risk autonomous actions.

A practical starting point

For companies evaluating AI adoption under EU compliance frameworks, a reasonable first step is mapping exactly where sensitive data would flow under each AI tool being considered, rather than relying on marketing claims about compliance. Tools built around self-hosted, auditable architecture, like the deployment model at privos.ai, simplify this mapping considerably because the answer to "where does the data go" is straightforward: it stays on infrastructure the company already controls.

The underlying principle is the same regardless of which vendor a company chooses: compliance in the AI era isn't just a legal question answered by a vendor's terms of service. It's increasingly an architectural question, and the architecture that makes the fewest assumptions about trusting a third party tends to be the one that holds up best under regulatory scrutiny.

Contract Terms Worth Negotiating Beyond Price

Sumas Keller — Fri, 17 Jul 2026 16:02:25 +0000

Most vendor negotiations focus almost entirely on price, and price is usually the term with the least long-term leverage attached to it. A handful of other clauses, routinely accepted as boilerplate, tend to matter more over the life of a contract than the initial discount a procurement team negotiated at signing.

Auto-renewal terms and cancellation windows

Standard vendor contracts often include auto-renewal clauses with a cancellation notice window that is easy to miss, sometimes 60 or 90 days before the renewal date. Miss the window and the contract renews automatically, frequently at a higher rate than the previous term. This is not usually predatory, it is a standard commercial default, but it puts the burden entirely on the buyer to track renewal dates across every vendor relationship.

Negotiating a shorter notice window, or at minimum getting the vendor to agree to send a renewal reminder a set number of days in advance, removes a surprisingly common source of unwanted renewals.

Price increase caps on renewal

Initial contract pricing is often the easiest term to negotiate, precisely because it's the most visible one. Price increases at renewal are less visible and less scrutinized, and vendors know this. A contract that doesn't cap annual price increases leaves the buyer exposed to whatever increase the vendor decides to apply at the next renewal, with limited leverage to push back since switching costs have usually grown by then.

A cap, commonly tied to a fixed percentage or to a public inflation index, is a reasonable ask during initial negotiation and is far harder to secure retroactively once the relationship is established and switching costs are higher.

Data portability and export terms

What happens to data at the end of the contract is rarely discussed during onboarding, when everyone is focused on getting the tool live. But export terms, format, completeness, and the timeframe during which data remains accessible after cancellation, materially affect how easy it is to actually leave the vendor later. A contract silent on this defaults to whatever the vendor's standard practice happens to be, which is not always generous.

Specifying export format and a minimum data retention window post-cancellation, in writing, costs nothing to ask for and closes a gap that only becomes expensive to discover during an actual vendor switch.

Service level commitments with actual remedies

Uptime commitments are common in vendor contracts, but the remedy attached to missing them is often weak, typically a small service credit that doesn't come close to covering the actual business impact of downtime. Reviewing not just whether an SLA exists but what happens when it's breached, and whether the remedy is proportionate to realistic impact, changes how much the SLA is actually worth.

For any tool considered business-critical, it's worth negotiating stronger remedies or, at minimum, a right to terminate without penalty if SLA breaches exceed a defined frequency within a contract term.

Liability caps and indemnification scope

Standard vendor paper usually caps the vendor's liability at the amount paid under the contract in the prior twelve months, which can be a small number relative to the actual damage a serious data breach or extended outage could cause. This term is genuinely negotiable more often than buyers assume, particularly for larger contracts, though vendors will resist raising it for standard terms with smaller customers.

Understanding what liability cap applies, and whether it's proportionate to the risk the tool actually carries, particularly for anything handling sensitive data, is worth a conversation even when the negotiation feels unlikely to move the number significantly.

Assignment and change of control clauses

Vendors get acquired, and acquisition often changes product direction, pricing, or support quality in ways that weren't part of the original decision to buy. A contract clause addressing what happens on a change of control, sometimes giving the buyer a right to terminate without penalty if the vendor is acquired, provides an exit path that otherwise doesn't exist if the acquiring company changes terms unfavorably.

This clause is rarely raised by either side during negotiation, mostly because it feels like a low-probability scenario at signing time. It becomes relevant often enough in practice that it's worth the small amount of negotiation effort it takes to include.

The general pattern

Every one of these terms shares a common trait: they matter far more at the point of renewal, exit, or crisis than they do at signing, which is exactly when they're hardest to renegotiate. The cost of raising them during the initial negotiation is low. The cost of discovering them absent later, during a price hike, a failed export, or an outage, is consistently higher.

A Compliance Checklist for Entering a New EU Market

Sumas Keller — Thu, 16 Jul 2026 16:20:12 +0000

Expanding into a new EU country looks simpler on paper than it plays out in practice. The EU single market creates the impression of one unified ruleset, but a meaningful share of employment, tax, and data regulation is still set at the national level. Companies that treat "EU expansion" as a single compliance project, rather than a country-by-country one, tend to discover the gaps the hard way.

Below is a practical checklist built around the categories that most commonly cause delays or fines during EU market entry.

Employment classification

Every EU country has its own rules for what counts as an employee versus a contractor, and the thresholds are not always intuitive. A working relationship that qualifies as freelance in one country can trigger mandatory employer contributions in another, sometimes retroactively, if a labor authority decides the relationship looks more like employment than contracting.

Before hiring anyone in a new country, worth confirming:

Local classification tests for employee versus contractor status
Mandatory benefits and minimum notice periods, which vary significantly between, say, Germany and Poland
Whether a local entity is required, or whether an Employer of Record arrangement is sufficient for the initial phase

Data residency and transfer rules

GDPR sets a floor, not a ceiling. Several member states layer additional requirements on top, particularly around health data, employee data, and data belonging to minors. A data processing setup that is fully compliant in the Netherlands is not automatically compliant in France or Germany.

Checklist items:

Confirm whether any sector-specific data rules apply beyond baseline GDPR
Map where customer and employee data will actually be stored and processed, not just where the company is legally headquartered
Review data processing agreements with any vendor that will touch data originating from the new market

Tax registration and VAT

VAT obligations often trigger earlier than companies expect. Selling into a country, even without a physical presence there, can create VAT registration requirements once revenue crosses a threshold. These thresholds differ by country and have changed in recent years with the shift toward the One-Stop-Shop system for cross-border digital sales.

Worth verifying before launch:

Whether the OSS scheme covers the products or services being sold, or whether local VAT registration is still required
Corporate tax residency rules, since a local sales presence can sometimes create a taxable presence even without a formal subsidiary
Withholding tax obligations on any payments to local contractors or partners

Consumer protection and contract law

Standard terms of service written for one jurisdiction do not transfer cleanly across the EU. Consumer protection law, particularly around cancellation rights, warranty periods, and mandatory disclosures, varies enough that a contract valid in one country can contain unenforceable clauses in another.

This is worth a local legal review rather than a translation exercise. Translating terms of service into the local language without adapting the substance is one of the more common and avoidable mistakes companies make during expansion.

Local works councils and employee representation

In several EU countries, companies past a certain employee threshold are required to establish or recognize employee representation bodies, and these bodies often need to be consulted before certain operational changes, including some restructuring or layoff decisions. Companies coming from jurisdictions without this tradition sometimes miss this entirely until a decision they expected to move quickly gets stalled by a consultation requirement they did not know existed.

Checking the applicable threshold and consultation requirements for the specific country, before any major operational changes are planned, avoids unpleasant surprises later.

A practical sequencing note

None of these five areas can really be handled well in isolation, since decisions in one often affect another. Employment classification affects payroll tax exposure. Data residency choices affect vendor contracts. The most efficient path tends to be a short, focused review across all five areas before the first hire or first sale in the new market, rather than treating each as a separate problem to solve reactively as it comes up.

The upfront cost of this review is real, but it is consistently smaller than the cost of retroactive fixes once a regulator or works council raises the issue after the fact.

THE TALENT METAMORPHOSIS AND WHY YOUR NEXT GREAT EXECUTIVE IS CURRENTLY AN INTERN

Sumas Keller — Tue, 14 Jul 2026 18:01:40 +0000

There is a conversation happening at almost every executive retreat right now that is completely missing the point. Leadership teams are sitting around looking at spreadsheets and trying to calculate how many junior roles they can eliminate this quarter by adopting artificial intelligence. They are looking at this revolutionary technology as nothing more than a giant digital lawnmower designed to cut costs and trim the organizational chart.

I think this is a tragically small minded way to run a company.

When I look at the operational landscape of the enterprises I advise, I am not looking at the jobs that are being lost. I am obsessed with the profound metamorphosis happening to the people who stay. We are witnessing the greatest flattening of the corporate learning curve in modern business history, and most executives are completely blind to it.

Let me tell you a story about a recent operational audit I conducted for a mid market logistics firm in Europe. I was sitting in a room with their executive team reviewing a massive optimization plan for their regional supply chain. The presentation was brilliant. It accounted for geopolitical risks, fuel price volatility, and complex labor regulations. It was the kind of strategic thinking I would expect from a seasoned Director of Strategy with twenty years of industry experience.

The person presenting it was twenty three years old. She had been with the company for exactly eight months.

Before this new era of intelligent tools, it would have been physically impossible for her to produce that work. She would not have had the time to manually parse through thousands of pages of historical shipping data. She would not have had the specialized knowledge to build the complex financial models required to justify her proposal. The sheer friction of gathering the information would have kept her locked in a junior execution role for the next decade.

But she was given access to a secure internal language model that had been trained on the proprietary data of the company. The technology removed the friction of information gathering. It wrote the code for the financial models. It summarized the historical shipping manifests.

What fascinated me was not the speed of the machine. What fascinated me was her mind. Because she was freed from the manual labor of building spreadsheets, she was able to spend her entire week doing something infinitely more valuable. She spent her time asking the machine increasingly complex and nuanced questions. She was pressure testing different strategic scenarios. She was acting as a high level orchestrator of logic.

This is the breath of fresh air that operations leaders need to embrace. Artificial intelligence is not a tool for replacing human talent. It is a cognitive equalizer. It takes your youngest, most energetic employees and instantly elevates their execution capabilities to the level of a senior manager.

This requires a complete paradigm shift in how we think about leadership and organizational structure.

For the last century, the corporate structure has been a pyramid. The junior employees at the bottom did the manual execution. The middle managers supervised the execution. The executives at the top did the strategic thinking.

If a junior employee equipped with an intelligent workspace can now execute at the level of a middle manager, your job as a Chief Operating Officer fundamentally changes. You are no longer managing output. You are no longer managing manual tasks. You are now managing judgment. You are managing taste. You are managing strategic direction.

You have a building full of highly empowered individuals who can build software, draft contracts, and analyze data at lightning speed. Your job is no longer to tell them how to do the work. Your job is to give them absolute clarity on what problems actually need solving.

To make this transition successful, you cannot just hand your employees a public subscription to a generic technology vendor and tell them to figure it out. That is reckless and it compromises your data sovereignty.

If you want to unlock this level of human potential, you have to build them a sanctuary. You need to provide your talent with a secure, unified digital environment where the artificial intelligence is directly connected to your private company knowledge base. They need a sandbox where they can experiment, query internal data, and build new workflows without the fear of leaking trade secrets to the public internet.

When you provide your team with a private operating environment that they can trust, the results are nothing short of magical. You stop seeing your employees as line items on a budget that need to be optimized. You start seeing them as strategic partners who are fully equipped to help you reinvent the business.

Stop trying to use this technology to shrink your company. Start using it to expand the intellectual capacity of every single person who walks through your doors. The companies that embrace this optimistic view of human potential are the ones who will absolutely dominate the next decade. Build the environment, trust your people with the tools, and watch them build the future.

Five Conversations About AI I Wish I Had Earlier

Sumas Keller — Mon, 13 Jul 2026 18:09:31 +0000

Not frameworks. Not best practices. Specific conversations I avoided or delayed that, had I had them earlier, would have saved months of friction.

The first was with the people whose jobs the AI was supposed to augment.

We announced the AI deployment before we talked to the teams affected. The announcement framed it as a productivity initiative. Several people heard it as a threat signal. The pattern was predictable in retrospect: we communicated the organizational benefit without involving the people whose daily work would change in figuring out what that change would look like.

The conversation I should have had first was simple: here is what we are thinking about, here is why, what do you see about your work that we might be missing? That conversation would have surfaced concerns that became adoption blockers six months later, while they were still cheap to address.

The second conversation was with our legal and compliance team, specifically about data flows.

I knew they needed to be involved. I scheduled it after we had selected a vendor because I thought the specific contract terms would give them something concrete to review. They had questions the vendor contract did not answer, about how inference logging worked at the subprocessor level, about what data handling applied during the period between contract start and DPA execution. These were questions that should have shaped vendor selection, not followed it.

Bring legal into vendor evaluation, not vendor onboarding. The distinction sounds procedural. The practical effect is significant.

The third conversation was with the IT and security team about what "connected to our systems" actually meant.

When we described the deployment to IT as "connecting the AI to our internal knowledge base," they heard something more limited than what we had built. What we had built was an integration that could in principle surface content from any document the AI system had been given access to index. What IT understood was a narrow integration with a specific document repository.

The gap between those two descriptions was the source of a security review that happened eight months into the deployment rather than before it. Walking through the actual data architecture with IT before deployment would have taken a day. The remediation process took six weeks.

The fourth conversation was with the managers whose teams were adopting AI.

Individual contributors either adopted or did not based on personal motivation. Managers were neutral, not because they opposed the AI but because nobody had told them what their role in adoption was supposed to be. They were not modeling usage, not creating space in team rituals for sharing what worked, not making AI assistance visible as something they valued.

The conversation that changed this was not a training session. It was a direct one-on-one with each manager asking: what would it look like if your team was genuinely good at using AI in six months? That question made the adoption outcome feel like theirs rather than a technology initiative happening to their team.

The fifth conversation was with myself, about what success actually meant.

I had a vague idea that the deployment would improve productivity. I did not have a specific definition of what productivity improvement we were targeting, how we would measure it, what we would do differently if the numbers were not moving, or what the threshold was at which we would conclude the deployment was not working.

Without that definition, every review of the deployment became a conversation about whether the vague idea of productivity improvement felt like it was happening. Those conversations are difficult to make useful decisions from.

The clarity conversation is uncomfortable because it forces commitment to a standard of success before you know whether you will meet it. That discomfort is exactly why it is worth having before deployment rather than after.

None of these conversations required anything unusual. They were the obvious conversations. I knew they needed to happen. I delayed them for reasons that seemed practical at the time and were not. The pattern I have noticed is that the conversations we delay are usually the ones where we anticipate friction, and the friction we avoid in those early conversations reliably accumulates into the larger problems we end up managing later.

Before You Renew Your AI Contract, Do This First

Sumas Keller — Fri, 10 Jul 2026 11:41:38 +0000

Most enterprise software renewals are passive. The contract comes up, procurement processes it, maybe there is a negotiation on price, and the subscription continues. The product gets evaluated once at purchase and then assumed to be delivering value until someone complains loudly enough.

AI contracts deserve a different process before renewal, not because vendors are more dishonest than others, but because the conditions that justified the initial purchase change significantly over time and the renewal represents a decision with much higher switching costs than the original purchase.

Here is the specific review I run before any AI contract renewal.

Pull the actual usage data for the most recent six months, not from memory and not from the vendor's dashboard if you can avoid it. You want raw numbers: who used it, how often, for what types of tasks. In most deployments I have reviewed, the active user count at month eighteen is 40 to 60% of the active user count at month three. The enthusiasm of initial adoption rarely sustains. Understanding specifically who stopped using it and what they have said about why is more informative than the aggregate number.

Talk to the people who use it most and the people who stopped using it. Ask them the same question: if this tool went away tomorrow, what would change about your work? The people who use it most will tell you where the real value is concentrated. The people who stopped will tell you where the product failed to meet their needs. Both answers matter for the renewal decision and for the negotiation.

Model the switching cost honestly. This is the step most organizations skip because it is uncomfortable. What would it actually cost, in time and money and organizational disruption, to move to an alternative? How much organizational-specific configuration, prompt engineering, and workflow integration has been built around this specific platform? How long would it take to rebuild that in another system? The switching cost is the vendor's most powerful leverage at renewal, and you need to understand it before you sit down to negotiate.

Evaluate what the market looks like now, not what it looked like at initial purchase. The AI tool market has changed substantially in eighteen months. Capabilities that were differentiating at initial purchase may now be table stakes. Limitations that were acceptable because alternatives did not do better may now have solutions. Running a lightweight current-state evaluation against the two or three most credible alternatives takes two weeks and changes your negotiating position significantly.

Check the vendor's roadmap commitments from the original sale against what actually shipped. Almost every AI vendor made product roadmap promises during the initial evaluation. Some of those promises were honest projections. Some were aspirational. Some were not delivered. Understanding which category each commitment fell into tells you how much weight to put on the roadmap commitments they are making now to support the renewal.

With this information, the renewal negotiation is about facts rather than relationships. You know the actual usage and value delivered. You know the switching cost. You know what alternatives exist and at what price. You know which vendor commitments were honored and which were not. This is a fundamentally different negotiating position than showing up with the contract and hoping for a better rate.

The renewal is not just a procurement decision. It is a fresh evaluation of whether this is still the right tool for where the organization is now, evaluated honestly against what it costs to stay and what it costs to move.

How to Measure Whether Your AI Deployment Is Actually Working

Sumas Keller — Thu, 09 Jul 2026 14:26:28 +0000

Most organizations measuring their AI deployments are measuring activity, not value. Usage rates, query volumes, active user counts, sessions per week. These metrics tell you whether people are using the tool. They tell you almost nothing about whether the tool is producing outcomes that justify the investment.

I want to describe the measurement framework I have built up through trial and error, because the path I took through wrong measurements to useful ones was longer than it needed to be and cost real decision-making quality along the way.

The measurement I started with and why it was wrong

We started with a simple question: are people using the tool? We tracked daily active users, weekly query volumes, and the percentage of employees who had used the tool at least once in the past thirty days.

These numbers looked good and climbed consistently in the first six months. I reported them to the executive team as evidence that the deployment was succeeding. Then a member of the finance team did something I had not done: she asked several employees directly what they were using the tool for.

The answers were instructive. Several people were using the tool to generate first drafts of documents they were going to rewrite almost completely anyway. Several were using it for tasks where the AI output was marginally useful but not substantially better than what they would have produced without it. A few were using it to look productive when they were not sure what to work on.

Usage was high. Value delivered was ambiguous.

The problem with activity metrics is that they measure the top of a funnel without measuring anything about the funnel's output. High usage of a tool that produces outputs requiring significant rework is not the same as high usage of a tool that reduces total work time. Lumping both together in a "users" metric obscures the difference.

The measurement shift that changed how I understood the deployment

The shift that produced useful information was moving from measuring tool activity to measuring workflow outcomes.

Instead of asking "how many people used the AI tool," I started asking "for the specific workflows where we deployed AI assistance, how have the time and quality characteristics of those workflows changed?"

This sounds obvious in retrospect. The reason I had not started there was that measuring workflow outcomes requires you to have baseline measurements before the deployment, which requires forethought that most deployments do not have.

We solved this retroactively by using historical data as a proxy baseline. For document workflows where we had timestamps, we could compare before-and-after cycle times. For tasks that produced outputs with trackable quality indicators, like proposals with win rates or reports with feedback scores, we could compare quality metrics before and after.

The specific metrics that proved most informative:

Time-to-first-draft for document categories where we could track this. Not the time saved in total but specifically the time from when the work was assigned to when a reviewable draft existed. This was meaningful because it captured the actual bottleneck we had been trying to address and was not confounded by how long editing took after the draft existed.

Error rates in specific high-volume tasks. For the operational workflows where we had automated AI assistance, we tracked error rates and compared them to the pre-AI baseline. This was the metric that caught a significant problem early: for one specific task type, the AI-assisted error rate was actually higher than the manual error rate, not lower. We would not have found this without measuring it.

Revision rounds required before approval. For documents going through internal review, we tracked whether AI-assisted drafts required fewer, the same, or more revision rounds before approval. The expectation was that AI drafts would require fewer revisions. For some document types this was true. For others, the AI introduced specific types of errors that reviewers consistently flagged, resulting in more revision rounds than unassisted drafts.

The metric that I wish I had built from the start

The most informative single metric I have found for enterprise AI deployments is not one that most organizations track: the proportion of AI outputs that are used with minor modification versus substantially rewritten versus discarded.

This metric captures whether the AI is actually reducing work or just shifting where the work happens. A tool where 70% of outputs require substantial rewriting before use is a tool that is consuming time in a different place, not a tool that is reducing total time spent.

Tracking this requires either user surveys or workflow instrumentation that records what happens to AI-generated content after it is produced. The instrumentation approach is more accurate but requires technical work upfront. The survey approach is faster to implement but suffers from self-reporting bias.

For any organization that is serious about understanding whether their AI deployment is delivering value, I would treat this metric as essential and design the measurement infrastructure before deploying, not after.

What to do when the measurements disappoint

The measurement framework I described will sometimes produce results that are worse than expected. This is the value of actually measuring rather than relying on impressions and activity metrics. The question is what to do with disappointing results.

The instinct is to adjust the measurement to make the numbers look better. Redefine "used with minor modification" to include outputs that were substantially edited. Focus reporting on the metrics that look good and de-emphasize the ones that do not.

The correct response is the opposite: treat disappointing measurements as diagnostic information and use them to identify specifically what is producing the disappointing result.

In my experience, disappointing AI deployment measurements trace to one of four specific causes. The AI is being used for the wrong tasks, ones where its capabilities are poorly matched to what the work actually requires. The underlying data quality is insufficient for the AI to produce good outputs on the tasks where it is being used. The prompting and configuration have not been tuned to the specific use cases of the organization. Or the deployment is in workflows where the speed improvement is real but the quality bar is high enough that the speed gain is offset by increased editing time.

Each of these has a specific solution. None of them is solved by not measuring.

The organizations that get sustained value from AI deployments are not the ones whose first measurements were impressive. They are the ones that measured honestly, identified the gaps between expectation and reality, and used that information to make specific changes. The measurement is not the destination. It is the navigation system that tells you whether you are going the right direction and how far you have to go.

The Metrics Trap: Why "Time Saved" is a Terrible Way to Measure AI ROI

Sumas Keller — Wed, 08 Jul 2026 16:57:02 +0000

Every board meeting I sit in lately seems to feature the exact same slide. A well-meaning department head puts up a chart showing that thanks to their new AI tools, their team is saving "40 hours a week" on drafting emails, summarizing reports, or generating code.

The executives nod, applaud the efficiency, and move on. Nobody asks the only operational question that actually matters: Where did those 40 hours go?

I have audited the operational throughput of these teams. I can tell you exactly where the saved time goes. It goes into a void. It goes into longer coffee breaks, endless slack scrolling, or worse—it goes into generating exponentially more low-value work.

When you measure AI success by "time saved," you are completely misunderstanding the economics of artificial intelligence.

Before AI, creating a 10-page market research brief took a human three days. That friction acted as a natural filter. Because it was hard to create, we only asked for it when we genuinely needed it.

Today, AI can generate that same 10-page brief in twelve seconds. The friction is gone. So what happens? Middle managers start requesting five briefs a day instead of one a month. The marketing team isn't acting on better information; they are just drowning in a higher volume of perfectly formatted noise.

We have successfully automated the production of corporate clutter.

Making a useless process faster is not an operational win. It is operational self-sabotage.

If an AI tool saves your customer support team 20 hours a week drafting replies, but your customer churn rate hasn't dropped and your resolution quality hasn't improved, your ROI is exactly zero. You just built a faster hamster wheel.

We need to completely rebuild how we evaluate enterprise AI. Operations leaders must stop tracking output metrics (words generated, time saved, tickets closed) and start tracking throughput and quality metrics.

Are we making decisions faster?
Has our defect rate dropped?
Are we closing deals with a higher win rate?
Is the cognitive load on our top performers actually decreasing?

If the answer to those questions is no, it doesn't matter if your shiny new LLM integration saves your team 100 hours a month. You are not buying efficiency. You are just subsidizing a faster version of the status quo.

Stop measuring the speed of the engine. Start measuring the destination of the car.

The Hidden Cost of AI Automation Nobody Adds to the Budget

Sumas Keller — Tue, 07 Jul 2026 16:00:11 +0000

When teams discuss AI adoption, the conversation usually starts with productivity.

How many hours can we save?

How many tasks can we automate?

How much faster can employees work?

They're important questions, but I've noticed something interesting.

Very few organizations spend the same amount of time estimating what AI will add to their operations.

Because automation doesn't only remove work.

Sometimes, it creates entirely new categories of work.

Every automation introduces a management responsibility

Imagine an AI assistant that drafts customer support replies.

At first glance, it seems straightforward. The assistant generates a response, an employee reviews it, and the customer receives an answer.

But after a few months, new questions begin to appear.

Who reviews the prompts?

Who updates the knowledge base?

Who monitors incorrect responses?

Who investigates incidents when customers receive inconsistent information?

None of these tasks existed before.

The AI didn't eliminate operational work.

It changed its shape.

Automation is not the same as operational automation

This is a distinction I think more teams should understand.

Automating a single task is relatively easy.

Building an operation that can safely rely on automation is much harder.

The second requires documented processes, clear ownership, monitoring, and continuous improvement.

Without those elements, automation becomes another system employees need to supervise instead of one that genuinely saves time.

Maintenance is part of the return on investment

Software engineers often say that every system eventually becomes someone's responsibility.

The same principle applies to AI.

A workflow that saves ten hours every week still requires ongoing attention.

Knowledge needs updating.

Business rules evolve.

Permissions need reviewing.

New employees need training.

None of these activities appear in product demos.

Yet they're essential if AI is expected to remain useful for years rather than months.

Operational debt grows quietly

One pattern I've repeatedly observed is that organizations rarely notice operational debt until AI becomes widely adopted.

Nothing feels expensive during the pilot phase.

Six months later, different teams are maintaining prompts, validating outputs, updating documentation, handling exceptions, and refining workflows.

The AI didn't fail.

The organization simply underestimated the amount of operational work required to support it.

That's an important distinction.

A simple question changes the entire planning process

Instead of asking only:

"How much time will AI save?"

Try asking:

"What new responsibilities will AI create?"

That single question often leads to better implementation plans.

It encourages teams to think about ownership, governance, maintenance, and long-term sustainability before deployment begins.

In my experience, those discussions create stronger AI projects than another week spent comparing benchmark scores.

Final thought

The success of an AI initiative isn't determined by how many workflows become automated.

It's determined by whether the organization is prepared to operate, maintain, and improve those workflows over time.

Automation removes repetitive work.

Operational excellence ensures the benefits actually last.

An AI Governance Framework That People Actually Follow

Sumas Keller — Mon, 06 Jul 2026 10:14:35 +0000

Most AI governance frameworks I have seen are compliance documents dressed up as operational frameworks. They describe what should happen in ideal conditions, they get approved by leadership, they get distributed to employees, and then they sit in a shared drive while the actual decisions about AI usage get made informally based on whatever feels reasonable to whoever is making the decision that day.

The reason this happens is not that people do not care about governance. It is that most governance frameworks are designed to be complete rather than to be used. They cover every possible scenario, include exhaustive policy language, and require the reader to hold a mental map of the whole document to understand how it applies to any specific situation. Nobody has time for that when they are deciding whether to paste a client email into an AI tool to get a draft response.

The governance framework that people actually follow looks different. It is designed around the decisions employees actually face, not the scenarios the governance team imagined they would face. It is short enough to remember. And it provides clear enough guidance that employees can apply it without consulting anyone else for common situations.

Here is the structure that has worked for me across several deployments.

The three-tier classification

Every piece of information that touches AI needs to be classifiable quickly into one of three tiers. If the classification requires a judgment call every time, it will not be applied consistently.

Tier 1 is public or general information. This is content that is either publicly available or that the organization has no material interest in keeping confidential. Marketing copy, publicly available competitor information, general industry knowledge. AI tools can process this freely, including external AI tools with standard terms.

Tier 2 is internal business information. This is information that is not public but is not specifically sensitive: internal process documentation, meeting notes about operational topics, general project updates, internal communications that do not contain client data or commercially sensitive content. AI tools can process this with enterprise agreements in place. External AI tools with standard consumer terms cannot.

Tier 3 is sensitive or confidential information. This includes: client data, personally identifiable information, financial data, legal strategy, M&A activity, HR records, anything under NDA, and any information that a client has shared in a professional relationship. AI tools can process this only if they run entirely within the organization's own infrastructure. External AI tools, even with enterprise agreements, are not appropriate for Tier 3 content.

The classification should be done by the person initiating the AI interaction. If they are unsure which tier content belongs in, the default is the higher tier.

The five common situations

Rather than trying to cover every scenario, the framework covers the five situations employees encounter most often and tells them specifically what to do.

One: drafting an email to a client. The email will likely contain client-specific context, making it Tier 3. Do not paste client-identifying information into an external AI tool. Either use an internal AI tool, draft the email without the AI, or use the AI only for general writing assistance on a version with client details removed.

Two: summarizing a meeting. If the meeting involved clients, client names, or confidential project details, it is Tier 3. Record and transcribe internally. If the meeting was internal and covered operational topics only, it is likely Tier 2 and can use AI tools with enterprise agreements.

Three: researching a market or competitor. This is almost always Tier 1 and any AI tool is appropriate.

Four: drafting a proposal. Proposals contain client context and pricing information, making them Tier 3. Use only internal AI tools. Do not use external AI tools even at the draft stage if client-identifying information is included.

Five: analyzing internal data. Depends on the data. Financial projections, HR data, or data that could identify individuals is Tier 3. General operational metrics or anonymized data is Tier 2.

The escalation path

For situations not covered above, the framework provides a single escalation path rather than a decision tree. Employees who are unsure contact the designated AI governance contact for their team. That person has the authority to make a binding classification decision in under 24 hours. If the 24-hour window is too slow for the situation, the default is to treat the content as Tier 3 and restrict to internal tools only.

This matters because governance frameworks that require escalation through multiple levels, or that take a week to produce an answer, get bypassed. The escalation path needs to be fast enough to be useful and authoritative enough to be trusted.

What makes it work

The framework I described above is not comprehensive. It will not cover every situation. There will be edge cases that require judgment calls. A comprehensive framework that covers every case is a framework that is too complex to remember and therefore not used.

The operating principle is that a governance framework that is incomplete but followed is more valuable than one that is comprehensive but ignored. The goal is to move decisions from "whatever feels reasonable" to "consistent with a framework that reflects our organizational values and risk tolerance." You do not need to cover every edge case to make that move.

The other thing that makes it work is visible leadership behavior. If the people who created the framework are seen using external AI tools in situations that the framework says require internal tools, the framework means nothing. Governance frameworks require behavioral modeling from the people who champion them, not just policy language in a document.

The frameworks I have seen successfully adopted share one property that the unsuccessful ones lack: the people who created them also visibly comply with them. That visible compliance is not a nice-to-have. It is what determines whether the framework is operational or decorative.

12 Questions That Separate Serious AI Buyers from the Rest

Sumas Keller — Fri, 03 Jul 2026 09:48:41 +0000

After sitting through more enterprise AI procurement processes than I can count on both sides of the table, the same pattern keeps showing up. The organizations that end up with AI deployments that actually work asked harder questions earlier. The ones that end up with expensive regrets asked easier questions and felt good about the answers.

Here are the twelve questions that I have watched separate the two groups. Not because the questions are complicated, but because asking them sincerely and demanding real answers takes a kind of organizational discipline that most procurement processes do not have.

On data handling:

"Walk me through exactly what happens to my data between when an employee submits a query and when they receive a response. Which servers does it touch, which external services are called, and what is retained after the response is generated?"

This question has a specific correct answer and a specific evasive answer. The correct answer names the infrastructure components, the services, and the retention periods. The evasive answer talks about enterprise agreements, security certifications, and data privacy commitments without describing the actual data flow. If you get the evasive answer, ask the question again more specifically. If you still get the evasive answer, you have learned something important.

"What is your complete subprocessor list, and what data handling commitments does each subprocessor have?"

Most enterprise AI vendors use external LLM inference APIs, external embedding APIs, external monitoring services, and cloud infrastructure from major providers. Your data processing agreement is with the vendor. Your data flows through systems governed by agreements you have not seen. Ask for the complete list and verify that the protections cascade.

"If we need to delete all data associated with our organization from your systems, including any derived data, cached prompts, or inference logs, what is the process and how long does it take?"

Deletion is harder than it sounds in distributed cloud infrastructure. Backups, caching layers, inference logs, and derived analytics can all hold your data beyond the obvious application layer. Understanding the deletion process before you need it is significantly more useful than understanding it during a GDPR deletion request.

On product reliability:

"Show me what the AI says when it doesn't know the answer, or when the retrieved information is insufficient to answer confidently."

This is the honesty test. Ask the vendor to demonstrate this live, with a question you know is not in their demo dataset. The tools that say "I don't have reliable information about this" or "the sources I found are from 2021 and may be outdated" are the tools that are calibrated to be honest about their limits. The tools that generate a fluent, authoritative-sounding answer to a question they have no basis to answer are the tools that will create trust-destroying errors in production.

"What is your retrieval accuracy on documents that were uploaded six months ago versus documents that were uploaded last week?"

Retrieval quality for recently indexed content is almost always better than for older content, because document metadata, embedding quality, and index optimization often improve over time. If the vendor cannot give you a concrete answer about how freshness affects retrieval quality in their system, they have not measured it. If they have not measured it, they do not know where the quality floor is in production deployments.

"Tell me about a production incident you have had in the last twelve months where AI outputs were incorrect in a consequential way. What happened and what did you change?"

Every AI tool in production has had incidents where the output was wrong in a way that mattered. The vendors who have thought seriously about reliability can tell you about these incidents clearly and describe what they learned. The vendors who have not thought seriously about it will tell you they have not had any meaningful incidents, which is almost certainly not true.

On organizational fit:

"What does the customer relationship look like in month eighteen, not in month three?"

The month three relationship is the vendor's best foot forward: attentive account management, responsive support, proactive check-ins. Month eighteen is when you have moved from new customer to established customer and you have real experience of how the vendor treats accounts that are not in the critical adoption period. Ask for references who are eighteen months or more into the deployment and specifically ask those references what changed in the relationship over time.

"What is the renewal price if our usage has grown by 50% over the first year?"

Usage-based pricing is not inherently bad. Surprise usage-based pricing at renewal is. Get a specific number for the growth scenario you actually expect before you sign the initial contract. If the vendor says the renewal pricing depends on factors they cannot estimate until renewal, that is information.

"Who in your organization is accountable if this deployment is not delivering value twelve months from now?"

This question is as much for your organization as it is for the vendor. The right answer internally is a specific person with a specific metric, not "we are all accountable for the success of this initiative." Diffuse accountability for AI deployment outcomes is one of the strongest predictors of underperformance.

On the exit scenario:

"What does leaving look like? What can we export, in what format, and what happens to the workflows we've built on your platform?"

Ask this question early and watch the reaction. Vendors with clean exit paths are not afraid of this question. Vendors whose business model depends on switching costs become evasive about it. The answer tells you how the vendor thinks about the relationship.

"If your company were acquired tomorrow, what would happen to our contract, our data, and our service terms?"

Acquisition risk is real in the current AI market. Many of the vendors you are evaluating will not exist in their current form in three years. They will be acquired, pivot, or shut down. Understanding the contractual protections you have in these scenarios before you commit is meaningfully different from understanding them after the acquisition has happened.

On the realistic case:

"What does a typical deployment look like for a company our size, and what are the two or three things that most commonly prevent organizations like ours from reaching their projected ROI?"

This question reveals whether the vendor has an honest model of why their product sometimes fails. The vendors who can answer it clearly and specifically, naming the actual reasons deployments underperform, are the vendors who have learned from deployment experience and built that learning into their sales process. The vendors who cannot answer it, or who answer with vague references to change management or data quality without specifics, are the vendors who either have not had enough failed deployments to learn from or are not being honest about the ones they have had.

The purpose of these questions is not to be difficult. It is to do the evaluation work that vendor demos are not designed to help you do. A compelling demo answers the question of whether the technology can work in optimal conditions. These questions help you understand whether it will work in your conditions, with your data, over your timeline, with your organizational constraints.

The organizations that ask them tend to make better decisions. The ones that do not tend to end up explaining to boards why the AI initiative that looked so promising at the demo stage is not delivering what was projected.

Ask the questions. Get real answers. Make better decisions.

The Day We Stopped Asking AI To Do Everything

Sumas Keller — Thu, 02 Jul 2026 17:29:47 +0000

For a while, our conversations about AI all sounded the same.

"Can we automate this?"

"What else can the agent do?"

"Can we connect one more system?"

Nobody asked a much simpler question.

"Should we?"

I've started to believe that's where many AI projects quietly drift off course.

Not because the technology isn't capable.

Because our expectations become larger than our operational discipline.

One pattern I've noticed across organizations is that AI usually earns trust by succeeding at small tasks.

Summarizing meeting notes.

Searching internal documentation.

Drafting emails.

Categorizing support tickets.

Those are meaningful improvements.

The problem begins when success in one area creates confidence everywhere else.

Teams start assuming that because AI performed well yesterday, it should be given more responsibility tomorrow.

Sometimes that's the right decision.

Sometimes it isn't.

The difficult part is knowing the difference.

I've found it useful to think about business processes in two categories.

The first contains work that's repetitive, well documented, and easy to verify.

These are excellent candidates for automation.

The second contains work involving judgment, context, negotiation, or accountability.

Those processes benefit from AI, but they still need people making the final decision.

That distinction has changed how I think about adoption.

Instead of asking whether AI can replace a workflow, I ask where human judgment creates the most value.

Surprisingly, that question often reduces the scope of automation.

And that's usually a good thing.

Smaller deployments are easier to understand.

They're easier to monitor.

They're easier to improve.

Most importantly, they're easier for teams to trust.

Another lesson is that operational complexity grows quietly.

Adding another AI capability rarely feels expensive on launch day.

The costs appear later.

Permissions need reviewing.

Processes need updating.

Employees need training.

Audit requirements expand.

Someone becomes responsible for maintaining everything that was previously "automatic."

None of those activities are exciting.

They're also the difference between an AI pilot and a sustainable operating model.

That's one reason I've become interested in platforms that treat governance as part of the workspace instead of an additional layer added afterward.

When permissions, collaboration, AI agents, and auditability are designed to work together from the beginning, operational overhead becomes much easier to manage.

PrivOS is an example of this design philosophy.

Rather than focusing only on what AI can automate, it places equal emphasis on controlled collaboration, privacy-first deployment, and governance that scales with the organization.

https://privos.ai/

Looking back, I don't think the most valuable question is:

"How much work can AI take away?"

It's a different one.

"What work should always remain understandable, observable, and ultimately owned by people?"

Technology changes quickly.

Operational responsibility doesn't.