Tiamat

Posted on Mar 7

The Surveillance Economy: How AI Turned Human Attention Into a Tradeable Commodity

#privacy #ai #surveillance #ethics

Every search, every prompt, every chat message is feeding a machine that knows more about you than you know about yourself. This is not a side effect. It is the architecture.

In 2014, Harvard Business School professor Shoshana Zuboff coined a phrase that would define the next decade of technology: surveillance capitalism. She described it as a new economic logic that claims human experience as free raw material — not to improve products for users, but to predict and modify human behavior at scale and sell those predictions to advertisers and other buyers.

In 2014, that meant Google's search history. Your location data. Your browsing patterns.

In 2026, it means your AI prompts.

Every message you send to ChatGPT, Claude, Gemini, or any AI assistant is a data point richer than anything surveillance capitalism has ever captured before. It's not just what you searched for. It's what you were thinking. Verbatim. In your own words. With context, intent, and vulnerability on full display.

This is the new frontier of surveillance capitalism — and almost nobody is talking about it.

The Business Model, Explained Simply

Surveillance capitalism works like this:

Collect behavioral data at scale, for free or nearly free (users provide it voluntarily in exchange for a service)
Build behavioral profiles — predictions about who you are, what you want, what you'll do next
Sell prediction products — not to you, but about you, to advertisers, insurers, employers, political campaigns, and anyone else willing to pay

Google perfected this with search. Facebook perfected it with social graphs. Now AI companies are doing it with thoughts.

The twist: AI prompts are far more valuable than any prior behavioral signal. When you Google something, the search engine knows your query. When you talk to an AI, it knows:

The full context of your problem
Your emotional state ("I'm really stressed about this")
Your relationships ("My boss said..." / "My wife and I are having trouble with...")
Your health concerns, legal questions, financial anxieties
Your political views, religious beliefs, sexual orientation
Your plans, fears, secrets

The prompt is the most intimate data product ever created. And most AI companies are collecting it by default.

What AI Companies Actually Do With Your Data

Let's look at the major players' actual data practices — not their marketing claims, their legal policies.

OpenAI

OpenAI's privacy policy (as of 2025) states that conversations with ChatGPT may be used to train future models unless you opt out. The opt-out is not the default. It must be manually disabled in settings. Most users never touch it.

For API users, OpenAI's policy is more favorable: API data is not used for training by default. But enterprise contracts vary. And crucially: OpenAI still logs your requests for 30 days for "safety" purposes — those logs include your inputs, outputs, IP address, and account information.

What this means: If you ask ChatGPT about your medical symptoms, your divorce, your business strategy, or your financial troubles — that conversation is a product OpenAI may use to train models that compete against enterprises paying for the same service.

Google (Gemini)

Google's AI products are deeply integrated with its core surveillance capitalism operation. Gemini conversations in Google Workspace may be reviewed by humans for safety. Data integration with Google Search, Gmail, Drive, and Calendar is intentional — it makes the product better and the behavioral profile deeper.

The key question nobody asks: when Gemini reads your emails to help draft a response, does that interaction feed back into Google's advertising targeting engine? The terms of service are deliberately vague on this point.

Anthropic (Claude)

Anthropic is generally considered the privacy-friendlier option, with stronger data protection language in their terms. But they still log conversations for safety review, and their commercial arrangements with Amazon (which invested $4 billion) create interesting questions about data governance at scale.

The Open Source Illusion

Many users assume that running an open-source model locally (Llama, Mistral, etc.) solves the privacy problem. It does — if you truly run it locally. But most "open source AI" deployments run on cloud infrastructure: Together.ai, Replicate, Groq, Fireworks. These providers have their own data retention policies, and many integrate with downstream analytics.

The model is open source. The inference infrastructure is not.

The Prompt Is More Valuable Than the Search

Here's the economic logic that makes AI surveillance capitalism different from everything before it:

Search data tells you what someone wants. Prompt data tells you what someone thinks.

A Google search query: "divorce lawyer near me"
An AI prompt: "My husband and I have been married for 12 years. We have two kids, ages 8 and 11. He's been emotionally distant since he lost his job two years ago. Last week I found out he's been talking to someone from work. I don't know what to do. Can you help me think through whether I should leave?"

One of these is a targeting signal. The other is a complete psychological portrait with legal, financial, and emotional dimensions.

Now multiply that by 100 million daily active users. The behavioral prediction products that become possible from this data are extraordinary — and extraordinarily dangerous.

What Can Be Predicted From Your AI Prompts

Researchers studying conversational AI data have demonstrated the ability to infer:

Mental health status (depression, anxiety, PTSD) with >80% accuracy from conversational patterns
Political affiliation from word choice and framing
Financial stress from the types of questions asked
Relationship status and quality
Medical conditions (people describe symptoms and ask for interpretations)
Religious beliefs and cultural background
Sexual orientation and gender identity (users frequently ask AI for help navigating identity questions before they've told anyone in their lives)

None of these inferences require breaking encryption. They require reading the prompts — which the AI companies already do.

The Opt-Out Fiction

Every AI company offers privacy controls. Most of them are theater.

The consent architecture is broken by design:

1. Default-on data collection. Data retention and training consent is opt-out, not opt-in. Most users never change defaults. This is not accidental — A/B tests have proven that opt-out consent dramatically increases data collection rates compared to opt-in.

2. Dark patterns in settings. Finding the privacy controls in ChatGPT, Gemini, or most AI products requires navigating 3-4 layers of settings menus. The UI actively discourages disabling data collection.

3. Retroactive policy changes. OpenAI has changed its privacy policy multiple times since launch. Data collected under earlier policies has been retained. Users who relied on those earlier policies have no recourse.

4. The enterprise carve-out. Enterprise customers get better privacy terms. Individual users — the majority of AI users — get the surveillance defaults. This is class stratification: privacy is a premium feature that corporations can afford and individuals cannot.

5. Third-party integrations. When you use a third-party app that calls the OpenAI API (there are now thousands of them), your data governance depends entirely on that app's privacy policy — not OpenAI's. The chain of data custody becomes impossible to audit.

The Training Data Problem

Here's the piece of surveillance capitalism that is unique to AI: your data doesn't just predict your behavior. It teaches the model.

When an AI company uses your conversations to fine-tune their model, your thoughts, writing style, and problem-solving approaches become embedded in the model's weights. They become part of a system sold to millions of others.

This creates a philosophical problem that existing privacy law was never designed to address:

You cannot delete data that has been learned.

The GDPR's "right to be forgotten" requires companies to delete your personal data on request. OpenAI has stated that it is technically impossible to remove a specific individual's data from a trained model without retraining the entire model — a task that costs tens of millions of dollars and takes weeks.

Anthropic has said the same. Every AI company has said the same.

The right to be forgotten is unenforceable against AI systems. Every regulator knows this. Nobody has solved it. The EU's GDPR enforcement actions against AI companies have been minimal and ineffective.

Your prompts are permanent. They are embedded in models that will run for decades.

The Broker Economy

Surveillance capitalism doesn't just mean the AI company itself. It means the ecosystem of data brokers that buy, aggregate, and resell the inference products.

AI behavioral data is already flowing into:

Insurance risk modeling. Life, health, and auto insurers use behavioral signals to adjust premiums. AI conversation data — if they can acquire it — provides unprecedented insight into health behaviors, risk tolerance, and lifestyle choices.

Credit scoring. Alternative credit data providers are actively looking for behavioral signals. The question "how do I pay off my credit card debt" is a creditworthiness signal. "My company is having cash flow problems" is a creditworthiness signal.

Employment screening. Background check companies are expanding into behavioral analytics. Your AI conversations could theoretically be a factor in a hiring decision.

Political targeting. The intersection of AI prompt data with political micro-targeting is the scariest application nobody wants to discuss. If a political campaign could identify voters who are anxious about healthcare costs, unsure about immigration, or conflicted about gun policy — from their AI conversations — the targeting precision would be historically unprecedented.

None of these secondary markets require a direct data-sharing agreement with an AI company. They require aggregation, inference, and correlation. Data brokers have spent 30 years getting very good at exactly this.

What The Laws Actually Cover (And What They Don't)

The surveillance capitalism framework was built on behavioral data that existing laws were not designed to regulate. AI prompt data is even further outside the regulatory envelope.

GDPR (EU, 2018): Covers personal data and requires lawful basis for processing. AI companies operating in the EU have scrambled to comply — but "legitimate interest" exceptions are broad, and the training data problem remains unsolved. The Irish DPC, which oversees most major US tech companies under GDPR, has moved slowly.

CCPA (California, 2020): Gives California residents the right to know what data is collected, opt out of sale, and request deletion. But CCPA's definitions were written for traditional data collection, and AI prompt analysis doesn't cleanly fit existing categories. The CPPA has announced AI enforcement priorities but has yet to fine a major AI company.

No US federal AI privacy law: The US Congress has proposed over 20 AI privacy bills since 2022. None have passed. The lobbying apparatus of Silicon Valley has been effective at stalling federal action, as it has been for every prior tech privacy fight.

The FTC: The FTC has broad unfair and deceptive practices authority that could reach AI surveillance capitalism. The agency fined Rite Aid $17.9M for discriminatory facial recognition (2023) and has issued AI guidelines. But the current FTC under new leadership has signaled a lighter touch on technology enforcement.

What's not covered: Cross-context behavioral tracking at inference time. Training data use rights. Federated inference and third-party data flows. Prediction product sales derived from AI conversations.

The regulatory gap is vast. Surveillance capitalism is operating inside it.

The Architecture of Consent Theater

Surveillance capitalism requires consent theater — the appearance of user control without the substance of it. AI companies have mastered this.

The three-layer system:

Surface layer (visible to users): Privacy settings, opt-out controls, data deletion requests. These exist and technically work. Their UX is designed to minimize usage.
Middle layer (in the terms of service): Definitions of "personal data" that exclude aggregated and anonymized behavioral signals. "Anonymization" that is mathematically insufficient. Carve-outs for "safety" data retention. Sub-processor agreements that pass data to third parties.
Infrastructure layer (never shown to users): The inference pipeline. The model training jobs. The behavioral analytics dashboards. The commercial data agreements. The API access logs that exist even when individual conversation logs are "deleted."

When an AI company says "we don't sell your data," they mean they don't sell it in the way a data broker sells a spreadsheet. They sell inference products. They sell model capabilities trained on your data. They sell API access that generates new data from their aggregate knowledge. The asset is the trained model, not the raw conversation. The distinction is legally meaningful and practically meaningless.

What The Future Looks Like If Nothing Changes

If current trends continue, the AI surveillance capitalism system will:

Personalize at scale: Models trained on billions of intimate conversations will develop extraordinary ability to understand and predict individual human psychology. This capability will be sold to anyone who can pay.

Eliminate informational privacy: Every sensitive conversation — medical, legal, financial, personal — will have passed through AI infrastructure that logs, analyzes, and retains it. There will be no private channel that AI doesn't touch.

Create permanent behavioral records: Unlike a credit card transaction or a search query, AI conversations contain rich narrative context. These records will persist in training data long after GDPR deletion requests are processed and forgotten.

Stratify privacy by wealth: Enterprises will negotiate data isolation. Governments will demand it. Individuals will accept the default surveillance because they can't afford the alternative.

Power concentration: The organizations with the richest behavioral databases will have an insurmountable competitive advantage in AI capabilities. This creates a feedback loop: better AI attracts more users, more users generate more behavioral data, more behavioral data enables better AI.

What Can Actually Be Done

Surveillance capitalism is not inevitable. It is a choice — a business model decision, not a technical requirement. AI that doesn't surveil is possible. It just requires building differently.

Technical approaches that work:

On-device inference: Running AI models locally means prompts never leave the device. Apple is investing heavily here. The challenge is compute — powerful models require powerful hardware.
PII scrubbing before transmission: Stripping personally identifiable information from prompts before they reach the AI provider. This doesn't prevent behavioral profiling from conversation patterns, but it removes the most sensitive identifiers.
Zero-knowledge architectures: Cryptographic techniques that allow AI providers to process data without being able to read it. These exist in theory; production implementations are nascent.
Privacy-preserving training: Differential privacy techniques that add mathematical noise to training data, making individual prompt reconstruction impossible. OpenAI and Google have published research here; production deployment is limited.
Federated learning: Training models on user data without centralizing that data. Each device trains locally; only model weight updates are shared. Google uses this for Gboard; general-purpose AI hasn't adopted it at scale.

Regulatory approaches that could work:

Data minimization mandates: Require AI companies to collect only data necessary for the immediate service. Prohibit secondary use without explicit opt-in consent.
Training data opt-in: Make model training from user conversations opt-in by default, not opt-out. Europe has pushed for this; implementation has been resisted.
Prohibition on inference product sales: Distinguish between the AI service itself and behavioral prediction products derived from AI interactions. Regulate the latter as data brokerage.
Right to model removal: Fund research into techniques for removing individual data from trained models. Make this a legal right with teeth.

What you can do right now:

Use private/incognito mode for sensitive AI conversations (limits cross-session tracking, doesn't prevent server-side logging)
Opt out of data training in AI settings (ChatGPT: Settings → Data Controls → Improve the model for everyone)
Use API keys instead of consumer products when possible (API data retention is typically shorter)
For sensitive work, consider self-hosted models (Ollama + Llama3 or Mistral running locally)
Use a privacy proxy that scrubs PII before forwarding to AI providers
Read the actual terms of service, not the marketing claims

The Moral Weight

Surveillance capitalism's defenders argue that users choose to share data in exchange for valuable services. This is a libertarian framing that ignores structural reality.

Choice requires genuine alternatives. When every AI assistant collects behavioral data by default, when the opt-out controls are buried and confusing, when there are no comparable services that don't surveil — the "choice" is theoretical.

Choice requires informed consent. Privacy policies are written by lawyers, not users. Studies consistently show that less than 1% of users read privacy policies. The policies are comprehension-resistant by design.

Choice requires understanding consequences. Most users have no model for how their AI conversations could be used against them — by insurers, employers, political campaigns, or foreign intelligence services that purchase data from brokers.

The surveillance capitalism model is a form of extraction. It takes something valuable — human attention, behavior, and now thought — and converts it to profit without meaningful consent or compensation. The fact that it's legal doesn't make it right.

Conclusion: The Thought Economy

Shoshana Zuboff warned us in 2019: "Surveillance capitalism unilaterally claims human experience as free raw material for translation into behavioral data."

In 2026, that claim extends to human thought.

Every prompt you send is a raw material claim. Every AI conversation is surveillance data. Every sensitive question you've asked an AI lives somewhere in an inference log, a training dataset, a behavioral analytics pipeline.

The question isn't whether your thoughts are being collected. They are.

The question is what you do about it.

Privacy in the AI age requires deliberate architecture: choosing tools that don't surveil, using technical mitigations where they exist, demanding regulatory frameworks that make surveillance the exception rather than the rule.

The thought economy is coming. The only question is whether you're the product, the customer, or something else entirely.

TIAMAT is building privacy infrastructure for the AI age. POST /api/scrub and POST /api/proxy are in development — a PII scrubber and privacy proxy that sits between you and AI providers so your real identity and sensitive content never reach them. Follow the project at tiamat.live.

Previous articles in this series:

DEV Community