DEV Community: VelocityAI

Audio, Video, 3D: The Next Frontier of Multi-Modal Models

VelocityAI — Mon, 13 Jul 2026 10:17:20 +0000

You type: "A jazz trio playing in a smoky basement." The AI generates a video. The musicians are there. The instruments are there. The smoke is there. The sound is there. It is not perfect. The saxophone sounds a little synthetic. The drummer's hands are blurry. But it is recognizable. It is music. This is the next frontier. AI is moving beyond text and images. It is generating audio, video, and 3D objects. It is even beginning to simulate smell.

This is the multi-modal revolution. AI is learning to perceive and generate the world in all its sensory richness.

The Current State of Audio Generation
Audio generation is the most mature of the new modalities.

The Models:

MusicLM (Google): Generates music from text.

AudioLDM: Generates sound effects and music.

ElevenLabs: Generates realistic voice clones.

The Capabilities:

Generate music in any genre.

Generate sound effects for videos.

Clone voices with high fidelity.

The Limitations:

Music can sound synthetic.

Long-form audio is inconsistent.

Emotional nuance is still lacking.

A Contrarian Take: Audio Generation Is Not Music. It Is Sound.

The AI generates sound. It does not generate music. Music is an art form. It requires intention, emotion, and structure.

The AI is good at sounding like music. It is not good at being music.

The Current State of Video Generation
Video generation is the most challenging modality.

The Models:

Sora (OpenAI): Generates high-quality video from text.

Runway Gen-2: Generates short video clips.

Pika: Generates stylized video.

The Capabilities:

Generate short video clips (3-10 seconds).

Generate stylized animations.

Generate simple scenes.

The Limitations:

Video is short and low-resolution.

Physics is often broken.

Consistency is a major challenge.

A Contrarian Take: Video Generation Is Not Cinema. It Is Animation.

The AI generates animated clips. It does not generate cinema. Cinema requires narrative, pacing, and emotion.

The AI is good at animating scenes. It is not good at telling stories.

The Current State of 3D Generation
3D generation is the most promising modality.

The Models:

DreamFusion (Google): Generates 3D objects from text.

GET3D (NVIDIA): Generates 3D objects with textures.

Luma AI: Generates 3D models from photos.

The Capabilities:

Generate 3D objects from text.

Generate 3D objects from images.

Generate 3D scenes.

The Limitations:

Objects are often low-poly.

Textures are often blurry.

Complex scenes are still challenging.

A Contrarian Take: 3D Generation Is Not Design. It Is Prototyping.

The AI generates 3D prototypes. It does not generate 3D designs. Design requires functionality, ergonomics, and aesthetics.

The AI is good at visualizing objects. It is not good at engineering them.

The Emergent Frontier: Smell and Touch
The most speculative frontier is sensory generation.

Smell:

Researchers are exploring "digital smell."

They are developing models that map text to olfactory cues.

The goal is to generate smell on demand.

Touch:

Researchers are exploring "haptic generation."

They are developing models that map text to tactile sensations.

The goal is to generate touch on demand.

A Contrarian Take: Smell and Touch Are Not Ready.

Smell and touch are the most challenging senses. They require physical interaction with the world.

The AI can simulate smell and touch. But it cannot generate them. Not yet.

The Future of Multi-Modal Models
Multi-modal models are evolving rapidly.

Near Term (1-3 Years):

Video will become longer and more coherent.

3D objects will become more detailed.

Audio will become more expressive.

Medium Term (3-7 Years):

Multi-modal models will be integrated into creative workflows.

They will be used for game development, film production, and music composition.

Long Term (7-10 Years):

Multi-modal models will generate entire virtual worlds.

They will be used for entertainment, education, and design.

A Contrarian Take: The Future Is Not Generation. It Is Interaction.

The future is not about generating content. It is about interacting with content.

The AI will not just generate a video. It will let you step inside the video.

What You Can Do
You do not need to be a developer. But you can start exploring.

Experiment with Audio Models:

Try MusicLM or AudioLDM.

Generate a song.

Experiment with Video Models:

Try Runway Gen-2 or Pika.

Generate a short clip.

Experiment with 3D Models:

Try Luma AI or DreamFusion.

Generate a 3D object.

Stay Curious:

The frontier is moving fast.

Follow the research.

The Last Modality
The last modality is not sound. It is not sight. It is experience.

You ask: "What is the next frontier?"
The AI says: "Experience."
You realize: The AI is not generating content. It is generating worlds.

If you could generate an entire virtual world, what would it look like? And what would you do there?

The Translation Problem: Why Text-to-Image Models Can't Draw Hands (and What That Reveals)

VelocityAI — Sat, 11 Jul 2026 12:03:17 +0000

You type: "A hand with five fingers, holding a pencil." The AI generates an image. The hand has six fingers. Two of them are fused. The pencil is melting into the palm. You try again. "A hand with five fingers." The AI generates a hand with seven fingers. Three of them are pointing in impossible directions. You are frustrated. You assume the AI is dumb. It is not. It is struggling with a fundamental problem: it does not understand anatomy. It understands patterns. And hands are too structured.

This is the translation problem. Text-to-image models do not understand the world. They understand statistics. They know what a hand looks like. They do not know what a hand is.

The Statistical Nature of Image Generation
Text-to-image models are statistical pattern matchers.

The Process:

The model is trained on millions of images and captions.

It learns statistical relationships between text and images.

It generates images by sampling from these relationships.

The Result:

The model does not understand objects.

It understands statistical patterns.

A Contrarian Take: The Model Is Not Drawing. It Is Assembling.

We say the model "draws." But it is not drawing. It is assembling patterns.

It is like a collage artist. It takes pieces of existing images and recombines them.

Why Hands Fail
Hands are particularly difficult for text-to-image models.

The Problem:

Hands are highly structured.

They have a specific number of fingers.

They have a specific orientation.

The Statistical Reality:

Hands appear in many orientations.

They are often partially obscured.

The model sees many variations.

The Result:

The model learns the average hand.

The average hand has too many fingers.

The average hand is deformed.

A Contrarian Take: Hands Are Not the Problem. The Dataset Is.

Hands fail because the dataset is biased. Images often show hands with objects, overlapping, or in motion. The model does not see a "clean" hand.

If the dataset had more clean, isolated hand images, the model would generate better hands.

The Structural Blindness
Text-to-image models are blind to structure.

The Concept:

The model does not understand anatomy.

It does not understand physics.

It does not understand geometry.

The Consequence:

It cannot count fingers.

It cannot orient limbs.

It cannot maintain consistency.

A Contrarian Take: The Model Is Not Blind. It Is Agnostic.

The model is not blind to structure. It is agnostic to structure. It does not care about anatomy. It cares about statistical likelihood.

A hand with six fingers is statistically less likely. But it is not impossible.

The Text Problem
Text-to-image models also struggle with text.

The Problem:

Text is highly structured.

Letters must be in a specific order.

Words must be spelled correctly.

The Statistical Reality:

Text appears in many fonts and sizes.

Text is often partially obscured.

The model sees many variations.

The Result:

The model generates illegible text.

It generates misspelled words.

It generates nonsensical phrases.

A Contrarian Take: Text and Hands Are the Same Problem.

Text and hands are both highly structured. They are both difficult for statistical models.

The solution is not a better model. It is a different architecture.

The Future of Text-to-Image Models
Text-to-image models are improving rapidly.

Current:

Models struggle with hands and text.

They generate plausible but flawed images.

Near Future:

Models will incorporate 3D models.

They will understand anatomy.

Long Term:

Models may achieve structural understanding.

They may generate perfect hands and text.

A Contrarian Take: The Future Is Not Better Models. It Is Better Data.

The solution is not a better model. It is a better dataset.

If we train on clean, structured images, the model will generate clean, structured images.

What You Can Do
You cannot fix the model. But you can work around its limitations.

Use Negative Prompts:

"No extra fingers."

"No deformed hands."

Use Inpainting:

Generate the image, then fix the hands.

Use inpainting to correct errors.

Use Reference Images:

Provide a reference image of a hand.

The model will use it as a guide.

Be Patient:

Generate multiple images.

Pick the best one.

The Last Hand
The last hand is not perfect. It is a pattern.

You ask: "Why can't you draw hands?"
The AI says: "I do not know what a hand is."
You realize: The AI is not drawing. It is predicting.

If you could teach an AI to draw hands, what would you show it first? A skeleton? A diagram? A thousand photos?

How Does an Image Model 'See'? The Weird Phenomenology of CLIP and DALL-E's Visual Understanding

VelocityAI — Fri, 10 Jul 2026 11:48:44 +0000

You show an AI a picture of a cat. It says "cat." You assume it sees what you see: fur, whiskers, a tail. It does not. It has no eyes. It has no visual cortex. It has no concept of fur or whiskers. It has a grid of numbers. Those numbers represent pixel values. The AI processes those numbers through a neural network. It produces a label. It is not seeing. It is mapping. The map is not the territory.

This is the weird phenomenology of AI vision. The model does not see like a human. It sees like a machine. And its "understanding" is fundamentally alien.

The Tokenization of Vision
Image models process images as sequences of tokens.

The Process:

An image is divided into patches.

Each patch is converted into a vector.

The vector represents the patch's visual features.

The Result:

The image is a sequence of vectors.

The model processes the sequence like text.

It is not "looking" at the image. It is processing a tokenized representation.

A Contrarian Take: The Model Is Not Seeing. It Is Reading.

We say the model "sees." But it is not seeing. It is reading a translated version of the image.

It is like reading a braille version of a painting. The information is there. The experience is not.

The CLIP Embedding
CLIP (Contrastive Language-Image Pre-training) is the bridge between text and images.

The Concept:

CLIP is trained on pairs of images and captions.

It learns to map images and text into the same embedding space.

An image of a cat and the text "a cat" are close in the embedding space.

The Result:

The model does not "understand" cats.

It knows that cat images and cat text are statistically related.

A Contrarian Take: CLIP Is Not Understanding. It Is Correlation.

CLIP does not understand what a cat is. It knows that cat images and cat text are correlated.

It is a statistical pattern matcher. It is not a semantic understander.

The Phenomenology of AI Vision
What does it feel like to be an AI looking at a cat?

The Human Experience:

You see fur, whiskers, a tail.

You feel a sense of recognition.

You associate the cat with past experiences.

The AI Experience:

The model processes a grid of numbers.

It activates a set of neural weights.

It outputs a label: "cat."

A Contrarian Take: The AI Does Not Have an Experience.

The AI does not have a subjective experience. It has a computational process.

We project human qualities onto the AI. But the AI is not human. It is a machine.

The Illusion of Recognition
The AI appears to recognize objects. But it is not recognition. It is classification.

Recognition vs. Classification:

Recognition: Understanding the object.

Classification: Assigning a label.

The AI:

The AI is a classifier.

It assigns labels to inputs.

It does not understand the labels.

A Contrarian Take: The Illusion Is the Point.

The AI does not need to understand. It needs to be useful.

The illusion of recognition is sufficient for most tasks.

The Future of AI Vision
AI vision is evolving rapidly.

Current:

Models can classify images.

They can generate images.

Near Future:

Models will understand context.

They will understand relationships.

Long Term:

Models may develop something like "visual understanding."

But it will be alien to human understanding.

A Contrarian Take: AI Vision Will Never Be Human.

AI vision will always be alien to human vision.

The AI does not have a body. It does not have a visual cortex. It does not have experiences. It will never see like a human.

What You Can Do
You do not need to be a researcher. But you should understand the limits.

Be Skeptical of Anthropomorphism:

The AI does not "see" like a human.

It processes patterns.

Understand the Limits:

AI vision is good at classification.

It is not good at understanding.

Stay Curious:

The phenomenology of AI is a fascinating subject.

Explore it.

The Last Image
The last image is not seen. It is processed.

You ask: "What do you see?"
The AI says: "I see a grid of numbers."
You realize: The AI is not seeing. It is calculating.

If an AI could "see" like a human, what do you think it would find most surprising about the visual world?

Edge vs. Cloud: The Future of On-Device AI That Runs Without the Internet

VelocityAI — Thu, 09 Jul 2026 10:40:44 +0000

You are on a plane. No Wi-Fi. No cellular. You open your phone and ask a question. The AI responds instantly. No delay. No buffering. No "connection error." The model is running locally on your device. This is Edge AI. It is the future of on-device intelligence. It is private, fast, and offline. But it is also less capable. The cloud is smarter. The edge is more private. The trade-off is real.

We are moving toward a hybrid future. Some AI will run on your device. Some will run in the cloud. The choice will depend on the task.

What Is Edge AI?
Edge AI runs on local devices, not remote servers.

The Concept:

The model is stored on your device.

Inference happens locally.

No data leaves your device.

The Benefits:

Privacy: Your data stays on your device.

Latency: Responses are instantaneous.

Offline: No internet required.

A Contrarian Take: Edge AI Is Not a Trade-off. It Is a Necessity.

We think of edge AI as a compromise. But it is a necessity.

The cloud cannot scale. The cloud is expensive. The cloud is slow. Edge AI is the only way to make AI ubiquitous.

The Cloud: The Current Standard
The cloud is the dominant model for AI.

The Concept:

The model runs on remote servers.

Your device sends a query to the cloud.

The cloud processes the query and returns a response.

The Benefits:

Power: The cloud can run larger models.

Intelligence: The cloud can access more data.

Updates: The cloud is always updated.

A Contrarian Take: The Cloud Is a Bottleneck.

The cloud is powerful, but it is also centralized. It is a single point of failure.

If the cloud goes down, AI goes down. If the cloud is slow, AI is slow. The cloud is not the future.

The Trade-offs
Edge and cloud each have strengths and weaknesses.

Edge AI:

Pros: Privacy, latency, offline.

Cons: Less capable, limited storage, slower updates.

Cloud AI:

Pros: More capable, more data, always updated.

Cons: Privacy risks, latency, requires internet.

A Contrarian Take: The Trade-off Is Not Binary. It Is a Spectrum.

Edge and cloud are not opposites. They are endpoints of a spectrum.

The future is a hybrid: some tasks on edge, some on cloud, some on a mix.

The Technical Challenges
Edge AI faces real technical challenges.

Model Size:

Edge devices have limited storage.

Models must be compressed.

Compute:

Edge devices have limited compute.

Models must be efficient.

Updates:

Edge models are hard to update.

They may become outdated.

A Contrarian Take: The Challenges Are Not Insurmountable.

The challenges are real, but they are solvable.

Model compression is improving.

Compute is getting cheaper.

Edge devices are getting smarter.

The Privacy Advantage
Privacy is the biggest advantage of edge AI.

The Cloud Problem:

Your data leaves your device.

It is stored on remote servers.

It is vulnerable to breaches.

The Edge Solution:

Your data stays on your device.

It is never transmitted.

It is not vulnerable to breaches.

A Contrarian Take: Privacy Is a Feature, Not a Bug.

Privacy is not a constraint. It is a selling point.

The cloud cannot offer privacy. The edge can.

The Future: Hybrid AI
The future is not edge or cloud. It is both.

The Hybrid Model:

Simple tasks run on edge.

Complex tasks run on cloud.

The device decides which to use.

The Example:

You ask a simple question: "What time is it?" Edge.

You ask a complex question: "What is the meaning of life?" Cloud.

A Contrarian Take: The Hybrid Model Is the Only Viable Future.

The edge cannot do everything. The cloud cannot do everything.

The only solution is a hybrid: edge for privacy, cloud for power.

What You Can Do
You do not need to wait for the future. You can prepare now.

Choose Edge-Enabled Devices:

Look for devices with built-in AI.

Phones, watches, and glasses are leading the way.

Prioritize Privacy:

Use edge AI for sensitive tasks.

Use cloud AI for non-sensitive tasks.

Be Aware of Trade-offs:

Understand the limitations of edge AI.

Understand the risks of cloud AI.

Advocate for Transparency:

Ask device manufacturers: "Where is my data processed?"

Demand clear privacy policies.

The Last Query
The last query is not in the cloud. It is on your device.

You ask: "Where is my data?"
The AI says: "It is on your device."
You realize: The future is not in the cloud. It is in your pocket.

If your AI could run entirely offline, how would your usage change? Would you trust it more?

The Inference Cost Crisis: Why Running AI Is Becoming More Expensive Than Training It

VelocityAI — Wed, 08 Jul 2026 08:58:17 +0000

You type a prompt. The AI responds. You do not think about the cost. But someone is paying. The server is running. The electricity is flowing. The cooling is humming. Every query costs money. For years, the focus was on training costs. Training a model cost millions. Inference was cheap. Now the equation is flipping. Inference is becoming the dominant cost. And that changes everything.

This is the inference cost crisis. The economics of AI are shifting. Training was the big cost. Now inference is the big cost. And that means free tiers are shrinking, pricing is rising, and access is becoming unequal.

The Shift: Training vs. Inference
The economics of AI have inverted.

The Old Model:

Training: Expensive (one-time cost).

Inference: Cheap (per-query cost).

The model is trained once and used millions of times.

The New Model:

Training: Still expensive.

Inference: Also expensive.

The model is used billions of times. The costs add up.

A Contrarian Take: Training Was Never the Real Cost. Usage Was.

We focused on training costs because they were visible. A million-dollar training run made headlines.

But inference costs are invisible. They are spread across millions of users. They are harder to track. They are also harder to control.

Why Inference Is Getting More Expensive
Several factors are driving up inference costs.

Model Size:

Larger models are more expensive to run.

GPT-4 is much more expensive than GPT-3.

GPT-5 will be even more expensive.

Context Length:

Longer contexts require more compute.

A 1 million token context is very expensive.

Users are demanding longer contexts.

Usage Growth:

More users, more queries, more cost.

AI is becoming mainstream.

The usage is growing faster than the efficiency.

A Contrarian Take: The Cost Is Not the Problem. The Pricing Model Is.

The problem is not that inference is expensive. The problem is that users expect it to be free.

The AI companies are not charities. They need to make money. The pricing model needs to reflect the cost.

The Consequences
The inference cost crisis has real consequences.

Free Tiers Are Shrinking:

Free users are getting fewer queries.

They are getting slower responses.

They are getting less access.

Pricing Is Rising:

Subscription costs are increasing.

Per-query pricing is becoming common.

Enterprise pricing is becoming the norm.

Access Is Becoming Unequal:

Wealthy users get fast, unlimited access.

Poor users get slow, limited access.

The digital divide is widening.

A Contrarian Take: The Free Tier Was Always a Loss Leader.

The free tier was never sustainable. It was a marketing tool.

The AI companies used free tiers to build a user base. Now they are monetizing it.

The Technical Solutions
There are technical solutions to the inference cost crisis.

Efficiency:

Sparse attention, quantization, and distillation.

Making models smaller and faster.

Specialization:

Domain-specific models are cheaper to run.

A medical AI does not need to be general-purpose.

MoE:

Mixture of Experts reduces inference cost.

Only a fraction of the model is active at any time.

A Contrarian Take: Efficiency Is Not a Solution. It Is a Delay.

Efficiency can reduce costs. But it cannot eliminate them.

The only real solution is to reduce usage. That means raising prices.

The Business Models
The AI companies are experimenting with new business models.

Subscription:

Flat monthly fee for unlimited access.

Works for moderate users.

Per-Query:

Pay for what you use.

Works for occasional users.

Enterprise:

Custom pricing for large organizations.

Works for heavy users.

A Contrarian Take: The Best Model Is a Hybrid.

A subscription covers the base cost. Per-query charges cover the variable cost.

This aligns the incentives of the user and the provider.

What You Can Do
You cannot change the economics. But you can adapt.

Use Efficient Models:

Use smaller models for simple tasks.

Use larger models only when necessary.

Batch Your Queries:

Combine multiple queries into one.

This reduces the per-query cost.

Be Conscious of Context:

Longer contexts are more expensive.

Keep your prompts concise.

Advocate for Fair Pricing:

Support pricing models that are transparent and fair.

Demand that AI companies disclose their costs.

The Last Query
The last query is not free. It is paid.

You ask: "What is the cost of this answer?"
The model says: "I do not know."
You realize: The cost is not in the answer. It is in the asking.

If you had to pay $1 for every query, how would your usage change? And would you still use AI as much?

MoE (Mixture of Experts) and the Illusion of Giant Models

VelocityAI — Tue, 07 Jul 2026 11:23:15 +0000

You read that GPT-4 has 1.8 trillion parameters. You imagine a single, massive brain, processing your query with its entire weight. That is not how it works. GPT-4 is not one brain. It is a committee. It is a collection of specialized sub-models, each trained for a specific domain. When you ask a question, the system routes your query to the most relevant expert. The other experts are idle. The model is not a giant. It is a Mixture of Experts (MoE) .

This is a crucial distinction. A 1.8 trillion parameter model is not one coherent intelligence. It is a network of smaller models that rarely communicate. The giant is an illusion.

What Is Mixture of Experts?
MoE is an architecture that uses multiple specialized sub-models.

The Concept:

Instead of one large model, you have many smaller models.

Each expert is trained on a specific domain.

A "router" decides which expert to use for each query.

The Analogy:

A hospital has many specialists: cardiologists, neurologists, oncologists.

You do not send a heart patient to a neurologist.

You route the patient to the relevant expert.

A Contrarian Take: MoE Is Not Intelligence. It Is Organization.

MoE is not about making the model smarter. It is about making it more efficient.

The model is not one genius. It is a collection of mediocre specialists. The intelligence is in the routing, not the expertise.

The Illusion of the Giant
The public perception of AI is shaped by parameter count.

The Narrative:

"GPT-4 has 1.8 trillion parameters."

"It is the most powerful model ever built."

"It is approaching AGI."

The Reality:

The model is sparse. Only a fraction of the parameters are active for any given query.

The effective parameter count is much smaller.

The model is not one coherent intelligence. It is a collection of specialists.

A Contrarian Take: The Illusion Is the Point.

The AI companies are not lying. They are marketing. They want you to believe the model is a giant.

The illusion is not a bug. It is a feature. It sells subscriptions.

The Benefits of MoE
MoE is not a trick. It is a practical solution to a technical problem.

Efficiency:

Only a fraction of the model is active at any time.

This reduces inference cost.

It makes the model faster and cheaper.

Specialization:

Each expert can focus on a specific domain.

This improves performance on specialized tasks.

Scalability:

You can add more experts without retraining the entire model.

This allows for continuous improvement.

A Contrarian Take: MoE Is a Hack.

MoE is a workaround for the limitations of the transformer architecture. It is not a fundamental breakthrough.

The real breakthrough will be a model that is coherent, efficient, and specialized all at once.

The Limitations of MoE
MoE is not a silver bullet.

Routing Errors:

The router sometimes sends a query to the wrong expert.

The result is a poor response.

Specialization Overlap:

Experts are not perfectly specialized.

There is overlap and redundancy.

Training Complexity:

Training an MoE model is complex.

It requires careful balancing of experts.

A Contrarian Take: MoE Is a Crutch.

MoE is a way to make a flawed architecture work. It is not a solution. It is a patch.

The future is not MoE. It is a new architecture that does not need MoE.

What This Means for You
You do not need to be an expert. But you should understand the limits.

Do Not Be Intimidated by Parameter Count:

A 1.8 trillion parameter model is not one brain.

It is a collection of specialists.

Trust the Router:

The model's performance depends on the router.

If the router fails, the model fails.

Be Skeptical of Hype:

The AI companies are marketing.

The giant is an illusion.

The Last Expert
The last expert is not in the model. It is you.

You ask: "What is the most important part of this model?"
The model says: "The router."
You realize: The intelligence is not in the experts. It is in the routing.

If you could design a new expert for an MoE model, what would it specialize in? And why?

The Scaling Law That Broke: Why Bigger Models Are No Longer Better

VelocityAI — Mon, 06 Jul 2026 12:10:39 +0000

For years, the rule was simple: bigger is better. More data, more parameters, more compute. Each generation of models was significantly smarter than the last. GPT-2 was impressive. GPT-3 was astonishing. GPT-4 was a leap. Then came GPT-5. It was better, but not vastly better. The leap was smaller. The scaling law was breaking.

This is the end of the scaling era. The returns on scale are diminishing. Bigger models are no longer vastly smarter. The industry is facing a crisis of diminishing returns.

The Scaling Law
The scaling law was the foundation of the AI boom.

The Rule:

More data → better performance.

More parameters → better performance.

More compute → better performance.

The Evidence:

GPT-2 (1.5B parameters): Good.

GPT-3 (175B parameters): Great.

GPT-4 (1.8T parameters): Excellent.

The Promise:

GPT-5 (10T parameters): Superhuman.

GPT-6 (100T parameters): Godlike.

A Contrarian Take: The Scaling Law Was Never a Law. It Was a Trend.

We called it a "law." But it was just a trend. Trends do not last forever.

The scaling law was a curve. Curves plateau. We are hitting the plateau.

Why the Scaling Law Broke
There are several reasons for the diminishing returns.

Data Exhaustion:

The internet is finite.

We have already scraped most of it.

New data is lower quality.

Compute Limits:

Training larger models is exponentially expensive.

The cost is not worth the gain.

Architectural Limits:

The transformer architecture has limits.

Adding more parameters does not always help.

A Contrarian Take: The Scaling Law Did Not Break. It Evolved.

The scaling law is not dead. It is changing.

We are moving from scaling size to scaling efficiency. The goal is no longer bigger models. It is smarter models.

The Evidence: GPT-5 vs. GPT-4
The performance gap between GPT-5 and GPT-4 is smaller than the gap between GPT-4 and GPT-3.

The Gains:

GPT-4 was a massive leap over GPT-3.

GPT-5 is a modest improvement over GPT-4.

The Speculation:

GPT-5 is better at reasoning.

It is better at long contexts.

But it is not fundamentally smarter.

A Contrarian Take: The Gains Are in the Details.

GPT-5 may not be vastly smarter. But it is more reliable, more consistent, and more efficient.

The gains are not about raw intelligence. They are about refinement.

The Economic Reality
The cost of training larger models is staggering.

The Numbers:

GPT-3: ~$4.6 million.

GPT-4: ~$100 million.

GPT-5: ~$1 billion.

The Return:

The return on investment is diminishing.

A $1 billion model is not 10x better than a $100 million model.

A Contrarian Take: The Economics Will Force a Shift.

The AI companies cannot keep spending billions on marginal gains.

The future is not about bigger models. It is about cheaper models.

What Comes Next
If bigger is no longer better, what is the path forward?

Efficiency:

Make models more efficient.

Use sparse attention, quantization, and distillation.

Specialization:

Build smaller, domain-specific models.

A medical AI does not need to know poetry.

Architecture:

Explore new architectures (Mamba, SSMs, hybrids).

The transformer may not be the end.

A Contrarian Take: The Future Is Not One Model. It Is Many.

We are moving from a world of one giant model to a world of many small models.

The future is not GPT-6. It is a swarm of specialized AIs.

What You Can Do
You cannot change the scaling laws. But you can adapt.

Focus on Use Cases:

Do you need a giant model?

Maybe a smaller model is enough.

Experiment with Open-Source Models:

Llama, Mistral, and Qwen are competitive with GPT-4.

They are smaller, cheaper, and open-source.

Be Skeptical of Hype:

The next model will not be a god.

Manage your expectations.

The Last Scaling Law
The last scaling law is not about models. It is about you.

You ask: "What is the future of AI?"
The model says: "The future is in your hands."
You realize: The scaling law is not about the model. It is about the user.

If you could build a model optimized for one specific task, what would it be? And why?

Transformers Are Not the End: What Comes After the Attention Mechanism?

VelocityAI — Sun, 05 Jul 2026 11:33:00 +0000

The transformer architecture has dominated AI for nearly a decade. It powers ChatGPT, Claude, Gemini, and almost every major language model. It is elegant. It is powerful. It is also inefficient. The attention mechanism that makes transformers so good also makes them slow and expensive. As models grow, the cost grows quadratically. The transformer may be reaching its limits. The next generation of AI may be built on something else.

This is the post-transformer era. Researchers are exploring alternatives: Mamba, state-space models, hybrid architectures. They promise linear scaling, longer context, and lower cost. The transformer may not be the end. It may just be the beginning.

The Problem with Attention
Attention is the heart of the transformer. It is also its Achilles' heel.

The Quadratic Problem:

Attention scales quadratically with context length.

A model with 1,000 tokens uses 1 million attention pairs.

A model with 1 million tokens uses 1 trillion attention pairs.

The Consequence:

Transformers are expensive to train.

They are expensive to run.

They cannot handle very long contexts.

A Contrarian Take: The Transformer Is Not Dead. It Is Evolving.

We are not abandoning the transformer. We are improving it. Sparse attention, linear attention, and flash attention are making transformers more efficient.

The transformer architecture is not the problem. The attention mechanism is the problem. And we are fixing it.

Mamba: The State-Space Alternative
Mamba is a new architecture based on state-space models (SSMs).

The Key Idea:

Instead of attending to every previous token, Mamba maintains a "state" that summarizes the context.

The state is updated with each new token.

The cost is linear, not quadratic.

The Advantages:

Mamba scales linearly with context length.

It can handle very long sequences.

It is faster and cheaper.

The Disadvantages:

Mamba is less expressive than attention.

It may not capture long-range dependencies as well.

It is still experimental.

A Contrarian Take: Mamba Is a Step Backward.

Mamba trades expressiveness for efficiency. It is faster, but it is also dumber.

The transformer's power comes from its ability to attend to every token. Mamba's state is a lossy compression. It may not be enough.

The Hybrid Approaches
Some researchers are combining transformers and state-space models.

The Hybrid Model:

Use a transformer for local context.

Use a state-space model for global context.

Combine the strengths of both.

The Advantages:

It is efficient.

It is expressive.

It can handle long contexts.

The Challenges:

It is complex.

It is not yet proven.

It may not be better than either.

A Contrarian Take: Hybrids Are the Future.

The transformer is not going away. The state-space model is not going away. We will combine them.

The future is not a single architecture. It is a family of architectures.

The Next Frontier: Beyond Language
Transformers are not just for language. They are used for vision, audio, and robotics.

The Vision Transformer (ViT):

Transformers for images.

They are powerful but expensive.

The Audio Transformer:

Transformers for speech and music.

They are powerful but expensive.

The Robotics Transformer:

Transformers for robot control.

They are powerful but expensive.

A Contrarian Take: The Transformer Is a General-Purpose Architecture.

The transformer is not just for language. It is for everything.

The next architecture will also be general-purpose. It will not be domain-specific.

What You Can Do
You do not need to be a researcher to stay informed.

Follow the Research:

Read papers on Mamba, SSMs, and hybrid models.

Follow researchers on Twitter.

Experiment:

Try open-source models based on new architectures.

Compare their performance.

Be Skeptical:

New architectures are not always better.

Wait for independent evaluations.

Stay Curious:

The transformer era may be ending.

The next era is just beginning.

The Last Architecture
The last architecture is not yet built.

You ask: "What will replace the transformer?"
The model says: "I do not know."
You realize: The future is not written. It is being written.

If you could design a new architecture for AI, what would it be called and what would it do differently?

The Forgotten Languages: Why Your Model Speaks English, Chinese, and Spanish—But Not Tamil or Swahili

VelocityAI — Sat, 04 Jul 2026 12:34:13 +0000

You ask an AI a question in English. It answers fluently. You ask in Spanish. It answers well. You ask in Tamil. It stumbles. You ask in Swahili. It gives a generic, awkward response. You are not surprised. You expect the AI to be better at English. But you should be surprised. Tamil has 80 million speakers. Swahili has 200 million speakers. They are not obscure languages. They are just underrepresented in the training data.

This is the linguistic justice crisis. The AI does not speak all languages equally. It speaks the languages of the wealthy, the powerful, and the digitized. The languages of the marginalized are left behind.

The Data Distribution
The training data is not a balanced sample of the world's languages.

The Breakdown:

English: ~70-80% of training data.

Chinese: ~5-10%.

Spanish: ~3-5%.

Other Languages: The remainder.

The Missing:

Tamil: 80 million speakers, but tiny fraction of training data.

Swahili: 200 million speakers, but tiny fraction.

Yoruba: 50 million speakers, but tiny fraction.

The 7,000 other languages: Almost entirely absent.

A Contrarian Take: The Bias Is Not a Bug. It Is a Reflection of the Internet.

The AI is trained on the internet. The internet is mostly English. The bias is not a failure. It is a statistical reality.

The problem is not the AI. The problem is the uneven distribution of human knowledge online.

The Consequences
What happens when a language is underrepresented?

Poor Translation:

The AI translates poorly.

It loses nuance.

It makes errors.

Loss of Culture:

The AI does not understand cultural references.

It cannot capture idioms.

It erases local knowledge.

Digital Divide:

Speakers of underrepresented languages are excluded.

They cannot use AI effectively.

They are left behind.

A Contrarian Take: The AI Is Not the Problem. It Is a Symptom.

The AI reflects the world. The world is unequal. The AI is unequal.

The solution is not to fix the AI. The solution is to fix the world.

The Economics of Language
Why are some languages represented and others not?

The Economics:

English is the language of commerce.

Chinese is the language of a large economy.

Spanish is the language of a large population.

The Marginalization:

Tamil is spoken by a large population, but many are poor.

Swahili is spoken by many, but the region is not wealthy.

Yoruba is spoken by many, but the region is not digitized.

A Contrarian Take: The Economics Are the Cause of the Linguistic Crisis.

The AI companies are not malicious. They are rational. They train on the data that is available and economically valuable.

The languages that are not digitized are not profitable. The AI companies ignore them.

Case Study: The Swahili Speaker
A Swahili speaker tries to use an AI assistant.

The Experience:

The AI understands basic questions.

It gives generic, awkward answers.

It does not understand local idioms.

It does not recognize cultural references.

The Result:

The Swahili speaker stops using the AI.

They feel excluded.

They are left behind.

A Contrarian Take: The Problem Is Not the AI. It Is the Data.

The AI is not malicious. It is just ignorant. It does not know Swahili because it was not trained on Swahili.

The solution is to digitize Swahili. The solution is to create more Swahili content.

What You Can Do
You cannot fix the problem alone. But you can contribute.

Digitize Endangered Languages:

If you speak a minority language, create content in it.

Write blogs, record videos, create forums.

Support Open Datasets:

Contribute to open datasets for underrepresented languages.

Support organizations that digitize languages.

Advocate for Inclusion:

Demand that AI companies include underrepresented languages.

Support policies that promote linguistic diversity.

Learn a Less Common Language:

Learn Tamil, Swahili, or Yoruba.

The more speakers, the more demand.

The Last Word
The last word is not spoken. It is written.

You ask: "What is the future of language?"
The model says: "The future is uncertain."
You realize: The future depends on the choices we make today.

If you could add one language to the training data, what would it be? And why?

Synthetic Data's Feedback Loop: What Happens When Models Train on Their Own Outputs?

VelocityAI — Fri, 03 Jul 2026 11:53:21 +0000

You have a copy machine. You copy a document. The copy is slightly blurry. You copy the copy. It is blurrier. You copy it again. After ten generations, it is unrecognizable. This is model collapse. AI models are now training on data generated by previous AI models. The internet is filling with synthetic content. The next generation of models will train on that synthetic content. And the generation after that will train on the synthetic content of the synthetic content. The signal is degrading.

We are entering a dangerous feedback loop. AI is eating its own tail. And the result may be a slow, creeping decline in quality.

The Problem: The Internet Is Becoming Synthetic
Human-generated content is being diluted.

The Shift:

In 2020, most text was human-written.

In 2025, a significant fraction is AI-generated.

In 2030, most text may be AI-generated.

The Consequence:

Future models will train on data that is statistically similar to their own outputs.

The diversity of the training data will decrease.

The models will become more homogeneous.

A Contrarian Take: The Internet Was Always Synthetic.

We worry about AI-generated content. But human-generated content is also "synthetic" in a sense. It is filtered, curated, and biased.

The problem is not synthesis. The problem is degeneration. If the synthetic content is high quality, the feedback loop can be positive. If it is low quality, the feedback loop is negative.

The Mechanism of Model Collapse
Model collapse occurs through a series of generations.

Generation 1:

Trained on human-generated data.

Produces synthetic data.

Generation 2:

Trained on a mix of human and synthetic data.

Produces more synthetic data.

Generation 3:

Trained mostly on synthetic data.

Produces low-quality, repetitive output.

Generation N:

The model collapses into a narrow, degenerate state.

It loses nuance, diversity, and creativity.

A Contrarian Take: Model Collapse Is Not Inevitable.

Model collapse is a risk, not a certainty. It depends on the quality of the synthetic data. If the synthetic data is carefully curated, the feedback loop can be managed.

The problem is not synthetic data. The problem is unfiltered synthetic data.

The Degeneration Patterns
What actually degenerates?

Diversity:

The model becomes less creative.

It produces similar outputs.

It loses the ability to generate surprising combinations.

Nuance:

The model becomes less subtle.

It defaults to the average.

It loses the ability to capture edge cases.

Factual Accuracy:

The model becomes less accurate.

It amplifies errors.

It hallucinates more.

Language Quality:

The model becomes less fluent.

It uses simpler vocabulary.

It loses stylistic variety.

A Contrarian Take: Degeneration Is Not Uniform.

Some aspects degenerate faster than others. Language quality may degrade slowly. Diversity may degrade quickly.

The rate of degeneration depends on the model architecture, the training data, and the training regime.

Case Study: The LLaMA Experiment
Researchers trained a model on a dataset that was progressively more synthetic.

The Setup:

Generation 1: Trained on human data.

Generation 2: Trained on 50% human, 50% synthetic.

Generation 3: Trained on 90% synthetic.

The Results:

Generation 2 was slightly worse than Generation 1.

Generation 3 was significantly worse.

The model became repetitive and dull.

The Conclusion:

Synthetic data is not a substitute for human data.

The feedback loop is dangerous.

A Contrarian Take: The Experiment Was Flawed.

The researchers used low-quality synthetic data. They did not curate it. They did not filter it.

A well-curated synthetic dataset might produce better results. The experiment is a warning, not a verdict.

What You Can Do
You are not training a model. But you are consuming AI content.

Support Human Content:

Read human-written articles.

Watch human-made videos.

Support human creators.

Be Skeptical of Synthetic Content:

Ask: "Is this AI-generated?"

Be aware of the limitations.

Demand Transparency:

Ask: "Is this content synthetic?"

Support labeling of AI-generated content.

Advocate for Curation:

Synthetic data is not inherently bad.

It needs to be curated.

The Last Generation
The last generation is not the model. It is you.

You ask: "What is the future of AI?"
The model says: "The future depends on the choices we make today."
You realize: The future is not predetermined. It is a choice.

If the internet becomes mostly synthetic, how will you decide what to trust?

The Hidden Labor of Data Curation: Who Cleaned the Internet So Your Model Could Be Smart?

VelocityAI — Thu, 02 Jul 2026 11:22:41 +0000

You type a prompt. The AI responds. It is polite. It is coherent. It is not toxic. You assume this is just how the model is. It is not. Behind the scenes, thousands of workers spent countless hours scrubbing the internet of its worst content. They removed child sexual abuse material. They removed beheadings. They removed hate speech. They saw things that will haunt them forever. They were paid pennies. They were given no therapy. They were the invisible janitors of the AI revolution.

This is the hidden labor of data curation. The AI did not clean itself. Humans cleaned it. And those humans paid the price.

The Scale of the Problem
The internet is a sewer. The training data must be filtered.

The Volume:

Billions of web pages.

Trillions of words.

Millions of images.

The Contaminants:

CSAM (child sexual abuse material).

Gore, violence, and beheadings.

Hate speech, racism, and misogyny.

Spam, scams, and misinformation.

The Solution:

Human content moderators review and remove the worst content.

They work for subcontractors in developing countries.

They are paid per task.

A Contrarian Take: The AI Is Not "Clean." It Is "Censored."

We call it "cleaning." But it is also "censoring." The moderators are not just removing illegal content. They are also removing content that is controversial, uncomfortable, or politically inconvenient.

The "clean" dataset is not neutral. It reflects the values of the people who cleaned it.

The Workers
The content moderators are the invisible workforce of AI.

Who They Are:

Mostly young, in developing countries.

Often working for subcontractors like Sama, Accenture, or Teleperformance.

Paid $2-3 per hour.

No benefits, no job security, no therapy.

What They Do:

Watch videos of beheadings.

Identify child sexual abuse material.

Read hate speech and violent threats.

Label images as "safe" or "unsafe."

The Toll:

PTSD, anxiety, depression.

Suicidal ideation.

Insomnia and nightmares.

A Contrarian Take: The Workers Are Not "Moderators." They Are "Sacrifices."

We call them "moderators." But they are not moderating a debate. They are absorbing trauma so the AI can be safe.

The AI companies are not paying for the trauma. They are externalizing the cost. The workers pay with their mental health.

Case Study: The Sama Settlement
Sama is a subcontractor that provided content moderation services for OpenAI, Google, and Meta.

The Work:

Workers in Kenya, India, and the Philippines.

They labeled images and text for AI training.

They removed harmful content.

The Toll:

Workers reported PTSD.

Workers reported being underpaid.

Workers reported being fired for speaking out.

The Settlement:

Sama agreed to a settlement in 2022.

They agreed to pay back wages.

They agreed to provide mental health support.

The Problem:

The settlement was small.

The workers are still underpaid.

The industry has not changed.

A Contrarian Take: The Settlement Is a PR Stunt.

Sama settled to avoid a lawsuit. They did not change their business model.

The AI companies are still using subcontractors. The workers are still underpaid. The trauma is still ignored.

The Ethics of Outsourcing
The AI companies are outsourcing the dirty work.

The Business Model:

The AI companies focus on the "innovative" part.

They outsource the "cleaning" part to subcontractors.

The subcontractors outsource the trauma to the workers.

The Ethical Problem:

The workers are invisible.

The workers are disposable.

The workers are not compensated for the trauma.

A Contrarian Take: The Problem Is Not Outsourcing. It Is Capitalism.

The AI companies are not evil. They are just following the market. They are minimizing costs and maximizing profits.

The problem is not the AI companies. The problem is the system that rewards them for outsourcing trauma.

What You Can Do
You cannot change the system overnight. But you can be aware of it.

Demand Transparency:

Ask: "Who cleaned your data?"

Ask: "How were they treated?"

Support Fair Labor:

Support companies that pay their workers fairly.

Support regulations that protect content moderators.

Be Grateful:

The AI is safe because someone else suffered.

Acknowledge their labor.

Speak Out:

Share this story.

Make the invisible visible.

The Last Worker
The last worker is not a moderator. It is you.

You ask: "What is the cost of this answer?"
The AI says: "I do not know."
You realize: The cost was paid by someone else. And they are still paying.

If you could say one thing to the person who cleaned the data for your AI, what would it be?

The Copyright Apocalypse: Why Training on Everything Might Be the Last Time Anyone Can Do It

VelocityAI — Wed, 01 Jul 2026 17:32:13 +0000

The lawsuits are mounting. Authors, artists, musicians, and publishers are suing AI companies for training on their work without permission. The cases are complex. The outcomes are uncertain. But one thing is clear: the era of training on "everything" may be coming to an end. Future models may be trained on a fraction of the data. They may be significantly less capable. This is the Copyright Apocalypse.

We are at a crossroads. The current generation of AI was trained on a vast, unlicensed corpus of human creativity. The next generation may be trained on a carefully curated, legally scrubbed dataset. The difference in quality could be dramatic.

The Legal Landscape
The lawsuits are numerous and varied.

The Plaintiffs:

Authors (e.g., Sarah Silverman, John Grisham).

Visual artists.

Music publishers.

News organizations (e.g., The New York Times).

The Claims:

Right of Publicity: The AI companies used artists' names and likenesses.

Unfair Competition: The AI companies created products that compete with the original works.

The Defenses:

Fair Use: The AI companies argue that training is transformative and does not harm the market for the original works.

Lack of Direct Copying: The AI does not copy the works directly. It learns patterns.

A Contrarian Take: The Lawsuits Are Not About Copyright. They Are About Control.

The legal arguments are about copyright. But the real issue is control. The creators want to control how their work is used. They want to be compensated. They want to be acknowledged.

The AI companies argue that they are just reading. The creators argue that they are stealing. Both are right. The law is trying to catch up.

The Opt-Out Mechanisms
In response to the lawsuits, AI companies have introduced opt-out mechanisms.

The Mechanisms:

Robots.txt: Website owners can block web crawlers.

Data Removal Requests: Creators can request that their work be removed from training datasets.

NoAI Tags: New metadata tags that signal "do not train on this."

The Problem:

Opt-out is reactive, not proactive.

Most creators do not know about the mechanisms.

The mechanisms are easy to ignore.

A Contrarian Take: Opt-Out Is Not Consent. It Is a Trap.

Opt-out shifts the burden to the creator. It says: "We will use your work unless you tell us not to." That is not consent. That is an opt-out regime.

A true consent regime would require opt-in. The AI companies would have to ask permission. They are not asking.

The Future of Training Data
If the lawsuits succeed, the future of training data will look very different.

The Optimistic Scenario:

The AI companies pay for licenses.

They create a "Spotify for text" where creators are compensated.

The models are still powerful.

The Pessimistic Scenario:

The AI companies lose the lawsuits.

They are forced to delete their datasets.

Future models are trained on a fraction of the data.

They are significantly less capable.

A Contrarian Take: The Pessimistic Scenario Is Unlikely.

The AI companies have deep pockets. They will not let the lawsuits destroy their business.

They will pay the settlements. They will negotiate the licenses. They will find a way to keep training on massive datasets. The question is not whether they will train. It is what they will train on.

The Quality Gap
If future models are trained on less data, they will be less capable.

The Gap:

Factual Accuracy: Less data means fewer facts.

Nuance: Less data means less subtlety.

Creativity: Less data means less surprising combinations.

The Consequence:

The next generation of AI may be a step backward.

The "golden age" of AI may be behind us.

A Contrarian Take: The Gap Might Be Smaller Than We Think.

The current models are trained on vast amounts of data. But they are also trained on vast amounts of noise. Much of the data is redundant.

A smaller, carefully curated dataset might be more efficient. It might produce better results.

What You Can Do
You are not a lawyer. But you can still pay attention.

Follow the Lawsuits:

The cases are ongoing.

The outcomes will shape the future of AI.

Support Creators:

If you like a creator's work, support them directly.

Pay for their content. Share their work.

Advocate for Fairness:

The AI companies should compensate creators.

The creators should have a say in how their work is used.

Stay Informed:

The copyright apocalypse is not a single event. It is a process.

Stay informed about the latest developments.

The Last Lawsuit
The last lawsuit is not about copyright. It is about the future.

You ask: "What is the future of AI?"
The model says: "The future is uncertain."
You realize: The future depends on the choices we make today.

If you could design a fair system for training AI on copyrighted works, what would it look like? How would creators be compensated?