DEV Community

Jean-Sebastien Beaulieu
Jean-Sebastien Beaulieu

Posted on

The Data Lake Is Not the Problem. The Missing Admission Layer Is.

#openscience #datascience #ai #dataengineering

In 2024, the data problem stopped being only a storage problem.

It became a meaning problem.

That is the part I want to look at today: not the romance of "big data", not another vendor promise around lakehouses, not another generic AI infrastructure post. I want to look at the pressure entering the pipe: global data creation, datacenter power, cloud migration, data lake adoption, and the gap between what we store and what we can honestly interpret.

My position is simple:

The data lake is not the problem. The missing admission layer is.

We keep building larger places to put data. That is useful. But if the data arrives without provenance, quality state, contradiction handling, units, policy version, and claim boundaries, then the lake becomes a place where uncertainty goes to become invisible.

1. The world is producing data faster than institutions can govern it

Public estimates vary because "data created, captured, copied, and consumed" is not measured by one universal public counter. But the order of magnitude is clear.

IDC's Global DataSphere program tracks worldwide data created, captured, and replicated. Older IDC-linked reporting projected roughly 175 zettabytes by 2025, while newer secondary summaries place the global datasphere around 149 zettabytes in 2024 and about 181 zettabytes in 2025.

Those numbers should be read carefully. They are not the same as "curated data", "trusted data", or "query-ready data". Most data does not become clean evidence. A zettabyte estimate tells us about volume pressure. It does not tell us that the data is admissible for decision-making.

That distinction matters.

In a data lake, raw data, logs, model outputs, embeddings, tables, documents, images, and scientific artifacts can all coexist. This is powerful because storage is flexible. But flexibility creates a second-order problem: when everything can be stored, the hard question becomes what each object is allowed to support.

Can this record support a dashboard?

Can this model output support a claim?

Can this simulation result support a paper?

Can this benchmark support a product decision?

Can this old file be reused under a newer policy?

If the lake does not preserve those boundaries, the system may look rich while becoming epistemically poor.

2. The infrastructure cost is no longer abstract

The International Energy Agency estimated that data centres consumed around 415 TWh of electricity in 2024, about 1.5% of global electricity consumption. The same IEA page says data-centre electricity consumption grew at about 12% per year over the previous five years.

That is the physical side of the dataflow.

Every pipeline has a material footprint. It needs power, cooling, servers, network equipment, storage media, replication, monitoring, backups, and human operations. When AI workloads enter the loop, the data lake is no longer just a passive archive. It becomes part of an inference, retrieval, enrichment, and training ecosystem.

This does not mean "stop building data centers." That is not a serious answer. Hospitals, research labs, schools, security teams, public agencies, and scientific infrastructure all depend on computation.

But it does mean we should stop pretending that every retained byte has equal value.

A classification layer should ask:

  • Is this object raw, derived, validated, deprecated, contradicted, or suspended?
  • What policy admitted it?
  • What evidence is attached?
  • What claims are permitted?
  • What claims are forbidden?
  • What would make the object useful enough to keep?
  • What would make it safe enough to publish?

The energy problem and the governance problem are connected. If storage and compute are expensive, then useless ambiguity is expensive too.

3. The data lake market is growing because the pain is real

Commercial market estimates should not be treated like scientific measurements, but they are useful signals of enterprise pressure.

Grand View Research reported the global data lake market at USD 13.6 billion in 2023, with projections toward USD 59.9 billion by 2030 and a 23.8% CAGR from 2024 to 2030. Other market reports forecast strong growth in cloud data warehouses, warehouse-as-a-service, and multi-cloud data warehouse systems.

The exact numbers will vary by analyst, scope, and definition. The signal is consistent: organizations are spending heavily because the old split between operational databases, warehouses, lakes, and ML stores is under pressure.

Data is moving from:

  • batch reports to streaming signals;
  • clean tables to semi-structured and unstructured objects;
  • local warehouses to cloud-native storage;
  • human-only analysis to AI-assisted interpretation;
  • isolated datasets to reused enterprise memory.

That movement is not automatically bad. It is the reason modern analytics works. It is also the reason weak provenance spreads quickly.

When an object moves across systems, it can lose its original context. A file becomes a table. A table becomes an embedding. An embedding becomes a retrieval result. A retrieval result becomes a generated answer. A generated answer becomes a decision. If no admission policy travels with it, the claim can become stronger than the evidence.

4. A lakehouse improves architecture, but it does not solve meaning by itself

The lakehouse pattern exists because teams wanted the scale and flexibility of a data lake with more warehouse-like governance, metadata, schema control, performance, and transactionality.

That is useful. It is a real architectural improvement.

But a lakehouse still needs a higher-level question:

What is this data allowed to mean?

Schema tells us shape.

Metadata tells us context.

Lineage tells us movement.

Quality checks tell us whether a rule passed.

But claim governance asks something slightly different: what public or operational statement is allowed to be made from this object?

For normal business analytics, that might mean whether a metric can support a quarterly planning decision.

For AI systems, it might mean whether a generated answer can be shown to a user without a warning.

For scientific workflows, it might mean whether a computational result can support a paper, a material candidate, a classification, or only an internal hypothesis.

The lakehouse can help preserve structure. It does not automatically preserve epistemic discipline.

5. The real dataflow is not "source to dashboard"

The old mental model is too simple:

source -> pipeline -> warehouse -> dashboard

The modern dataflow is closer to:

source -> raw lake -> transformations -> features -> embeddings -> model context -> generated output -> review -> publication -> reuse

That is a longer chain. Every step can change meaning.

A unit can be dropped.

A timestamp can cross a timezone boundary.

A join can multiply rows.

A filter can silently remove failures.

A model can summarize uncertainty out of existence.

A dashboard can round away a warning.

A human can copy the strongest sentence and forget the caveat.

This is why I think data teams need a concept that sits between observability and publication: an admission layer.

Not every object should be admitted for every use.

Some should be admitted for storage but suspended for claims.

Some should be admitted for internal exploration but blocked from external publication.

Some should be admitted for narrow claims but rejected for broader ones.

Some should expire unless refreshed by new evidence.

This is not bureaucracy. It is data hygiene at the level of meaning.

6. What changed from 2024 to now

From 2024 onward, three pressures became harder to ignore.

First, AI made interpretation cheaper. That sounds good, but cheap interpretation creates more opportunities to over-interpret. A model can produce a convincing summary faster than a human can verify the lineage behind it.

Second, infrastructure pressure became visible. Data centres are no longer invisible rooms behind the internet. Their electricity, water, cooling, land, and grid requirements have become public-policy issues.

Third, enterprise architectures shifted toward more reusable data surfaces. Lakes, lakehouses, warehouses, vector stores, catalogs, and governance layers are no longer separate conversations. They are becoming one data operating system.

That convergence is exactly why classification matters now.

If data is reused more often, each reuse needs a memory of what the data was allowed to support before. If AI sits on top of the lake, the model needs more than retrieval. It needs evidence boundaries. If scientific workflows produce larger output sets, the publication layer needs to know what is admitted, suspended, and rejected.

7. The admission layer I want to see

The practical version does not need to be mystical.

It can start with simple fields:

  • source identity;
  • method and software version;
  • raw result reference;
  • unit map;
  • lineage hash;
  • quality score;
  • uncertainty state;
  • contradiction state;
  • policy version;
  • permitted claims;
  • forbidden claims;
  • decision state;
  • reason codes;
  • replay instructions.

Those fields do not solve science. They make the decision inspectable.

The most important output is not a beautiful score. The most important output is a boundary:

This can be used for X.

This cannot be used for Y.

This is suspended until Z is provided.

That boundary is what prevents a lake from becoming a machine that stores ambiguity and exports confidence.

8. What the statistics really say

The numbers do not prove that any one architecture is correct.

They prove that the pressure is real.

Global data volumes are in the zettabyte range. Data-centre electricity consumption is already a measurable part of global electricity use. The data lake and lakehouse markets are growing because organizations need flexible, large-scale data systems. AI is increasing the speed at which stored data becomes interpreted data.

The conclusion is not "panic."

The conclusion is "instrument the passage from data to claim."

If a system can store more, retrieve more, summarize more, and generate more, then it must also classify more carefully.

9. A better lake is not only bigger

A better lake is not only cheaper storage.

A better lake is not only faster queries.

A better lake is not only a new table format.

A better lake is a system where the data carries enough memory that future users can understand what it was, where it came from, how it changed, and what it is allowed to support.

That is the line I want to keep testing:

Data without provenance is not ready.

Data without uncertainty is not honest.

Data without claim boundaries is not safe to amplify.

The next data platform should not only help us ask, "What can we compute?"

It should help us ask, "What do we have the right to say?"

Sources

Validation Note

The global data volume numbers are directional estimates from market and analyst sources, not direct physical measurements. I treat them as scale signals, not exact truth. The IEA electricity figure is stronger for infrastructure context because it comes from an energy agency report. Market-size projections are used only to show adoption pressure, not to prove technical superiority.

Top comments (0)