Part 2: The Data Engineer of 2028 - Three Career Paths in the Age of Autonomous Platforms
Introduction
I'm Taro Matsui, Head of Technology Strategy at CCCMK Holdings, the company behind V Point, Japan's largest loyalty platform. I'm also a Snowflake Data Superhero—a recognition for community members who actively share knowledge and insights, particularly around data platform strategies.
Part 1 mapped Snowflake's three-phase evolution toward autonomous platforms. But what does this actually mean for you as a data engineer? And more importantly—how should you prepare?
In Part 2, we answer that question by diving into Phase 3's concrete capabilities and the three career trajectories they unlock.
Future Scenarios: The Phase 3 World
Let me share three specific capabilities I anticipate in Phase 3 (2027–2028).
Prediction 1: Natural Language Data Platform Design
The Transformation
Business users describe requirements in plain natural language (no SQL or technical syntax required), and AI automatically proposes data architectures.
In Practice
A request like "I want to analyze customer purchase history every Monday morning and generate reports" will be fulfilled within minutes:
- Identifying required data sources
- Proposing pipeline designs
- Recommending optimal data models
- Estimating costs and performance
The Shifting Role of Data Engineers
Traditional approach: Listen to requirements, spend days to weeks on design and implementation
Future approach: Review AI proposals and validate business logic—a shift to oversight and approval
Why This Is Feasible
Cortex Analyst already generates SQL from natural language. Snowflake Copilot handles DDL/DML operations. Integrating these capabilities makes this technically achievable.
That said, capturing all organizational context in Snowflake remains challenging. Full autonomy won't be achieved—data engineers will retain critical decision-making responsibilities.
Prediction 2: Self-Evolving Data Models
The Transformation
Data platforms learn workload changes and autonomously evolve their data models.
In Practice
Traditional approach: Humans periodically decide, "Query patterns are increasing—let's create a materialized view."
Future approach: AI detects query pattern changes and automates responses:
- Proposing optimizations (automatic MV generation)
- Estimating cost reduction impacts
- Validating post-implementation effectiveness
- Automatically rolling back when benefits don't materialize
The Revolutionary Impact
This marks a shift from "static data models" to "dynamic architectures that evolve continuously with business needs."
Data engineers move from "daily optimization tasks" to "long-term data strategy."
Prediction 3: Automated Semantic Layer Construction
The Transformation
Business terminology automatically maps to data, enabling everyone to work with data naturally.
In Practice
For a question like "What were the top 10 products by revenue in last year's Q4?":
- Automatically interprets "revenue," "quarter," and "products"
- Applies organization-specific definitions (fiscal year, etc.)
- Generates consistent answers
Why This Is Feasible
This foundation is expected to emerge by 2026.
Beyond that, these definitions will unify across all tools—Tableau, Power BI, Looker—eliminating the "numbers vary by tool" problem.
Open Semantic Interface (OSI) standardized in 2025, and Snowflake Horizon Catalog already provides metadata management. Integration should enable this by 2026–2027.
Read more about Open Semantic Interface (OSI)
Organizational Impact
Data engineers shift from "terminology translators" to strategic value creators.
Upstream data modeling—before platform integration—becomes increasingly critical. Data engineers will govern data architecture and modeling across the entire organization and business, not just the platform.
This shift elevates data engineering's strategic importance to the business.
What This Phase Delivers
The New Role of Data Engineers
Traditional: "Data maintainers" (focused on keeping systems running)
2027: "Data strategists" (focused on driving business outcomes)
Specifically:
- Translators: Communicating business vision to AI
- Strategists: Evaluating and approving AI decisions
- Innovators: Creating new business value through data
The following outlines an ideal scenario. In practice, organizational maturity and technical constraints will likely extend the timeline.
Yet within a few years, pioneering organizations will begin realizing this vision.
A Data Engineer's Week (2027–2028 Vision)
Operations (~20%):
- Reviewing and approving AI-generated optimization proposals (not implementing them)
Strategy (~80%):
- Engaging with business units to clarify new requirements
- Evaluating new technologies and business applicability
- Building and operating data governance frameworks
- Proposing data strategies to leadership
These percentages illustrate a fundamental inversion: from "operations-heavy, strategy-light" to strategy-dominant workloads.
This operational-to-strategic shift transforms data engineers from cost centers to profit centers.
"Data is the oil of the 21st century," as the saying goes. But like crude oil, raw data doesn't drive business. It must be refined, processed, and productized to create value.
Managing the production process? That's a cost center.
Conceiving the product itself and managing its creation? That's a profit center.
The Evolution of Data Engineer Value:
Traditional: Evaluated by "technical skills"
2027: Evaluated by "business impact"
- "How many pipelines built?" → "How much revenue generated?"
- "Achieved zero incidents?" → "Created new business opportunities?"
- "Reduced costs?" → "Established competitive advantage through data?"
Snowflake's Ultimate Question:
Will you transform the world with data—or merely manage it?
Our answer:
We will transform business and the world through data.
Long-term Strategy (Beyond 2028): Evolution to Data Strategist
Vision: From "Data Gatekeepers" to "Business Creators"
In 2028, the data engineering profession transforms fundamentally.
Traditional Responsibilities:
- Building and operating data platforms
- Performance optimization
- Incident response
Post-2028 Responsibilities:
- Proposing data-driven business strategies
- Creating value through AI agent collaboration
- Planning and launching data-enabled new ventures
This represents elevated status for data engineers.
Emerging Career Paths
I once viewed data engineering as "backstage work." But participating in executive meetings changed my perspective. Watching executives analyze data to formulate strategy and observing diverse approaches to generating new data shifted my understanding: data engineers should partner with leadership.
This realization led me to identify three emerging career patterns.
Path A: Data Visionary
Role: Designing future businesses powered by data
Core Responsibilities:
- Formulating data strategies through executive dialogue
- Planning new data-enabled ventures
- Discovering and leveraging data assets for competitive advantage
Required Skills:
- Business strategy comprehension
- Data-driven insight extraction
- Executive and business communication skills
Path B: AI Orchestrator
Role: Overseeing multiple AI agents and optimizing the entire data platform
Core Responsibilities:
- Supervising AI agent decisions (Optima, Adaptive Warehouse, Cortex, etc.)
- Managing anomalies and edge cases
- Proposing AI decision logic improvements
Required Skills:
- Foundational AI/ML understanding
- Deep data architecture knowledge
- Anomaly detection and analysis
- A holistic understanding of system integration
Path C: Data Product Manager
Role: Treating data as a product for internal and external delivery
Core Responsibilities:
- Planning, developing, and operating data products
- Advancing Data as a Service
- Operating data marketplaces
Required Skills:
- Product management
- User experience design
- Business model development
- Marketing and sales capabilities
Recommended Team Structure (2028)
Data Strategy Team (10 members):
-
Data Visionaries: 2 members
- Business strategy, new venture planning
-
AI Orchestrators: 3 members
- Platform oversight, edge case handling
-
Data Product Managers: 2 members
- Internal/external data product operations
-
Data Analysts: 2 members
- Business analysis, dashboard development
-
Platform Architect: 1 member
- Deep technical expertise, new technology evaluation
Note: In this autonomous future, dedicated roles for traditional infrastructure operations and performance tuning shift toward AI oversight and business strategy. Organizations in transition may maintain hybrid models.
The Essential Mindset Shift
Question: Will autonomy eliminate our jobs?
Answer: No. Your role gets elevated.
Traditional: "Optimize this slow query" → Takes days
2026: AI autonomously optimizes → Time freed for new business planning
Traditional: "Pipeline broke" → Half-day investigation and fix
2027: AI automatically remediates → Time freed for strategy formulation
Traditional: Incident response consumes all bandwidth
2028: Delegate to AI, discuss future with executives
Dispelling a Critical Misconception
"Autonomy makes data engineers obsolete" is false. Autonomy drives the evolution of the profession, not its extinction.
Data engineers aren't disappearing. They're moving toward more strategic, higher-value work.
How We Measure Value Is Changing
Traditional: "Technical Skills" × "Implementation Speed"
2028: "Business Acumen" × "AI Collaboration" × "Strategic Thinking"
Autonomy isn't a threat—it's an ally liberating you from routine work to focus on what truly matters.
Autonomy's Limits and the Human Role
Autonomy has boundaries.
What AI Still Can't Decide:
- Deep business context understanding
- Data quality judgment
- Stakeholder interest reconciliation
- Ethical and legal decisions
- Security risk assessment
- Creative problem-solving
These remain human domains beyond 2028. Data engineers’ value doesn’t vanish—it shifts to higher-level, more creative territories across the data landscape.
If I paraphrase the three message pillars from Snowflake World Tour Tokyo:
Trust Snowflake, liberate yourself from complexity, connect across boundaries—and transform the world.
A Note on These Predictions
As a Data Superhero, I have the privilege of early access to some features, regular conversations with Snowflake's product teams, and deep engagement with the global data community. These insights inform my predictions—but they remain educated speculation, not guaranteed timelines.
The value isn't in being precisely correct about "when." It's in having a directional framework to:
- Prepare your skills and career
- Advocate for the right capabilities at your organization
- Engage productively with platform evolution
Whether Snowflake, Databricks, BigQuery, or others lead specific capabilities, the industry trajectory toward autonomy is clear.
Conclusion
I've worked as a data engineer for many years. One thought persisted:
"Performance tuning matters. Incident response matters. Data management matters. But what I truly want is to transform business and society through data."
Snowflake's "Simplicity" philosophy—hiding complexity—sends us a message: "Focus on what truly matters."
I'm an optimist who embraces change. For better or worse, I'm a dreamer—and my wild ideas occasionally confuse those around me.
These timelines are speculation. Actual releases may shift due to technical challenges or market dynamics. What matters isn't 'when' but 'direction'.
AI's evolution is remarkable. While we feel our own productivity gains, mega-platforms are accelerating development even faster.
Predictable futures become achievable. They arrive faster than anticipated.
The Future This Article Envisions
Snowflake's features are being reverse-engineered from their vision of data engineering's future. This trend has accelerated significantly this year.
This is the dawn of a new data engineering era.
Phase 1 (~2026):
Autonomous performance tuning → Liberation from technical optimization
Phase 2 (2026–2027):
Autonomous operations → Minimizing data management tasks
Phase 3 (2027–2028):
Autonomous data strategy → Platforms automatically built from business requirements
The ultimate destination:
Data engineers fully liberated from reactive work, empowered to drive proactive innovation.
This forecast may prove inaccurate. But this kind of directional thinking deepens our understanding of what's possible. And if I'm wrong? I'll have plenty of specific feature requests ready to share with Snowflake!
To You, the Reader
If you worry autonomy will eliminate your job, I hope this article offers hope.
If daily operations and tuning prevent you from doing what you truly want:
Autonomy isn't your enemy. It's your ally.
To drive greater business impact within limited time, we're delegating more to AI—and that's strategic.
Drive meaningful change with data.
Build successful businesses through data.
Improve lives with data.
We need time for this. Snowflake is giving us that time.
So here's my final question to you:
Data engineers will become more valuable than ever—will you join me in shaping that future?
One Last Thing: A Request for Snowflake
If anyone from Snowflake reads this—one request:
Please accelerate Adaptive Warehouse's public preview!
(We're waiting eagerly!)
Author's Note: As a Snowflake Data Superhero, I engage regularly with Snowflake's product teams and community. However, all predictions, timelines, and interpretations in this article represent my personal analysis, not Snowflake's official roadmap or strategy. I write as an independent technologist and community contributor, not as a company representative.
I'd love to hear your thoughts—share your perspective in the comments below, or connect with me on LinkedIn.
Top comments (0)