Migrating Legacy ETL Pipelines to Palantir Foundry: Incremental Processing, Schema Evolution, and Reconciliation
Replacing a legacy ETL platform is rarely a simple exercise in rewriting SQL or moving transformation code.
The difficult part is preserving data correctness while changing how data is ingested, transformed, validated, and consumed. Existing pipelines may support dashboards, operational reports, financial processes, APIs, machine learning workloads, and applications that nobody wants to interrupt.
For a Palantir Foundry migration, three concerns deserve particular attention:
- Incremental data processing
- Schema evolution
- Data reconciliation
A fourth concern connects all three: how to operate the legacy and Foundry pipelines safely during the transition.
The objective should not be to move every pipeline as quickly as possible. A better objective is to create a controlled migration in which differences can be detected before they affect consumers.
This article describes one practical migration pattern. It is an architectural approach, not a mandatory Palantir methodology.
Why Legacy ETL Migrations Are Difficult
Older ETL estates often contain much more business logic than their diagrams suggest.
A nightly job might appear to perform four steps:
Extract -> Transform -> Load -> Report
In practice, the process may depend on:
- Stored procedures maintained by another team
- File drops from external systems
- Hard-coded business mappings
- Historical correction rules
- Scheduled dependencies
- Manual restart procedures
- Temporary tables
- Reporting-specific transformations
- Downstream jobs that assume a particular schema
Some dependencies are documented. Others exist only because one job happens to finish before another begins.
This is why a direct "lift and shift" can be dangerous.
The technical transformation may be reproduced correctly while an operational dependency is missed.
A big-bang migration increases that risk. If extraction, transformation, storage, schedules, downstream interfaces, and reporting consumers all move together, diagnosing a discrepancy becomes harder.
A safer approach reduces the migration boundary.
A Safer Migration Pattern
Instead of migrating an entire ETL estate at once, divide it into data products or logical pipeline groups.
A possible migration lifecycle is:
- Inventory existing pipelines.
- Identify source and downstream dependencies.
- Select a manageable migration boundary.
- Rebuild the required pipeline in Foundry.
- Keep the existing ETL process operating.
- Execute the new pipeline in parallel.
- Reconcile legacy and Foundry outputs.
- Investigate unexplained differences.
- Move consumers gradually.
- Retire the corresponding legacy jobs.
For example, suppose a reporting warehouse receives customer, product, order, invoice, and payment data.
Migrating all five domains simultaneously makes reconciliation difficult. Instead, the team might begin with the order domain while maintaining its existing interfaces to customer and product data.
The boundary should be large enough to deliver something useful but small enough to validate thoroughly.
This staged approach is an implementation strategy. Foundry provides capabilities that can support pipeline construction, data validation, and incremental processing, but organizations still need to design the migration sequence around their own dependencies.
Incremental Transforms in Foundry
Full-refresh pipelines are common in legacy environments.
Consider a source table containing 500 million transaction records. A nightly pipeline might reread and transform the entire table even though only a fraction changed during the day.
A simplified process could look like this:
Read all transactions
|
Apply transformation rules
|
Recalculate entire output
|
Replace reporting table
That design may be acceptable for smaller datasets, but it becomes increasingly expensive and operationally awkward as data grows.
Foundry supports incremental transforms that use build history to avoid recomputing the entire output when appropriate. Palantir documents its incremental transform model as using previous build information to determine the state of inputs and process eligible changes incrementally.
Conceptually, the migration could instead become:
Identify new or changed records
|
Process affected data
|
Merge or append results
|
Publish updated dataset
Change detection
A migration team first needs to understand how the source exposes changes.
Possible mechanisms include:
- Append-only transactions
- Modification timestamps
- Change-data-capture feeds
- Source sequence numbers
- Partition dates
- Snapshot comparisons
The correct choice depends on the source architecture.
A timestamp such as last_modified_at can sometimes act as a watermark:
previous_watermark = last_successful_timestamp
read records where:
last_modified_at > previous_watermark
This is an architectural concept rather than a universal Foundry requirement.
Partitioning
Partitioning can help narrow processing boundaries.
A transaction pipeline might partition data by:
transaction_date
region
source_system
If the source changes only recent partitions, the pipeline may avoid unnecessary work on older data.
Partition design should follow access and update patterns rather than being added only for migration purposes.
Late-arriving data
Watermarks introduce another challenge.
Suppose the pipeline processed data through 02:00, but an upstream application later sends a transaction created at 01:55.
A strict timestamp boundary could miss it.
Teams may address late arrivals with strategies such as:
- Reprocessing a recent time window
- Tracking source sequence identifiers
- Maintaining correction feeds
- Comparing recent partitions
- Designing source ingestion around append semantics where possible
The correct approach depends on the source system.
Idempotency matters
Incremental processing should also consider retries.
If yesterday's batch runs twice, the output should not unintentionally duplicate transactions.
A useful design question is:
Can this transform safely process the same source change more than once?
Keys, merge rules, deduplication logic, and transaction semantics should be considered before an incremental design is promoted.
Palantir's documentation also notes that certain input changes can cause an incremental transform to fall back to non-incremental execution, so teams should understand how their selected transform implementation behaves rather than assuming every run will always remain incremental.
Managing Schema Evolution
Schema changes are one of the easiest ways to break a migration unexpectedly.
Typical changes include:
- Adding a column
- Renaming a field
- Removing a field
- Changing a data type
- Modifying nullability
- Changing nested structures
- Introducing a new source-system version
Foundry datasets can have schemas describing fields and their types, and schemas can change as the structure of a dataset changes over time.
That does not mean every schema change should automatically propagate through a production pipeline.
Use explicit schema contracts
Assume a customer source currently provides:
customer_id: string
customer_name: string
credit_limit: decimal
A source upgrade changes it to:
customer_id: string
customer_name: string
credit_limit: string
customer_segment: string
An automatic mapping could hide a significant problem.
credit_limit may now contain values such as:
"50000"
"25K"
"NOT_AVAILABLE"
The field name stayed the same, but its meaning and validation requirements changed.
A safer transformation explicitly handles the new schema:
validate source type
|
normalize credit_limit
|
reject or quarantine invalid values
|
map customer_segment
|
publish stable consumer schema
Separate source schemas from consumer contracts
A useful pattern is to avoid forcing every upstream schema change directly onto downstream consumers.
For example:
Raw source schema
|
Normalization layer
|
Canonical customer schema
|
Reporting / analytics / applications
This creates a controlled boundary.
Source systems can evolve while the canonical output remains stable until consumers are ready for a new version.
Treat incremental schema changes carefully
Incremental pipelines deserve additional attention because historical and newly written records may have been produced under different assumptions.
Palantir's incremental transform documentation describes schema compatibility considerations when reading previous output. It also documents patterns such as changing a transform's semantic version when a logic or schema change requires existing data to be recomputed.
The implementation decision should therefore answer:
- Can old and new records coexist safely?
- Does historical data require recomputation?
- Will downstream consumers understand the new field?
- Does a type change require migration logic?
- Should the new schema be introduced through a versioned dataset?
Schema evolution should be treated as an interface change, not merely a column change.
Reconciliation Between Legacy and Foundry Pipelines
Building the new pipeline is only half the migration.
The next question is more important:
Does it produce the correct result?
A common mistake is checking only row counts.
Suppose both systems produce exactly:
10,000,000 rows
That does not prove equivalence.
The pipelines could contain different records, incorrect values, duplicate keys, or different revenue totals while still having identical row counts.
Reconciliation should happen at several levels.
1. Row counts
Compare overall counts and meaningful subsets.
legacy_count = 10,000,000
foundry_count = 10,000,000
Then compare by business dimensions:
count by date
count by region
count by product
count by status
2. Key coverage
Compare business keys.
legacy_only_keys
foundry_only_keys
matching_keys
duplicate_keys
Unexpected missing keys usually deserve investigation.
3. Record-level comparison
For shared keys, compare significant fields.
Conceptually:
SELECT
customer_id,
legacy.amount,
foundry.amount
FROM legacy
JOIN foundry
ON legacy.customer_id = foundry.customer_id
WHERE legacy.amount <> foundry.amount;
Real implementations may require null handling, rounding rules, timestamps, or tolerance thresholds.
4. Aggregate reconciliation
Business aggregates often expose problems faster than row-level inspection.
Compare:
SUM(invoice_amount)
COUNT(DISTINCT customer_id)
AVG(order_value)
SUM(quantity)
Then calculate them across meaningful dimensions.
For example:
| Metric | Legacy | Foundry | Difference |
|---|---|---|---|
| Order count | 1,004,218 | 1,004,218 | 0 |
| Revenue | 87,412,552.10 | 87,412,552.10 | 0 |
| Unique customers | 124,911 | 124,911 | 0 |
The numbers above are purely illustrative.
5. Data quality indicators
Compare:
- Null percentages
- Duplicate rates
- Invalid codes
- Referential integrity
- Date ranges
- Unexpected values
- Key uniqueness
Foundry provides mechanisms for defining data expectations and health checks. Depending on the implementation, these can validate requirements such as row counts, non-null values, permitted values, primary-key conditions, and custom row-level conditions.
These platform capabilities can support validation, but migration-specific reconciliation logic still has to reflect the organization's business rules.
Running Both Pipelines Without Downtime
Parallel execution gives teams time to establish confidence before changing consumers.
The pattern may look like this:
+-> Legacy ETL -> Legacy output -> Current consumers
Source systems --|
+-> Foundry pipeline -> New output -> Validation
Users continue reading from the existing production path while the Foundry output is evaluated.
This is often called shadow processing or parallel validation.
Move consumers gradually
Once reconciliation is stable, consumers can be migrated separately.
For example:
Week 1:
Finance dashboard -> Legacy
Operations dashboard -> Legacy
Risk model -> Legacy
Week 2:
Finance dashboard -> Foundry
Operations dashboard -> Legacy
Risk model -> Legacy
Week 3:
Finance dashboard -> Foundry
Operations dashboard -> Foundry
Risk model -> Legacy
This creates smaller cutover events.
If a problem appears in one consumer, the entire migration does not necessarily need to be reversed.
Define rollback before cutover
A rollback plan should answer:
- What conditions trigger rollback?
- Can the consumer reconnect to the legacy output?
- How long will the legacy pipeline remain operational?
- Which team makes the rollback decision?
- How will data generated during the transition be handled?
Do not define these rules during an incident.
The goal is not to promise literal zero downtime. The goal is to minimize disruption and preserve a controlled path back to a known state.
A Practical Migration Example
Consider a hypothetical distributor with a nightly ETL pipeline.
Three operational systems provide:
- Customer records
- Orders
- Payment transactions
Every night, the legacy platform reloads the data warehouse. Finance and sales dashboards depend on the output by 7:00 AM.
The migration could proceed in six stages.
Stage 1: Understand the legacy pipeline
The team documents:
CRM -----------+
|
ERP -> Orders -+-> Legacy ETL -> Warehouse -> BI
|
Payments ------+
Transformation rules, schedules, keys, error handling, and downstream dependencies are identified.
Stage 2: Build the Foundry version
The equivalent processing path is implemented in Foundry.
The objective is not yet optimization. The first priority is producing a result that can be compared with the existing pipeline.
Stage 3: Introduce incremental processing
Instead of refreshing ten years of orders nightly, the new design processes changed data where the source and pipeline architecture allow it.
Historical data is loaded during initial construction.
Subsequent processing focuses on eligible new or changed records.
Stage 4: Run both pipelines
For several production cycles:
Legacy output ----+
+-> Reconciliation process
Foundry output ----+
The migration team compares counts, keys, aggregates, duplicates, nulls, and important business rules.
Stage 5: Explain every material difference
Imagine the Foundry result reports 214 more orders.
The difference should not immediately be classified as an error.
Investigation might find that the legacy job silently excluded late-arriving orders after its extraction window.
Now the team must decide what the correct business behavior should be.
Reconciliation is therefore not simply about making the new system match old defects. It is about identifying and explaining differences.
Stage 6: Move consumers and retire legacy jobs
After agreed validation thresholds are met, consumers are migrated individually.
The legacy process remains available during the defined rollback period.
Only after downstream dependencies, schedules, reconciliations, and support responsibilities have been confirmed should the corresponding legacy jobs be retired.
Common Migration Mistakes
Migrating everything at once
Large migration boundaries increase the number of possible causes when something fails.
Start with well-understood domains.
Ignoring downstream dependencies
A dataset is not isolated simply because no other ETL job reads it.
Reports, spreadsheets, APIs, applications, data science workflows, and manual processes may depend on it.
Treating schema changes as trivial
Changing integer to string is not merely metadata if consumers perform arithmetic on the value.
Evaluate semantic impact.
Validating only row counts
Matching counts can hide incorrect records and aggregates.
Use multiple reconciliation dimensions.
Switching consumers too early
A successful pipeline run does not prove production readiness.
Reconciliation should reach an agreed confidence level first.
Failing to define rollback criteria
Teams need measurable cutover and rollback conditions.
"Rollback if something looks wrong" is not sufficient.
Leaving ownership unclear
During parallel operation, determine who owns:
- Legacy failures
- Foundry failures
- Reconciliation issues
- Source-system changes
- Consumer migration
- Final cutover approval
Retiring legacy pipelines too early
Removing the previous processing path immediately after cutover eliminates a useful recovery option.
Retirement should follow validated operational stability and dependency confirmation.
Where Palantir Consulting Can Fit
A Palantir consulting team or experienced internal implementation group can be useful when the main challenge is architectural rather than simply writing transforms.
Typical areas of support may include:
- Legacy dependency analysis
- Migration boundary design
- Foundry pipeline architecture
- Incremental processing strategy
- Schema compatibility planning
- Reconciliation design
- Data quality controls
- Cutover planning
- Operational handover
The important qualification is domain knowledge.
A technically elegant Foundry implementation can still fail a migration if the team does not understand the meaning of the legacy data, hidden consumer dependencies, or operational recovery requirements.
For that reason, Palantir Consulting Services should be evaluated partly on their ability to reason about the existing environment, not only their familiarity with the destination platform.
Migration Checklist
Before moving a pipeline into production, confirm the following.
Source and dependencies
- [ ] Source systems are documented.
- [ ] Source update behavior is understood.
- [ ] Downstream consumers are identified.
- [ ] Scheduling dependencies are known.
- [ ] Hidden manual processes have been investigated.
Transformation design
- [ ] Existing transformation rules are understood.
- [ ] Incremental versus full processing has been decided.
- [ ] Late-arriving data is handled.
- [ ] Retry behavior is understood.
- [ ] Idempotency has been considered.
Schema
- [ ] Source schemas are documented.
- [ ] Consumer schema contracts are defined.
- [ ] Type changes are handled explicitly.
- [ ] Compatibility risks are identified.
- [ ] Historical recomputation requirements are understood.
Validation
- [ ] Row counts are compared.
- [ ] Key coverage is compared.
- [ ] Important fields are reconciled.
- [ ] Business aggregates are validated.
- [ ] Null and duplicate rates are checked.
- [ ] Business rules are tested.
- [ ] Material differences are explained.
Cutover
- [ ] Parallel execution has completed successfully.
- [ ] Monitoring is in place.
- [ ] Cutover criteria are documented.
- [ ] Rollback criteria are documented.
- [ ] Ownership is assigned.
- [ ] Consumers are migrated deliberately.
- [ ] Legacy retirement occurs only after validation.
Final Takeaway
A successful legacy ETL migration to Palantir Foundry is not primarily a code-conversion project.
It is a controlled change to the way enterprise data is produced and trusted.
Incremental processing can reduce unnecessary recomputation, but it requires careful handling of source changes, retries, historical state, and late-arriving data.
Schema evolution can support changing business systems, but only when compatibility and consumer impact are managed deliberately.
Reconciliation provides the evidence needed to determine whether the new pipeline is actually correct.
Finally, parallel execution creates room to validate those decisions before production consumers are fully committed to the new path.
The strongest migration plans therefore focus on three questions:
Is the data correct?
Can we explain every important difference?
Can we recover safely if the cutover fails?
If those questions can be answered confidently, the organization is doing more than moving ETL workloads. It is building a safer foundation for long-term enterprise data transformation.
Top comments (0)