DEV Community: Kavi Writes

Grounding Palantir AIP Workflows in the Ontology: Tool Design, Guardrails, and Evaluation Patterns

Kavi Writes — Wed, 14 Oct 2026 10:24:46 +0000

Migrating Legacy ETL Pipelines to Palantir Foundry: Incremental Processing, Schema Evolution, and Reconciliation

Replacing a legacy ETL platform is rarely a simple exercise in rewriting SQL or moving transformation code.

The difficult part is preserving data correctness while changing how data is ingested, transformed, validated, and consumed. Existing pipelines may support dashboards, operational reports, financial processes, APIs, machine learning workloads, and applications that nobody wants to interrupt.

For a Palantir Foundry migration, three concerns deserve particular attention:

Incremental data processing
Schema evolution
Data reconciliation

A fourth concern connects all three: how to operate the legacy and Foundry pipelines safely during the transition.

The objective should not be to move every pipeline as quickly as possible. A better objective is to create a controlled migration in which differences can be detected before they affect consumers.

This article describes one practical migration pattern. It is an architectural approach, not a mandatory Palantir methodology.

Why Legacy ETL Migrations Are Difficult

Older ETL estates often contain much more business logic than their diagrams suggest.

A nightly job might appear to perform four steps:

Extract -> Transform -> Load -> Report

In practice, the process may depend on:

Stored procedures maintained by another team
File drops from external systems
Hard-coded business mappings
Historical correction rules
Scheduled dependencies
Manual restart procedures
Temporary tables
Reporting-specific transformations
Downstream jobs that assume a particular schema

Some dependencies are documented. Others exist only because one job happens to finish before another begins.

This is why a direct "lift and shift" can be dangerous.

The technical transformation may be reproduced correctly while an operational dependency is missed.

A big-bang migration increases that risk. If extraction, transformation, storage, schedules, downstream interfaces, and reporting consumers all move together, diagnosing a discrepancy becomes harder.

A safer approach reduces the migration boundary.

A Safer Migration Pattern

Instead of migrating an entire ETL estate at once, divide it into data products or logical pipeline groups.

A possible migration lifecycle is:

Inventory existing pipelines.
Identify source and downstream dependencies.
Select a manageable migration boundary.
Rebuild the required pipeline in Foundry.
Keep the existing ETL process operating.
Execute the new pipeline in parallel.
Reconcile legacy and Foundry outputs.
Investigate unexplained differences.
Move consumers gradually.
Retire the corresponding legacy jobs.

For example, suppose a reporting warehouse receives customer, product, order, invoice, and payment data.

Migrating all five domains simultaneously makes reconciliation difficult. Instead, the team might begin with the order domain while maintaining its existing interfaces to customer and product data.

The boundary should be large enough to deliver something useful but small enough to validate thoroughly.

This staged approach is an implementation strategy. Foundry provides capabilities that can support pipeline construction, data validation, and incremental processing, but organizations still need to design the migration sequence around their own dependencies.

Incremental Transforms in Foundry

Full-refresh pipelines are common in legacy environments.

Consider a source table containing 500 million transaction records. A nightly pipeline might reread and transform the entire table even though only a fraction changed during the day.

A simplified process could look like this:

Read all transactions
        |
Apply transformation rules
        |
Recalculate entire output
        |
Replace reporting table

That design may be acceptable for smaller datasets, but it becomes increasingly expensive and operationally awkward as data grows.

Foundry supports incremental transforms that use build history to avoid recomputing the entire output when appropriate. Palantir documents its incremental transform model as using previous build information to determine the state of inputs and process eligible changes incrementally.

Conceptually, the migration could instead become:

Identify new or changed records
        |
Process affected data
        |
Merge or append results
        |
Publish updated dataset

Change detection

A migration team first needs to understand how the source exposes changes.

Possible mechanisms include:

Append-only transactions
Modification timestamps
Change-data-capture feeds
Source sequence numbers
Partition dates
Snapshot comparisons

The correct choice depends on the source architecture.

A timestamp such as last_modified_at can sometimes act as a watermark:

previous_watermark = last_successful_timestamp

read records where:
last_modified_at > previous_watermark

This is an architectural concept rather than a universal Foundry requirement.

Partitioning

Partitioning can help narrow processing boundaries.

A transaction pipeline might partition data by:

transaction_date
region
source_system

If the source changes only recent partitions, the pipeline may avoid unnecessary work on older data.

Partition design should follow access and update patterns rather than being added only for migration purposes.

Late-arriving data

Watermarks introduce another challenge.

Suppose the pipeline processed data through 02:00, but an upstream application later sends a transaction created at 01:55.

A strict timestamp boundary could miss it.

Teams may address late arrivals with strategies such as:

Reprocessing a recent time window
Tracking source sequence identifiers
Maintaining correction feeds
Comparing recent partitions
Designing source ingestion around append semantics where possible

The correct approach depends on the source system.

Idempotency matters

Incremental processing should also consider retries.

If yesterday's batch runs twice, the output should not unintentionally duplicate transactions.

A useful design question is:

Can this transform safely process the same source change more than once?

Keys, merge rules, deduplication logic, and transaction semantics should be considered before an incremental design is promoted.

Palantir's documentation also notes that certain input changes can cause an incremental transform to fall back to non-incremental execution, so teams should understand how their selected transform implementation behaves rather than assuming every run will always remain incremental.

Managing Schema Evolution

Schema changes are one of the easiest ways to break a migration unexpectedly.

Typical changes include:

Adding a column
Renaming a field
Removing a field
Changing a data type
Modifying nullability
Changing nested structures
Introducing a new source-system version

Foundry datasets can have schemas describing fields and their types, and schemas can change as the structure of a dataset changes over time.

That does not mean every schema change should automatically propagate through a production pipeline.

Use explicit schema contracts

Assume a customer source currently provides:

customer_id: string
customer_name: string
credit_limit: decimal

A source upgrade changes it to:

customer_id: string
customer_name: string
credit_limit: string
customer_segment: string

An automatic mapping could hide a significant problem.

credit_limit may now contain values such as:

"50000"
"25K"
"NOT_AVAILABLE"

The field name stayed the same, but its meaning and validation requirements changed.

A safer transformation explicitly handles the new schema:

validate source type
        |
normalize credit_limit
        |
reject or quarantine invalid values
        |
map customer_segment
        |
publish stable consumer schema

Separate source schemas from consumer contracts

A useful pattern is to avoid forcing every upstream schema change directly onto downstream consumers.

For example:

Raw source schema
      |
Normalization layer
      |
Canonical customer schema
      |
Reporting / analytics / applications

This creates a controlled boundary.

Source systems can evolve while the canonical output remains stable until consumers are ready for a new version.

Treat incremental schema changes carefully

Incremental pipelines deserve additional attention because historical and newly written records may have been produced under different assumptions.

Palantir's incremental transform documentation describes schema compatibility considerations when reading previous output. It also documents patterns such as changing a transform's semantic version when a logic or schema change requires existing data to be recomputed.

The implementation decision should therefore answer:

Can old and new records coexist safely?
Does historical data require recomputation?
Will downstream consumers understand the new field?
Does a type change require migration logic?
Should the new schema be introduced through a versioned dataset?

Schema evolution should be treated as an interface change, not merely a column change.

Reconciliation Between Legacy and Foundry Pipelines

Building the new pipeline is only half the migration.

The next question is more important:

Does it produce the correct result?

A common mistake is checking only row counts.

Suppose both systems produce exactly:

10,000,000 rows

That does not prove equivalence.

The pipelines could contain different records, incorrect values, duplicate keys, or different revenue totals while still having identical row counts.

Reconciliation should happen at several levels.

1. Row counts

Compare overall counts and meaningful subsets.

legacy_count = 10,000,000
foundry_count = 10,000,000

Then compare by business dimensions:

count by date
count by region
count by product
count by status

2. Key coverage

Compare business keys.

legacy_only_keys
foundry_only_keys
matching_keys
duplicate_keys

Unexpected missing keys usually deserve investigation.

3. Record-level comparison

For shared keys, compare significant fields.

Conceptually:

SELECT
    customer_id,
    legacy.amount,
    foundry.amount
FROM legacy
JOIN foundry
  ON legacy.customer_id = foundry.customer_id
WHERE legacy.amount <> foundry.amount;

Real implementations may require null handling, rounding rules, timestamps, or tolerance thresholds.

4. Aggregate reconciliation

Business aggregates often expose problems faster than row-level inspection.

Compare:

SUM(invoice_amount)
COUNT(DISTINCT customer_id)
AVG(order_value)
SUM(quantity)

Then calculate them across meaningful dimensions.

For example:

Metric	Legacy	Foundry
Order count	1,004,218	1,004,218
Revenue	87,412,552.10	87,412,552.10
Unique customers	124,911	124,911

The numbers above are purely illustrative.

5. Data quality indicators

Compare:

Null percentages
Duplicate rates
Invalid codes
Referential integrity
Date ranges
Unexpected values
Key uniqueness

Foundry provides mechanisms for defining data expectations and health checks. Depending on the implementation, these can validate requirements such as row counts, non-null values, permitted values, primary-key conditions, and custom row-level conditions.

These platform capabilities can support validation, but migration-specific reconciliation logic still has to reflect the organization's business rules.

Running Both Pipelines Without Downtime

Parallel execution gives teams time to establish confidence before changing consumers.

The pattern may look like this:

                 +-> Legacy ETL -> Legacy output -> Current consumers
Source systems --|
                 +-> Foundry pipeline -> New output -> Validation

Users continue reading from the existing production path while the Foundry output is evaluated.

This is often called shadow processing or parallel validation.

Move consumers gradually

Once reconciliation is stable, consumers can be migrated separately.

For example:

Week 1:
Finance dashboard -> Legacy
Operations dashboard -> Legacy
Risk model -> Legacy

Week 2:
Finance dashboard -> Foundry
Operations dashboard -> Legacy
Risk model -> Legacy

Week 3:
Finance dashboard -> Foundry
Operations dashboard -> Foundry
Risk model -> Legacy

This creates smaller cutover events.

If a problem appears in one consumer, the entire migration does not necessarily need to be reversed.

Define rollback before cutover

A rollback plan should answer:

What conditions trigger rollback?
Can the consumer reconnect to the legacy output?
How long will the legacy pipeline remain operational?
Which team makes the rollback decision?
How will data generated during the transition be handled?

Do not define these rules during an incident.

The goal is not to promise literal zero downtime. The goal is to minimize disruption and preserve a controlled path back to a known state.

A Practical Migration Example

Consider a hypothetical distributor with a nightly ETL pipeline.

Three operational systems provide:

Customer records
Orders
Payment transactions

Every night, the legacy platform reloads the data warehouse. Finance and sales dashboards depend on the output by 7:00 AM.

The migration could proceed in six stages.

Stage 1: Understand the legacy pipeline

The team documents:

CRM -----------+
               |
ERP -> Orders -+-> Legacy ETL -> Warehouse -> BI
               |
Payments ------+

Transformation rules, schedules, keys, error handling, and downstream dependencies are identified.

Stage 2: Build the Foundry version

The equivalent processing path is implemented in Foundry.

The objective is not yet optimization. The first priority is producing a result that can be compared with the existing pipeline.

Stage 3: Introduce incremental processing

Instead of refreshing ten years of orders nightly, the new design processes changed data where the source and pipeline architecture allow it.

Historical data is loaded during initial construction.

Subsequent processing focuses on eligible new or changed records.

Stage 4: Run both pipelines

For several production cycles:

Legacy output  ----+
                   +-> Reconciliation process
Foundry output ----+

The migration team compares counts, keys, aggregates, duplicates, nulls, and important business rules.

Stage 5: Explain every material difference

Imagine the Foundry result reports 214 more orders.

The difference should not immediately be classified as an error.

Investigation might find that the legacy job silently excluded late-arriving orders after its extraction window.

Now the team must decide what the correct business behavior should be.

Reconciliation is therefore not simply about making the new system match old defects. It is about identifying and explaining differences.

Stage 6: Move consumers and retire legacy jobs

After agreed validation thresholds are met, consumers are migrated individually.

The legacy process remains available during the defined rollback period.

Only after downstream dependencies, schedules, reconciliations, and support responsibilities have been confirmed should the corresponding legacy jobs be retired.

Common Migration Mistakes

Migrating everything at once

Large migration boundaries increase the number of possible causes when something fails.

Start with well-understood domains.

Ignoring downstream dependencies

A dataset is not isolated simply because no other ETL job reads it.

Reports, spreadsheets, APIs, applications, data science workflows, and manual processes may depend on it.

Treating schema changes as trivial

Changing integer to string is not merely metadata if consumers perform arithmetic on the value.

Evaluate semantic impact.

Validating only row counts

Matching counts can hide incorrect records and aggregates.

Use multiple reconciliation dimensions.

Switching consumers too early

A successful pipeline run does not prove production readiness.

Reconciliation should reach an agreed confidence level first.

Failing to define rollback criteria

Teams need measurable cutover and rollback conditions.

"Rollback if something looks wrong" is not sufficient.

Leaving ownership unclear

During parallel operation, determine who owns:

Legacy failures
Foundry failures
Reconciliation issues
Source-system changes
Consumer migration
Final cutover approval

Retiring legacy pipelines too early

Removing the previous processing path immediately after cutover eliminates a useful recovery option.

Retirement should follow validated operational stability and dependency confirmation.

Where Palantir Consulting Can Fit

A Palantir consulting team or experienced internal implementation group can be useful when the main challenge is architectural rather than simply writing transforms.

Typical areas of support may include:

Legacy dependency analysis
Migration boundary design
Foundry pipeline architecture
Incremental processing strategy
Schema compatibility planning
Reconciliation design
Data quality controls
Cutover planning
Operational handover

The important qualification is domain knowledge.

A technically elegant Foundry implementation can still fail a migration if the team does not understand the meaning of the legacy data, hidden consumer dependencies, or operational recovery requirements.

For that reason, Palantir Consulting Services should be evaluated partly on their ability to reason about the existing environment, not only their familiarity with the destination platform.

Migration Checklist

Before moving a pipeline into production, confirm the following.

Source and dependencies

[ ] Source systems are documented.
[ ] Source update behavior is understood.
[ ] Downstream consumers are identified.
[ ] Scheduling dependencies are known.
[ ] Hidden manual processes have been investigated.

Transformation design

[ ] Existing transformation rules are understood.
[ ] Incremental versus full processing has been decided.
[ ] Late-arriving data is handled.
[ ] Retry behavior is understood.
[ ] Idempotency has been considered.

Schema

[ ] Source schemas are documented.
[ ] Consumer schema contracts are defined.
[ ] Type changes are handled explicitly.
[ ] Compatibility risks are identified.
[ ] Historical recomputation requirements are understood.

Validation

[ ] Row counts are compared.
[ ] Key coverage is compared.
[ ] Important fields are reconciled.
[ ] Business aggregates are validated.
[ ] Null and duplicate rates are checked.
[ ] Business rules are tested.
[ ] Material differences are explained.

Cutover

[ ] Parallel execution has completed successfully.
[ ] Monitoring is in place.
[ ] Cutover criteria are documented.
[ ] Rollback criteria are documented.
[ ] Ownership is assigned.
[ ] Consumers are migrated deliberately.
[ ] Legacy retirement occurs only after validation.

Final Takeaway

A successful legacy ETL migration to Palantir Foundry is not primarily a code-conversion project.

It is a controlled change to the way enterprise data is produced and trusted.

Incremental processing can reduce unnecessary recomputation, but it requires careful handling of source changes, retries, historical state, and late-arriving data.

Schema evolution can support changing business systems, but only when compatibility and consumer impact are managed deliberately.

Reconciliation provides the evidence needed to determine whether the new pipeline is actually correct.

Finally, parallel execution creates room to validate those decisions before production consumers are fully committed to the new path.

The strongest migration plans therefore focus on three questions:

Is the data correct?

Can we explain every important difference?

Can we recover safely if the cutover fails?

If those questions can be answered confidently, the organization is doing more than moving ETL workloads. It is building a safer foundation for long-term enterprise data transformation.

Migrating Legacy ETL to Palantir Foundry: Incremental Transforms, Schema Evolution, and Reconciliation Without Downtime

Kavi Writes — Tue, 06 Oct 2026 07:13:40 +0000

Legacy ETL replacements rarely fail because the new platform can't run the transformations. They fail because nobody can prove the new outputs are correct, or because a finance report three teams away breaks on the first Monday after cutover.

A legacy pipeline has usually been around long enough to collect undocumented fixes, silent type coercions, and consumers nobody remembers onboarding. Rewriting its SQL in a new tool is the easy part. Keeping the behavior that the business actually depends on is the hard part.

This article covers three concerns that decide whether a migration to Palantir Foundry goes smoothly:

Incremental transformation: processing only what changed, instead of rebuilding everything nightly.
Schema evolution: handling source and output schema changes without breaking consumers.
Reconciliation: proving, with evidence, that the new pipeline produces equivalent results.

Running the legacy and Foundry pipelines side by side ties these together. Parallel operation turns cutover from a leap of faith into a decision backed by data.

Why Legacy ETL Migrations Are Difficult

Most legacy ETL estates share a familiar set of problems:

Monolithic jobs. A single workflow extracts, cleans, joins, aggregates, and loads, so you can't migrate one piece without understanding all of it.
Hidden dependencies. A downstream job reads an intermediate staging table that was never meant to be public.
Hard-coded logic. Currency rates, region mappings, and exclusion lists live inside stored procedures.
Fragile schedules. Job B starts at 02:30 because job A "usually finishes by then."
Inconsistent schemas. The same customer ID is a string in one system and an integer in another.
Limited observability. Failures surface as a stale dashboard, not as an alert.
Locked-in consumers. Reports, extracts, and applications depend on exact table names, column orders, and even rounding behavior.

A big bang migration bets that every one of these has been found and handled before go-live. In practice, some will only be found after go-live, when the old system is gone and there's nothing to compare against.

A Safer Migration Pattern

An incremental approach reduces that risk by migrating in bounded slices. One common lifecycle:

Inventory existing jobs, sources, targets, and schedules.
Identify critical data products and who consumes them.
Select a migration boundary small enough to validate fully, such as one subject area or one fact table and its dimensions.
Build the corresponding Foundry pipeline.
Run both pipelines in parallel on the same inputs.
Compare outputs systematically.
Resolve differences, classifying each as a new-pipeline bug, a legacy bug, or an accepted change.
Shift consumers gradually.
Retire the legacy pipeline once nothing reads from it.

This is an architectural pattern, not a mandatory Palantir methodology. Its value is that every step produces evidence, and the blast radius of a mistake stays limited to one slice.

Incremental Transforms in Foundry

Many legacy jobs follow a truncate-and-reload pattern: delete the target table, rebuild it from full source extracts. It's simple, but cost and runtime grow with total data volume rather than with the volume of change.

What Foundry provides

Foundry datasets are versioned through transactions. A build either commits a transaction or it doesn't, so a failed run doesn't leave a half-written output visible to consumers. Python transforms in Code Repositories support an @incremental decorator. When a transform runs incrementally, reading an input returns only the rows added since the last successful build, and the output can append rather than replace. Bumping the decorator's semantic_version forces a full recompute, which is useful when logic changes. Data Connection also supports incremental syncs for some source types, typically by tracking a monotonically increasing column such as an update timestamp or sequence ID. Pipeline Builder has its own incremental options; what's available can depend on your enrollment and pipeline type.

An important constraint: incremental transforms are built around appended data. If an upstream dataset is fully replaced by a snapshot, a downstream incremental transform generally can't run incrementally. Foundry datasets also aren't key-based tables with native upserts, so "latest state per key" has to be modeled explicitly.

Design concerns

Change detection. Prefer reliable change signals (CDC feeds, sequence numbers) over updated_at columns that applications forget to maintain.
Partitioning. Partition outputs by a column consumers filter on, such as business date, so reprocessing stays targeted.
Watermarks. Ingestion-side watermarks decide what gets pulled. Transform-side incrementality decides what gets processed. Keep the two concepts separate in your design.
Late-arriving data. An order updated today for last month's date lands in today's increment. Any aggregate by business date must recompute the affected dates, not just append.
Idempotency. Transactions make commits atomic, but your logic must still be deterministic. Avoid current_timestamp() inside business logic and deduplicate on stable keys.

Example: from full refresh to changelog plus current state

Legacy: a nightly job truncates fact_orders and reloads from a full ERP extract.

Redesign: ingest only changed rows into an append-only changelog, then derive the current state per order.

from pyspark.sql import functions as F, Window
from transforms.api import transform, incremental, Input, Output

@incremental(semantic_version=1)
@transform(
    out=Output("/Sales/clean/orders_current"),
    changes=Input("/Sales/clean/orders_changelog"),
)
def compute(changes, out):
    new_rows = changes.dataframe()  # only rows appended since last build
    previous = out.dataframe("previous", schema=new_rows.schema)

    w = Window.partitionBy("order_id").orderBy(
        F.col("updated_at").desc(), F.col("ingested_at").desc()
    )
    latest = (
        previous.unionByName(new_rows)
        .withColumn("rn", F.row_number().over(w))
        .filter("rn = 1")
        .drop("rn")
    )
    out.set_mode("replace")
    out.write_dataframe(latest)

The trade-off is visible: the extraction and changelog are incremental, but this current-state dataset is still rewritten each run. For very large tables, teams may partition the current-state output, or have consumers read from the changelog through a view. The decorator style above is the classic API; check your repository template for the syntax your Foundry version uses.

Managing Schema Evolution

Schema changes cause more migration incidents than transformation logic does, because they often fail silently.

Change	Typical risk
Column added	Low, unless consumers use `SELECT *` or positional reads
Field renamed	Breaks consumers; can look like a dropped column plus a new null column
Type changed	Silent nulls on failed casts, precision loss, timezone shifts
Field removed	Breaks consumers and business rules that depend on it
Nested structure changed	Breaks flattening logic, often deep inside the pipeline
Source version upgrade	Several of the above at once, often undocumented

Practices that help:

Schema contracts. Define the canonical output schema explicitly and treat changes to it as reviewed changes.
Explicit mappings. Map source fields to canonical fields by name, never by position.
Version-aware transforms. Handle known source versions deliberately instead of hoping the shape stays stable.
Validation before promotion. Foundry Data Expectations let a transform fail a build when checks don't pass, for example Check(E.primary_key("order_id"), "orders pk", on_error="FAIL"). Foundry also supports branching, so schema changes can be built and checked on a branch before merging.
Consumer impact analysis. Use Foundry's data lineage to see which downstream datasets and applications read a column before you change it.

Example: a source rename and type change

The ERP upgrade renames cust_id to customer_id and changes amount from string to decimal.

from pyspark.sql import functions as F, types as T

CANONICAL = {
    "order_id": T.StringType(),
    "customer_id": T.StringType(),
    "amount": T.DecimalType(18, 2),
    "updated_at": T.TimestampType(),
}
RENAMES = {"cust_id": "customer_id"}  # known source v1 -> v2 mapping

def to_canonical(df):
    for old, new in RENAMES.items():
        if old in df.columns and new not in df.columns:
            df = df.withColumnRenamed(old, new)
    missing = [c for c in CANONICAL if c not in df.columns]
    if missing:
        raise ValueError(f"Schema contract violated, missing: {missing}")
    return df.select([F.col(c).cast(t).alias(c) for c, t in CANONICAL.items()])

One catch: with Spark's default (non-ANSI) settings, a failed cast returns null instead of raising an error. Pair the mapping with a null-rate check on amount. Also, if the output schema of an incremental transform changes, appending to the existing output will typically conflict. Bumping semantic_version to force a full rebuild is the cleaner path.

Reconciliation Between Legacy and Foundry Pipelines

Matching row counts proves very little. Two tables can have identical counts while one has duplicated keys, swapped values, or nulls where the other has data.

Reconcile at several levels:

Level	What it catches
Row counts	Gross load failures, filter differences
Key coverage	Records present in only one output
Duplicate detection	Join fan-out, broken dedup logic
Aggregate totals	Sums by date, region, product; rounding drift
Null rates per column	Failed casts, broken mappings
Record-level comparison	Value differences on matching keys
Business-rule validation	Rules that must hold regardless of either pipeline
Sample-based review	Human inspection where full comparison is impractical

A common approach is to ingest the legacy output into Foundry via Data Connection, then compare in a dedicated reconciliation transform:

COMPARE_COLS = ["customer_id", "amount", "status"]

def row_hash(df):
    normalized = [F.coalesce(F.trim(F.col(c).cast("string")), F.lit("<null>"))
                  for c in COMPARE_COLS]
    return df.select("order_id", F.sha2(F.concat_ws("||", *normalized), 256).alias("h"))

legacy, foundry = row_hash(legacy_df).alias("l"), row_hash(foundry_df).alias("f")
result = legacy.join(foundry, "order_id", "full_outer").select(
    "order_id",
    F.when(F.col("f.h").isNull(), "only_legacy")
     .when(F.col("l.h").isNull(), "only_foundry")
     .when(F.col("l.h") != F.col("f.h"), "value_mismatch")
     .otherwise("match").alias("status"),
)

Normalization rules matter more than the comparison itself: decimal scale, timestamp timezones, trailing whitespace, empty strings versus nulls. Agree on documented tolerances up front, and record every accepted difference with a reason. Some mismatches turn out to be legacy bugs, and the business has to decide whether to fix or preserve them.

Running Both Pipelines Without Downtime

"Without downtime" here means consumers keep getting correct data on schedule throughout the migration. Whether cutover itself is invisible depends on your architecture.

Dual processing. Both pipelines run on the same inputs and schedule.
Shadow validation. Foundry outputs are reconciled every run, but nobody consumes them yet.
Consumer-by-consumer migration. Move low-risk consumers first, critical ones last.
Read-path switching. Give consumers a stable interface, such as a warehouse view, so switching means redirecting the view, not editing every report. Where Foundry needs to feed an external system, Data Connection exports may handle this, depending on the target.
Rollback planning. Define measurable rollback triggers, like mismatch rate above threshold or a missed freshness SLA, and keep the legacy path warm until the window closes.
Monitoring. Foundry health checks can cover freshness, build status, and schema. Monitor the reconciliation results as well.
Ownership. Name who fixes a broken build on each side during the transition. Ambiguity here causes most parallel-run incidents.

A Practical Migration Example

A hypothetical scenario. A retailer runs a nightly ETL job that pulls customers from a CRM, orders from an ERP, and payments from a payment gateway, then loads a reporting warehouse star schema used by dozens of reports.

Legacy pipeline. Full extracts land in staging, stored procedures build dimensions and fact_orders, and reports read the warehouse directly.

Foundry pipeline. Data Connection syncs land raw data, incremental where the source exposes a reliable change column. Python transforms build changelogs, apply the schema contracts, and produce current-state customer, order, and payment datasets. Data Expectations enforce primary keys and non-negative amounts.

Parallel validation. Both pipelines run nightly through at least one month-end close, since period-end adjustments and late payments often surface edge cases that daily runs don't.

Reconciliation. Early runs show value mismatches on amount. The cause is a legacy procedure that rounds at line level while the Foundry transform rounds at order level. Finance chooses line-level rounding, and the Foundry logic is adjusted to match.

Consumer migration. Operational dashboards move first through view redirection, then finance reports after a clean month-end.

Legacy retirement. Once query logs show no reads from the legacy tables for an agreed period, the jobs are disabled, archived, and eventually removed.

Common Migration Mistakes

Migrating everything at once, which removes the ability to compare and roll back.
Ignoring downstream dependencies, especially consumers of intermediate tables.
Treating schema changes as column edits instead of contract changes with consumers.
Validating only row counts.
Switching consumers before reconciliation is stable across a full business cycle.
No rollback criteria, so rollback becomes a debate during an incident.
Unclear ownership during the parallel period.
Retiring legacy jobs too early, before quarterly or annual processes have run.

Where Palantir Consulting Fits

Some organizations bring in Palantir consulting or implementation partners for specific parts of the program: migration architecture, legacy dependency analysis, incremental pipeline design, schema strategy, reconciliation frameworks, deployment planning, and operational handover. The most useful engagements leave the internal team able to own and extend the pipelines afterward. If you use outside help, make knowledge transfer an explicit deliverable.

Migration Checklist

Source inventory: systems, extraction methods, volumes, change signals
Dependencies: all consumers, including intermediate-table readers
Transformation logic: documented, including undocumented fixes in legacy code
Incremental strategy: append versus snapshot, late data, recompute triggers
Schema compatibility: contracts, explicit mappings, versioning rules
Data quality: expectations enforced at build time
Reconciliation: multi-level checks with agreed tolerances
Parallel execution: same inputs, same schedule, through a full business cycle
Monitoring: freshness, build health, reconciliation trends
Cutover: consumer order, read-path switching, communication plan
Rollback: measurable triggers and a warm legacy path
Legacy retirement: read-log evidence, archival, decommission date

Final Takeaway

Moving ETL to Palantir Foundry is not mainly a code-translation exercise. The real work is controlling correctness, dependencies, and operational risk while the business keeps running. Incremental processing keeps the new pipeline efficient, schema contracts keep it stable, and reconciliation gives you the evidence to cut over. Migrate in slices, compare relentlessly, and retire the old system only when the data says it's safe.

Designing a Palantir Foundry Ontology That Holds Up in Production: Object Types, Link Types, and Action Governance

Kavi Writes — Tue, 29 Sep 2026 10:40:04 +0000

A Foundry pilot usually works. A small team maps a few datasets into the Ontology, builds an operational app, and a planner can finally see open orders next to supplier delays in one place.

Then the second and third use cases arrive. A Customer object means "billing account" to one team and "legal entity" to another. A link that looked like obvious double-counts shipments. An Action that updates a status field has no clear owner, and nobody can say who changed what last Tuesday, or why.

These are rarely platform limitations. They are design decisions made quickly during a proof of concept and then inherited by production. This article covers how to design a Palantir Foundry Ontology that survives that transition: Object Types, Link Types, properties, and the Action governance that connects the model to real workflows.

What a Production-Ready Foundry Ontology Actually Needs

The Foundry Ontology is a semantic layer over your datasets. It maps rows and columns to business concepts through four main building blocks:

Object Types: the entities people work with, such as a Work Order, Supplier, or Aircraft.
Properties: the attributes of those entities.
Link Types: the relationships between Object Types.
Action Types: defined, permissioned ways to change objects and links.

Applications built on Foundry, from operational apps to SDK-based custom software to AI workflows, read and write through this layer. That is what makes it powerful, and it is also why design mistakes spread quickly.

In a pilot, the Ontology is a convenience. In production, it behaves like a public API with many consumers. It needs stable meaning, named owners, controlled change, secure access, and writes you can audit. The rest of this article is about building those properties in from the start.

Start from what people act on

A useful test for a candidate Object Type: would someone open it, assign it, approve it, or track it? If the answer is no, it is probably a property, a lookup, or a pipeline detail.

Avoid creating one Object Type per source table. ERP systems often split one business entity across header, item, and status tables. The Ontology should present the entity users recognize, with the joining done upstream in pipelines.

Granularity takes judgment. Should a Work Order and its individual operations be one Object Type or two? Split them when they have different lifecycles, owners, or access rules. Keep them together when users only ever treat them as one unit.

Choose primary keys that never move

The primary key is the most consequential decision on an Object Type. Links, user edits, and downstream references all depend on it, so changing it later is expensive.

A good key is unique, deterministic, and stable across pipeline rebuilds. Avoid row numbers, surrogate IDs that are regenerated with each run, and natural keys that are reused, e.g., reissued badge numbers. When objects come from multiple source systems, prefix the key with the system and site (for example, SAP-DE01-4500012345) to prevent collisions after a merge.

Be selective with properties

Include the properties people filter on, decide with, or need an AI workflow to reason about. Leave the other 180 source columns in the dataset. Use business names, explicit units, UTC timestamps, and a clear title property so objects are readable in search results and apps.

Getting Link Types Right

A Link Type connects two Object Types with a defined cardinality. One-to-many links are typically resolved through a foreign key property, while many-to-many links are backed by a join dataset.

The goal is to model relationships the business talks about, not every foreign key in the source schema. "Supplier supplies Part" is meaningful. "Table A references column B in a staging table" is not. Name both directions clearly, such as Supplied Parts from the Supplier side and Approved Suppliers from the Part side.

When a link should become an object

Some relationships carry their own data: effective dates, contract prices, approval status. If you need that, consider modeling the relationship as its own Object Type, like a Supplier Part Approval between Supplier and Part. You pay for an extra hop in queries, but you gain history, a lifecycle, and separate permissions. For a simple association with no attributes, a plain link is the better choice.

Verify cardinality against real data

A link declared one-to-many that is actually many-to-many in the data will not always fail loudly. It can produce misleading counts and incomplete traversals instead. Test cardinality assumptions in the backing pipelines before you publish the link.

From Data Model to Operational Model

Data quality belongs upstream

The Ontology reflects whatever its backing datasets contain. Duplicate keys, null foreign keys, and orphaned link targets show up as confusing objects in front of users. Put checks where the data is produced: key uniqueness, required fields, and referential integrity for every link, using Foundry's data health checks and pipeline expectations.

Ownership and lineage

Give every Object Type a named business owner who decides what it means and a technical owner who maintains its pipeline. Foundry's lineage shows how data flows from source to backing dataset, which helps with impact analysis. Lineage does not explain meaning, though, so write real descriptions for Object Types and properties.

Change over time

Decide early whether an Object Type represents current state, history, or both. A common pattern keeps the entity as current state and adds a separate event Object Type, such as Status Change, for history.

Treat schema changes like API versioning. Adding a property is usually safe. Renaming or removing properties, changing types, or altering a primary key can break apps and SDK consumers. Deprecate first, notify consumers, and use review workflows such as ontology branching or proposals where your environment supports them.

Designing for AI applications

AI workflows that reason over the Ontology depend on the same clarity humans do: precise names, useful descriptions, and unambiguous links. When an AI workflow is allowed to propose or make changes, the Action becomes the safety boundary. It should get the same permissions, validation, and audit trail as any human user, not a shortcut around them.

Designing Actions With Governance in Mind

An Action Type is a parameterized, permissioned transaction against the Ontology. It can create, modify, or delete objects and links, and it can trigger side effects such as notifications or webhooks. Simple Actions are configured with rules; complex ones can be backed by functions. Actions are how the Ontology stops being a read-only view and starts supporting decisions.

They are also where most production risk lives. Ontology governance that controls reads but not writes is only half done.

Keep Actions narrow

A single "Update Work Order" Action with 30 optional fields is hard to secure and harder to audit. Separate Actions like Reassign Work Order and Close Work Order each have one purpose, one set of eligible users, and one clear validation rule.

The five governance questions

Concern	What to define
Permissions	Who can submit, separate from who can view. Grant through groups, not individuals.
Validation	Submission criteria on the user and parameters, enforced server-side, not just in the UI.
Ownership	A named owner, a documented purpose, and the expected callers: people, automations, or AI.
Auditability	Who changed what, when, and why. Add a required justification parameter for sensitive changes.
Approval	Whether a second person must confirm before the change takes effect.

Approvals need explicit design

For changes that need sign-off, a portable pattern is a request object with a status, plus a separate approval Action limited to approvers. Submission criteria can block self-approval by checking that the approver is not the requester. If your environment offers a native approval capability, evaluate it against this pattern, but make the approval path visible in the model either way.

A Practical Enterprise Example

Consider a hypothetical manufacturer that wants to manage supplier quality problems across several plants. Today, incidents live in spreadsheets, and putting a supplier on hold means an email thread with procurement.

Object Types and Link Types

Supplier, keyed by a global vendor ID, not a per-plant ERP code.
Part, linked many-to-many to Supplier through an approved supplier list.
Plant, the site where incidents are raised.
Quality Incident, linked to one Part, one Plant, and one Supplier.
Supplier Hold Request, linked to one Supplier and to the incidents that justify it.

Actions

Action	Who can submit	Key validation	Effect
Log Quality Incident	Plant quality engineers	Part is active at the selected Plant; severity required	Creates an incident and its links
Request Supplier Hold	Quality leads	At least one linked incident; justification required	Creates a request with status Pending
Approve Supplier Hold	Procurement leads	Approver is not the requester; request is Pending	Sets the Supplier's hold status and notifies buyers
Release Supplier Hold	Procurement leads	Supplier is currently on hold; reason required	Clears the hold and records the release

An AI assistant could summarize open incidents for a supplier and draft a hold request. Only a procurement lead can submit the approval, so the most consequential step stays with a human.

One decision remains outside the Ontology: which system owns the hold status. If the ERP blocks purchase orders, the approved hold must reach it, for example through a webhook side effect or an outbound sync. Define the system of record explicitly, or two systems will disagree about whether a supplier is blocked.

Common Ontology Design Mistakes

Mirroring the source schema. The Ontology becomes a harder-to-query copy of the ERP instead of a model of the business.
Unstable primary keys. Keys that shift on rebuild break links and orphan user edits.
Catch-all Object Types. A single Asset type covering vehicles, buildings, and software licenses forces dozens of mostly-null properties and muddled permissions.
Generic write Actions. "Edit Object" style Actions make validation and audit nearly meaningless.
Governing reads but not writes. Row-level access is carefully designed, while any editor can submit any Action.
No owners or descriptions. Six months later, nobody knows whether status means ERP status or operational status.
Using the Ontology as a reporting layer. Heavy aggregations usually belong in datasets and analytics tools, with the Ontology focused on entities people act on.
Breaking changes without a consumer inventory. Renaming a property can quietly break apps, automations, and SDK clients you did not know existed.

Production Readiness Checklist

Use this before promoting an Object Type, Link Type, or Action to production.

Model

Each Object Type represents something users open, assign, approve, or track.
Primary keys are unique, deterministic, and stable across rebuilds and source systems.
Properties include explicit units and business names. Timezones are all the same.
The link cardinality is tested on real data and both directions are explicitly stated.

Data and ownership

Backing datasets have health checks for key uniqueness, required fields, and link integrity.
Every Object Type has a business owner, a technical owner, and a written description.
History requirements are decided: current state, event objects, or both.
Consumers are known, and there is a deprecation process for breaking changes.

Actions

Each Action has a single purpose and a named owner.
Submission permissions are group-based and distinct from read access.
Validation is performed server-side via submission criteria or function logic.
Sensitive Actions require a justification and, where needed, a second approver.
AI-initiated changes go through the same Actions and controls as human changes.
The system of record is defined for any value that is written back to external systems.

Final Takeaways

A production Foundry Ontology is a contract, not a diagram. Object Types should reflect the entities people act on, Link Types should capture relationships the business actually uses, and primary keys should be treated as permanent from day one.

Actions deserve the most care, because they are where the model changes the business. Keep them narrow, permission them deliberately, validate server-side, and make approvals and audit trails explicit.

None of these rules is absolute. Granularity, history, and the choice between a link and an intermediate object depend on your workflows. Whether the work is done by an internal platform team or with a Palantir consulting partner, the useful test is the same: can a new team understand, trust, and safely change this Ontology without asking the people who built it?