Gowtham Potureddi

Posted on Aug 12

Talend & Apache Hop: Open-Source ETL Studios and Their Interview Questions

#python #sql #interview #dataengineering

talend and Apache Hop are the two names that still come up whenever a data engineer says "we built it in a visual ETL studio" — a drag-and-drop canvas where boxes are transformations, arrows are streams of rows, and the whole pipeline is metadata you can version, parameterise, and hand to an operator who has never written a line of Java. For twenty years the visual-ETL lineage — Kettle, Pentaho Data Integration, Talend Open Studio, and now Apache Hop — has quietly run a large slice of the world's warehouse loads, master-data flows, and nightly batch jobs, and it keeps appearing in interviews precisely because so many production systems are still built on it. The question an interviewer is really asking when they put "Talend" or "Hop" on the table is not "can you drag a component" but "do you understand how a visual studio models extract, transform, and load, where it generates code versus interprets metadata, and when a graphical etl studio beats a hand-written pipeline."

This guide is the walkthrough you wished existed the first time someone asked "explain a Talend job and the role of tmap", or "what's the difference between an Apache Hop pipeline and a workflow", or "your team runs Kettle in production — should you migrate to Hop or rewrite in code?" It walks the full arc: why open source etl studios still matter in 2026, how Talend models talend jobs, components, contexts, and Java code generation inside Talend Studio, how Apache Hop models hop pipelines, workflows, and transforms on a metadata-driven engine descended from kettle pentaho, how the same extract-transform-load job looks built in both studios side by side, and how to frame Talend versus Hop versus a code-first stack when the talend interview questions turn into architecture questions. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse on the data-transformation practice library →, and sharpen the load side with the database practice library →.

On this page

Why open-source ETL studios still matter
Talend — jobs, components, tMap, contexts, code generation
Apache Hop — pipelines, workflows, the metadata-driven engine
Building a pipeline — a worked ETL job in both studios
Talend vs Hop vs code-first + interview questions
Cheat sheet — Talend and Apache Hop recipes
Frequently asked questions
Practice on PipeCode

1. Why open-source ETL studios still matter

The visual-ETL lineage from Kettle to Talend to Hop is still load-bearing infrastructure — and interviewers know it

The one-sentence invariant: an open source etl studio is a graphical modelling environment where you assemble an extract-transform-load pipeline as a directed graph of pre-built components — each box a transformation, each arrow a stream of typed rows — and the studio either generates runnable code from that graph (Talend generates Java) or interprets the graph as metadata at run time (Apache Hop and its Kettle ancestor), so that the pipeline becomes a versionable, parameterisable artifact that a data engineer, an analyst, and an operator can all reason about without reading imperative code. The reason interviewers still probe it in 2026 — long after "just write Spark" became the fashionable answer — is that a huge installed base of warehouse loads, master-data-management flows, healthcare and finance batch jobs, and government data exchanges runs on these studios, and someone has to maintain, migrate, or interview about them.

The lineage every senior engineer should be able to draw.

Kettle (2001). The original open-source visual ETL engine, created by Matt Casters. Two artifact types — transformations (row-level data flow) and jobs (orchestration). Row-based, thread-per-step streaming engine.
Pentaho Data Integration (PDI). Kettle rebranded after Pentaho acquired it (2006); "Kettle" and "PDI" are used interchangeably. Later owned by Hitachi Vantara.
Talend Open Studio (2006–2024). A different lineage — an Eclipse-based studio that generates Java. The free/open edition (Talend Open Studio for Data Integration) was the on-ramp; the paid platform (now under Qlik) added orchestration, data quality, and cloud. Talend ended the free Open Studio line in early 2024.
Apache Hop (2020+). A clean-room fork/rewrite of the Kettle codebase, donated to the Apache Software Foundation; became a top-level Apache project in 2021. Modernises Kettle — projects/environments, a plugin-first metadata architecture, and an Apache Beam integration that can run the same pipeline on Spark, Flink, or Google Dataflow.

Why visual studios keep winning a slice of the market.

Non-coder accessibility. An analyst who knows the data but not Java or Python can read, and often edit, a canvas of labelled boxes. This is the single biggest reason ETL studios persist in enterprises with mixed-skill teams.
Rich connector catalogue. Hundreds of pre-built components / transforms for databases, files, SaaS APIs, message queues, and cloud storage — no glue code to write a JDBC reader or a CSV parser.
Visible lineage. The graph is the documentation. You can point at a box and say "this is where the currency conversion happens" in a compliance review — something a 2,000-line Python script does not give you for free.
Operational tooling. Built-in logging, run-history, restartability, and a server component (Talend Administration Center / Hop Server) for scheduling and monitoring without wiring up Airflow.

Where code-first stacks push back — and the honest trade-off.

Version control and diffs. A Talend .item file or a Hop .hpl is XML; it diffs poorly compared to SQL or Python. Hop improved this (cleaner XML, git-friendly projects) but it is still not as reviewable as code.
Testing. Unit-testing a visual pipeline is harder than testing a pure function. Hop added a native unit-testing framework; Talend leans on job-level integration tests.
Scale ceiling. The classic single-JVM row engine (Kettle, Talend standalone) tops out on one machine. Hop's Beam runners and Talend's Spark/data integration cloud jobs address this, but a team already fluent in Spark often skips the studio.

What interviewers listen for.

Do you name the Kettle → PDI → Hop lineage and the separate Talend lineage without conflating them? — senior signal.
Do you know that Talend generates Java while Hop interprets metadata at run time? — required answer.
Do you say "pipeline = data flow, workflow/job = orchestration" as the two-artifact model both families share? — required answer.
Do you push back on "just rewrite it in Spark" with the maintenance-cost and non-coder-accessibility argument? — senior signal.

Worked example — mapping the ETL vocabulary across the three tools

Detailed explanation. The single most useful artifact for a visual-ETL interview is a translation table across Kettle/PDI, Talend, and Apache Hop, because interviewers routinely switch vocabulary mid-question ("you said 'step' — is that a Talend component?"). Being fluent in all three vocabularies signals you have actually worked across the lineage, not just one tool.

The unit of data flow. Kettle "transformation", Talend "job (data flow portion)", Hop "pipeline".
The unit of orchestration. Kettle "job", Talend "job / parent job with tRunJob", Hop "workflow".
The processing box. Kettle "step", Talend "component", Hop "transform".
The connecting arrow. Kettle/Hop "hop", Talend "row / connection (Main, Lookup, Reject)".

Question. Build the cross-tool vocabulary map an interviewer can quiz you on in any direction.

Input.

Concept	Kettle / PDI	Talend	Apache Hop
Row-level data flow	transformation (.ktr)	job data flow	pipeline (.hpl)
Orchestration	job (.kjb)	job + tRunJob	workflow (.hwf)
Processing box	step	component	transform
Connecting arrow	hop	row (Main/Lookup/Reject)	hop
Central mapper	"Stream lookup" / "Merge join"	tMap	"Stream lookup" / "Join rows"
Reusable sub-flow	mapping (sub-transformation)	joblet	(metadata-injection pipeline)

Code.

Vocabulary translation cheat card (memorise both directions)
============================================================

Talend says ..............  Hop / Kettle says
  job (orchestration) ....... workflow / job
  subjob ..................... (a chain of actions)
  component .................. transform / action
  Main row .................. hop (data)
  Lookup row ................ info-stream hop into a lookup transform
  Reject row ................ error-handling hop
  tMap ...................... Join rows + Filter + Select values + JavaScript
  context group ............. project + environment (variables/parameters)
  routine (Java) ............ User Defined Java Class (UDJC)
  Repository metadata ....... Hop metadata (relational DB connection, etc.)

Step-by-step explanation.

The data-flow artifact is the heart of the tool: a Kettle transformation, the data-flow portion of a Talend job, and a Hop pipeline are the same idea — a graph of processing boxes through which rows stream. Rows move box-to-box; each box transforms, filters, joins, or routes them.
The orchestration artifact — Kettle job, Talend parent job, Hop workflow — is sequential and control-flow-oriented: run this pipeline, if it succeeds send mail, if it fails abort. Do not run bulk data transformation here; orchestration steps pass control, not row streams.
The processing box has three names (step / component / transform) for one concept. Interviewers use whichever the tool uses; you must map on the fly.
tMap has no single equivalent in Hop/Kettle — it fuses join, filter, expression, and multi-output routing into one dialog. In Hop you decompose it into Join rows / Stream lookup + Filter rows + Select values + optional JavaScript. Knowing this decomposition is the classic "have you used both?" tell.
Reusable logic maps as joblet (Talend) ↔ mapping/sub-transformation (Kettle) ↔ metadata-injection pipeline (Hop). Each lets you factor a repeated sub-flow into one reusable unit.

Output.

You are asked about ...	Correct one-line answer
"What's a Talend job vs a Hop pipeline?"	Talend job = orchestration + data flow in one; Hop splits them into workflow + pipeline
"Kettle 'step' — Talend name?"	component
"Hop equivalent of tMap?"	no single transform; Join rows + Filter + Select values (+ JavaScript)
"Where do variables live?"	Talend context groups; Hop project + environment configs

Rule of thumb. Learn the vocabulary in both directions before any visual-ETL interview. The fastest way to sound junior is to call a Hop transform a "component" or a Talend component a "step" — small slips that signal you have only touched one tool.

Worked example — deciding studio vs code for a new pipeline

Detailed explanation. A senior engineer does not reach for a visual studio reflexively, nor dismiss it reflexively. They run a short decision on team skills, connector needs, governance, and scale. Codifying that decision makes the interview answer reproducible.

Team composition. Mixed analysts + engineers → studio wins on accessibility. All-engineer team fluent in Spark/dbt → code often wins.
Connector surface. Many exotic sources (SAP, Salesforce, legacy DBs, flat-file EDI) → studio's connector catalogue saves weeks.
Governance. Auditors need visible lineage and column-level documentation → studio's graph is an asset.
Scale. Sub-terabyte nightly batch → single-engine studio is fine. Petabyte streaming → Beam runner or native Spark.

Question. For three scenarios, decide studio-or-code and name the specific tool.

Input.

Scenario	Team	Sources	Scale	Governance
Hospital nightly warehouse load	analysts + 1 engineer	6 DBs + HL7 files	40 GB/night	strict audit
Ad-tech clickstream enrichment	all engineers	Kafka + S3	5 TB/hour	light
Retail MDM consolidation	mixed	SAP + Salesforce + CSV	200 GB/night	moderate

Code.

# Illustrative decision helper — studio vs code
def pick_etl_approach(has_non_coders: bool,
                      exotic_connectors: bool,
                      needs_visible_lineage: bool,
                      hourly_volume_tb: float) -> str:
    if hourly_volume_tb >= 1.0 and not has_non_coders:
        return "code-first (Spark / Beam) or Hop-on-Beam"
    if exotic_connectors or needs_visible_lineage or has_non_coders:
        return "visual ETL studio (Talend or Apache Hop)"
    return "code-first (dbt / Python)"

print(pick_etl_approach(True,  True,  True,  0.04))   # hospital
# → visual ETL studio (Talend or Apache Hop)
print(pick_etl_approach(False, False, False, 5.0))    # ad-tech
# → code-first (Spark / Beam) or Hop-on-Beam
print(pick_etl_approach(True,  True,  True,  0.2))    # retail MDM
# → visual ETL studio (Talend or Apache Hop)

Step-by-step explanation.

The hospital load has non-coder maintainers, exotic sources (HL7), and strict audit — three independent reasons the studio wins. Apache Hop is the modern open-source pick; Talend if the shop is already licensed.
The ad-tech clickstream is all-engineer, high-volume, low-governance — the studio's accessibility advantage evaporates and the scale requirement favours a code-first Spark/Beam job (or Hop configured to run on a Beam Spark runner, which keeps the visual model but scales out).
The retail MDM consolidation mixes skills and leans on SAP/Salesforce connectors — the studio's catalogue alone can save weeks of connector engineering, so it wins despite a competent engineering team.
The helper encodes the priority order: extreme scale + all-engineers pushes to code; non-coders, exotic connectors, or lineage requirements pull to a studio; everything else defaults to code-first.
The senior nuance is that Hop-on-Beam is not an either/or — you keep the visual model and get Spark/Flink scale-out, which is the answer that most impresses when scale is the objection.

Output.

Scenario	Decision	Tool
Hospital nightly load	studio	Apache Hop (or Talend if licensed)
Ad-tech clickstream	code / hybrid	Spark, or Hop-on-Beam
Retail MDM	studio	Talend or Hop (connector-driven)

Rule of thumb. Pick the visual studio for accessibility, connector breadth, and visible lineage; pick code for all-engineer teams at extreme scale. When scale is the only objection to a studio, reach for Hop-on-Beam before abandoning the visual model entirely.

Common beginner mistakes.

Conflating the two lineages. Talend and Kettle/Hop are different codebases with different execution models. Saying "Hop is the open-source version of Talend" is wrong and interviewers catch it.
Assuming a studio can't scale. The single-JVM ceiling is real for classic Kettle and standalone Talend, but Hop-on-Beam and Talend Spark jobs break it.
Treating the canvas as throwaway. These graphs are versioned artifacts; committing .item / .hpl files to git and parameterising them is the professional baseline.
Ignoring the free-edition end-of-life. Talend Open Studio's free line ended in 2024 — recommending it for a new open-source project in an interview signals stale knowledge; Apache Hop is the live open-source answer.

ETL-architecture interview question on choosing a visual studio

A senior interviewer might ask: "A regional bank runs 300 nightly Kettle transformations on Pentaho Data Integration 8. Support is ending and the team is half analysts. Walk me through evaluating a migration — to Apache Hop, to Talend, or to a code-first stack — and how you'd stage it without a big-bang cutover."

Solution Using a lineage-aware, staged migration from Kettle to Apache Hop

Migration decision + staging plan
=================================

Step 0 — inventory
  - Count transformations (.ktr) and jobs (.kjb): 300 + 45.
  - Classify by risk: 220 "simple" (input→transform→output),
    80 "complex" (JavaScript steps, sub-mappings, DB procedures).

Step 1 — pick the target
  - Team is half analysts  -> keep a VISUAL studio (accessibility).
  - Source is Kettle/PDI    -> Apache Hop is the natural successor
                               (shared lineage; Hop imports Kettle files).
  - Rewriting 300 flows in Spark = months + retrains half the team -> reject
    for the bulk; reserve code-first for the ~10 highest-volume flows.

Step 2 — automated import
  - Use Hop's Kettle/PDI import: hop-import converts .ktr -> .hpl,
    .kjb -> .hwf, mapping most steps to Hop transforms 1:1.
  - Flag steps with no clean Hop equivalent (deprecated / custom).

Step 3 — stage the cutover (strangler pattern)
  - Wave 1: 220 simple flows, imported + smoke-tested in Hop, run in
    parallel with Kettle for 2 weeks (dual-run, compare row counts).
  - Wave 2: 80 complex flows, hand-review JavaScript -> UDJC / Select values.
  - Wave 3: promote the ~10 highest-volume flows to Hop-on-Beam (Spark runner).

Step 4 — verify + decommission
  - Reconcile: SELECT COUNT(*) and checksums per target table, Kettle vs Hop.
  - Cut scheduler to Hop Server; keep Kettle read-only for 1 release, then retire.

-- Wave-1 dual-run reconciliation query (run against each target table)
SELECT 'kettle' AS engine, COUNT(*) AS rows, SUM(hashtext(t::text)) AS checksum
FROM   warehouse_kettle.dim_customer t
UNION ALL
SELECT 'hop'    AS engine, COUNT(*) AS rows, SUM(hashtext(t::text)) AS checksum
FROM   warehouse_hop.dim_customer t;
-- Rows and checksum must match before promoting the Hop flow to primary.

Step-by-step trace.

Step	Input	Action	Result
Inventory	300 .ktr + 45 .kjb	classify simple vs complex	220 simple, 80 complex, 45 jobs
Target pick	half-analyst team + Kettle source	prefer visual + shared lineage	Apache Hop
Import	.ktr / .kjb files	`hop-import`	.hpl / .hwf, flagged exceptions
Wave 1	220 simple flows	dual-run 2 weeks	row + checksum parity
Wave 2	80 complex flows	hand-review JS steps	UDJC / Select values equivalents
Wave 3	10 hot flows	Hop-on-Beam	Spark-scale execution

After the staged migration, the bank keeps its visual model (analysts stay productive), the shared Kettle→Hop lineage makes the bulk import nearly mechanical, and only the highest-volume flows take on the extra complexity of a Beam runner. No big-bang cutover; every wave is reconciled before it becomes primary.

Output:

Metric	Before (Kettle 8, EOL)	After (Apache Hop)
Studio support	ending	active Apache project
Team retraining	none needed	minimal (same visual model)
Bulk migration effort	N/A	mostly automated import
Scale ceiling	single JVM	Beam runners on hot flows
Cutover risk	—	low (staged, dual-run reconciled)

Why this works — concept by concept:

Shared Kettle-Hop lineage — Hop is a fork of the Kettle codebase, so hop-import maps most .ktr/.kjb steps to Hop transforms/actions almost one-to-one. This is why migrating Kettle→Hop is an order of magnitude cheaper than Kettle→Talend or Kettle→Spark.
Keep the visual model — half the team are analysts; preserving a drag-and-drop etl studio keeps them productive and avoids a costly retraining project. Accessibility is a first-class migration constraint.
Strangler / dual-run — running Hop beside Kettle and reconciling row counts + checksums per wave means every flow is proven equivalent before it becomes primary, so there is never a moment where correctness is unverified.
Selective Beam promotion — only the ~10 highest-volume flows pay the Beam-runner complexity tax; the other 290 keep the simple native engine. Scale is applied surgically, not universally.
Cost — mostly-automated import + a few weeks of dual-run per wave, versus months to rewrite 300 flows in Spark and retrain analysts. The migration is O(flows) in review effort but near-O(1) in per-flow engineering for the simple majority.

ETL
Topic — etl
ETL problems on pipeline design and migration

Practice →

Data Processing Topic — database Database problems on batch loads

Practice →

2. Talend — jobs, components, tMap, contexts, and code generation

A Talend job is a canvas of `components` that the studio compiles into Java — and `tmap` is where the real transformation logic lives

The mental model in one line: talend models an ETL pipeline as a job — an Eclipse-canvas graph of components connected by typed rows (Main, Lookup, Reject) and control triggers (OnSubjobOk, OnComponentOk, Run If) — where each component is a pre-built building block (tFileInputDelimited, tDBInput, tMap, tAggregateRow, tDBOutput), the central tmap component does joins, filtering, expressions, and multi-output routing in one dialog, contexts parameterise the job across Dev/Test/Prod, and the studio generates Java source from the whole graph so the deliverable is a compiled, standalone Java program rather than an interpreted metadata file. Understanding that last clause — Talend generates code — is the single most important thing that distinguishes it from Hop/Kettle.

The job anatomy — what every Talend canvas is made of.

Components. The boxes. Prefixed by family: tFile* (files), tDB* / tMysql* / tPostgresql* (databases), tMap / tXMLMap (mapping), tAggregateRow, tSortRow, tFilterRow, tJava / tJavaRow / tJavaFlex (custom code), tRunJob (call a child job), tLogRow (debug).
Rows (data connections). Main carries the primary row stream; Lookup feeds a reference stream into tMap; Reject carries rows that failed a component (e.g. rows a tDBOutput couldn't insert). Row schemas are typed and propagate down the flow.
Triggers (control connections). OnSubjobOk / OnSubjobError sequence whole subjobs; OnComponentOk / OnComponentError sequence single components; Run If branches on a condition. These are orchestration, not data flow.
Subjobs. A connected group of components sharing one data flow; the orange-bordered region. Triggers link subjobs into an execution order.

tMap — the transformation heart.

Joins (lookups). The main input on the left; one or more lookup inputs joined on key expressions. Join model is Left Outer or Inner; match model is Unique, First, or All matches.
Expressions. Per-output-column Java expressions: row1.amount * context.fx_rate, StringHandling.UPCASE(row1.name), Relational.ISNULL(lookup.email) ? "unknown" : lookup.email.
Filters. A boolean expression per output table gates which rows reach it — this is how one tMap fans a single input into "valid" and "rejected" outputs.
Variables. tMap internal variables (the Var panel) compute intermediate values once and reuse them across output columns — the studio equivalent of a local variable.

Contexts — parameterising the job.

Context variables. Named parameters (context.db_host, context.fx_rate, context.input_dir) referenced anywhere an expression is allowed.
Context groups. A named set of variables with per-environment values — a Default group with columns for Dev, Test, Prod. Switching the active context repoints the whole job at a different environment.
Load at run time. tContextLoad reads context values from a file or database at start-up, so you can change parameters without recompiling.
Implicit context load. Project-level setting to auto-load contexts from a file/DB for every job.

Code generation — the Talend differentiator.

Java under the hood. Each component maps to a Java code template; the studio stitches them into one .java class per job. You can open the "Code" tab and read it.
Routines. Reusable static Java methods (StringHandling, Numeric, TalendDate, plus your own) callable from any expression.
Build / export. "Build Job" produces a standalone .zip with the compiled classes, dependencies, and run.sh/run.bat — deployable on any JVM with no Talend Studio present.
Consequence. Because the artifact is Java, Talend jobs run anywhere Java runs, integrate with Java libraries, and are debuggable as Java — but the generated code is verbose and diffs poorly in git.

Worked example — a tMap join with a lookup, an expression, and a reject flow

Detailed explanation. The canonical Talend job: read orders from a delimited file, enrich each order with customer data via a tMap lookup, compute a converted total with an expression, route unmatched orders to a reject output, and load the good rows into a database. This one job exercises components, rows, tMap joins, expressions, filters, and contexts.

Main input. tFileInputDelimited reading orders.csv (order_id, customer_id, amount_usd).
Lookup input. tDBInput reading customers (customer_id, name, country, fx_rate).
tMap. Left-outer join on customer_id; compute amount_local; split matched vs unmatched.
Outputs. tDBOutput for enriched orders; tFileOutputDelimited for rejects.

Question. Specify the tMap join, the output-column expressions, and the reject filter for this enrichment job.

Input.

tMap element	Setting
Main input	row1 (order_id, customer_id, amount_usd)
Lookup input	cust (customer_id, name, country, fx_rate)
Join key	row1.customer_id = cust.customer_id
Join model	Left Outer, match = Unique
Output "enriched"	filter: cust.customer_id != null
Output "rejects"	filter: cust.customer_id == null

Code.

// tMap output "enriched" — per-column expressions (Java, evaluated per row)
enriched.order_id     = row1.order_id;
enriched.customer_id  = row1.customer_id;
enriched.customer     = StringHandling.UPCASE(cust.name);
enriched.country      = cust.country;
enriched.amount_usd   = row1.amount_usd;
enriched.amount_local = row1.amount_usd * cust.fx_rate;    // FX conversion
enriched.loaded_at    = TalendDate.getCurrentDate();

// tMap "enriched" output-table filter (only matched rows)
cust.customer_id != null

// tMap "rejects" output-table filter (unmatched lookups)
cust.customer_id == null

// A tMap Var (computed once, reused): normalise the country code
// Var.country_code = cust.country == null ? "XX" : cust.country.toUpperCase()

Job canvas (rows and triggers)
==============================

  tFileInputDelimited(orders.csv) --Main(row1)--> tMap --Main(enriched)--> tDBOutput(orders_enriched)
                                                    |
  tDBInput(customers) -----------Lookup(cust)-------+
                                                    |
                                                    +--Main(rejects)------> tFileOutputDelimited(rejects.csv)

  tDBOutput --Reject--> tLogRow   (rows the DB refused, e.g. constraint violations)

Step-by-step explanation.

tFileInputDelimited reads orders.csv and emits a typed Main row stream (row1) into tMap. The schema (order_id INT, customer_id INT, amount_usd DECIMAL) is defined once and propagates.
tDBInput runs SELECT customer_id, name, country, fx_rate FROM customers and feeds it into tMap as a Lookup stream (cust). By default the lookup is loaded into memory once ("Load once") for fast hash-join matching.
Inside tMap, the join key row1.customer_id = cust.customer_id with Left Outer / Unique means every order flows through even if no customer matches — unmatched rows carry cust.* == null. The amount_local expression multiplies by the looked-up fx_rate.
Two output tables split the stream by filter: enriched keeps cust.customer_id != null; rejects keeps the nulls. This is tMap's signature move — one input, expression-gated fan-out to many outputs.
The tDBOutput Reject row (distinct from the tMap reject output) catches rows the database refuses — duplicate keys, constraint violations — and sends them to tLogRow. Two different "reject" concepts: tMap-filter rejects (business rule) vs component rejects (runtime error).

Output.

order_id	customer	country	amount_usd	amount_local	routed to
1001	ACME LTD	GB	100.00	78.50	enriched
1002	NORD AB	SE	50.00	520.00	enriched
1003	(no match)	—	40.00	—	rejects.csv

Rule of thumb. Use one tMap for join + expression + fan-out; put business-rule routing in tMap output filters and runtime-error handling on component Reject rows. Keep FX rates and hosts in context variables, never hard-coded in expressions.

Worked example — context groups for Dev/Test/Prod promotion

Detailed explanation. A job hard-coded to a dev database is unshippable. Talend contexts solve this: define a context group with per-environment values, reference context.* everywhere, and switch the active context (or load it at run time) to promote the same job across environments without touching a single component.

Context group. Env with variables db_host, db_name, db_user, fx_rate.
Environments. Columns Dev, Test, Prod each holding different values.
Run-time load. tContextLoad overrides values from a properties file so Ops can retune Prod without a rebuild.

Question. Define the context group and show how the job references it and loads overrides at run time.

Input.

Variable	Dev	Test	Prod
db_host	localhost	test-db.internal	prod-db.internal
db_name	orders_dev	orders_test	orders_prod
fx_rate	0.80	0.80	0.785

Code.

// tDBInput / tDBOutput "Use an existing connection" fields reference context:
//   Host      : context.db_host
//   Database  : context.db_name
//   Username  : context.db_user
//   (never a literal "prod-db.internal" — always context.*)

// tMap FX expression stays environment-agnostic:
enriched.amount_local = row1.amount_usd * context.fx_rate;

# prod.properties — loaded at run time by tContextLoad (overrides compiled defaults)
db_host=prod-db.internal
db_name=orders_prod
db_user=etl_writer
fx_rate=0.785

Run-time promotion (no recompile)
=================================
  tFileInputDelimited(prod.properties) --Main--> tContextLoad --OnSubjobOk--> main ETL subjob

  # Or from the CLI on the built job:
  ./orders_enrich_run.sh --context=Prod \
      --context_param fx_rate=0.782      # last-minute override wins

Step-by-step explanation.

The context group turns four environment-specific values into named variables. Because Dev, Test, and Prod are columns of the same group, the identical job binary can target any of them.
Every component references context.db_host etc. instead of a literal. This is the discipline that makes a job promotable — a single hard-coded host anywhere breaks the promotion story.
tContextLoad reads prod.properties at the start of the run and overwrites the compiled context values. This lets operators retune Prod (e.g. change fx_rate) without opening Talend Studio or rebuilding.
The built job's run.sh accepts --context=Prod to select the environment and --context_param key=value to override any single variable — the override precedence is CLI param > loaded file > compiled default.
The net effect: build once, run anywhere. The same exported job artifact runs in Dev during testing and Prod in production, differing only by which context is active and which properties file is loaded.

Output.

Run	Active context	fx_rate used	Target DB
Local dev	Dev	0.80	orders_dev @ localhost
CI test	Test	0.80	orders_test @ test-db
Production	Prod + prod.properties	0.785	orders_prod @ prod-db
Prod hotfix	Prod + CLI override	0.782	orders_prod @ prod-db

Rule of thumb. Never hard-code hosts, credentials, or business constants in a Talend component — put them in a context group, reference context.* everywhere, and layer tContextLoad + CLI overrides so the same built job promotes cleanly across environments.

Worked example — reading the generated Java and adding a routine

Detailed explanation. Because Talend generates Java, you can (and in interviews should be able to) reason about what the studio produces. A routine is reusable static Java you call from any expression. Walk through a custom routine and where it lands in the generated code.

Routine. A static method maskEmail(String) in a project routine class.
Call site. A tMap expression MyRoutines.maskEmail(cust.email).
Generated code. The job class calls the routine inline per row.

Question. Write a Talend routine that masks emails and show how it's invoked and roughly what the generated per-row code looks like.

Input.

Element	Value
Routine class	MyRoutines
Method	maskEmail(String email)
Call in tMap	enriched.email = MyRoutines.maskEmail(cust.email)

Code.

// Project routine — MyRoutines.java (reusable across all jobs in the project)
package routines;

public class MyRoutines {
    /**
     * {talendTypes} String
     * {Category} User Defined
     * {param} string("a@b.com") input: the email to mask
     * {example} maskEmail("alice@corp.com") -> "a***@corp.com"
     */
    public static String maskEmail(String email) {
        if (email == null || !email.contains("@")) return "unknown";
        int at = email.indexOf('@');
        String user = email.substring(0, at);
        String masked = user.charAt(0) + "***";
        return masked + email.substring(at);
    }
}

// Roughly what the studio generates inside the job's tMap block (per row):
if (cust_email != null) {
    enriched_email = routines.MyRoutines.maskEmail(cust_email);
} else {
    enriched_email = "unknown";
}
// ... this runs once per input row inside the generated while(rs.next()) loop.

Step-by-step explanation.

The routine lives in the project's routines package. The Javadoc {talendTypes} / {param} / {example} comments are what make it appear in the studio's expression auto-complete with documentation — a small but professional touch.
maskEmail is a plain static method: null-safe, returns "unknown" for malformed input, otherwise keeps the first character plus *** and the domain. It is ordinary Java you can unit-test outside Talend.
In tMap, the expression MyRoutines.maskEmail(cust.email) calls it. The studio resolves MyRoutines to routines.MyRoutines at generation time.
The generated code inlines that call inside the job's main row loop — a while over the input result set. There is no reflection or interpretation; it is a direct Java method call, which is why Talend jobs are fast per-row and debuggable as Java.
This is the concrete meaning of "Talend generates code": your visual job becomes a Java class, and your routines become normal method calls within it. Contrast this with Hop, where the same masking would be a JavaScript or UDJC transform interpreted by the engine at run time.

Output.

cust.email	enriched.email
alice@corp.com	a***@corp.com
bob@x.io	b***@x.io
(null)	unknown
not-an-email	unknown

Rule of thumb. Factor repeated expression logic into a routine so it is written once, unit-tested as plain Java, and reused across jobs. Reading the generated code is the fastest way to understand — and to explain in an interview — that Talend is a code generator, not an interpreter.

Common beginner mistakes.

Confusing the two reject concepts. A tMap output filter reject (business rule) is not the same as a component Reject row (runtime error). Interviewers ask this to test depth.
Loading a huge lookup with "Load once" into memory. For a multi-million-row lookup, the "Reload at each row" / "Store on disk" options or a tDBInput with a join pushed to the database avoid an out-of-memory crash.
Hard-coding environment values. Any literal host, path, or credential breaks context-based promotion.
Forgetting subjob ordering. Without OnSubjobOk triggers, two subjobs may run in an undefined order; sequencing is explicit, not implied by layout.

Talend interview question on the tMap and code generation

A senior interviewer might ask: "In Talend, you have a 5-million-row main flow that must be enriched from a 20-million-row reference table and split into matched, unmatched, and over-limit outputs. Design the tMap (join model, lookup strategy, outputs) and explain what the studio generates so it doesn't run out of memory."

Solution Using a database-side lookup, tMap multi-output filters, and reject handling

// tMap configuration (described as code)
// -------------------------------------
// Main input : row1(order_id, customer_id, amount_usd)   -- 5M rows, streamed
// Lookup     : cust(customer_id, name, country, fx_rate, credit_limit)  -- 20M rows
//
// Lookup model: "Reload at each row" is WRONG here (20M x 5M).
// Better: push the join to the DB OR use "Load once" with a filtered lookup query.

// tDBInput feeding the lookup uses a filtered/keyed query, not SELECT *:
//   SELECT customer_id, name, country, fx_rate, credit_limit
//   FROM   customers
//   WHERE  customer_id IN (SELECT DISTINCT customer_id FROM staging_orders)

// tMap Var (computed once per row):
//   Var.local = row1.amount_usd * (cust.fx_rate == null ? 0.0 : cust.fx_rate)

// Output "matched"    filter: cust.customer_id != null && Var.local <= cust.credit_limit
// Output "over_limit" filter: cust.customer_id != null && Var.local >  cust.credit_limit
// Output "unmatched"  filter: cust.customer_id == null

// matched.amount_local    = Var.local;
// over_limit.amount_local = Var.local;
// over_limit.reason       = "exceeds credit_limit";

Canvas
======
  tDBInput(staging_orders) --Main(row1)--> tMap --matched-----> tDBOutput(orders_ok)
                                             |--over_limit----> tDBOutput(orders_hold)
  tDBInput(customers, keyed) --Lookup(cust)--+--unmatched-----> tFileOutputDelimited(unmatched.csv)

  tDBOutput(orders_ok) --Reject--> tLogRow   (DB constraint failures)

Step-by-step trace.

Row	customer match?	Var.local	credit_limit	Output
order 1 (cust 7)	yes	78.50	1000	matched
order 2 (cust 9)	yes	5200.00	3000	over_limit
order 3 (cust 0)	no	0.00	—	unmatched
order 4 (cust 7)	yes	120.00	1000	matched

The 5M main flow is streamed row by row from the DB via a server-side tDBInput — it never fully materialises in memory.
The 20M lookup is not blindly loaded with "Reload at each row" (which would be 5M×20M lookups). Instead the lookup query is filtered to only the customers that appear in staging, shrinking it to fit "Load once" in memory as a hash table.
Var.local computes the converted amount once per row and is reused by both the matched and over_limit output expressions — avoiding recomputation.
Three output-table filters fan the single input into matched / over_limit / unmatched purely by boolean expression — the studio generates three if branches inside the row loop.
The tDBOutput Reject row catches database-level failures separately from the business-rule outputs, so a constraint violation is logged, not silently dropped.

Output:

Output table	Rows	Meaning
orders_ok	matched, within limit	load to warehouse
orders_hold	matched, over credit_limit	route to review
unmatched.csv	no customer found	data-quality queue
tLogRow	DB rejected	constraint failures

Why this works — concept by concept:

tMap multi-output filters — one component fans a single input into three destinations by per-output boolean expressions; the studio compiles each into an if branch, so there is no extra pass over the data.
Filtered lookup query — pushing a WHERE customer_id IN (...) into the lookup's tDBInput shrinks a 20M reference table to only the keys in play, letting "Load once" fit in memory instead of triggering an OOM or a per-row reload.
tMap Var reuse — computing Var.local once and referencing it from two outputs avoids duplicate multiplication and keeps the expressions consistent between matched and over_limit rows.
Two reject planes — tMap output filters handle business routing (unmatched, over-limit); the component Reject row handles runtime errors (DB constraints). Separating them prevents a data-quality issue from masking an infrastructure error.
Cost — one streamed pass over 5M main rows plus one in-memory hash lookup over the filtered reference set; roughly O(main + filtered_lookup) time and O(filtered_lookup) memory, versus the O(main × lookup) blow-up of a naive per-row reload.

Data Transformation
Topic — data-transformation
Data-transformation problems on joins and mapping

Practice →

ETL Topic — etl ETL problems on enrichment and reject handling

Practice →

3. Apache Hop — pipelines, workflows, and the metadata-driven engine

Apache Hop splits work into `hop pipelines` (data flow) and `workflows` (orchestration) and interprets metadata at run time — no code generation, Kettle DNA modernised

The mental model in one line: Apache Hop models ETL as two artifact types — a pipeline (.hpl, a graph of transforms through which rows stream in parallel, one thread per transform) and a workflow (.hwf, a sequence of actions connected by conditional hops for orchestration) — where everything from database connections to run configurations is metadata the engine reads and executes at run time (there is no generated Java), the design is deliberately decoupled from the runtime so the same pipeline runs on the native Hop engine or on Apache Beam over Spark/Flink/Dataflow, and the whole thing is the Apache-governed modern successor to kettle pentaho. Where Talend compiles, Hop interprets — that inversion drives every other difference.

Pipelines — the data-flow artifact.

Transforms. The processing boxes (formerly Kettle "steps"): CSV file input, Table input, Table output, Filter rows, Sort rows, Group by, Stream lookup, Merge join, Join rows, Select values, Value mapper, Calculator, JavaScript, User Defined Java Class, Dummy.
Parallel streaming. Each transform runs on its own thread; rows flow through hops as they're produced — a pipeline is not sequential, it is a dataflow where all transforms are live at once.
Hops. The arrows between transforms carry the row stream. A transform with multiple outgoing hops can copy rows to all, or distribute (round-robin) across them.
Row metadata. The schema (field names + types) propagates and is editable at any transform via Select values (rename, reorder, retype, remove).

Workflows — the orchestration artifact.

Actions. The orchestration boxes (formerly Kettle "job entries"): Start, Pipeline (run a pipeline), Workflow (run a child workflow), SQL, Shell, Mail, Evaluate, Success, Abort, Wait for file.
Conditional hops. Green (on success), red (on failure), and unconditional hops sequence actions — this is control flow, not data flow.
Looping. Actions can iterate (e.g. run a pipeline once per row of a result set) via the "Execute for every input row" option.

The metadata-driven engine.

Metadata objects. Relational database connections, pipeline run configurations, workflow run configurations, servers, and partition schemas are all first-class metadata, stored separately from the pipelines that use them.
Projects and environments. A project holds pipelines/workflows/metadata; an environment supplies variable values (Dev/Test/Prod). Switching environment repoints every variable — Hop's answer to Talend context groups.
No code generation. The engine reads the .hpl XML and executes each transform's plugin directly. Nothing is compiled ahead of time; the pipeline is the runnable artifact.
Metadata injection. A template pipeline whose transform settings are injected from another pipeline's data at run time — build one generic loader, drive it with a table of table-names. This is Hop's superpower for "load 300 tables the same way."

Runtimes — native and Beam.

Native Hop engine. The default single-JVM row engine, descended from Kettle — fast, simple, great for gigabytes-per-night batch.
Apache Beam runners. The same pipeline can run on Beam via Spark, Flink, or Google Cloud Dataflow run configurations — horizontal scale without redesigning the flow.
Hop Server + hop-run. hop-run is the CLI executor; Hop Server (the successor to Kettle's Carte) hosts pipelines/workflows for remote execution and scheduling.

Worked example — a Hop pipeline with Table input, Filter, Stream lookup, and Table output

Detailed explanation. The canonical Hop pipeline: read orders from a table, filter out cancelled ones, enrich with a customer lookup via Stream lookup, compute a converted amount with Calculator, and load to a target table. This exercises transforms, hops, parallel streaming, and row-metadata editing — the Hop equivalent of the Talend tMap job from section 2.

Table input. SELECT order_id, customer_id, amount_usd, status FROM orders.
Filter rows. Keep status <> 'cancelled'; send the rest to a Dummy (or reject) transform.
Stream lookup. Enrich with customers (customer_id → name, country, fx_rate).
Calculator / Select values. amount_local = amount_usd * fx_rate; trim the schema.

Question. Describe the transforms, hops, and the key settings for this Hop pipeline.

Input.

Transform	Type	Key setting
read_orders	Table input	SELECT ... FROM orders
drop_cancelled	Filter rows	condition status <> 'cancelled'
enrich_customer	Stream lookup	key customer_id → name,country,fx_rate
convert_fx	Calculator	amount_local = amount_usd * fx_rate
load_target	Table output	orders_enriched

Code.

<!-- Fragment of the pipeline .hpl (Hop stores everything as metadata XML) -->
<transform>
  <name>drop_cancelled</name>
  <type>FilterRows</type>
  <send_true_to>enrich_customer</send_true_to>
  <send_false_to>cancelled_dummy</send_false_to>
  <condition>
    <leftvalue>status</leftvalue>
    <function>&lt;&gt;</function>
    <constant>cancelled</constant>
  </condition>
</transform>

<transform>
  <name>convert_fx</name>
  <type>Calculator</type>
  <calculation>
    <field_name>amount_local</field_name>
    <calc_type>MULTIPLY</calc_type>
    <field_a>amount_usd</field_a>
    <field_b>fx_rate</field_b>
    <value_type>Number</value_type>
  </calculation>
</transform>

-- The Table input transform's query (parameterised by an environment variable)
SELECT order_id, customer_id, amount_usd, status
FROM   ${SRC_SCHEMA}.orders
WHERE  updated_at >= '${RUN_DATE}';

Step-by-step explanation.

Table input runs the parameterised SQL and emits rows into the pipeline. ${SRC_SCHEMA} and ${RUN_DATE} are resolved from the active environment/variables at run time — no recompile, just a different variable set.
Filter rows evaluates status <> 'cancelled' per row and routes true-rows to enrich_customer and false-rows to a cancelled_dummy (or a reject/log transform). Unlike Talend's tMap filters, filtering is its own transform.
Stream lookup holds the customers result set in a hash map keyed on customer_id and appends name, country, fx_rate to each order row — the Hop analogue of a tMap lookup join. (Merge join is the sorted-input alternative for very large lookups.)
Calculator computes amount_local = amount_usd * fx_rate as a new field with an explicit output type. Select values (not shown) would then trim/rename fields to match the target schema.
Table output writes to orders_enriched. Because every transform runs on its own thread, read_orders is still emitting rows while load_target is already writing — parallel streaming, the defining behaviour of the Hop engine.

Output.

order_id	name	country	amount_usd	amount_local	status
1001	Acme Ltd	GB	100.00	78.50	open
1002	Nord AB	SE	50.00	520.00	shipped
1009	(dropped)	—	12.00	—	cancelled

Rule of thumb. In Hop, decompose what Talend does in one tMap into explicit transforms — Filter rows for gating, Stream lookup/Merge join for enrichment, Calculator for math, Select values for schema shaping. The pipeline reads like a data-flow diagram because that is exactly what the engine executes.

Worked example — a workflow that orchestrates the pipeline with success/failure branches

Detailed explanation. A pipeline transforms rows; a workflow decides when and whether pipelines run and what happens on failure. The canonical workflow: create the target table if needed, run the enrichment pipeline, and branch — on success send a completion mail, on failure abort and alert.

Start. The entry action.
SQL. CREATE TABLE IF NOT EXISTS ... for the target.
Pipeline. Run the enrichment .hpl.
Branches. Green hop → Mail (success); red hop → Abort/alert (failure).

Question. Lay out the workflow actions and the conditional hops.

Input.

Action	Type	On success →	On failure →
start	Start	prep_table	—
prep_table	SQL	run_enrich	fail_alert
run_enrich	Pipeline	ok_mail	fail_alert
ok_mail	Mail	success	—
fail_alert	Mail + Abort	—	—

Code.

<!-- Fragment of the workflow .hwf: conditional hops carry control, not rows -->
<hop>
  <from>run_enrich</from>
  <to>ok_mail</to>
  <evaluation>Y</evaluation>   <!-- green: on success -->
  <unconditional>N</unconditional>
</hop>
<hop>
  <from>run_enrich</from>
  <to>fail_alert</to>
  <evaluation>N</evaluation>   <!-- red: on failure -->
  <unconditional>N</unconditional>
</hop>

# Execute the workflow from the CLI with an environment (no GUI, CI-friendly)
hop-run \
  --project=sales-etl \
  --environment=Prod \
  --runconfig=local \
  --file='workflows/nightly_enrich.hwf' \
  --parameters=RUN_DATE=2026-08-03

Step-by-step explanation.

Start kicks the workflow. Actions run sequentially (control flow), unlike a pipeline's parallel transforms — this is the key mental split between the two artifacts.
prep_table (an SQL action) idempotently ensures the target exists. Its green hop leads to run_enrich; its red hop leads to fail_alert, so a failed DDL never proceeds to load.
run_enrich (a Pipeline action) executes the .hpl from section 3's first example. The workflow evaluates its exit status and picks the green or red hop accordingly.
On success, the green hop fires ok_mail; on failure, the red hop fires fail_alert, which mails on-call and Aborts the workflow with a non-zero exit code — so a scheduler (cron, Airflow, Hop Server) sees the failure.
hop-run executes the whole thing headless with --environment=Prod supplying variable values and --parameters passing the run date. The same workflow runs from the GUI during development and from hop-run in production — design/runtime decoupling in action.

Output.

Outcome	Path taken	Exit code
All good	start → prep_table → run_enrich → ok_mail → success	0
DDL fails	start → prep_table → fail_alert (abort)	non-zero
Pipeline fails	start → prep_table → run_enrich → fail_alert (abort)	non-zero

Rule of thumb. Keep row-level work in pipelines and control-flow (DDL, sequencing, notifications, retries) in workflows. Wire explicit green/red hops so failure never silently proceeds, and drive production runs with hop-run --environment so the same artifact is portable from laptop to server.

Worked example — metadata injection to load many tables with one template

Detailed explanation. Hop's metadata injection is the answer to "I have 200 source tables to load the same way." You build one template pipeline with placeholder transform settings, and a driver pipeline injects the real settings (table name, columns, key) per iteration — one pipeline maintains 200 loads.

Template. A generic Table input → Table output pipeline whose SQL and target table are injectable.
Driver. Reads a control table of (source_table, target_table, key_col) and injects each row into the template via the Metadata Injection transform.
Result. Adding a 201st table is a new control-table row, not a new pipeline.

Question. Sketch the template, the control table, and the injection driver.

Input.

Control-table row	source_table	target_table	load_mode
1	crm.customers	wh.dim_customer	full
2	crm.products	wh.dim_product	full
3	ops.orders	wh.fact_orders	incremental

Code.

Template pipeline: load_template.hpl
  [Table input: SELECT * FROM ${SRC}]  -->  [Table output: ${TGT}]
      ^ query is an injection target        ^ target table is an injection target

Driver pipeline: load_all.hpl
  [Table input: SELECT source_table, target_table, load_mode FROM etl.load_control]
      --> [Metadata Injection: template = load_template.hpl]
             inject:  ${SRC} <- source_table
                      ${TGT} <- target_table
             run the injected pipeline for each control row

-- The control table that drives every load (add a row = add a table)
CREATE TABLE etl.load_control (
    id            SERIAL PRIMARY KEY,
    source_table  TEXT NOT NULL,
    target_table  TEXT NOT NULL,
    load_mode     TEXT NOT NULL DEFAULT 'full'
);
INSERT INTO etl.load_control(source_table, target_table, load_mode) VALUES
  ('crm.customers', 'wh.dim_customer', 'full'),
  ('crm.products',  'wh.dim_product',  'full'),
  ('ops.orders',    'wh.fact_orders',  'incremental');

Step-by-step explanation.

The template pipeline is generic: its Table input query and Table output target are left as injection targets rather than fixed values. On its own it does nothing useful — it is a shape waiting for parameters.
The driver pipeline reads etl.load_control and, for each row, uses the Metadata Injection transform to push source_table and target_table into the template's injectable fields, then executes the now-concrete pipeline.
Because the mapping is data-driven, adding a table is an INSERT into load_control, not a new pipeline — the maintenance surface stays at one template regardless of table count.
This is impossible to express cleanly in Talend without generating jobs; Hop's metadata-first design makes injecting settings a native transform. It is the standard Hop interview flex for "how do you avoid 200 near-identical pipelines?"
The load_mode column can drive a branch inside the template (full truncate-load vs incremental updated_at > filter), keeping even the two load strategies in one maintained artifact.

Output.

Iteration	Injected SRC	Injected TGT	Effect
1	crm.customers	wh.dim_customer	full load
2	crm.products	wh.dim_product	full load
3	ops.orders	wh.fact_orders	incremental load
+1 (future)	new control row	new target	zero new pipelines

Rule of thumb. When you catch yourself copy-pasting a pipeline for the Nth table, stop and reach for metadata injection: one template + a control table scales to hundreds of loads with a single artifact to maintain. This is the metadata-driven engine's biggest payoff over a code-generation studio.

Common beginner mistakes.

Thinking a pipeline runs sequentially. All transforms run in parallel on their own threads; row order across transforms is not guaranteed unless you sort.
Putting orchestration in a pipeline. DDL, notifications, and sequencing belong in a workflow; a pipeline is for row streaming only.
Ignoring environments. Hard-coding schema names instead of ${SRC_SCHEMA} breaks Dev/Test/Prod portability — the Hop analogue of hard-coding a Talend host.
Choosing Stream lookup for a giant reference set. Stream lookup loads the lookup into memory; for very large or unsorted references, Merge join (sorted) or a database-side join scales better.

Apache Hop interview question on pipelines vs workflows and the engine model

A senior interviewer might ask: "In Apache Hop, explain the difference between a pipeline and a workflow, why Hop has no code-generation step, and how you'd run the same pipeline first on the native engine and later on Spark when volume grows — without redesigning the flow."

Solution Using pipeline/workflow separation and swappable Beam run configurations

Design: one pipeline, two run configurations
============================================

pipeline: enrich_orders.hpl   (Table input -> Filter -> Stream lookup -> Table output)
workflow: nightly.hwf         (Start -> SQL prep -> Pipeline(enrich_orders) -> Mail)

Run configuration A (dev / small):   "local"      -> native Hop engine, single JVM
Run configuration B (prod / large):  "beam-spark" -> Apache Beam on a Spark cluster

The pipeline XML does NOT change between A and B. Only the run configuration
selected at execution time changes.

# Small volume: native engine
hop-run --project=sales-etl --environment=Prod \
        --runconfig=local \
        --file='workflows/nightly.hwf' --parameters=RUN_DATE=2026-08-03

# Large volume later: same files, Beam-on-Spark run configuration
hop-run --project=sales-etl --environment=Prod \
        --runconfig=beam-spark \
        --file='workflows/nightly.hwf' --parameters=RUN_DATE=2026-08-03

<!-- Metadata: a Beam-Spark pipeline run configuration (stored, not generated) -->
<pipeline-run-configuration>
  <name>beam-spark</name>
  <engine-plugin-id>BeamSpark</engine-plugin-id>
  <configuration>
    <spark_master>yarn</spark_master>
    <streaming>false</streaming>
  </configuration>
</pipeline-run-configuration>

Step-by-step trace.

Concern	Answer	Why
pipeline vs workflow	pipeline = parallel row flow; workflow = sequential control flow	two artifact types, two engines
code generation	none; engine interprets metadata	Hop reads `.hpl`/`.hwf` and runs plugins
scale-out	swap run configuration to Beam-Spark	design decoupled from runtime
pipeline changes	zero	only the run-config metadata differs
orchestration	workflow calls the pipeline	DDL + mail stay out of the data flow

After deployment, the identical enrich_orders.hpl runs on the native single-JVM engine while volumes are modest, and — when nightly volume outgrows one machine — the operator changes --runconfig to beam-spark and the same pipeline executes distributed on a Spark cluster. No transform is edited; the design/runtime decoupling does the work.

Output:

Run config	Engine	Good for	Change to pipeline
local	native Hop, 1 JVM	up to ~tens of GB/night	none
beam-spark	Beam on Spark	TB-scale, cluster	none
beam-flink	Beam on Flink	streaming	none
beam-dataflow	Beam on Dataflow	managed GCP	none

Why this works — concept by concept:

Pipeline vs workflow — a pipeline is a parallel dataflow (all transforms live at once, one thread each); a workflow is a sequential control-flow graph with green/red hops. Separating them keeps row processing and orchestration in the right engine.
Metadata interpretation (no codegen) — Hop executes each transform's plugin against the .hpl metadata at run time, so there is no compile step; the pipeline file is the runnable artifact, which is what makes swapping runtimes trivial.
Design/runtime decoupling — the run configuration is separate metadata from the pipeline, so pointing the same flow at native, Spark, Flink, or Dataflow is a configuration change, not a redesign.
Apache Beam runners — Beam abstracts the execution engine; a Hop pipeline lowered to Beam runs wherever Beam runs, giving horizontal scale without leaving the visual model.
Cost — near-zero incremental cost to scale: no rewrite, no new artifact, just a stored run-configuration and a cluster to run it on. The trade is a heavier runtime (Spark) and its operational overhead — paid only when volume justifies it.

ETL
Topic — etl
ETL problems on pipeline and workflow design

Practice →

Design Topic — design Design problems on metadata-driven systems

Practice →

4. Building a pipeline — a worked ETL job in both studios

The same extract → transform → load job, built once in Talend and once in Hop, shows the two execution models side by side

The mental model in one line: the fastest way to internalise the difference between the two etl studio families is to build the same job in both — extract from a source, transform (join a lookup, derive a column, filter), load to a target, and orchestrate it — because the data-flow shape is nearly identical (a graph of boxes and arrows) while the mechanics diverge sharply: Talend fuses join+expression+routing into one tmap and generates Java, whereas Hop uses several explicit transforms and interprets metadata, and the orchestration layer is a Talend parent job with tRunJob versus a Hop workflow calling a pipeline. Seeing both makes the trade-offs concrete instead of abstract.

The shared job specification.

Extract. Read sales.csv (sale_id, product_id, qty, unit_price) and a products reference table (product_id, name, category, active).
Transform. Join sales to products on product_id; derive revenue = qty * unit_price; keep only active = true products; drop products with category = 'internal'.
Load. Write enriched sales to wh.fact_sales; route inactive/internal rows to a rejects file.
Orchestrate. Ensure the target exists, run the load, notify on completion or failure.

How Talend expresses it.

Extract. tFileInputDelimited (sales) as Main; tDBInput (products) as the tMap Lookup.
Transform. One tMap: inner-join model on product_id, expression revenue = row1.qty * row1.unit_price, output filters products.active == true && !"internal".equals(products.category) for enriched vs the negation for rejects.
Load. tDBOutput (fact_sales) + tFileOutputDelimited (rejects).
Orchestrate. A parent job: tDBRow (DDL) → OnSubjobOk → tRunJob (the load child) → OnSubjobOk → tSendMail.

How Hop expresses it.

Extract. CSV file input (sales) + Table input (products).
Transform. Stream lookup (join products), Calculator (revenue), Filter rows (active + not internal → true to load, false to rejects), Select values (shape schema).
Load. Table output (fact_sales) + Text file output (rejects).
Orchestrate. A workflow: SQL action (DDL) → green → Pipeline action (the load) → green → Mail; red hops → Mail + Abort.

What the comparison reveals.

Density vs explicitness. Talend packs join+derive+route into one dense tMap; Hop spreads it across four legible transforms. Neither is "better" — dense is faster to build, explicit is easier to review and diff.
Codegen vs interpret. The Talend deliverable is compiled Java (runs on any JVM, integrates Java libs); the Hop deliverable is metadata the engine runs (swappable to Beam, no compile).
Orchestration. Both cleanly separate "run this data flow" from "sequence and notify" — tRunJob+triggers in Talend, workflow+hops in Hop.

Worked example — the transform step in both tools, expression for expression

Detailed explanation. The transform is where the two studios look most different. Put the Talend tMap expressions next to the Hop transforms that compute the identical result so the mapping is unambiguous.

Derive revenue. Talend: one tMap output expression. Hop: a Calculator field.
Join products. Talend: tMap lookup. Hop: Stream lookup.
Filter active/internal. Talend: tMap output-table filter. Hop: Filter rows.

Question. Show the revenue derivation, the join, and the filter in both tools side by side.

Input.

Logic	Talend (tMap)	Apache Hop
join	Lookup on product_id	Stream lookup, key product_id
revenue	expression	Calculator field
active filter	output-table filter	Filter rows condition

Code.

// TALEND — inside tMap
// Lookup: prod, joined on row1.product_id = prod.product_id (Inner, Unique)
enriched.sale_id    = row1.sale_id;
enriched.product    = prod.name;
enriched.category   = prod.category;
enriched.revenue    = row1.qty * row1.unit_price;             // derive
// Output "enriched" filter:
//   prod.active == true && !"internal".equals(prod.category)
// Output "rejects" filter:
//   prod.active == false || "internal".equals(prod.category) || prod.product_id == null

<!-- APACHE HOP — three transforms doing the same work -->
<!-- 1) Stream lookup: key product_id -> name, category, active -->
<transform><name>lookup_product</name><type>StreamLookup</type>
  <lookup><keyfield><name>product_id</name></keyfield></lookup>
</transform>

<!-- 2) Calculator: revenue = qty * unit_price -->
<transform><name>calc_revenue</name><type>Calculator</type>
  <calculation><field_name>revenue</field_name><calc_type>MULTIPLY</calc_type>
    <field_a>qty</field_a><field_b>unit_price</field_b><value_type>Number</value_type>
  </calculation>
</transform>

<!-- 3) Filter rows: active AND not internal -->
<transform><name>keep_active</name><type>FilterRows</type>
  <send_true_to>load_target</send_true_to>
  <send_false_to>rejects_out</send_false_to>
  <compare><condition>
    <leftvalue>active</leftvalue><function>=</function><constant>Y</constant>
    <operator>AND</operator>
    <subcondition><leftvalue>category</leftvalue><function>&lt;&gt;</function><constant>internal</constant></subcondition>
  </condition></compare>
</transform>

Step-by-step explanation.

The join is a single tMap lookup line in Talend versus a dedicated Stream lookup transform in Hop — same hash-join semantics, different granularity of the building block.
The derivation revenue = qty * unit_price is one tMap output expression in Talend (arbitrary Java) versus a Calculator MULTIPLY field in Hop (a typed calculation from a fixed catalogue; use JavaScript/UDJC for anything the Calculator can't express).
The filter is a tMap output-table boolean in Talend (fusing routing into the mapper) versus an explicit Filter rows transform in Hop with send_true_to / send_false_to hops — Hop makes the fan-out a visible edge in the graph.
Talend's version is denser: one component holds join + derive + route. Hop's version is more explicit: four transforms, each doing one thing, which diffs and reviews better.
Both compute byte-identical output for the same input — the difference is purely how the logic is packaged and executed (compiled Java expression vs interpreted transform plugin).

Output.

sale_id	product	category	revenue	routed to
5001	Widget	retail	240.00	fact_sales
5002	Gadget	internal	90.00	rejects
5003	Bolt (inactive)	hardware	15.00	rejects
5004	Widget	retail	480.00	fact_sales

Rule of thumb. Map Talend tMap logic to Hop as: lookup → Stream lookup/Merge join, expression → Calculator/JavaScript/UDJC, output filter → Filter rows. When translating between the tools in an interview, decompose the one dense tMap into the several explicit Hop transforms and you will never be caught out.

Worked example — orchestration in both tools with failure handling

Detailed explanation. A production load is never just the data flow — it is "make sure the target exists, run the load, and do the right thing on failure." Show the Talend parent-job orchestration next to the Hop workflow.

DDL. Ensure wh.fact_sales exists.
Run. Execute the load flow.
Notify. Mail on success; alert + fail on error.

Question. Express the orchestration and failure handling in both tools.

Input.

Stage	Talend	Apache Hop
DDL	tDBRow (CREATE IF NOT EXISTS)	SQL action
run load	tRunJob (child)	Pipeline action
success	tSendMail via OnSubjobOk	Mail via green hop
failure	tSendMail via OnComponentError	Mail + Abort via red hop

Code.

TALEND parent job (triggers = orange control links)
===================================================
  tDBRow(DDL) --OnSubjobOk--> tRunJob(load_sales) --OnSubjobOk--> tSendMail(success)
       |                            |
       +--OnComponentError-->  tSendMail(alert) --OnComponentOk--> tDie("load failed")

# HOP workflow: nightly_sales.hwf, run headless in CI/prod
hop-run --project=sales-etl --environment=Prod --runconfig=local \
        --file='workflows/nightly_sales.hwf' --parameters=RUN_DATE=2026-08-03
# Inside the workflow:
#   Start -> SQL(create fact_sales) --green--> Pipeline(load_sales) --green--> Mail(success)
#                     \--red--> Mail(alert) -> Abort
#                                    Pipeline --red--> Mail(alert) -> Abort

Step-by-step explanation.

DDL first, idempotently. Talend tDBRow runs CREATE TABLE IF NOT EXISTS; Hop uses an SQL action. Both gate the load behind a successful DDL via their success links.
Run the child flow. Talend calls the load with tRunJob (a child job, its own context); Hop calls the load with a Pipeline action. Both keep the data flow in a separate, reusable artifact.
Success path. Talend fires tSendMail on OnSubjobOk; Hop fires Mail on the green hop. Only a fully successful load notifies success.
Failure path. Talend routes OnComponentError to an alert mail then tDie (non-zero exit); Hop routes the red hop to Mail then Abort (non-zero exit). A scheduler sees the failure in both.
The structures are isomorphic: trigger-based sequencing in Talend maps to conditional-hop sequencing in Hop. Once you see this, orchestration in either tool is the same mental model with different labels.

Output.

Scenario	Talend path	Hop path	Exit
success	DDL → load → success mail	SQL → pipeline → success mail	0
DDL fails	DDL → alert → tDie	SQL → alert → Abort	non-zero
load fails	DDL → load → alert → tDie	SQL → pipeline → alert → Abort	non-zero

Rule of thumb. Orchestrate identically in both studios: idempotent DDL, then run the data flow as a separate child artifact, then branch on success/failure with explicit links. tRunJob+triggers (Talend) and workflow+hops (Hop) are the same pattern; keep row work out of the orchestration layer in both.

Common beginner mistakes.

Putting the whole job in one giant flow. Separate the data flow (job data portion / pipeline) from orchestration (parent job / workflow) in both tools; monolith flows are unschedulable and unreusable.
Assuming byte-identical output guarantees identical performance. Talend's compiled expression and Hop's interpreted transform can differ in throughput; benchmark on real volume.
Skipping the reject path. Both examples route failed rows somewhere — dropping them silently is the classic data-quality bug.
Forgetting non-zero exit codes on failure. tDie/Abort are what make a scheduler notice a failed run; without them a red path can still exit 0.

ETL interview question on porting a job between studios

A senior interviewer might ask: "You inherit a Talend job that reads a file, enriches it via a tMap lookup with a filter and a reject output, and loads a table. Reproduce it in Apache Hop transform-for-transform, and explain any behaviour that does not port cleanly."

Solution Using a transform-for-transform Talend→Hop port

Talend job                          Apache Hop pipeline
==========                          ===================
tFileInputDelimited(sales.csv)  ->  CSV file input(sales.csv)
tDBInput(products)  [tMap Lookup] ->  Table input(products)  -> Stream lookup(key product_id)
tMap
  - join products                 ->  Stream lookup (above)
  - revenue = qty*unit_price      ->  Calculator (MULTIPLY qty,unit_price -> revenue)
  - output filter active&!internal->  Filter rows (active=Y AND category<>internal)
  - reject output                 ->  Filter rows send_false_to -> Text file output(rejects)
tDBOutput(fact_sales)           ->  Table output(fact_sales)
tDBOutput Reject -> tLogRow     ->  Table output "Error handling" hop -> Text file output(db_rejects)

<!-- The one thing that does not port 1:1: tMap's in-mapper Var + null-safe Java -->
<!-- Talend:  Var.rev = row1.qty * row1.unit_price;  enriched.rev = Relational.ISNULL(...) ? 0 : Var.rev -->
<!-- Hop:     Calculator cannot do arbitrary null-coalescing + branching in one field, -->
<!--          so use a JavaScript / UDJC transform for the null-safe branch:            -->
<transform><name>null_safe_rev</name><type>ScriptValueMod</type>  <!-- JavaScript transform -->
  <jsScripts><jsScript><jsScript_script>
    var rev = (qty == null || unit_price == null) ? 0 : qty * unit_price;
  </jsScript_script></jsScript></jsScripts>
</transform>

Step-by-step trace.

Talend element	Hop equivalent	Ports cleanly?
tFileInputDelimited	CSV file input	yes
tMap lookup	Stream lookup	yes
tMap expression (simple)	Calculator	yes
tMap expression (null-safe branch)	JavaScript / UDJC	needs a code transform
tMap output filter	Filter rows	yes
tMap reject output	Filter rows false branch	yes
tDBOutput Reject row	Table output error hop	yes (different UI)

File and DB reads port one-to-one: tFileInputDelimited → CSV file input, tDBInput → Table input.
The tMap lookup becomes a Stream lookup keyed on the same field — identical hash-join semantics.
Simple expressions (qty * unit_price) become a Calculator field; complex/null-safe expressions with branching do not fit the Calculator's fixed catalogue and must become a JavaScript/UDJC transform.
The output filters become Filter rows transforms whose true/false hops replace tMap's per-output filters; the reject output is just the false branch.
The component Reject row (DB errors) maps to Hop's Table output error-handling hop — same concept (runtime-error stream), different place in the UI.

Output:

Aspect	Talend original	Hop port
Extract	file + DB input	CSV + Table input
Enrich	tMap lookup	Stream lookup
Derive	tMap expression	Calculator (+ JS for null-safe)
Route	tMap filters	Filter rows
Load	tDBOutput + Reject	Table output + error hop
Non-porting bit	in-mapper Java branching	needs JavaScript/UDJC

Why this works — concept by concept:

One dense component → several explicit transforms — Talend's tMap fuses join, derive, and route; the faithful port spreads those responsibilities across Stream lookup, Calculator, and Filter rows, which is exactly Hop's design intent.
Calculator vs arbitrary Java — Hop's Calculator covers a fixed set of typed operations; anything with branching or null-coalescing that Talend did in inline Java must move to a JavaScript/UDJC transform. This is the single most common non-clean port.
Filter rows as the routing primitive — tMap output filters become Filter rows true/false hops, turning implicit in-mapper routing into visible edges in the Hop graph.
Error hops mirror Reject rows — both tools separate runtime-error streams from business-rule routing; the port preserves that separation.
Cost — a transform-for-transform port is mostly mechanical (O(components)); the only real engineering is re-expressing inline-Java expressions as Hop code transforms, so budget effort proportional to the number of non-trivial tMap expressions, not the number of components.

Data Processing
Topic — database
Database problems on extract-transform-load

Practice →

ETL Topic — etl ETL problems on end-to-end job building

Practice →

5. Talend vs Hop vs code-first + interview questions and signals

Picking between `talend`, Apache Hop, and a code-first stack is a four-axis decision — and knowing the axes is what the `talend interview questions` really test

The mental model in one line: choosing an ETL approach is a picking exercise across four axes — team skill mix (non-coders vs all-engineers), governance and lineage requirements, connector breadth, and scale/runtime — where talend wins on enterprise connector depth and its compiled-Java + commercial-support story, Apache Hop wins on being a live Apache-governed open source etl studio with a metadata-driven engine and Beam scale-out, and a code-first stack (dbt/Spark/Airflow) wins on version control, testing, and all-engineer velocity — and the senior interview signal is naming the axes and the trade-offs rather than defending one tool as universally best. Every "which would you use" question is really "do you know the four axes."

The four axes, tool by tool.

Team skills. Talend and Hop both serve mixed analyst+engineer teams (visual canvas). Code-first assumes fluency in SQL/Python/Spark. If half the maintainers can't code, a studio wins by default.
Governance / lineage. Both studios give visible graph lineage; Talend's paid tiers add data-quality and stewardship tooling. Code-first needs external lineage (dbt docs, OpenLineage) bolted on.
Connectors. Talend has the deepest enterprise catalogue (SAP, mainframe, hundreds of SaaS). Hop covers the common estate well. Code-first writes/imports connectors as libraries.
Scale / runtime. Code-first (Spark) and Hop-on-Beam scale horizontally; classic Talend/native-Hop are single-JVM unless you use their distributed options. Talend Cloud/Spark jobs also scale out.

Where each tool is the right default.

Talend. Large enterprise already licensed; deep/exotic connector needs; requirement for vendor support and integrated data quality; teams comfortable with the Java-generation model.
Apache Hop. New open-source projects; Kettle/PDI shops modernising; mixed teams wanting a live, free, Apache-governed studio; need for metadata injection and optional Beam scale-out.
Code-first (dbt/Spark/Airflow). All-engineer teams; ELT-in-warehouse patterns; strong CI/testing/version-control culture; extreme scale where a studio's model adds no value.

The honest weaknesses to name unprompted.

Talend. The free Open Studio line ended in 2024, so "open-source Talend" is no longer a live recommendation; generated Java diffs poorly; licensing cost for the platform tiers.
Apache Hop. Smaller ecosystem/community than Spark or dbt; XML diffs are better than Talend's but still not code; fewer managed-cloud offerings.
Code-first. Excludes non-coders; you build connectors and lineage yourself; more upfront engineering before the first pipeline ships.

What senior interviewers listen for.

Do you name all four axes (skills, governance, connectors, scale) unprompted? — senior signal.
Do you know Talend generates Java, Hop interprets metadata, code-first is native code? — required answer.
Do you mention that Talend Open Studio's free line ended in 2024 and Apache Hop is the live open-source studio? — senior signal.
Do you describe the choice as "match the tool to team + governance + connectors + scale" rather than "X is best"? — required answer.
Do you cite Hop-on-Beam or Talend Spark when scale is the objection to a studio? — senior signal.

Worked example — the four-axis scoring table

Detailed explanation. The most useful interview artifact for this section is a scoring table across the three approaches on the four axes. Having it memorised lets you answer any "which would you pick for X" by reading down the relevant column.

Score. High / Medium / Low fit per axis.
Read it. For a given scenario, weight the axes and pick the highest total.

Question. Build the four-axis scoring table across Talend, Apache Hop, and code-first.

Input.

Axis	Talend	Apache Hop	Code-first
Non-coder team fit	High	High	Low
Governance / visible lineage	High	High	Medium
Connector breadth	High	Medium	Medium
Scale / runtime	Medium (High on Cloud/Spark)	Medium (High on Beam)	High

Code.

# Weighted picker — turns axis weights + fit scores into a recommendation
FIT = {"High": 3, "Medium": 2, "Low": 1}
TOOLS = {
    "Talend":     {"skills": "High",   "governance": "High",   "connectors": "High",   "scale": "Medium"},
    "Apache Hop": {"skills": "High",   "governance": "High",   "connectors": "Medium", "scale": "Medium"},
    "Code-first": {"skills": "Low",    "governance": "Medium", "connectors": "Medium", "scale": "High"},
}

def recommend(weights: dict[str, float]) -> str:
    scores = {t: sum(FIT[axis[k]] * weights[k] for k in weights) for t, axis in TOOLS.items()}
    return max(scores, key=scores.get)

# Mixed team, heavy governance, many connectors, modest scale:
print(recommend({"skills": 2, "governance": 2, "connectors": 2, "scale": 1}))  # -> Talend
# All-engineer team, extreme scale, light governance:
print(recommend({"skills": 0.5, "governance": 1, "connectors": 1, "scale": 3}))  # -> Code-first
# Kettle shop modernising, mixed team, open-source mandate:
print(recommend({"skills": 2, "governance": 2, "connectors": 1.5, "scale": 1.5}))  # -> Apache Hop (tie-break: open-source + Kettle lineage)

Step-by-step explanation.

Each tool gets a fit score per axis (High/Medium/Low). Talend and Hop tie on skills and governance; Talend edges connectors; code-first leads scale.
A scenario supplies weights per axis — how much each matters for this team. The recommendation is the weighted-sum maximum.
Heavy governance + connectors + mixed team favours Talend (its connector depth breaks the tie with Hop).
Extreme scale + all-engineers favours code-first — the studio's accessibility advantage is weighted near zero and scale dominates.
Kettle-modernisation + open-source mandate favours Apache Hop; when Talend and Hop tie numerically, the open-source-license and shared-Kettle-lineage considerations break it toward Hop.

Output.

Scenario	Weights (skills/gov/conn/scale)	Pick
Enterprise, exotic sources	2 / 2 / 2 / 1	Talend
All-engineer, TB-scale	0.5 / 1 / 1 / 3	Code-first
Kettle shop, OSS mandate	2 / 2 / 1.5 / 1.5	Apache Hop
Startup, mixed team, cost-sensitive	2 / 1 / 1 / 1	Apache Hop (free)

Rule of thumb. Do not memorise a verdict; memorise the four axes and score them for the scenario in front of you. The interviewer is testing whether you weight their constraints correctly, not whether you like a particular tool.

Worked example — the rapid-fire signal answers

Detailed explanation. Visual-ETL interviews often finish with rapid-fire questions probing depth. Prepare crisp one-liners so you never stall on the fundamentals.

Codegen vs interpret. Talend compiles to Java; Hop interprets metadata.
Two artifacts. Pipeline/transformation = data flow; workflow/job = orchestration.
tMap decomposition. Join + filter + expression + fan-out in one component.
Scale story. Hop-on-Beam / Talend-Spark break the single-JVM ceiling.

Question. Draft the rapid-fire answer set an interviewer can fire in any order.

Input.

Probe	Weak answer	Senior answer
"Talend vs Hop engine?"	"both are ETL tools"	"Talend generates Java; Hop interprets metadata at run time"
"Pipeline vs workflow?"	"not sure"	"pipeline = parallel row flow; workflow = sequential orchestration"
"What is tMap?"	"a Talend box"	"join + filter + expression + multi-output in one component"
"Is Kettle related to Hop?"	"no idea"	"Hop is a fork of the Kettle/PDI codebase, modernised under Apache"
"Studio at scale?"	"studios don't scale"	"Hop-on-Beam or Talend-Spark scale horizontally"

Code.

Rapid-fire answer card
======================
Q: Talend vs Hop — core difference?
A: Execution model. Talend is a CODE GENERATOR (job -> Java class, compiled,
   runs on any JVM). Hop is METADATA-DRIVEN (engine interprets .hpl/.hwf at
   run time; no compile). Everything else follows from that.

Q: The two artifacts?
A: Data flow (Talend job data portion / Kettle transformation / Hop pipeline)
   runs transforms in PARALLEL; orchestration (Talend parent job / Kettle job /
   Hop workflow) runs actions SEQUENTIALLY with conditional hops.

Q: tMap in one sentence?
A: Talend's central mapper: inner/outer joins to lookups, per-output boolean
   filters, per-column Java expressions, and multi-output fan-out — all in one
   dialog. In Hop it's Stream lookup + Filter rows + Calculator + Select values.

Q: Kettle / Pentaho relationship?
A: Kettle = the original OSS engine; PDI = Kettle under Pentaho/Hitachi; Hop =
   a 2020 fork of Kettle donated to Apache. Same DNA, modern architecture.

Q: Open-source status in 2026?
A: Talend Open Studio's FREE line ended in 2024 (Talend/Qlik). Apache Hop is
   the live, free, Apache-governed open-source studio. Recommend Hop for new OSS.

Step-by-step explanation.

Lead with the execution model — codegen vs interpret is the root difference; naming it first frames every follow-up correctly.
The two-artifact model (data flow vs orchestration) is shared across all three tools; stating it shows you understand the family, not just one product.
tMap answered as "join + filter + expression + fan-out in one" plus its Hop decomposition proves cross-tool experience.
Kettle/PDI/Hop lineage answered crisply signals you know the history — a common senior differentiator.
Open-source status is the currency check: knowing Talend's free line ended in 2024 and Hop is the live OSS studio shows your knowledge is current, not five years stale.

Output.

Probe	One-line senior answer
engine model	Talend compiles Java; Hop interprets metadata
two artifacts	data flow (parallel) vs orchestration (sequential)
tMap	join+filter+expression+fan-out in one component
lineage	Kettle → PDI → Hop; Talend is a separate line
OSS status	Hop is the live OSS studio; Talend Open Studio ended 2024

Rule of thumb. Rehearse the five rapid-fire answers until they are reflexive. Leading every answer with the execution-model distinction keeps you coherent no matter what order the interviewer fires the questions in.

Common beginner mistakes.

Declaring one tool universally best. The correct answer is always "it depends on the four axes"; a flat verdict signals shallow thinking.
Recommending Talend Open Studio as a live OSS option. Its free line ended in 2024; Apache Hop is the current open-source studio.
Claiming studios can't scale. Hop-on-Beam and Talend-Spark scale out; the single-JVM ceiling is a classic-config limitation, not a law.
Ignoring team skills. The strongest tool for an all-engineer Spark team is often the wrong tool for a half-analyst team; skills are a first-class axis.

Interview question on tool selection and trade-offs

A senior interviewer might ask: "A mid-size retailer runs 150 nightly loads on Talend Open Studio, which is now end-of-life for free updates. The team is two engineers and three analysts, sources include SAP and Salesforce, nightly volume is ~300 GB, and leadership wants to avoid new licensing cost. Recommend a target and justify it across every axis."

Solution Using a four-axis evaluation landing on Apache Hop with selective Beam scale-out

Four-axis evaluation
====================
Constraints: EOL free Talend, mixed team (2 eng + 3 analysts), SAP+Salesforce,
             ~300 GB/night, no new license budget.

Axis 1 — Team skills:
  3 analysts can't maintain Spark/Python -> KEEP A VISUAL STUDIO. (Hop or paid Talend)
Axis 2 — Governance/lineage:
  Both studios give visible lineage -> neutral; slight edge to studio over code.
Axis 3 — Connectors:
  SAP + Salesforce -> Talend catalogue is deeper, but Hop has Salesforce +
  JDBC/SAP options that cover this estate. Not decisive.
Axis 4 — Scale/runtime:
  300 GB/night -> native Hop engine handles this on one strong box; Beam-Spark
  in reserve if it grows. Talend standalone also fine.

Cost gate: "no new license budget" -> paid Talend platform is OUT.
           Apache Hop is free + Apache-governed + shares Kettle-style model
           Talend engineers already understand.

Decision: Apache Hop (native engine now; Beam-Spark run config reserved).

# Proof-of-scale run: same pipeline, native now, Beam-Spark if volume grows
hop-run --project=retail-etl --environment=Prod --runconfig=local \
        --file='workflows/nightly_all.hwf' --parameters=RUN_DATE=2026-08-03
# If nightly volume later triples:
hop-run --project=retail-etl --environment=Prod --runconfig=beam-spark \
        --file='workflows/nightly_all.hwf' --parameters=RUN_DATE=2026-08-03

-- Migration acceptance gate: per-target reconciliation, Talend vs Hop
SELECT table_name,
       talend_rows, hop_rows,
       (talend_rows = hop_rows) AS row_match,
       (talend_checksum = hop_checksum) AS checksum_match
FROM   etl.migration_reconcile
WHERE  NOT (talend_rows = hop_rows AND talend_checksum = hop_checksum);  -- must return 0 rows

Step-by-step trace.

Axis	Finding	Effect on decision
Team skills	3 analysts	must stay visual (Hop or paid Talend)
Governance	both studios visible lineage	neutral
Connectors	SAP + Salesforce	Talend deeper, but Hop covers it
Scale	300 GB/night	native Hop fine; Beam in reserve
Cost gate	no new license	paid Talend eliminated
Winner	Apache Hop	free + visual + covers estate + scalable

After the evaluation, Apache Hop is chosen: it keeps the analysts productive (visual studio), covers the SAP/Salesforce estate, runs 300 GB/night on the native engine with a Beam-Spark run configuration held in reserve, and — critically — carries zero new licensing cost while being Apache-governed. Each migrated load is gated on a row + checksum reconciliation against the old Talend output before it goes primary.

Output:

Requirement	Met by Apache Hop?	How
Keep analysts productive	yes	visual, drag-and-drop studio
SAP + Salesforce	yes	native connectors / JDBC
300 GB/night	yes	native engine; Beam-Spark in reserve
No new license	yes	free, Apache-governed
Safe cutover	yes	per-table reconciliation gate

Why this works — concept by concept:

Four-axis evaluation — scoring team skills, governance, connectors, and scale turns a vague "which tool" into a defensible recommendation; the cost gate then eliminates paid options the budget forbids.
Visual studio for a mixed team — three analysts cannot maintain a Spark rewrite, so keeping a drag-and-drop studio is a hard constraint, not a preference; this eliminates code-first immediately.
Apache Hop as the live OSS studio — with Talend's free line ended, Hop is the open-source studio that satisfies the no-license constraint while preserving the visual model the team already knows from Talend/Kettle.
Beam scale-out in reserve — 300 GB runs natively today, and a stored Beam-Spark run configuration means future growth is a config change, not a re-platforming — de-risking the scale axis without paying for it now.
Cost — the migration cost is dominated by per-load review and reconciliation (O(loads)), not licensing (zero) or retraining (minimal, same visual paradigm); scale cost is deferred until volume justifies the Beam runtime.

Design
Topic — design
Design problems on ETL tool selection

Practice →

Data Transformation
Topic — data-transformation
Data-transformation problems across tools

Practice →

Cheat sheet — Talend and Apache Hop recipes

The lineage in one line. Kettle (2001) → Pentaho Data Integration (PDI) → Apache Hop (2020, Apache top-level 2021) is one codebase family; talend (2006, Eclipse-based, Java code generation) is a separate lineage. Hop interprets metadata; Talend compiles Java. Both share the two-artifact model: data flow + orchestration.
Two artifacts, both families. Data flow = Kettle transformation / Talend job data portion / Hop pipeline (transforms run in parallel, one thread each). Orchestration = Kettle job / Talend parent job + tRunJob / Hop workflow (actions run sequentially with conditional hops). Never put bulk transformation in the orchestration layer.
Talend job vocabulary. Components (boxes: tFileInputDelimited, tDBInput/tDBOutput, tMap, tAggregateRow, tFilterRow, tSortRow, tJava, tRunJob, tLogRow); rows (Main / Lookup / Reject data connections); triggers (OnSubjobOk, OnComponentOk, Run If control connections); subjobs (connected component groups).
tMap in one card. Left = main input; right = lookup input(s) joined on key expressions (Left Outer/Inner × Unique/First/All). Per-output-table boolean filters fan one input into many outputs; per-column Java expressions (row1.amount * context.fx_rate, StringHandling.UPCASE(...)); Var panel computes intermediates once. Distinguish tMap output-filter rejects (business rule) from component Reject rows (runtime error).
Talend contexts. Define a context group with Dev/Test/Prod columns; reference context.* everywhere (never hard-code hosts/paths/constants); tContextLoad overrides from a properties file at run time; run.sh --context=Prod --context_param key=value selects env + overrides at launch. Build once, run anywhere.
Talend code generation. Each component → a Java template; the job → one .java class; routines → reusable static Java (StringHandling, TalendDate, custom). "Build Job" exports a standalone .zip with run.sh for any JVM. Generated code is fast and debuggable but diffs poorly in git.
Hop pipeline transforms. CSV file input / Table input (extract), Filter rows (gate, true/false hops), Stream lookup (in-memory join) / Merge join (sorted, large), Calculator (typed math), Select values (rename/reorder/retype/drop), JavaScript / User Defined Java Class (custom), Table output / Text file output (load). All transforms stream in parallel.
Hop workflow actions. Start, SQL (DDL), Pipeline (run a .hpl), Workflow (child), Shell, Mail, Evaluate, Success, Abort, Wait for file. Green hop = on success, red hop = on failure, unconditional = always. Abort/tDie give the non-zero exit a scheduler needs.
Hop metadata + environments. Database connections, run configurations, and servers are first-class metadata stored separately from pipelines. A project holds artifacts; an environment supplies ${VAR} values per Dev/Test/Prod. Reference ${SRC_SCHEMA}/${RUN_DATE} in queries; never hard-code.
Hop metadata injection. One template pipeline with injectable transform settings + a driver pipeline that injects (source_table, target_table, ...) from a control table = hundreds of loads from one maintained artifact. Adding a table is an INSERT into the control table, not a new pipeline.
Hop runtimes. local = native single-JVM engine (great to tens of GB/night); beam-spark / beam-flink / beam-dataflow = same pipeline lowered to Apache Beam for horizontal scale — swap the --runconfig, do not edit the pipeline. hop-run for CLI/CI, Hop Server for hosted/scheduled runs.
Talend→Hop port map. tMap lookup → Stream lookup/Merge join; simple expression → Calculator; complex/null-safe expression → JavaScript/UDJC; output filter → Filter rows; component Reject → Table output error hop; context group → project+environment; routine → UDJC. Budget effort by the number of non-trivial tMap expressions, not component count.
Tool-selection axes. Team skills (non-coder vs all-engineer), governance/lineage, connector breadth, scale/runtime. Talend = deepest connectors + Java + commercial support; Hop = live Apache OSS + metadata-driven + Beam scale-out; code-first = version control + testing + all-engineer velocity. Match the tool to the constraints; never declare one universally best. Talend Open Studio's free line ended in 2024 — recommend Apache Hop for new open-source projects.

Frequently asked questions

What is the difference between Talend and Apache Hop?

The core difference is the execution model: talend is a code generator — a Talend job is compiled into a standalone Java class you can run on any JVM — while Apache Hop is metadata-driven — its engine reads the pipeline/workflow XML (.hpl/.hwf) and interprets it at run time with no compile step. They also come from different lineages: Hop is a 2020 fork of the Kettle/Pentaho Data Integration codebase donated to the Apache Software Foundation, whereas Talend is a separate Eclipse-based product now under Qlik. Practically, both share the two-artifact model (a data-flow artifact plus an orchestration artifact) and a drag-and-drop etl studio experience, but Talend centralises transformation in the dense tmap component while Hop spreads it across explicit transforms like Stream lookup, Filter rows, and Calculator.

Is Talend Open Studio still free and open-source in 2026?

No — the free Talend Open Studio line was discontinued in early 2024 by Talend/Qlik, so it is no longer a live open-source recommendation for a new project. The commercial Talend platform (data integration, data quality, and cloud tiers) continues under Qlik, but it is licensed, not free. If your requirement is specifically a free, open-source visual ETL studio in 2026, Apache Hop is the live answer — it is an Apache Software Foundation top-level project, actively developed, and shares the Kettle/PDI heritage that many existing teams already understand. In interviews, recommending Talend Open Studio as a current open-source option signals stale knowledge; naming Apache Hop instead signals you are current.

What is tMap in Talend and what is its Apache Hop equivalent?

tmap is Talend's central transformation component: in one dialog it performs inner or outer joins against one or more lookup inputs, applies per-output-table boolean filters to fan a single input stream into multiple outputs, evaluates per-column Java expressions (for example row1.amount * context.fx_rate or StringHandling.UPCASE(row1.name)), and computes reusable intermediate variables. Apache Hop has no single equivalent — it deliberately decomposes that work into separate transforms: Stream lookup or Merge join for the join, Filter rows for the routing, Calculator for typed math, Select values for schema shaping, and JavaScript/User Defined Java Class for arbitrary logic. Knowing this decomposition both ways is a reliable signal that you have actually worked across both studios rather than just one.

How does Apache Hop relate to Kettle and Pentaho?

Apache Hop is a fork and modernisation of the Kettle codebase. Kettle was the original open-source visual ETL engine (2001); after Pentaho acquired it, the same engine was distributed as Pentaho Data Integration (PDI), and the names "Kettle" and "PDI" are used interchangeably. In 2019–2020 a fork of that codebase was created and donated to the Apache Software Foundation, becoming the top-level Apache Hop project in 2021. Hop keeps Kettle's DNA — the pipeline (transformation) and workflow (job) artifact split, the transform/step catalogue, the row-streaming engine — while adding a plugin-first metadata architecture, projects and environments, native unit testing, and an Apache Beam integration that runs the same pipeline on Spark, Flink, or Google Dataflow. This shared lineage is why migrating a Kettle/PDI shop to Hop is far cheaper than migrating to Talend or a code-first stack — hop-import maps most .ktr/.kjb files almost one-to-one.

When should I use a visual ETL studio instead of writing code?

Use a visual open source etl studio (Talend or Apache Hop) when your team includes non-coders who must maintain the pipelines, when you need a broad connector catalogue for exotic sources (SAP, Salesforce, mainframe, flat-file EDI) without writing integration code, when auditors require visible lineage you can point at in a review, or when built-in logging, restartability, and scheduling matter more than raw flexibility. Reach for a code-first stack (dbt, Spark, Airflow, plain Python) when the team is all-engineers with strong version-control and testing discipline, when the pattern is ELT-in-warehouse, or when you operate at a scale where the studio's model adds no value. The nuance that impresses interviewers: when scale is the only objection to a studio, you do not have to abandon the visual model — Apache Hop on an Apache Beam Spark/Flink runner (or Talend's Spark jobs) keeps the drag-and-drop design while scaling out horizontally.

What are the most common Talend and Apache Hop interview questions?

The talend interview questions and Hop questions cluster around a few themes. Expect vocabulary: explain a job vs a pipeline vs a workflow, a component vs a transform, and Main/Lookup/Reject rows. Expect tMap depth: describe join models (Left Outer/Inner, Unique/First/All), the difference between an output-filter reject and a component Reject row, and how you avoid loading a giant lookup into memory. Expect contexts/environments: how you parameterise a job across Dev/Test/Prod without recompiling. Expect the engine model: why Talend generates Java and Hop interprets metadata, and what that means for git diffs, debugging, and scale. Expect lineage and history: how Hop relates to Kettle/PDI and why Talend Open Studio's free line ending in 2024 matters. And expect tool selection: a scenario where you weigh Talend vs Hop vs code-first across team skills, governance, connectors, and scale — the interviewer is testing whether you name the axes, not whether you have a favourite.

Practice on PipeCode

Drill the ETL practice library → for the pipeline-design, extract-transform-load, and studio-migration problems that Talend and Hop interviews lean on.
Rehearse on the data-transformation practice library → for the join, lookup, expression, and mapping problems that tmap and Stream lookup questions are built from.
Sharpen the load side with the database practice library → for the batch-load, enrichment, and reject-handling patterns both studios implement.
Work the design practice library → for the tool-selection and metadata-driven-architecture questions senior interviewers open with when Talend or Apache Hop is on the table.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the four-axis tool-selection matrix against real graded inputs.

Turn ETL-studio theory into interview reps

Docs explain components and transforms. PipeCode drills explain the decision — when a dense tMap beats explicit Hop transforms, when metadata injection saves 200 pipelines, when code generation helps and when it hurts, and how to weigh Talend vs Apache Hop vs code-first across team, governance, connectors, and scale. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs data engineers actually face.

Practice ETL problems →
Practice data-transformation problems →

1. Why open-source ETL studios still matter

The visual-ETL lineage from Kettle to Talend to Hop is still load-bearing infrastructure — and interviewers know it

Worked example — mapping the ETL vocabulary across the three tools

Worked example — deciding studio vs code for a new pipeline

ETL-architecture interview question on choosing a visual studio

Solution Using a lineage-aware, staged migration from Kettle to Apache Hop

2. Talend — jobs, components, tMap, contexts, and code generation

A Talend job is a canvas of components that the studio compiles into Java — and tmap is where the real transformation logic lives

Worked example — a tMap join with a lookup, an expression, and a reject flow

Worked example — context groups for Dev/Test/Prod promotion

Worked example — reading the generated Java and adding a routine

Talend interview question on the tMap and code generation

Solution Using a database-side lookup, tMap multi-output filters, and reject handling

3. Apache Hop — pipelines, workflows, and the metadata-driven engine

Apache Hop splits work into hop pipelines (data flow) and workflows (orchestration) and interprets metadata at run time — no code generation, Kettle DNA modernised

Worked example — a Hop pipeline with Table input, Filter, Stream lookup, and Table output

Worked example — a workflow that orchestrates the pipeline with success/failure branches

Worked example — metadata injection to load many tables with one template

Apache Hop interview question on pipelines vs workflows and the engine model

Solution Using pipeline/workflow separation and swappable Beam run configurations

4. Building a pipeline — a worked ETL job in both studios

The same extract → transform → load job, built once in Talend and once in Hop, shows the two execution models side by side

Worked example — the transform step in both tools, expression for expression

Worked example — orchestration in both tools with failure handling

ETL interview question on porting a job between studios

Solution Using a transform-for-transform Talend→Hop port

5. Talend vs Hop vs code-first + interview questions and signals

Picking between talend, Apache Hop, and a code-first stack is a four-axis decision — and knowing the axes is what the talend interview questions really test

Worked example — the four-axis scoring table

Worked example — the rapid-fire signal answers

Interview question on tool selection and trade-offs

Solution Using a four-axis evaluation landing on Apache Hop with selective Beam scale-out

Cheat sheet — Talend and Apache Hop recipes

Frequently asked questions

What is the difference between Talend and Apache Hop?

Is Talend Open Studio still free and open-source in 2026?

What is tMap in Talend and what is its Apache Hop equivalent?

How does Apache Hop relate to Kettle and Pentaho?

When should I use a visual ETL studio instead of writing code?

What are the most common Talend and Apache Hop interview questions?

Practice on PipeCode

Turn ETL-studio theory into interview reps

A Talend job is a canvas of `components` that the studio compiles into Java — and `tmap` is where the real transformation logic lives

Apache Hop splits work into `hop pipelines` (data flow) and `workflows` (orchestration) and interprets metadata at run time — no code generation, Kettle DNA modernised

Picking between `talend`, Apache Hop, and a code-first stack is a four-axis decision — and knowing the axes is what the `talend interview questions` really test