Sling: CLI-First Database-to-Database & File Replication for Data Teams

sling data replication is what you reach for when the job is not "transform the data" but simply "get this table, or this folder of files, from over here to over there — correctly typed, incrementally, and without writing a connector." Sling is a single self-contained binary. You install it, name your connections once, and run sling run --src-conn ... --tgt-conn ... — or point it at a replication.yaml — and it extracts from the source, infers the schema, creates or migrates the target, and loads the rows. There is no cluster to operate, no Python runtime to pin, no vendor row-based bill.

That is a different shape from the tools data teams reached for before it: a managed connector platform (Fivetran, Airbyte) you configure but cannot drop into a shell script, a library-first loader (dlt) you embed in Python, or a hand-rolled SELECT + INSERT job that breaks the first time a column is added. This guide walks through the four ideas an interviewer will actually probe — the sling run and connections model, the replication.yaml structure, incremental mode with primary_key and update_key, and the load modes plus file-to-database replication — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse the load-shape decisions on the data-transformation practice set →, and harden your incremental logic on the idempotency practice set →.

---

On this page

• Why Sling changes database & file replication in 2026
• sling run, connections & the CLI-first model
• replication.yaml — source, target, defaults & streams
• Incremental mode with primary_key + update_key
• Load modes & file replication
• Cheat sheet — Sling recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Sling changes database & file replication in 2026

Sling is a single binary you run, not a service you operate — that one fact decides where it fits

The one-sentence invariant: Sling is a CLI-first replication tool that ships as one dependency-free binary, so data movement becomes a command you run rather than a platform you provision. Everything that makes Sling attractive to a small data team follows from that. There is no connector cluster, no separate control plane, no per-row vendor bill; a Sling job is a command that runs anywhere a binary runs — a cron entry, an Airflow BashOperator, a GitHub Action, a container, your laptop.

The EL split — what Sling does and deliberately does not do.

• Extract. Sling reads from databases (Postgres, MySQL, SQL Server, Oracle, Snowflake, BigQuery, Redshift, DuckDB, and more), from files (CSV, JSON/JSON-lines, Parquet), and from object storage (S3, GCS, Azure, SFTP, local disk).
• Load. Sling owns the tedious part: inferring column types, creating or migrating the target table, staging into a temporary table, and swapping or merging it into place atomically per the load mode you chose.
• Transform is out of scope on purpose. Sling is the EL in ELT. Joins, dimensional models, and metrics belong downstream in dbt or SQL, against the clean tables Sling landed. Keeping transform out is what keeps Sling a small, predictable tool. (Sling does offer lightweight per-column transforms and casting, but not business logic.)

Where Sling sits against the alternatives.

• vs a hand-rolled script. A SELECT loop plus INSERT statements has no schema management, no watermark state, no atomic swap, and no mode switch. The first schema drift or partial failure corrupts the table. Sling handles all of that behind one flag.
• vs Fivetran / Airbyte. Managed connectors are excellent when a prebuilt connector exists and you want a hosted, zero-code service. Sling wins when you want data movement versioned in your repo as a replication.yaml, when you cannot route data through a third-party plane for compliance, or when you just want a binary in a cron job rather than an account to manage.
• vs dlt. dlt is a Python library you import and embed in your own code; Sling is a CLI you invoke. dlt fits when ingestion logic lives inside a Python application; Sling fits when replication is an operational task you want to declare in config and schedule, independent of any one language runtime.

What interviewers listen for.

• Do you say "Sling is a CLI-first binary, not a platform" in the first sentence? — senior signal.
• Do you place Sling as "the EL, with transform left to dbt" unprompted? — required framing.
• Do you distinguish sling run for ad-hoc flags vs replication.yaml for reusable, versioned pipelines? — the core of the model.
• Do you name primary_key + update_key for incremental and full-refresh / truncate / snapshot / backfill as the load modes without prompting? — that is the whole feature surface.

Worked example — one command that replaces a hundred lines

Detailed explanation. The canonical Sling "hello world" copies one table from a source database to a target database. It looks trivial, and that is the point: the same command that moves a five-row table moves a five-million-row table, because Sling streams the rows and manages the schema for you — no CREATE TABLE, no type mapping, no connection boilerplate you maintain.

Question. Copy public.orders from a Postgres connection named MY_PG into a Snowflake connection named MY_SNOWFLAKE, letting Sling create the target table, and show what it does.

Input.

order_id	amount	customer
1	42.50	ada
2	17.00	linus

Code.

sling run \
  --src-conn MY_PG \
  --src-stream 'public.orders' \
  --tgt-conn MY_SNOWFLAKE \
  --tgt-object 'raw.orders' \
  --mode full-refresh

Step-by-step explanation. --src-conn / --src-stream name what to read (the orders table in MY_PG); --tgt-conn / --tgt-object name where to write (raw.orders in MY_SNOWFLAKE); --mode full-refresh (the default) says drop and recreate. Sling connects, reads the source rows while inferring each column's type, stages them into a temporary table in Snowflake, creates raw.orders if it is absent, and swaps the staged data into place in one atomic step so a reader never sees a half-loaded table. You wrote no DDL.

Output.

Sling did	value
target table	raw.orders created (columns: order_id number, amount number, customer varchar)
load path	source → temp table → atomic swap
rows loaded	2
exit	execution succeeded

Rule of thumb. If the job is "move this object from A to B," Sling can do it in one sling run; the toy example and the production replication differ only in the connections and the stream, never in the command shape.

---

2. sling run, connections & the CLI-first model

sling run plus named connections is the whole entry point — learn the four flags and one config file and the rest is options

Sling has exactly one command you run for data movement — sling run — and exactly two ways to configure it. An interviewer who asks "walk me through how you'd use Sling" wants both paths and the connection model in order. Get this vocabulary crisp and the tool snaps into focus.

The two ways to run.

• CLI flags — ad-hoc. sling run --src-conn ... --src-stream ... --tgt-conn ... --tgt-object ... --mode .... Best for quick one-off moves, shell scripts, and experimentation from the terminal.
• replication.yaml — reusable. sling run -r replication.yaml. Streams and defaults declared in a YAML (or JSON) file you commit to your repo. Best for anything scheduled or shared, because it is versioned config, not a shell history entry.

The four flags you always reach for.

• --src-conn / --tgt-conn. The source and target connections — a name you defined (MY_PG), or an inline connection string/URL. For a database source with a file target (or vice versa) one side may be a file path instead.
• --src-stream. The source table (schema.table), a file path (file:///tmp/data/), a .sql file, or inline SQL to run as the query.
• --tgt-object. The target table (schema.table) or file path. Supports runtime variables like {stream_table}.
• --mode. The load mode: full-refresh (default), incremental, truncate, snapshot, backfill, definition-only, or change-capture.

Connections — named once, referenced everywhere.

• Set / list / test / discover. sling conns set MY_PG url='postgresql://...' stores a credential; sling conns list shows every detected connection; sling conns test MY_PG verifies it; sling conns discover MY_PG --pattern public.* lists the streams (tables or files) available.
• Where credentials come from. Sling reads, in order, the global env file ~/.sling/env.yaml, a project-local .env.sling in the working directory, your ~/.dbt/profiles.yml, and plain environment variables. Naming a connection keeps secrets out of process listings and out of your shell history.

CLI vs library — an important nuance.

• Sling is fundamentally the CLI binary. The pip install sling package, and the wrappers for other languages, are thin shells that invoke that same binary — so a Python Replication(...).run() call and a sling run -r ... command execute identical logic. This is the opposite of dlt, where the Python library is the engine.

Worked example — set a connection, then run a move

Detailed explanation. Real usage is two steps: register the connections once, then run moves against them by name. Here we set a Postgres and a Snowflake connection, confirm they work, then copy a table. The connections persist in ~/.sling/env.yaml, so every later run just refers to the names.

Question. Register MY_PG and MY_SNOWFLAKE, test them, then load public.customers from Postgres into raw.customers in Snowflake.

Input. Two connection URLs and one source table public.customers.

Code.

sling conns set MY_PG url='postgresql://user:[e-mail]:5432/db'   # register in ~/.sling/env.yaml
sling conns set MY_SNOWFLAKE url='snowflake://user:pass@acct/db'

sling conns test MY_PG                                    # verify the credential
sling conns discover MY_PG --pattern 'public.customer*'   # inspect available streams

sling run \
  --src-conn MY_PG --src-stream 'public.customers' \
  --tgt-conn MY_SNOWFLAKE --tgt-object 'raw.customers' \
  --mode full-refresh

Step-by-step explanation. sling conns set writes each credential to the env file under a name; nothing is stored in the command history in plaintext beyond that one set. sling conns test MY_PG opens a connection and returns success! if the credential is valid. sling conns discover lists matching tables so you confirm the stream name before moving data. The final sling run references both connections by name, so the same command is safe to paste into a scheduler.

Output.

command	result
conns set MY_PG	connection saved to ~/.sling/env.yaml
conns test MY_PG	success!
conns discover	lists public.customers (20 columns)
sling run	raw.customers created, rows loaded

Rule of thumb. Set connections once by name, then never put a raw credential in a sling run command again — named connections are what make your runs safe to schedule and safe to share.

Sling interview question on the run model

Question. An interviewer asks you to load every table in a Postgres schema sales into Snowflake in one invocation, each target table named after the source table, using three parallel workers. Show the command.

Solution Using a wildcard stream with runtime variables

Code.

export SLING_THREADS=3

sling run \
  --src-conn MY_PG \
  --src-stream 'sales.*' \
  --tgt-conn MY_SNOWFLAKE \
  --tgt-object 'raw.{stream_table}' \
  --mode full-refresh

Step-by-step trace.

step	resolved	effect
1	sales.* expands	Sling lists every table in sales
2	sales.orders → raw.orders	{stream_table} fills in per table
3	sales.customers → raw.customers	one task per table
4	SLING_THREADS=3	up to 3 tables move in parallel

1. The wildcard stream sales.* tells Sling to enumerate every table in the schema and generate one task per table — internally the same as writing them all out by hand.
2. The runtime variable {stream_table} in --tgt-object substitutes each source table's name, so sales.orders lands as raw.orders and sales.customers as raw.customers.
3. SLING_THREADS=3 runs up to three of those per-table tasks concurrently; the rest queue.
4. Each task is still an independent full-refresh — one failing table does not corrupt the others.

Output:

source table	target table	mode
sales.orders	raw.orders	full-refresh
sales.customers	raw.customers	full-refresh
sales.refunds	raw.refunds	full-refresh

Why this works — concept by concept:

• Wildcard stream — schema.* expands into one task per table, so a whole schema replicates from a single command instead of a hand-maintained list.
• Runtime variables — {stream_table} (and siblings like {stream_schema}, {target_schema}) template the target name so the mapping is a rule, not N lines of config.
• Named connections — because MY_PG and MY_SNOWFLAKE are pre-registered, the command carries no secrets and is safe to schedule.
• Thread parallelism — SLING_THREADS moves independent tables concurrently, turning a serial schema copy into a parallel one with one env var.
• Cost — wall-clock is O(largest table / threads) rather than O(sum of all tables); memory stays flat because each stream is read incrementally, not buffered whole.

ETL
Topic — etl
ETL extract-and-load pipeline problems

Practice →

Pipelines Topic — pipelines Pipeline-design and schema-replication problems

Practice →

---

3. replication.yaml — source, target, defaults & streams

One YAML file replaces a wall of flags — source, target, defaults, streams is the entire structure

Once a move is more than a one-off, you promote it from CLI flags to a replication.yaml you run with sling run -r. The file has a tiny, fixed top-level shape, and an interviewer who asks "how do you make a Sling job reusable" wants these keys in order. Learn four keys and you can express an entire multi-table replication as versioned config.

The four top-level keys.

• source and target. The names of the connections this replication reads from and writes to — the same names you set with sling conns set.
• defaults. Settings applied to every stream unless a stream overrides them: mode, object (the target name, usually templated), primary_key, source_options, target_options.
• streams. A map whose keys are source streams (tables, wildcards, or custom names) and whose values are per-stream overrides. An empty value means "use the defaults."
• env. Run-wide variables: SLING_THREADS (parallelism), SLING_RETRIES (retry failed streams), SLING_LOADED_AT_COLUMN (stamp _sling_loaded_at), SLING_STREAM_URL_COLUMN (record the source file path).

Streams — the flexible part.

• A plain table. finance.accounts: with no body loads that table with the defaults.
• A wildcard. finance.*: expands to one task per table — a feature available only in replications, not in CLI flags for some sources.
• An override. Give a stream a body to override object, mode, primary_key, or update_key for just that table.
• disabled: true. Keep a stream in the file but skip it this run.
• A custom-SQL stream. Provide a sql: block to shape the extract, and add an object: key because Sling cannot infer the target name from a free-form query.

Runtime variables template the target.

• object: '{target_schema}.{stream_schema}_{stream_table}' builds a deterministic target name for every stream, so a 40-table schema needs one object rule, not 40 lines.

Worked example — a replication with defaults and per-stream overrides

Detailed explanation. The everyday replication.yaml sets a default mode and target-naming rule, then lists a few streams — most taking the defaults, one overriding the mode and keys, one disabled. Running it generates one task per active stream and executes them in order (or in parallel with SLING_THREADS).

Question. Write a replication.yaml that full-refreshes finance.accounts and finance.departments into a raw schema, but loads finance.transactions incrementally on last_updated_at keyed by id, and skips finance.users.

Input. Four source tables in schema finance; target connection MY_SNOWFLAKE.

Code.

source: MY_PG
target: MY_SNOWFLAKE

defaults:
  mode: full-refresh
  object: 'raw.{stream_table}'

streams:
  finance.accounts:
  finance.departments:

  finance.transactions:
    mode: incremental
    primary_key: [id]
    update_key: last_updated_at

  finance.users:
    disabled: true

env:
  SLING_THREADS: 3
  SLING_LOADED_AT_COLUMN: true

Step-by-step explanation. defaults.mode: full-refresh and defaults.object: 'raw.{stream_table}' apply to every stream that does not override them, so finance.accounts → raw.accounts and finance.departments → raw.departments, both dropped and recreated. finance.transactions overrides mode to incremental and supplies primary_key + update_key, so it upserts only new rows. finance.users is kept in the file but disabled, so it is skipped. SLING_THREADS: 3 runs the active streams in parallel and SLING_LOADED_AT_COLUMN stamps each row with a load timestamp.

Output.

stream	target	mode	ran?
finance.accounts	raw.accounts	full-refresh	yes
finance.departments	raw.departments	full-refresh	yes
finance.transactions	raw.transactions	incremental	yes
finance.users	—	—	skipped

Rule of thumb. Put everything shared in defaults and let each stream override only what differs — a good replication.yaml reads as "here is the rule, and here are the three exceptions."

Sling interview question on scaling a replication

Question. You must replicate all ~60 tables in a crm schema, but four large event tables should load incrementally while the rest full-refresh nightly. You do not want to enumerate 60 tables by hand. How do you structure the replication.yaml?

Solution Using a wildcard default plus targeted overrides

Code.

source: MY_PG
target: MY_SNOWFLAKE

defaults:
  mode: full-refresh
  object: 'raw.{stream_table}'

streams:
  # every table in crm, full-refresh by default
  crm.*:

  # four large tables overridden to incremental
  crm.events:
    mode: incremental
    primary_key: [id]
    update_key: updated_at
  crm.page_views:
    mode: incremental
    primary_key: [id]
    update_key: updated_at
  crm.clicks:
    mode: incremental
    primary_key: [id]
    update_key: updated_at
  crm.sessions:
    mode: incremental
    primary_key: [id]
    update_key: updated_at

env:
  SLING_THREADS: 4
  SLING_RETRIES: 1

Step-by-step trace.

stream entry	matches	mode used
crm.*	all ~60 tables	full-refresh (default)
crm.events	one table	incremental (override wins)
crm.page_views	one table	incremental (override wins)
crm.clicks, crm.sessions	two tables	incremental (override wins)

1. The wildcard stream crm.* enumerates every table and applies the full-refresh default, so you never list 56 small tables individually.
2. A more specific stream key (crm.events) overrides the wildcard for that exact table, flipping it to incremental with its own keys — explicit entries win over the wildcard match.
3. SLING_THREADS: 4 moves four streams at a time; SLING_RETRIES: 1 re-attempts any stream that fails once, so a transient network blip does not fail the whole run.
4. The result is 60 tables replicated from ~50 lines of config, with the four heavy tables loading only their deltas.

Output:

group	count	mode
small tables (via crm.*)	~56	full-refresh
large event tables	4	incremental

Why this works — concept by concept:

• Wildcard default — crm.* plus a templated object covers the long tail of tables with one rule, so config scales sub-linearly with table count.
• Specific-over-wildcard — an explicit stream key beats the wildcard match, which is exactly how you carve four exceptions out of sixty tables without duplication.
• Per-stream keys — primary_key + update_key live on each incremental stream, so the heavy tables move deltas while the rest stay simple.
• Retries and threads — SLING_RETRIES and SLING_THREADS turn a fragile serial job into a resilient parallel one with two env lines.
• Cost — config size is O(number of exceptions), not O(number of tables); runtime is dominated by the incremental deltas rather than a nightly full re-pull of the large tables.

Pipelines
Topic — pipelines
Config-as-code pipeline-design problems

Practice →

ETL Topic — data-transformation Stream-shaping and target-mapping problems

Practice →

---

4. Incremental mode with primary_key + update_key

Sling remembers the last value it loaded — update_key sets the boundary and primary_key decides upsert vs append

A nightly full-refresh is fine for a 200-row dimension and ruinous for a 500-million-row event table. Incremental mode means each run pulls only rows newer than the last run, and the two keys you supply decide exactly how. Say the invariant in one breath: the update_key sets the watermark, and the primary_key decides whether new rows are upserted or merely appended.

The mechanism.

• Watermark from the target. Sling computes the watermark as max(update_key) in the target table, then pulls source rows where update_key > watermark. The state lives in the destination data itself, so it survives crashes and machine changes — there is no separate state service to babysit.
• First run reads everything. On the very first run the target is empty or missing, so max(update_key) is null and the incremental WHERE is effectively 1=1: Sling reads the whole source once, creates the target with inferred types, applies the keys, and records the watermark. Every later run is cheap.
• Auto-created target. You never pre-create the table; Sling creates it on that first run with the primary_key (and any target_options.table_keys) applied at creation time.

The three incremental strategies.

• primary_key + update_key → new-data upsert. Pull only rows after max(update_key), then update matching keys and insert the rest. This is the common case for a mutable, growing table.
• primary_key only → full-data upsert. Read the full source every run, but upsert on the key so the target has no duplicates. Correct when the source has no reliable increasing column but you still want current-state semantics.
• update_key only → append-only. Pull rows after max(update_key) and insert them; no updates. Correct for an immutable event log.

Failure modes interviewers probe.

• Late-arriving / backdated rows are missed. A row whose update_key is at or below the current watermark is not selected next run — its key is not greater than the last maximum. To catch late data, write a custom-SQL stream with the {incremental_value} placeholder and subtract a lookback interval; with a primary_key, the re-scanned rows are upserted, so re-reading them is idempotent.
• Bad cursor choice. An update_key a client can backdate (an app-set timestamp with clock skew) can drop rows below the watermark. Prefer a database-assigned commit time or a monotonically increasing id.

Worked example — an incremental run keyed on updated_at

Detailed explanation. The everyday incremental job supplies --mode incremental with both keys. On the first run Sling reads the whole table and seeds the watermark; on the second it reads only rows past that watermark and upserts them. The same command runs every night unchanged.

Question. Load public.orders incrementally into raw.orders, keyed by id and cursored on updated_at, so run 2 only moves rows changed since run 1.

Input. Source rows across two runs (target starts empty).

Code.

sling run \
  --src-conn MY_PG --src-stream 'public.orders' \
  --tgt-conn MY_SNOWFLAKE --tgt-object 'raw.orders' \
  --mode incremental \
  --primary-key 'id' \
  --update-key 'updated_at'

Step-by-step explanation. On run 1 the target raw.orders does not exist, so max(updated_at) is null: Sling reads every source row, creates the table with id as the primary key, loads the rows, and records the maximum updated_at it saw. On run 2 Sling reads max(updated_at) from raw.orders and pulls only source rows where updated_at is greater, then upserts them on id — existing orders update in place, new ones insert. The watermark is data, so a crash between runs changes nothing.

Output.

run	watermark at start	rows read	target rows after
1	null (full read)	4	4
2	2026-03-02 10:15	1	5 (one upsert-insert)

Rule of thumb. Pick an update_key the source controls and only increases — a commit timestamp or an auto-increment id — and always pair it with a primary_key when rows can change, or the boundary row can slip through twice.

Sling interview question on incremental correctness

Question. Your source occasionally receives backdated corrections — a row whose updated_at is set a few hours in the past. A plain incremental run misses them because their updated_at is below the watermark. How do you make Sling pick them up without re-reading the whole table every night?

Solution Using a custom-SQL stream with an incremental lookback

Code.

source: MY_PG
target: MY_SNOWFLAKE

streams:
  # custom SQL stream: re-scan the last 3 days on every run
  orders_incremental:
    mode: incremental
    primary_key: [id]
    update_key: updated_at
    object: raw.orders
    sql: |
      select *
      from public.orders
      where updated_at > coalesce({incremental_value}, '2001-01-01')::timestamp
                         - interval '3 days'

Step-by-step trace.

run	{incremental_value}	effective filter	rows re-scanned
1	null	> '2001-01-01' - 3d (all)	full table
2	2026-03-05	> 2026-03-02	last 3 days
2	backdated row @ 2026-03-04	inside window	picked up + upserted

1. The custom-SQL stream replaces the auto-generated query; Sling injects the last watermark wherever you write {incremental_value}.
2. Subtracting interval '3 days' widens the boundary so the query re-reads a trailing window, catching any correction whose updated_at landed up to three days behind the watermark.
3. Because a primary_key is set, every re-scanned row is upserted, not appended — re-reading a row that was already loaded updates it in place instead of duplicating it, so the lookback is idempotent.
4. coalesce({incremental_value}, '2001-01-01') handles the first run, where the watermark is null, by falling back to a floor date that reads everything once.

Output:

table	duplicates	late rows captured
raw.orders	0 (upsert on id)	yes, within the 3-day window

Why this works — concept by concept:

• Watermark boundary — Sling's default update_key > max is exact but blind to anything at or below the last maximum; the lookback deliberately relaxes that boundary.
• {incremental_value} placeholder — exposing the watermark inside custom SQL lets you shape the WHERE clause instead of accepting the generated one.
• Lookback window — subtracting an interval trades a little extra read each run for correctness on late-arriving data; size the window to your worst-case lateness.
• Primary-key upsert — the primary_key makes re-reading the window idempotent, turning "scan more" into "scan more, safely, with no duplicates."
• Cost — extra work is O(rows in the lookback window) per run, far cheaper than a nightly full-refresh, in exchange for late-data safety.

Idempotency
Topic — idempotency
Idempotent incremental-load problems

Practice →

ETL Topic — etl Incremental extract-load problems

Practice →

---

5. Load modes & file replication

One flag decides how rows hit the target, and Sling moves files as easily as tables — modes plus any-direction replication

Every Sling run writes with one load mode, and the choice is the single most consequential correctness decision in the job. And because Sling treats a database and a file the same way, the same command that copies table-to-table also copies table-to-Parquet or CSV-folder-to-table. Say the modes in one breath: full-refresh drops and recreates, truncate empties but keeps DDL, snapshot appends with a timestamp, backfill loads a bounded range, and incremental merges deltas.

The load modes.

• full-refresh (default). The target table is dropped and recreated from the source. Correct for small, fully re-pullable tables.
• truncate. Like full-refresh, but the target is truncated instead of dropped — so grants, indexes, and special DDL survive. Correct when downstream depends on the table's DDL staying put.
• snapshot. Appends the full dataset each run and stamps a _sling_loaded_at column. Correct when you want a periodic point-in-time history of a whole table.
• backfill. Like incremental, but bounded by a --range on the update_key (e.g. a date or id range) so you can replay a specific window deterministically.
• change-capture. Reads row-level inserts/updates/deletes from the source database's transaction log (CDC); does an initial snapshot, then streams changes. Requires a primary_key.

File replication — the second half of the tool.

• Database → file. Point --tgt-object at a path (s3://bucket/orders/ or file:///tmp/orders/) and set --tgt-options '{"format": "parquet"}' to export a table as Parquet, CSV, or JSON-lines. file_max_rows splits the output into chunks.
• File → database. Point --src-stream at a file or folder (file:///tmp/csvs/, s3://bucket/data/) and Sling reads every file, infers the schema, and loads it into the target table.
• Formats and stdin. Sling handles CSV, JSON/JSON-lines, and Parquet, and can read piped stdin (cat data.json | sling run ...) or write to --stdout, so it slots into Unix pipelines.

Choosing a mode.

• Small, re-pullable → full-refresh (or truncate to preserve DDL/grants).
• Growing, mutable, keyed → incremental with primary_key + update_key.
• Immutable events → incremental with update_key only (append-only).
• Point-in-time history of a whole table → snapshot.
• Replay a specific window → backfill with --range.

Worked example — export a table to partitioned Parquet on S3

Detailed explanation. The clearest way to see file replication is a database-to-file export. Here Sling reads a Postgres table and writes it to an S3 folder as Parquet, splitting the output into files of at most 100k rows. The command shape is identical to a database-to-database move — only the target is a path and format.

Question. Export public.orders from MY_PG to s3://analytics/orders/ as Parquet, capped at 100,000 rows per file.

Input. A Postgres table public.orders with ~250,000 rows, and a configured MY_S3 connection.

Code.

sling run \
  --src-conn MY_PG \
  --src-stream 'public.orders' \
  --tgt-conn MY_S3 \
  --tgt-object 's3://analytics/orders/' \
  --tgt-options '{"format": "parquet", "file_max_rows": 100000}'

Step-by-step explanation. --src-stream 'public.orders' reads the table; --tgt-object 's3://analytics/orders/' with MY_S3 names an object-storage folder as the destination; --tgt-options sets format: parquet and caps each file at file_max_rows: 100000. Sling streams the source rows, encodes them as Parquet with inferred column types, and writes multiple part files under the folder — three files for ~250k rows. No intermediate database is involved.

Output.

target	files written	format	rows total
s3://analytics/orders/	3 part files (100k, 100k, ~50k)	parquet	~250,000

Rule of thumb. A file target is just another --tgt-object — swap a schema.table for a path and add --tgt-options for format; everything else about the command stays the same.

Sling interview question on choosing a load mode

Question. You ingest a small dim_country reference table that has GRANTs and an index the BI tool depends on, and separately you need a nightly point-in-time copy of a positions table so analysts can see how it looked each day. Which load modes, and why?

Solution Using truncate for the dimension and snapshot for the history

Code.

source: MY_PG
target: MY_SNOWFLAKE

streams:
  # small reference table: refresh contents but keep DDL/grants/index
  ref.dim_country:
    mode: truncate
    object: raw.dim_country

  # daily point-in-time history of positions
  trading.positions:
    mode: snapshot
    object: raw.positions_history

env:
  SLING_THREADS: 2

Step-by-step trace.

stream	mode	target behaviour
ref.dim_country	truncate	rows cleared, table + grants + index kept, reloaded
trading.positions (day 1)	snapshot	full rows appended, _sling_loaded_at = day 1
trading.positions (day 2)	snapshot	full rows appended again, _sling_loaded_at = day 2

1. truncate on dim_country empties and reloads the table without dropping it, so the GRANTs and the index the BI tool relies on survive every run — a full-refresh would drop the table and lose them.
2. snapshot on positions appends the entire table each run and stamps _sling_loaded_at, so each day's full state is preserved and query-able by that timestamp.
3. Over time raw.positions_history accumulates one dated copy per run, giving analysts a periodic point-in-time view without any SCD logic.
4. SLING_THREADS: 2 runs both streams together; they are independent, so mode choice per stream is isolated.

Output:

table	mode	shape after 2 runs
raw.dim_country	truncate	current rows only, DDL intact
raw.positions_history	snapshot	two dated copies, keyed by _sling_loaded_at

Why this works — concept by concept:

• truncate vs full-refresh — both give current-state, but truncate preserves DDL, grants, and indexes because it never drops the table; that is the difference downstream tooling cares about.
• snapshot semantics — appending the full dataset with _sling_loaded_at turns a mutable table into a periodic-snapshot history, the cheapest way to answer "what did it look like on day X."
• Per-stream modes — because the mode lives on each stream, one replication mixes a truncate dimension and a snapshot history in a single run.
• No hand-written history logic — snapshot replaces a bespoke insert-with-timestamp job, and truncate replaces a delete-and-reload script, with one keyword each.
• Cost — truncate is O(table size) per run on a small table (cheap); snapshot grows storage O(rows × runs), so it fits low-volume tables you need dated, not high-churn ones.

ETL
Topic — data-transformation
Load-mode, snapshot and merge problems

Practice →

Pipelines Topic — pipelines File-and-database replication problems

Practice →

---

Cheat sheet — Sling recipes

Ad-hoc database-to-database move.

sling run --src-conn MY_PG --src-stream 'public.orders' \
  --tgt-conn MY_SNOWFLAKE --tgt-object 'raw.orders' --mode full-refresh

Incremental run with keys.

sling run --src-conn MY_PG --src-stream 'public.orders' \
  --tgt-conn MY_SNOWFLAKE --tgt-object 'raw.orders' \
  --mode incremental --primary-key 'id' --update-key 'updated_at'

replication.yaml skeleton.

source: MY_PG
target: MY_SNOWFLAKE
defaults:
  mode: full-refresh
  object: 'raw.{stream_table}'
streams:
  sales.*:
  sales.events:
    mode: incremental
    primary_key: [id]
    update_key: updated_at
env:
  SLING_THREADS: 3

Run it with sling run -r replication.yaml.

Database-to-file export (Parquet on S3).

sling run --src-conn MY_PG --src-stream 'public.orders' \
  --tgt-conn MY_S3 --tgt-object 's3://analytics/orders/' \
  --tgt-options '{"format": "parquet", "file_max_rows": 100000}'

File folder-to-database load.

sling run --src-stream 'file:///tmp/csvs/' \
  --tgt-conn MY_PG --tgt-object 'raw.imported' --mode full-refresh

Manage connections.

sling conns set MY_PG url='postgresql://user:pass@host:5432/db'
sling conns list
sling conns test MY_PG
sling conns discover MY_PG --pattern 'public.*'

Mode picker.

Situation	Mode
Small, re-pullable table	full-refresh
Re-pullable but must keep DDL / grants	truncate
Growing, mutable, keyed table	incremental + primary_key + update_key
Immutable event log	incremental + update_key only
Periodic point-in-time history	snapshot
Replay a bounded window	backfill + --range

---

Frequently asked questions

What is Sling?

Sling is an open-source, CLI-first data replication tool that moves data between databases, data warehouses, files, and object storage. It ships as a single self-contained binary: you install it, register connections by name, and run sling run with a few flags or a replication.yaml file. Sling infers the schema, creates or migrates the target, and loads the rows in the mode you choose — with no cluster to operate and no runtime to pin. It is the EL in ELT; transformations stay downstream in dbt or SQL.

How is Sling different from Fivetran, Airbyte, or dlt?

Fivetran and Airbyte are managed connector platforms you configure and run as a hosted service; dlt is a Python library you embed in your own application. Sling is a CLI binary you invoke — from a shell, a cron job, or a scheduler. Use managed connectors when a prebuilt connector exists and you want zero infrastructure; use dlt when ingestion logic lives inside Python code; use Sling when you want replication declared as versioned replication.yaml, a binary you can drop into any pipeline, or a tool that moves both database tables and files without a third-party plane.

How do I run a database-to-database replication with Sling?

For a one-off, use CLI flags: sling run --src-conn MY_PG --src-stream 'public.orders' --tgt-conn MY_SNOWFLAKE --tgt-object 'raw.orders' --mode full-refresh. For anything reusable, put the source, target, and streams in a replication.yaml and run sling run -r replication.yaml. Connections are registered once with sling conns set NAME url='...' and referenced by name, so commands carry no raw credentials and are safe to schedule.

What is a replication.yaml in Sling?

A replication.yaml is Sling's config file for reusable, multi-stream jobs. It has four top-level keys: source and target (connection names), defaults (settings applied to every stream, such as mode and a templated object), and streams (a map of source tables to per-stream overrides). Wildcards like finance.* expand to one task per table, runtime variables like {stream_table} template the target name, and an env block sets run-wide options like SLING_THREADS and SLING_RETRIES.

How does incremental mode work in Sling?

In incremental mode Sling computes a watermark as max(update_key) in the target table and pulls only source rows where update_key is greater. If you also set a primary_key, matching rows are upserted (updated in place) and new rows inserted; with only an update_key, new rows are appended. The first run reads the whole source to seed the watermark, and every later run is cheap. To catch backdated or late-arriving rows, use a custom-SQL stream with the {incremental_value} placeholder and a lookback interval — the primary_key keeps the re-scan idempotent.

Can Sling replicate files (CSV / Parquet) to and from databases?

Yes — file replication is half of what Sling does. Point --tgt-object at a path (s3://bucket/folder/ or file:///tmp/out/) with --tgt-options '{"format": "parquet"}' to export a table to Parquet, CSV, or JSON-lines, using file_max_rows to split the output. Point --src-stream at a file or folder to load files into a database — Sling reads every file, infers the schema, and loads it. It also reads piped stdin and writes to --stdout, so it fits into Unix pipelines.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Sling idea above, from the wildcard `replication.yaml` to the `primary_key` + `update_key` incremental watermark and the truncate-versus-snapshot mode choice, maps to a hands-on practice room where you build the load against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you make this replication idempotent and catch late data?" holds up under a senior interviewer's depth probes.

Practice ETL problems now →
Pipeline-design drills →