Spark JDBC Parallel Reads on Azure Databricks: partitionColumn, Bounds, Predicates, and Not Knocking Over Your Database

Pulling a big table out of Azure SQL Database (or any relational source) with Spark looks like a one-liner. Then the job runs for two hours on a 16-node cluster and the Spark UI shows exactly one task doing all the work. This post walks through why that happens and how to read and write over JDBC in parallel without hurting the source database. It's also a favourite interview topic, because it tests whether you understand how Spark actually splits work.

Why the default JDBC read uses one task

jdbc_url = (
    "jdbc:sqlserver://myserver.database.windows.net:1433;"
    "database=sales;encrypt=true;trustServerCertificate=false;"
)
props = {
    "user": dbutils.secrets.get("kv-scope", "sql-user"),
    "password": dbutils.secrets.get("kv-scope", "sql-password"),
}

df = (spark.read.format("jdbc")
      .option("url", jdbc_url)
      .option("dbtable", "dbo.orders")
      .options(**props)
      .load())

print(df.rdd.getNumPartitions())  # 1

Spark has no idea how to split an arbitrary SQL table, so without extra options it opens one connection and runs one SELECT over the whole table. One partition means one task on one core, no matter how big the cluster is. Everything downstream inherits that single partition until a shuffle happens.

(Keep credentials in a Databricks secret scope backed by Key Vault, as above. Never hard-code them in a notebook.)

Parallel reads with partitionColumn

To get parallelism, you give Spark four options together:

bounds = (spark.read.format("jdbc")
          .option("url", jdbc_url)
          .option("query", "SELECT MIN(order_id) AS lo, MAX(order_id) AS hi FROM dbo.orders")
          .options(**props)
          .load()
          .first())

orders = (spark.read.format("jdbc")
          .option("url", jdbc_url)
          .option("dbtable", "dbo.orders")
          .option("partitionColumn", "order_id")
          .option("lowerBound", bounds["lo"])
          .option("upperBound", bounds["hi"] + 1)
          .option("numPartitions", 16)
          .option("fetchsize", 10000)
          .options(**props)
          .load())

print(orders.rdd.getNumPartitions())  # 16

Spark divides the range between lowerBound and upperBound into numPartitions equal strides and runs one query per stride, each over its own connection. Roughly:

-- partition 0
SELECT ... FROM dbo.orders WHERE order_id < 625001 OR order_id IS NULL
-- partitions 1..14
SELECT ... FROM dbo.orders WHERE order_id >= 625001 AND order_id < 1250001
-- partition 15
SELECT ... FROM dbo.orders WHERE order_id >= 9375001

A few rules that trip people up:

• The bounds don't filter anything. The first partition gets everything below the first stride (plus NULLs) and the last gets everything above the final one. If your bounds are wrong you don't lose rows; you get one enormous partition instead. That's why querying MIN/MAX first is worth the extra round trip.
• partitionColumn must be numeric, date, or timestamp. Strings don't work.
• You can't combine query with partitionColumn. If you need a filtered or joined source, wrap it as a subquery in dbtable: .option("dbtable", "(SELECT ... FROM dbo.orders WHERE status = 'SHIPPED') AS src").
• Pick an indexed column. Each partition is a range predicate. On a clustered index key it's a cheap seek; on an unindexed column, 16 partitions means 16 full scans of the table.

The skew problem: equal ranges, unequal rows

Strides are equal in value, not in row count. If order_id has a big gap (say IDs jumped from 2 million to 50 million after a migration), most strides return nothing and a couple return everything. You'll see it in the Spark UI as a handful of long tasks while the rest finish in seconds.

When the key isn't evenly distributed, use explicit predicates instead. Each predicate becomes exactly one partition, so you control the split:

predicates = [
    f"order_date >= '{m}-01' AND order_date < DATEADD(month, 1, '{m}-01')"
    for m in ["2026-01", "2026-02", "2026-03", "2026-04", "2026-05", "2026-06"]
]

orders_h1 = spark.read.jdbc(
    url=jdbc_url,
    table="dbo.orders",
    predicates=predicates,
    properties=props,
)

print(orders_h1.rdd.getNumPartitions())  # 6

Two things to check with predicates: they must not overlap (or you duplicate rows), and together they must cover everything you want (or you silently drop rows). Unlike the stride approach, there's no automatic catch-all for NULLs or out-of-range values. A quick count() against a source-side COUNT(*) is a cheap sanity test.

Don't knock over the source

numPartitions is also the number of concurrent connections Spark opens against the database. Sixty-four partitions against a small Azure SQL tier will throttle on DTU/vCore limits or hit the session cap, and the production app sharing that database will feel it. In practice:

• Start at 8 to 16 partitions and watch the database's CPU and log IO metrics, not just Spark's runtime.
• Read from a readable secondary or a replica when one exists (for Azure SQL, add applicationIntent=ReadOnly to the URL on tiers that support read scale-out).
• Schedule big extracts outside business hours.

fetchsize, pushdown, and reading only what you need

fetchsize controls how many rows the driver pulls per round trip. Small values mean thousands of network round trips per partition; some drivers (Oracle's defaults to 10) are painfully conservative. Values in the 5,000 to 20,000 range are a reasonable start for typical row widths.

Filters you apply after load() are pushed into the SQL by default (pushDownPredicate is true), and only the selected columns are fetched. You can confirm it with explain():

(orders
 .filter("status = 'SHIPPED'")
 .select("order_id", "customer_id", "amount")
 .explain())
# Look for PushedFilters: [*IsNotNull(status), *EqualTo(status,SHIPPED)]
# and ReadSchema with only the three columns

If a filter doesn't appear under PushedFilters (for example, one built on a Python UDF), Spark pulls every row and filters it afterwards. For incremental loads, filter on a watermark column such as modified_at > last_run so each run moves only changed rows, then land them in Delta with MERGE.

Writing back over JDBC

Writes are parallel by default: each DataFrame partition writes over its own connection. That's the risk here, because 400 partitions after a shuffle means 400 connections hammering the target.

(summary_df
 .write.format("jdbc")
 .option("url", jdbc_url)
 .option("dbtable", "dbo.daily_summary")
 .option("numPartitions", 8)     # Spark coalesces down to 8 before writing
 .option("batchsize", 10000)     # rows per INSERT batch, default 1000
 .option("truncate", "true")     # with overwrite: TRUNCATE instead of DROP/CREATE
 .options(**props)
 .mode("overwrite")
 .save())

truncate matters more than people expect. Without it, overwrite drops and recreates the table, losing indexes, permissions, and column types you defined in the database. Also remember JDBC writes aren't transactional across partitions: if task 5 of 8 fails, partitions 1 to 4 are already committed. For anything that must be all-or-nothing, write to a staging table and swap or MERGE on the database side in one statement.

When not to use plain JDBC

On Databricks, Lakehouse Federation lets you query SQL Server and other sources through Unity Catalog without managing JDBC options per notebook, and Lakeflow Connect covers managed ingestion for some sources. For large one-off extracts, exporting to files in ADLS and reading those can be faster than any number of JDBC partitions. JDBC is still the right tool for targeted incremental pulls and small write-backs, and it's what most interview questions are about.

Interview questions to practise

1. Why does a default JDBC read produce one partition, and what four options fix it?
2. Do lowerBound and upperBound filter rows? What happens to IDs outside them?
3. Your parallel JDBC read has two tasks taking 90% of the time. What's going on and how do you fix it?
4. How do you stop a Spark write from opening hundreds of connections to Azure SQL?
5. What does truncate=true change about mode("overwrite") over JDBC?

If you can explain the generated WHERE clauses for question 2 from memory, you understand this better than most candidates.

---

If you want more practice like this, there's a free 10-question sample of Azure Databricks and PySpark interview questions with worked answers: free sample. The full set is in the Azure Databricks & PySpark Interview Pack.