DEV Community: Md Mohiuddin

How to Query Databricks from Salesforce Apex (Without Copying a Billion Rows)

Md Mohiuddin — Tue, 21 Jul 2026 09:05:23 +0000

The problem

We had a reference dataset in Databricks with 1B+ rows. Users needed to look up a handful of those rows from a Salesforce record page.

The obvious idea — sync it into a Salesforce custom object — falls apart fast:

Salesforce data storage is priced for business records, not reference datasets
The source refreshed monthly, so we'd re-load a billion rows every month
The copy immediately starts drifting from the source

So instead: query Databricks on demand. The user enters a key, we fetch the few matching rows over HTTP, display them, and store a small snapshot on the record.

This post covers exactly how to wire that up — and the four gotchas that aren't in the docs.

Architecture

Lightning Web Component  (user enters a search key)
  → Apex Controller
    → Apex Service
      → Databricks Client ──HTTPS──▶  Databricks SQL Statement Execution API
                            ◀── JSON rows

One important rule up front: all filtering happens in Databricks. Apex has a 6 MB
heap limit, so pulling rows into Salesforce to filter them there fails almost immediately. Push the WHERE clause down.

Why HTTP (and not an SDK)

Databricks ships SDKs for Python, Java, Go, and JS. You can't use any of them from Apex — Salesforce doesn't allow third-party libraries.

That costs you nothing. Every Databricks SDK is a wrapper around the same REST API.
POST /api/2.0/sql/statements is what the SDK sends anyway. You're doing by hand what the SDK does for you.

Step 1 — Collect three values from Databricks

You need exactly three things:

Value	Where to find it
Workspace URL	Your workspace address, e.g. `https://dbc-xxxxxxxx-xxxx.cloud.databricks.com`
Warehouse ID	SQL Warehouses → your warehouse → Connection details
Access token	Settings → Developer → Access tokens

💡 Tip: if someone hands you a JDBC/Python connection string, the warehouse ID is the last segment of the HTTP path:

http_path = "/sql/1.0/warehouses/abc123def456"
                                  ^^^^^^^^^^^^ this is your warehouse ID

For production, use a service principal token, not a personal access token. A PAT expires and dies with the person who created it.

Step 2 — Create the External Credential

In Salesforce Setup → Named Credentials → External Credentials → New:

Label / Name: Databricks
Authentication Protocol: Custom

Add a Principal:

Parameter Name: Sandbox (or Production)
Identity Type: Named Principal
Authentication Parameter: Name Token, Value = your Databricks token

Add a Custom Header:

Name	Value
`Authorization`	`Bearer {!$Credential.Databricks.Token}`

The Token in that formula must match your authentication parameter name exactly, including case.

Step 3 — Create the Named Credential

Setup → Named Credentials → New:

Label: Databricks Sandbox
Name: Databricks_Sandbox
URL: your workspace URL (no path)
External Credential: Databricks

⚠️ Gotcha #1: Allow Formulas in HTTP Header

Check the "Allow Formulas in HTTP Header" box.

If you don't, Salesforce sends the literal string Bearer {!$Credential...} instead of resolving it, and Databricks returns 403. The error gives you no hint that a checkbox is the cause. This one cost me an hour.

Step 4 — Grant principal access

Create a permission set with External Credential Principal Access for your principal (e.g. Databricks-Sandbox), and assign it to the running user.

Without this, callouts fail even though the credential looks correctly configured.

Step 5 — The Apex client

The API is asynchronous. You POST a statement; if the warehouse is cold, you get back PENDING and a statement_id to poll.

public with sharing class DatabricksClient {
  private static final String STATEMENTS_PATH = '/api/2.0/sql/statements';
  private static final String WAIT_TIMEOUT = '50s';
  private static final Integer MAX_POLLS = 15;

  private final String endpoint;      // 'callout:Databricks_Sandbox'
  private final String warehouseId;
  private final String tableName;

  public DatabricksClient(String endpoint, String warehouseId, String tableName) {
    this.endpoint = endpoint;
    this.warehouseId = warehouseId;
    this.tableName = tableName;
  }

  public List<Map<String, Object>> query(String keyA, String keyB) {
    HttpResponse response = post(buildBody(keyA, keyB));
    Map<String, Object> payload =
      (Map<String, Object>) JSON.deserializeUntyped(response.getBody());

    payload = awaitResult(payload);
    return toRows(payload);
  }

  private String buildBody(String keyA, String keyB) {
    return JSON.serialize(new Map<String, Object>{
      'warehouse_id' => warehouseId,
      'statement' =>
        'SELECT col_a, col_b, amount FROM ' + tableName +
        ' WHERE key_a = :keyA AND key_b = :keyB',
      'parameters' => new List<Object>{
        new Map<String, Object>{ 'name' => 'keyA', 'value' => keyA },
        new Map<String, Object>{ 'name' => 'keyB', 'value' => keyB }
      },
      'wait_timeout' => WAIT_TIMEOUT
    });
  }

  private HttpResponse post(String body) {
    HttpRequest request = new HttpRequest();
    request.setEndpoint(endpoint + STATEMENTS_PATH);
    request.setMethod('POST');
    request.setHeader('Content-Type', 'application/json');
    request.setTimeout(60000);
    request.setBody(body);
    return new Http().send(request);
  }
}

Always use bound parameters

Notice the parameters array. Never concatenate user input into the SQL string:

// ❌ SQL injection
'... WHERE key_a = \'' + userInput + '\''

// ✅ bound parameter
'parameters' => new List<Object>{
  new Map<String, Object>{ 'name' => 'keyA', 'value' => userInput }
}

Parsing the response

Databricks returns columns and rows separately. Build a name→index map from the manifest so column order changes can't silently break you:

private Map<String, Integer> columnIndex(Map<String, Object> payload) {
  Map<String, Integer> index = new Map<String, Integer>();
  Map<String, Object> manifest = (Map<String, Object>) payload.get('manifest');
  Map<String, Object> schema = (Map<String, Object>) manifest.get('schema');
  List<Object> columns = (List<Object>) schema.get('columns');

  for (Integer i = 0; i < columns.size(); i++) {
    Map<String, Object> column = (Map<String, Object>) columns[i];
    index.put((String) column.get('name'), i);
  }
  return index;
}

Rows then live in result.data_array as arrays of strings — everything comes back as
a string, so cast explicitly (Decimal.valueOf, Date.valueOf).

The gotchas nobody tells you

⚠️ Gotcha #2: `wait_timeout` maxes out at 50 seconds

The valid range is 5–50 seconds (default 10). Use 50s to minimise polling.

⚠️ Gotcha #3: Apex has no `sleep()` — so 50s is your real ceiling

This is the subtle one. Apex cannot pause between polls. Your polling loop fires all its requests back-to-back in a couple of seconds and gives up — it can't actually wait for a warehouse to finish starting.

So the only place Salesforce can genuinely wait is Databricks holding the initial POST open, capped at 50s. Apex's own callout budget (120s per callout) is irrelevant here.

If a query doesn't finish in 50 seconds, your user gets an error. Which leads to…

⚠️ Gotcha #4: cluster your table, or point lookups will time out

Querying a few rows from a billion-row Delta table only works if Databricks can skip the irrelevant files. That's what clustering does:

ALTER TABLE my_catalog.my_schema.my_table
  CLUSTER BY (key_a, key_b);

OPTIMIZE my_catalog.my_schema.my_table;

ALTER TABLE only records the rule — OPTIMIZE does the actual reorganisation. Run only the first and nothing gets faster.

Delta stores min/max stats per file, so a clustered lookup reads a few MB instead of the whole table — and stays roughly flat whether the table holds 1M or 1B rows.

Two follow-ups:

Re-run OPTIMIZE after each bulk load, or new unclustered rows degrade lookups over time. Better: enable Predictive Optimization.
Don't partition on a high-cardinality column. ~41k distinct values = ~41k tiny fragments, and per-file overhead makes reads slower.

Cold starts are real

A serverless SQL warehouse that has idled out takes 10–30 seconds to wake. Options:

Set auto-stop to 15–30 min so normal traffic keeps it warm
Tell users the first search of the day may be slower

Result size limits

Limit	Value
Databricks `INLINE` response	25 MiB
Databricks `EXTERNAL_LINKS`	100 GiB
Apex heap (synchronous)	6 MB ← binds first

Salesforce is ~4× tighter than Databricks, so you'll hit the heap limit long before the API limit. Heap use also multiplies as you deserialize. Keep result sets small — which is the whole point of pushing filters down.

Testing

Inject an HTTP sender so tests never make real callouts:

@IsTest
private static void query_parsesResponse() {
  Test.setMock(HttpCalloutMock.class, new MyMock());

  Test.startTest();
  List<Map<String, Object>> rows = newClient().query('a', 'b');
  Test.stopTest();

  Assert.areEqual(2, rows.size());
}

Worth covering explicitly:

A SUCCEEDED response parses correctly
A PENDING response triggers a poll
A non-200 throws
A FAILED statement state throws (note: HTTP 200 but the query failed — these are different failure modes and deserve different exception types)

Wrapping up

The integration itself is ~150 lines of Apex. The hard parts were never the code — they were:

The "Allow Formulas in HTTP Header" checkbox
Apex having no sleep(), making 50s the real ceiling
Clustering being mandatory, not an optimisation
Remembering the 6 MB heap, not the 25 MiB API limit, is what binds

If you keep filtering in Databricks and your table clustered, point lookups land in 1–2 seconds and stay there as the table grows.

Have you built something similar? I'd be curious whether you went direct like this or put middleware in between — drop a comment.

🐳 Docker Basics Every Beginner Should Know

Md Mohiuddin — Sun, 19 Jul 2026 16:01:06 +0000

When I started learning Docker, I quickly realized that understanding a few core concepts and commands makes working with containers much easier. Instead of memorizing commands, I wanted to understand what Docker does and why each command is useful.

In this article, I'm sharing my beginner notes. If you're just getting started with Docker, this guide should help you build a solid foundation.

What is Docker?

Docker is a containerization platform that allows you to package an application together with all of its dependencies. This ensures the application runs consistently across different environments—whether it's your local machine, a testing server, or production.

One of Docker's biggest advantages is the famous phrase:

"It works on my machine." → With Docker, it works on every machine.

Docker Architecture

Before learning the commands, it's helpful to understand how Docker works behind the scenes.

Docker Client

The Docker Client is the command-line interface (CLI) that we use every day.

Whenever we run commands like:

docker run
docker ps
docker pull

the Docker Client sends those requests to the Docker Daemon.

Docker Daemon

The Docker Daemon (dockerd) runs in the background.

It is responsible for:

Building images
Running containers
Managing images
Managing networks
Managing volumes

Think of it as the engine that powers Docker.

Docker Image

A Docker Image is a read-only template that contains everything required to run an application:

Application code
Runtime
Libraries
Dependencies
Configuration

Images are used to create containers.

Docker Container

A Container is a running instance of a Docker image.

Containers are:

Lightweight
Isolated
Portable
Fast to start

You can create multiple containers from the same image.

Docker Registry

A Docker Registry is where Docker images are stored and shared.

The most commonly used public registry is Docker Hub, where thousands of official and community images are available.

What is Containerization?

Containerization means packaging an application together with all of its dependencies so that it runs the same way everywhere.

Instead of worrying about different operating systems or missing libraries, Docker provides a consistent environment.

Essential Docker Commands

1. Check Docker Installation

Check Docker Version

docker -v

Displays the installed Docker version.

Show Docker Client & Daemon Version

docker version

Shows both the Docker Client and Docker Daemon versions.

Display Docker Information

docker info

Displays detailed information about your Docker installation.

2. Working with Images

Download an Image

docker pull hello-world:latest

Downloads the image from Docker Hub.

Note: If the image already exists locally, Docker won't download it again.

View Local Images

docker images

docker image ls

Lists all images stored locally.

Remove an Image

docker image rm <image_name_or_id>

docker rmi <image_name_or_id>

Deletes an image from your local machine.

3. Creating and Running Containers

Create a Container

docker create ubuntu

Creates a container but does not start it.

Start an Existing Container

docker start <container_id>

Starts a previously created container.

Create and Run a Container

docker run hello-world

If the image doesn't exist locally, Docker downloads it first and then creates and starts the container.

Run an Interactive Ubuntu Container

docker run -it ubuntu

Downloads the Ubuntu image (if needed), starts the container, and opens an interactive terminal where you can execute Linux commands.

Run a Container in Detached Mode

docker run -d nginx

Runs the container in the background.

4. Managing Containers

View Running Containers

docker ps

Lists only running containers.

View All Containers

docker ps -a

Lists both running and stopped containers.

Stop a Container

docker stop <container_id>

Stops a running container.

Restart a Container

docker restart <container_id>

Restarts a container.

Remove a Container

docker rm <container_id>

Deletes a stopped container.

5. Execute Commands Inside a Running Container

Open a Bash shell inside a running container:

docker exec -it <container_id> bash

For example, if you're running Redis:

docker exec -it <container_id> redis-cli

6. Viewing Logs

Display Container Logs

docker logs <container_id>

Shows logs generated by the container.

Follow Logs in Real Time

docker logs -f <container_id>

Streams logs as they are generated.

7. Cleaning Up Docker Resources

Remove Unused Resources

docker system prune

This removes:

Stopped containers
Unused networks
Dangling images
Build cache

Remove All Unused Images

docker system prune -a

Removes all unused images, stopped containers, unused networks, and build cache.

Quick Summary

Command	Purpose
`docker -v`	Check Docker version
`docker version`	Show Docker Client & Daemon version
`docker info`	Display Docker information
`docker pull`	Download an image
`docker images`	List local images
`docker run`	Create and run a container
`docker create`	Create a container
`docker start`	Start an existing container
`docker ps`	List running containers
`docker ps -a`	List all containers
`docker exec -it`	Execute commands inside a running container
`docker logs`	View container logs
`docker image rm`	Remove an image
`docker rm`	Remove a container
`docker system prune`	Clean up unused Docker resources

Final Thoughts

Docker is one of the most important tools in modern software development and DevOps. While there are many advanced topics to explore—such as Dockerfiles, Docker Compose, volumes, networking, Kubernetes, and container orchestration—mastering these fundamentals will make your learning journey much smoother.

I'm currently learning Docker as part of my DevOps roadmap, and I'll continue sharing practical notes and examples as I progress.

If you have suggestions, best practices, or beginner tips, feel free to share them in the comments.

Happy Learning!