Gaurang Bhatt

Posted on May 28

How I Added dbt Cloud to Coral — My Open Source Hackathon Journey

#dataengineering #opensource #showdev #sql

What is Coral?

Coral is an open-source tool that lets you query any API, database,
or file as SQL — no ETL, no warehouse, no glue code.

The Problem I Solved

dbt Cloud is used by thousands of data engineers. But there was no
way to query dbt Cloud jobs, runs, and environments via SQL. I built that.

What I Built

PR #627 — dbt Cloud Community Source

Added 3 tables to Coral:

dbt_cloud.jobs — all jobs with schedules and state
dbt_cloud.runs — run history with status codes
dbt_cloud.environments — deployment environments

The Code

# manifest.yaml (simplified)
name: dbt_cloud
backend: http
tables:
  - name: jobs
  - name: runs  
  - name: environments

Live Demo

-- Query 890 open PRs from Coral repo
SELECT number, title, state 
FROM github.pulls 
WHERE owner='withcoral' AND repo='coral' 
AND state='open' 
LIMIT 1000

Result: 890 rows in ~10 seconds. No ETL. Just SQL.

Status Codes I Learned

dbt Cloud run status codes:

1 = Queued
2 = Starting
3 = Running
10 = Success
20 = Error
30 = Cancelled

Challenges

Pagination — dbt Cloud uses offset pagination, had to implement correctly
Timestamps — ISO8601 format needed special handling in Coral DSL
Virtual columns — filter echo columns needed for debugging

What I Learned

How to write a Coral source manifest (DSL v3)
HTTP backend configuration
Open source contribution workflow — fork, branch, PR, review
Real API pagination patterns

The Result

PR #627 merged into withcoral/coral ✅

Now any Coral user can:

SELECT j.name, r.status, r.started_at
FROM dbt_cloud.runs r
JOIN dbt_cloud.jobs j ON j.id = r.job_id
WHERE r.status = 20  -- Error
ORDER BY r.started_at DESC
LIMIT 20;

Try It Yourself

coral source add --file sources/community/dbt_cloud/manifest.yaml
coral sql "SELECT * FROM dbt_cloud.jobs LIMIT 10"

DEV Community