DuckDB: Simplifying data analytics for happier data professionals and reduced hosting costs

Simon Pantzare — Mon, 24 Apr 2023 07:38:47 +0000

The world of data analytics is dominated by large platforms that only sometimes offer the flexibility of using your own hardware. When possible, it's often meant to run on clusters, not your laptop.

Enter DuckDB, an in-process database management system perfect for analytics while keeping your favorite tools and hardware in play. No more waiting for jobs to complete on remote clusters! Plus, it scales well beyond the needs of many organizations, even on a tight budget.

Intrigued, I tested DuckDB with a 107 GB, 5 billion row dataset of Parquet files, a standard data lake format, and looked into what it could mean for hosting bills:

DuckDB offers an enticing alternative for those seeking a more flexible and cost-effective data analytics solution. Give it a try and see the difference it can make for your organization!

ClojureScript on Cloudflare Workers

Simon Pantzare — Sun, 18 Apr 2021 19:36:05 +0000

I am a long-time user of AWS. Lambda is great, but for many use cases I feel like it brings too much complexity. You need to be aware of the surrounding AWS landscape and how Lambda ties into IAM, CloudWatch Events if you run cron-like jobs or API Gateway if you want to expose HTTP endpoints. Serverless framework and similar make things easier by hiding details. The hoops you need to go through to avoid cold starts are annoying.

I am yet to use Cloudflare Workers professionally and can't speak to how the service fares in practice but I'm excited by the operational simplicity it promises. I'm not using Durable Objects in this post but access to persistent memory in V8 isolates at edge nodes seems like a strong approach to scale out web services. It seems like you could avoid introducing many cache layers commonly found in web services.

Clojure I'm just fascinated by. I've been keeping tabs on it but have never done anything substantial with it. I have worked mostly in statically typed languages over the last years in code bases that follow an imperative style. While I appreciate types and compile-time errors and how these things bring at least some level of structure to any pile of code, there seems to be a sweet-spot where, if you can keep things simple enough, the type system adds more overhead than it brings clarity. I also appreciate that in Clojure, immutability is something that you need to actively opt out of.

Let's get started.

Project setup and worker source

First, we create an empty npm project that depends on shadow-cljs.

mkdir lispmachine && cd $_

package.json:

{
  "name": "lispmachine",
  "version": "0.0.1",
  "private": true,
  "dependencies": {
    "shadow-cljs": "^2.12.5"
  },
  "scripts": {
    "build": "shadow-cljs compile worker"
  }
}

shadow-cljs.edn:

{:source-paths ["src/main"]
 :dependencies []
 :builds       {:worker {:target     :esm
                         :output-dir "dist"
                         :modules    {:worker {:exports {default lispmachine.worker/eventHandlers}
                                               }}}}}

Next, add our worker code. src/main/lispmachine/worker.cljs:

(ns lispmachine.worker)

(def eventHandlers
 #js {:fetch
      (fn [request env]
          (js/Response. (str "Clojure <3 Cloudflare Workers!\n\n" (js/JSON.stringify request))))
      })

Now we should be able to compile.

npm install
npm run build

Cloudflare Workers setup

Next, install and configure Wrangler if you haven't already.

You need to be on 1.16.0 or later with support for ES modules. For now, to use ES modules on Workers you also need to opt-in to the Durable Objects open beta. (You can avoid ES modules and add a bundler but I didn't want that extra step. If you do, esbuild will do the trick.)

Add wrangler.toml:

name = "lispmachine"
workers_dev = true
type = "javascript"

[build]
cwd = "./dist"
command = "npm run build"
upload.format = "modules"
upload.main = "worker.js"
upload.rules = [{ type = "ESModule", globs = ["**/*.js"] }]

I had to add upload.rules because shadow-cljs generates files with .js extensions and Wrangler assumes such files are CommonJS. I also had to use cwd = "./dist" after getting errors about the main module not being found with upload.main = "./dist/worker.js".

Now we are ready to take this live and see that it works!

CF_ACCOUNT_ID=... wrangler publish

curl https://lispmachine.yoursubdomain.workers.dev

Clojure <3 Cloudflare Workers!

{"fetcher":{},"redirect":"manual","headers":{},"url":"https://lispmachine.yoursubdomain.workers.dev/","method":"GET","bodyUsed":false,"body":null}

(If anyone knows why request.cf isn't part of the output from (js/JSON.stringify request) please let me know.)

DEV Community: Simon Pantzare

DuckDB: Simplifying data analytics for happier data professionals and reduced hosting costs

ClojureScript on Cloudflare Workers

Project setup and worker source

Cloudflare Workers setup