Clojure Query DSLs

Domain Specific Languages (DSLs) are a popular technique for writing database queries. There are a few reasons for this, including:

• They help ensure queries are syntactically correct
• They make programmatic construction of queries more tractable
• Fragments of queries are easier to reuse

Some languages implement DSLs in libraries, some allow the language to be extended to include the DSL, and some can use already existing syntax and data structures to implement the DSL. While most languages can implement DSLs in libraries, Clojure also has the option of extending the language via macros, as well as providing syntax for regular data structure that can also be convenient to use.

Macros

While basic macros are relatively easy to write in Clojure, they can become quite complex. In general, when it comes to Clojure macros, I think Sandra Sierra's 2010 advice holds up well: You do not write macros.

That said, some libraries may use macros to make writing queries easier for developers. This can make it easier to incorporate names and labels into expressions that would otherwise lead to errors on "unbound" values. Many Clojure DSL libraries eschew this, since keywords often work just as well. As an example, a SELECT expression for reading the name and age columns of a table would need a macro if a developer wanted to write:

(select name age)

However, the complexity of a macros can be avoided by switching to keywords instead:

(select :name :age)

Structures

Clojure code is almost always written to use the regular data structures that are built into the syntax of the language. These are:

• Maps: {key value …} Also called a "Dictionary" in languages like Python.
• Sets: #{data …}
• Vectors: [data …] Called "Lists" in Python.
• Lists: (data …) Implemented as Linked Lists.

Lists are a little different, as they are "executed" by defautl in Lisp dialect like Clojure. This is avoided by introducing the list with a ' quote character. e.g. '(1 2 3)

Since these structures are part of the language, they can be a simple way to build a DSL for querying. For instance, Sean Corfield's HoneySQL can represent an SQL query using a map and vectors:

{:select [:a :b :c]
 :from   [:foo]
 :where  [:= :foo.a "baz"]}

Datomic does something similar, using either a map or a vector for the queries:

'[:find ?title 
  :where [?e :movie/title ?title] 
         [?e :movie/release-year 1985]]

There is an issue here though. Unlike previous examples, this last query is using variables (marked with a ? prefix) as a part of the query language. This is especially common in graph languages like Datomic or SPARQL.

Quoting

Using a : to turn these variables into keywords makes it harder to distinguish variables from actual values stored in the database (since Datomic stores keywords directly, and SPARQL libraries assume keywords to be CURIEs). Instead, Clojure Symbols are used. The problem is that symbols are the mechanism that Clojure uses for associating data with a name, so using a symbol usually results in Clojure looking for that data, which will be an error if the symbol is not bound, and inserts a value where you wanted a variable if it is found. This is avoided using the ' quote character.

To illustrate this, let's look at that Datomic query again, this time without the quote. I'll show what happens at a REPL (the Clojure prompt), where the prompt includes the current namespace (user by default, though it can be something else):

user=> [:find ?title 
        :where [?e :movie/title ?title] 
               [?e :movie/release-year 1985]]
Syntax error compiling at (REPL:0:0).
Unable to resolve symbol: ?title in this context

user=> (def ?title "not a variable")
user=> (def ?e "also not a variable")
user=> [:find ?title 
        :where [?e :movie/title ?title] 
               [?e :movie/release-year 1985]]
[:find "not a variable" :where ["also not a variable" :movie/title "not a variable"] ["also not a variable" :movie/release-year 1985]]

This is read, but we can see that the use of the symbols has placed their saved values into the query, rather than a variable like we wanted.

We can avoid this problem by quoting the symbols that we want to keep as symbols:

user=> [:find '?title 
        :where ['?e :movie/title '?title] 
               ['?e :movie/release-year 1985]]
[:find '?title :where ['?e :movie/title '?title] ['?e :movie/release-year 1985]]

We can also quote entire structures that contain multiple variables:

user=> '[:find ?title 
         :where [?e :movie/title ?title] 
                [?e :movie/release-year 1985]]
[:find ?title :where [?e :movie/title ?title] [?e :movie/release-year 1985]]

Note that the result is printing the data structure, and not trying to evaluate it. Because of this, the symbols are not printed with a quote.

Mixed Symbols

Quoting entire structures makes it easy to include multiple symbols, but it also makes it hard to include values from a program. For instance, a user may be asking to get all titles from a year that they provide in a user-interface:

user=> (let [release-year (get-user-input)] ;; user input 1985
         '[:find ?title 
           :where [?e :movie/title ?title] 
                  [?e :movie/release-year release-year]])
[:find ?title :where [?e :movie/title ?title] [?e :movie/release-year release-year]]

That's put a symbol into the query when we wanted a number associated with that symbol.

There are lots of ways to address this. One is to only quote the parts we need to:

user=> (let [release-year (get-user-input)] ;; user input 1985
         [:find '?title 
          :where '[?e :movie/title ?title] 
                  ['?e :movie/release-year release-year]])
[:find ?title :where [?e :movie/title ?title] [?e :movie/release-year 1985]]

This works, but it can (and frequently does, at least in my code) lead to lots of quote characters everywhere, which can be a bit bug prone too (LLMs will usually catch it, but I'm here to discuss how WE write code, not LLMs).

Another way is to bind the names to symbols (binding to a symbol with the same name would reduce confusion):

(let [release-year (get-user-input)  ;; user input 1985
      ?e (symbol "?e")
      ?title (symbol "?title")]
   [:find ?title 
    :where [?e :movie/title ?title] 
           [?e :movie/release-year release-year]])

But now we're asking the developer to increase their code significantly. That's not really helpful.

For completeness, I'll also mention that sometimes it works to build the parts you need, and create the query structure with code:

(let [release-year (get-user-input)]  ;; user input 1985
   (conj '[:find ?title :where [?e :movie/title ?title]]
          ['?e :movie/release-year release-year]])

This is sometimes useful, but is too cumbersome for basic querying.

Unquoting

Another option is the "unquote". This tells the Clojure reader that is reading quoted data that the next item is not to be considered as quoted. This is done with the ~ character. However, it does not work quite as you might expect.

For simplicity, I will reduce the evaluation to just the last part:

user=> (def release-year 1985)
user=> '[?e :movie/release-year ~release-year]
[?e :movie/release-year (clojure.core/unquote release-year)]

Unfortunately, this has shown us what the ~ unquoting gets translated to. Quoting actually gets translated like that too. We just haven't seen it before:

user=> '['?e :movie/release-year ~release-year]
[(quote ?e) :movie/release-year (clojure.core/unquote release-year)]

Instead, we need to use a different kind of quoting: syntax quoting. This is done with a single "back-quote" or .
clojure
user=> (def release-year 1985)
user=> `[?e :movie/release-year ~release-year]
[user/?e :movie/release-year 1985]

This embedded the value ofrelease-yearas we wanted, but it has also change the?esymbol. Now it tells us that it's the the symbol?ein the namespaceuser`. That's the current namespace, so that's correct, but we want to embed the symbol without the namespace.

This is a tricky form. We want to "unquote" from the syntax-quote, but then we want to immediately "quote" again:
clojure
user=>[~'?e :movie/release-year ~release-year]
[?e :movie/release-year 1985]
`

Going back to the complete query, we can see the full form:
clojure
(let [release-year (get-user-input)] ;; user input 1985
[:find '?title
:where '[?e :movie/title ?title]
[~'?e :movie/release-year ~release-year]])
`

This is not entirely satisfactory, and it explains why macros may be attractive, but it does show an approach.

Wrap Up

This post demonstrates some of the approaches of using Domain Specific Languages (DSLs) in Clojure, focusing on data structures to represent database queries. Query languages like Datomic and SPARQL use symbols in their queries, and we looked at a few ways that these can be embedded into a query structure.