DEV Community: Toni

Clojure for Node Developers: File System

Toni — Thu, 23 Nov 2023 06:10:11 +0000

This will be a straightforward post with Javascript to Clojure (on JVM ) examples of filesystem operations: listing, reading, writing, renaming, and deleting files. Let me know if you're interested in how to use Node packages from Clojurescript.

So, let's get to it.

In Javascript, we'd import the fs-module.

const fs = require('node:fs/promises')

I'll be running all of the Javascript blocks inside a (async () => { ...code... })() closure.

In Clojure, we'd reach out for clojure.java.io

(require '[clojure.java.io :as io])

Node Filehandle and Java File

In Javascript, we can get a reference to a file with a Filehandle.

let directory = await fs.open(".");

console.log(directory)

// FileHandle {
// close: [Function: close],
// [Symbol(kHandle)]: FileHandle {},
// [Symbol(kFd)]: 20,
// [Symbol(kRefs)]: 1,
// [Symbol(kClosePromise)]: null
// }

FileHandle can be either a directory or a file.

let directoryStats = await fs.stat(".");
directoryStats.isFile(); // false
directoryStats.isDirectory(); // true

let fileStats = await fs.stat("files/a.txt");
fileStats.isFile()); // true
fileStats.isDirectory(); // false

The basic building block in Clojure is java.io.File.

User interfaces and operating systems use system-dependent pathname strings to name files and directories. This class presents an abstract, system-independent view of hierarchical pathnames.

Docs: java.io.File

(io/file ".")
;; => #object[java.io.File 0x54162701 "."]

Similarly to Filehandle in JS, java.io.File can be either a directory or a file. The File has isDirectory and isFile methods that can be called via Java-interop.

(.isDirectory (io/file "."))
;; => true

(.isFile (io/file "."))
;; => false

(.isDirectory (io/file "deps.edn"))
;; => false

(.isFile (io/file "deps.edn"))
;; => true

Let's see what other properties the java.io.File has with the bean function.

bean

Takes a Java object and returns a read-only implementation of the map abstraction based upon its JavaBean properties.

Source

(bean (io/file "."))
;; => 
{:path ".",
 :freeSpace 146245873664,
 :parent nil,
 :directory true,
 :parentFile nil,
 :name ".",
 :file false,
 :canonicalFile
 #object[java.io.File 0x4f2abe7e "/home/tvaisanen/projects/tmp"],
 :absolute false,
 :absoluteFile
 #object[java.io.File 0x5594224b "/home/tvaisanen/projects/tmp/."],
 :hidden true,
 :class java.io.File,
 :canonicalPath "/home/tvaisanen/projects/tmp",
 :usableSpace 124331769856,
 :totalSpace 429923737600,
 :absolutePath "/home/tvaisanen/projects/tmp/."}

Listing Files

Given we have the following file structure.

 tree files
files
 a.txt
 b.txt
 c.txt
 nest
     a.txt

In Javascript, we'd typically write.

let dir = await fs.readdir("files");
console.log(dir);
// ['a.txt', 'b.txt', 'c.txt', 'nest']

And in Clojure, we'll take the folder we want to list as java.io.File and list all the files with file-seq.

(file-seq (io/file "files"))
;; =>
(#object[java.io.File 0x7e2b40d7 "files"]
 #object[java.io.File 0x22aaad8f "files/a.txt"]
 #object[java.io.File 0x3730fe92 "files/b.txt"]
 #object[java.io.File 0x62aaf1b "files/c.txt"]
 #object[java.io.File 0x22236af1 "files/nest"]
 #object[java.io.File 0x55126def "files/nest/a.txt"])

I'd expect this to return files similarly fs.readdir, but now we have the nested files in the listing. Let's see why that happens and if we could get it to work similarly to the listing in Javascript.

If we take a look at the file-seq source code

(defn file-seq
  "A tree seq on java.io.Files"
  {:added "1.0"
   :static true}
  [dir]
    (tree-seq
     (fn [^java.io.File f] (. f (isDirectory)))
     (fn [^java.io.File d] (seq (. d (listFiles))))
     dir))

The docstring says that the function returns a tree seq, which sounds like a non-flat structure. And if we look deeper into tree-seq function and get the docstring.

Returns a lazy sequence of the nodes in a tree, via a depth-first walk. ...

And here's the explanation of what is happening: "a depth-first walk." This function will go first till the end of the file tree and start coming back towards the root directory. The first argument is a function that returns a boolean value to decide whether the node (read file or directory) needs to be traversed (read looked into). If it returns true, the same file is passed to the second function to list its nodes (read files and subdirectories). Based on this, the tree-seq will do a depth-first search and, therefore, read all the subdirectories.

You might have already noticed that the second function argument calls (. d (listFiles)) to get the files for our directory d. Let's use this to mimic the JS version readdir.

(seq (. (io/file "files") listFiles))
;; =>
(#object[java.io.File 0x356bf4d3 "files/a.txt"]
 #object[java.io.File 0x13c28931 "files/b.txt"]
 #object[java.io.File 0x444349ab "files/c.txt"]
 #object[java.io.File 0x2e64f09c "files/nest"])

At this point, the expected result is almost identical to the Javascript's fs.readdir. We have the files listed in the directory, but instead of file names, we have java.io.File instances. Let's iterate over the files and once again use Java-interop with the getName method.

(for [file (seq (. (io/file "files") listFiles))]
  (.getName file))
;; => ("a.txt" "b.txt" "c.txt" "nest")

The fs.readdir function will throw if called with a filename that is not a directory. If we try to create a file sequence from a java.io.File that is not a directory, we get only the file itself in a list.

(file-seq (io/file "files/a.txt"))
;; => (#object[java.io.File 0x37eeb5e7 "files/a.txt"])

Let's combine what we've learned from the previous examples into a new function read-dir that takes a path, checks that it is not a file, and then lists the files and directories in the folder but not the subpages.

(defn read-dir [d]
  (let [directory (io/file d)]
    (when (. directory isFile)
      (throw (ex-info "Path is not a directory" {:path directory})))
    (doall
     (for [file (seq (. directory listFiles))]
       (.getName file)))))

(read-dir "files")
;; => ("a.txt" "b.txt" "c.txt" "nest")

(read-dir "files/a.txt")
;; Unhandled clojure.lang.ExceptionInfo
;; Path is not a directory
;; {:path #object[java.io.File 0x7c7a2da9 "files/a.txt"]}

That's enough on listing files for now. Next, let's look into reading the files.

Read Files

Let's see the content for a couple of the example files.

 cat files/a.txt
AAAAA
 cat files/b.txt
BBBB
1111
2222
3333

In Javascript, we'd typically use the fs.readFile.

let content = await fs.readFile("files/a.txt", {encoding: "utf8"});
console.log(content);
// AAAAA

In Clojure, we default to slurp.

(slurp (io/file "files/a.txt"))
;; => "AAAAA\n"

Slurp also works with the filename since if the argument is a string; it's first coerced (read interpreted as) as a URI and second as a local file.

(slurp "files/a.txt")
;; => "AAAAA\n"

Both fs.readdir and slurp read the file into memory at once; therefore, you might want to avoid it in production!

The more responsible way is to read the files as streams line by line.

let file = await fs.open('files/b.txt',);

for await (const line of file.readLines()) {
    console.log(line);
}
// BBBB
// 1111
// 2222
// 3333

In Clojure, we can use the java.io.Reader to create a buffered reader for streaming over the files. slurp uses the same mechanism, but instead of reading a file line by line, it reads it all simultaneously.

(io/reader "files/a.txt")
;; => #object[java.io.BufferedReader 0x4754cc18 "java.io.BufferedReader@4754cc18"]

We can open this stream (read file) to create a lazy line sequence that can be processed one line at a time.

(with-open [reader (io/reader "files/b.txt")]
  (doall (for [line (line-seq reader)]
           (str "read: " line))))
;; => ("read: BBBB" "read: 1111" "read: 2222" "read: 3333")

Next, to writing files.

Writing Files

With Javascript, we'd typically do this with fs.writeFile or use fs.createFileStream for extra control.

await fs.writeFile("files/write-with-js.txt", "writing from JS");

let content_01 = await fs.readFile("files/write-with-js.txt", 
                                   {encoding: "utf8"});
console.log(content); 
// writing from JS

In Clojure, similarly to slurp we have spit to write data.

(spit "files/new.txt" "New content here")
;; => nil
(slurp "files/new.txt")
;; => "New content here"

By default, spit overrides the file with the given content. To add to the file, use arguments :append true.

(spit "files/new.txt" "Append to file")
;; => nil
(slurp "files/new.txt")
;; => "Append to file"
(spit "files/new.txt" " Try again" :append true)
;; => nil
(slurp "files/new.txt")
;; => "Append to file Try again"

spit wraps the java.io.writer, which can be used directly. Similarly to slurp java.io.writer replaces the file content if :append true is not passed as an option.

(with-open [writer (io/writer "files/other.txt")]
  (.write writer "text here"))

(slurp "files/other.txt")
;; => "text here"

(with-open [writer (io/writer "files/other.txt" :append true)]
  (.write writer " more text"))
;; => nil

(slurp "files/other.txt")
;; => "text here more text"

Let's go through a couple of more use cases.

Renaming Files

In Javascript, we rename files with fs.rename.

await fs.rename("files/c.txt", "files/d.txt");

In Clojure, we use the renameTo method.

(.renameTo (io/file "files/new.txt")
           (io/file "files/newer.txt"))
;; => true
(.isFile (io/file "files/new.txt"))
;; => false
(.isFile (io/file "files/newer.txt"))
;; => true

Deleting Files

In Javascript, we delete files with fs.unlink

 await fs.unlink("files/d.txt");

In Clojure, we have clojure.java.io/delete-file for the same task.

(.isFile (io/file "files/other.txt"))
;; => true
(io/delete-file "files/other.txt")
;; => true
(.isFile (io/file "files/other.txt"))
;; => false

Conclusion

Clojure can do the same tasks as Javascript on the filesystem level since clojure.java.io provides utility functions as a layer on top of java.io classes. If there's no function matching your need, you can use Java-interop by, for example, using the java.io.File methods. If you hit a roadblock, check the following resources for more hints.

Once again, I hope you found this helpful. Feel free to reach out and let me know what pain points you might have encountered when learning Clojure and what type of posts would help make the jumpsocial links in the menu.

Thank you for reading.

Take Your Linting Game to the Next Level!

Toni — Tue, 14 Nov 2023 18:25:22 +0000

TLDR: It is possible to declare types for Clojure functions and make them available to the library users by creating clj-kondo configs (type exports). The configs can be created with Malli by collecting function schemas and then transforming these into clj-kondo types in a config file. If the config files are committed into resources, they can be imported by the host project when used as a dependency. Having Malli schemas has the added benefit of enabling the run-time checks via Malli instrumentation.

I like imagining an ecosystem where there'd be a clj-kondo config available for the most used libraries. I thought I would chime in my two cents on spreading the know-how to spread the word on how to do this. I'm by no means an expert on the topic, but I've learned a thing or two that might also be helpful to others.

Suppose you are unfamiliar with Malli and clj-kondo and how they can help you with static code analysis. In that case, I recommend first reading Data Validation in Clojure and Typescript Like Intellisense for Clojure Functions With Malli for background.

This time, I want to dig deeper into how these tools can improve the developer experience by making the types available to our code's human and CI consumers. I don't know if this is already yesterday's news, but this trick is worth knowing.

Let's explore the topic by creating a library, publishing it to Github, then using it as a dependency for another project, and making the library types available to benefit from the extended clj-kondo features.

Publish Library With Types

The only required dependency is Malli, which will be used to generate the clj-kondo types.

Note that in a "real" project Malli would be added as an extra dependency for an alias when creating the types if the library itself is not depending on Malli. If this is something that doesn't make sense to you let me know in the comments.

{:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        metosin/malli {:mvn/version "0.13.0"}}}

My library will have two functions: adding (x,y) coordinate points and rendering points as a string. If you need help defining the schemas for your use case, check out Malli function schema docs for reference.

(ns tvaisanen.types-export-sample.core
  "Example on how to create and export type clj-kondo
  type definitions that can be installed in a similar way
  that @types/lib-name is used in Typescript.")

(def Point [:map
            [:x :int]
            [:y :int]])

(defn add-points
  {:malli/schema [:=> [:cat Point Point] Point]}
  [p1 p2]
  (merge-with + p1 p2))

(defn render-point
  {:malli/schema [:=> [:cat Point] :string]}
  [p]
  (str p))

The only difference from what you'd typically write is the added metadata key pair. This will be enough for us to get the benefits in later stages when we want to provide the type hints for the library users. Let's take a look at how to make that happen next.

Export Configuration

Exporting is the process of creating the library's clj-kondo configuration under the resources folded as the documentation instructs. To do this, I created a naive exporting function that:

Also collects the malli/schema schemas from the functions,
generates the type definitions under the Malli clj-kondo-types,
copies the config.edn files into resources,
and cleans the clj-kondo-types cache.

(require '[malli.dev :as dev]
         '[clojure.java.io. :as io])

(defn export-types []
  ;; collect schemas and start instrumentation
  (dev/start!)

  ;; create export file
  (def export-file
    (io/file "resources/clj-kondo/clj-kondo.exports/tvaisanen/export-types-sample/config.edn"))

  ;; make parents if not exist
  (io/make-parents export-file)

  ;; copy the configs
  (io/copy
   (io/file ".clj-kondo/metosin/malli-types-clj/config.edn")
   export-file)

  ;; clear the cache and stop instrumentation
  (dev/stop!))

If we inspect the .clj-kondo cache folder, we can see that there's a config for malli-types-clj which is used to store the temporary cache while the Malli instrumentation is active (dev/start!) and it's cleared on (dev/stop!).

 tree .clj-kondo
.clj-kondo
 metosin
     malli-types-clj
         config.edn

What we just did with the export function was copy this config.edn file from the clj-kondo cache into our libs resources for persistence.

 tree resources
resources
 clj-kondo
     clj-kondo.exports
         tvaisanen
             export-types-sample
                 config.edn

5 directories, 1 file

This is the folder structure that clj-kondo expects to see when it's looking for available configurations to be imported. In my case the config.edn looks like this.

{:linters
 {:unresolved-symbol {:exclude [(malli.core/=>)]},
  :type-mismatch
  {:namespaces
   {export-types-sample.core

    {render-point
     {:arities
      {1 {:args [{:op :keys,
                  :req {:x :int,
                        :y :int}}],
          :ret :string}}},

     add-points
     {:arities
      {2 {:args [{:op :keys,
                  :req {:x :int,
                        :y :int}}
                 {:op :keys,
                  :req {:x :int,
                        :y :int}}],
          :ret {:op :keys,
                 :req {:x :int, :y :int}}}}}}}}}}

From here, we can see that argument and return types are defined for two functions render-point and add-points that have. This tells clj-kondo how the function arguments and return value should look when it's doing the static code analysis.

We are almost done with the first step: creating our library with typehints. Next, commit and push the changes to your Git repo. Here's mine for a reference.

Load the Published Library as a Dependency

Now that we have "published" a library with clj-kondo configs, we can use this as a dependency for another project.

{:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}

        metosin/malli {:mvn/version "0.13.0"}

        tvaisanen/export-types-sample
        {:git/sha "29f9b259975bde304300e4ca69c70d61622cabab"
         :git/url "https://github.com/tvaisanen/export-types-sample.git"}}}

If you didn't use Github see the available gitlib config options from the docs.

When we fetch the source code for the dependencies, we receive the resources (read configs) in our local cache and the malli function schemas within the source code. We don't need to install additional libs or types to access them. Next, we must collect these configs to our project's .clj-kondo cache. Clj-kondo docs provide the steps on how to do this. Remember to create the .clj-kondo folder if it doesn't exist before copying the configs.

First, we copy the configs.

 clj-kondo --lint "$(clojure -Spath)" --copy-configs --skip-lint

Imported config to .clj-kondo/tvaisanen/export-types-sample. 
To activate, add "tvaisanen/export-types-sample" to 
:config-paths in .clj-kondo/config.edn.

Then update the .clj-kondo/config.edn (you might need to create the file) as instructed by clj-kondo output.

{:config-paths ["tvaisanen/export-types-sample"]}

And I am then populating the cache with.

$ clj-kondo --lint $(clojure -Spath) --dependencies --parallel

After this, we should be good to jump into the editor and see if everything worked as expected.

Benefits of Types in the Editor

I created a core namespace for the new project where I've required the library I set up. You can see that when add-points is called with an invalid sample/Point it tells us that we are missing a required key :y. Without the extra type config, we should have an issue with "duplicate key :x", so it looks like the lib configs trump the default linting issues in priority.

As we extend the code, we can see that add-points return a value that the render-point is okay with according to clj-kondo, but if we pass a map that is not a sample/Point we get another linting error on the screen.

These red squiggly lines can save you a lot of headaches by letting you know where the types don't match the expected value. But this is just the first benefit that comes from using the types. So far, we haven't looked into how Malli expands on this.

Benefits of Types in the REPL

If a library has Malli schema function definitions, we can start instrumentation (monitoring with what values the functions are called and what they return) and get run-time type checking. In practice, this means we get a report on "invalid function call" in the REPL every time a function is called with unexpected arguments.

Let me demonstrate this.

We can use this to our advantage since we created the library with the malli schemas instead of writing the type hints manually into the clj-kondo config. Let's require malli.dev in our namespace and run dev/start! as we did when we first exported the types. Now, try running the sample/render-point with an invalid point argument.

When we evaluate line 19 while the malli instrumentation is active, we can find something new in the REPL, a schema error nicely documenting what is happening. The REPL provides helpful information about what went wrong with the function call.

Of course, we must remember that if there are a lot of functions being called, having the instrumentation on all the time can have a performance penalty.

Just keep in mind when running dev/start! the types are created into .clj-kondo/metosin/malli-types-clj/config.edn and when the dev/stop! is run. These types will be cleaned up.

Benefit of Types in the Terminal / CI

Last but least, we can also use the generated types outside our editor by using clj-kondo from the command line.

 clj-kondo --lint "src"
src/core.clj:6:2: error: Missing required key: :y
src/core.clj:6:8: error: duplicate key :x
src/core.clj:13:22: error: Missing required key: :x
src/core.clj:13:22: error: Missing required key: :y
linting took 5ms, errors: 4, warnings: 0

Adding this step can help keep many unnecessary bugs out of production when added to the continuous integration checks.

Where to Contribute

If you find that your particular use case is not supported for some reason. Providing patches is encouraged. Malli defines the malli->clj-kondo transformations in malli.clj-kondo namespace, which is where you want to look if you want to learn how the transformation process works or if you're going to extend the functionality. On the clj-kondo side, read the status of types and this source code file listing the available type mappings.

Conclusion

That was probably a lot to digest in one go. It took me some time to understand the relationship between clj-kondo and Malli, but slowly and surely, it's making more sense. I wanted to write about this topic because I think if more people knew how to add the types to their projects, it would improve the developer experience across the whole ecosystem (at least, this is what I'd like to imagine).

This is somewhat what happened with Javascript and Typescript. Little by little, the majority of the libs started offering type definitions. I don't know about you, but I'd appreciate a little extra help from my editor in debugging the expected types for a given function and having the change occasionally to turn on the Malli instrumentation to provide some run-time info on what's going wrong.

Some tools can provide the observability to run-time values like Flowstrom or just good old clojure/tools.trace. However, these tools do not tell me what type of data the original author had intended to receive.

Once again, I hope you found this helpful. Feel free to reach out and let me know what you thinksocial links in the menu.

Thank you for reading.

Deploying Clojure Like a Seasoned Hobbyist

Toni — Tue, 07 Nov 2023 18:05:12 +0000

This is a tutorial on how to set up a minimal Clojure app development environment with Postgres on DigitalOcean using Terraform. Although there will be room for further optimization and improvement in the implementation, this post focuses on providing the most straightforward template for setting up a project.

I prefer configuring deployments in declarative Terraform files over clicking through dashboards. This helps me to get back on track when I return to my abandoned side projects.

It's worth mentioning that I will be using a local terraform state, although I don't recommend it. Lately, I have been using Terraform Cloud for my projects. If you want to learn how to set up a Github CI with Terraform Cloud for this project, please let me know in the comments.

Most platforms like AWS, GCP, Azure, Linode, Fly, and Digitalocean support Terraform. The big three, AWS, GCP, and Azure, feel excessive for hobby projects. Additionally, managing virtual servers or utilizing Kubernetes is not something I want to deal with, so that crosses Linode off the list, leaving me with Fly and Digitalocean as the options. While I haven't tested Fly yet, DigitalOcean's app platform has become my go-to choice out of habit and because the development experience feels suitable.

Requirements

We need Terraform, Doctl, Docker, and Docker Compose installed on our development machine and a DigitalOcean account with an API token with write permissions that we can use to authenticate ourselves from the command line.

You should be able to follow up on the examples without prior knowledge of Terraform. Still, if you're interested in learning more, the official website is an excellent place to start.

Application

To get started, create the following folder structure for the project. Clojure-related code and required Docker files are in the api folder, where docker-compose.yml is for running the application locally with a Postgres database, and the DigitalOcean-related Terraform code will be in the terraform folder.

project-root
 api
    deps.edn
    docker-compose.yml
    Dockerfile
    src
        main.clj
 terraform
     main.tf
     output.tf
     setup.tf

Clojure

First, let's start by defining the Clojure dependencies for the application in the api/deps.edn. We need ring dependencies for serving the API and next.jdbc and postgresql for the database access.

{:paths ["src"]

 :deps {org.clojure/clojure {:mvn/version "1.11.0"}
        ring/ring-core {:mvn/version "1.6.3"}
        ring/ring-jetty-adapter {:mvn/version "1.6.3"}
        com.github.seancorfield/next.jdbc {:mvn/version "1.2.659"}
        org.postgresql/postgresql {:mvn/version "42.2.10"}}

 :aliases {:run {:main-opts ["-m" "main"]
                 :exec-fn main/run!}}}

Next, create the code to serve the API, connect to the database, and make a database query in api/src/main.clj.

(ns main
  (:require [ring.adapter.jetty :as jetty]
            [next.jdbc :as jdbc])
  (:gen-class))

(def port (Integer/parseInt (System/getenv "PORT")))

(def db {:dbtype "postgres"
         :jdbcUrl (System/getenv "JDBC_DATABASE_URL")})

(def ds (jdbc/get-datasource db))

(defn app [_request]
  (let [db-version (jdbc/execute! ds ["SELECT version()"])]
    {:status 200
     :headers {"Content-Type" "application/edn"}
     :body (str db-version)}))

(defn run! [& _args]
  (jetty/run-jetty #'app {:port port}))

This is all of the application code. The next step is to create a Dockerfile and set up the local development environment to validate that the database connection and the API are working as expected when provided with the expected environment variables.

Dockerfile

Let's keep the api/Dockerfile as simple as possible for now.

FROM clojure:openjdk-17-tools-deps-buster

WORKDIR app
COPY . .

RUN clojure -P

CMD clj -X:run

In general it's better to create the Dockerfile in stages to avoid unnecessary build delay and minimize the image size. If you're interested in learning more about that let me know in the comments. You can also read more Docker: Multi-Stage Builds for further information.

Finally, let's set up a api/docker-compose.yml file for mimicking our deployment environment to test that the API is working as expected.

version: '3.1'

services:

  api:
    build: .
    environment:
      JDBC_DATABASE_URL: "jdbc:postgresql://postgres:5432/db?user=user&password=password"
      PORT: 8000
    ports:
      - 8000:8000

  postgres:
    image: postgres
    environment:
      POSTGRES_DB: db
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    ports:
      - 5432:5432

Validate Application Code

Start the API and the database with docker-compose up and HTTP GET localhost:8000.

 http localhost:8000
HTTP/1.1 200 OK
Content-Type: application/edn
Date: Mon, 30 Oct 2023 16:50:53 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked

[{:version "PostgreSQL 14.2 (Debian 14.2-1.pgdg110+1) 
            on x86_64-pc-linux-gnu, compiled by gcc 
            (Debian 10.2.1-6) 10.2.1 20210110, 64-bit"}]

If you did get something similar to the above, we are good to continue forward. The next step is to set up the application to run in DigitalOcean.

In the case you're wondering what's the http command. It is httpie.

DigitalOcean and Terraform

The first thing we must do is to set up the DigitalOcean Terraform provider. This is analogous to adding a Clojure dependency in a deps.edn file. Open terraform/setup.tf and add the following content to it.

terraform {
  required_providers {
    digitalocean = {
      source = "digitalocean/digitalocean"
      version = "~> 2.0"
    }
  }
}

provider "digitalocean" {}

When a new provider is added, Terraform needs to be explicitly initialized with terraform init. After running the command, you should see the following message with additional output.

Terraform has been successfully initialized!

So far, we have downloaded the provider but have not yet configured the authentication. We'll get there in a moment.

Project

Now it's time to set up the project to which we'll assign all our resources. Add the following content to terraform/main.tf and feel free to name your project however you wish.

resource "digitalocean_project" "project" {
  name = "my-sample-project"
  description = "description"
  environment = "development"
  purpose = "template-app"
}

If we try to apply the changes before configuring the API token, we should see something like this:

 terraform apply

...

digitalocean_project.project: Creating...

 Error: Error creating Project: POST https://api.digitalocean.com/v2/projects: 
| 401 (request "f2774cfc-66fc-4f34-98d9-0935f9dcd33d") 
| Unable to authenticate you with digitalocean_project.project,
        on main.tf line 1, in resource "digitalocean_project" "project":
        1: resource "digitalocean_project" "project" {

To apply the changes, we must authenticate ourselves with a token. There are a couple of different options on how to do this.

token - (Required) This is the DO API token. Alternatively, this can also be specified using environment variables ordered by precedence:

DIGITALOCEAN_TOKEN

DIGITALOCEAN_ACCESS_TOKEN

DigitalOcean Terraform Provider

I'll do this step by exporting my API token in an environment variable in the terminal.

 export DIGITALOCEAN_TOKEN=${YOUR_API_TOKEN}

After this, we should be able to create the project by running terraform apply.

 terraform apply

Terraform used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # digitalocean_project.project will be created
  + resource "digitalocean_project" "project" {
      + created_at = (known after apply)
      + description = "description"
      + environment = "development"
      + id = (known after apply)
      + is_default = false
      + name = "my-sample-project"
      + owner_id = (known after apply)
      + owner_uuid = (known after apply)
      + purpose = "template-app"
      + resources = (known after apply)
      + updated_at = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

digitalocean_project.project: Creating...
digitalocean_project.project: Creation complete after 1s [id=22276173-77de-4e41-ab47-fae4a86ec414]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

And confirm that the project was created by logging in to the DigitalOcean dashboard.

Container Registry

The next step is to set up the container registry to publish the application image. Add the following to terraform/main.tf.

 resource "digitalocean_container_registry" "app_registry" {
  name = "clojure-sample-app"
  subscription_tier_slug = "starter"
}

After applying the configuration, confirm that the registry was created. Let's do it with the doctl command line tool this time around.

 doctl registry get clojure-sample-app
Name Endpoint Region slug
clojure-sample-app registry.digitalocean.com/clojure-sample-app blr1

Here, we need to pick up the endpoint value clojure-sample-app and use that to tag the application image. Add the build section to the api/docker-compose.yml file and run docker-compose build.

services:
  api:
    build:
      context: .
      tags:
        - "registry.digitalocean.com/clojure-sample-app/dev"
    environment:
      JDBC_DATABASE_URL: "jdbc:postgresql://postgres:5432/db?user=user&password=password"
      PORT: 8000
    ports:
      - 8000:8000

After building, confirm that the image exists with the expected name.

 docker image ls | grep clojure-sample-app
registry.digitalocean.com/clojure-sample-app/dev latest 20266c1dfcda 12 seconds ago 715MB

It looks like the build produced the expected outcome. Next, we must log in to the DigitalOcean container registry to publish the Docker image.

 doctl registry login
Logging Docker in to registry.digitalocean.com

And then push the image to the registry.

 docker push registry.digitalocean.com/clojure-sample-app/dev
...
latest: digest: sha256:d8a5c01cd95ab5c0981c454866ec52203feba529f169e2da93cb6a99f3af5d88 size: 2840

Let's confirm with doctl that the image is available in DigitalOcean.

 doctl registry repository list-v2
Name Latest Manifest Latest Tag Tag Count Manifest Count Updated At
dev sha256:d8a5c01cd95ab5c0981c454866ec52203feba529f169e2da93cb6a99f3af5d88 latest 1 1 2023-11-01 16:07:59 +0000 UTC

At this point, we have created everything required for the app deployment. Let's do that next.

Application

Create a digitalocean_app resource and update the digitalocean_project from before by adding the app to the resources. This way, DigitalOcean can group the resources (in this case, just the app) under the project.

Let's use the least expensive configuration for the dev environment by utilizing a basic-xss instance with a development database. You can modify these based on your needs. See the rest of the available configuration options from the docs.

resource "digitalocean_project" "project" {
  ...
  # Add this line
  resources = [digitalocean_app.app.urn]
}

resource "digitalocean_app" "app" {
  spec {
    name = "sample-app"
    region = "ams"

    alert {
      rule = "DEPLOYMENT_FAILED"
    }

    service {
      name = "api"
      instance_count = 1
      instance_size_slug = "basic-xxs"

      image {
        registry_type = "DOCR"
        repository = "dev"
        tag = "latest"
        deploy_on_push {
          enabled = true
        }
      }

      env {
        key = "JDBC_DATABASE_URL"
        value = "$${starter-db.JDBC_DATABASE_URL}"
      }

      source_dir = "api/"
      http_port = 8000

      run_command = "clj -X:run"
    }

    database {
      name = "starter-db"
      engine = "PG"
      production = false
    }
  }
}

You probably noticed that we don't have a PORT variable for the application. This is because, in this case, we had configured it. Digitalocean would override it with the http_port variable.

The image block of the configuration ties the container registry we created earlier to the application. After, whenever we push a new image with the dev tag, the application will be redeployed.

Validate

Finally, it's time to check if we did everything correctly. Let's fetch the live URL for our new application. We can do this with doctl apps list.

 doctl apps list -o json | jq ".[0].live_url"
"https://sample-app-xeoo8.ondigitalocean.app"

However, I prefer using the resource available attributes via Terraform outputs. These are usually declared in the output.tf file, but I'll skip this step for the example and leave it in terraform/main.tf. Add the following to the file and run terraform apply once more.

output "app_url" {
  value = digitalocean_app.app.live_url
}

Afterwards, the outputs can be viewed with terraform output

 terraform output
app_url = "https://sample-app-xeoo8.ondigitalocean.app"

Now, we are ready for the last step. Validate that we get the Postgres version as a result, as we did with the local setup.

 http https://sample-app-xeoo8.ondigitalocean.app

HTTP/1.1 200 OK
CF-Cache-Status: MISS
CF-RAY: 81f5801c8898d92e-HEL
Connection: keep-alive
Content-Encoding: br
Content-Type: application/edn
Date: Wed, 01 Nov 2023 16:26:37 GMT
Last-Modified: Wed, 01 Nov 2023 16:26:36 GMT
Server: cloudflare
Set-Cookie: __cf_bm=SHREkz9ddLOs1EoTzuNj7DSPYfazEy_bpdi7y_Kb9BE-1698855996-0-AVgy3TNvUDITYonrci23K+dW1BHozbNCLeNX0FMHMpI3miRJBq8I4HlQE5nMA5YjoZ4dEXLatcef+AE+gjq4xeQ=; path=/; expires=Wed, 01-Nov-23 16:56:36 GMT; domain=.ondigitalocean.app; HttpOnly; Secure; SameSite=None
Transfer-Encoding: chunked
Vary: Accept-Encoding
cache-control: private
x-do-app-origin: 35eedf3b-e0a9-4376-863e-c5ba59ef5d6a
x-do-orig-status: 200

[{:version "PostgreSQL 12.16 on x86_64-pc-linux-gnu, 
            compiled by gcc, a 47816671df p 0b9793bd75, 64-bit"}]

Looks like everything is working as expected.

Destroy

If you don't want to keep the application running, remember to destroy all of the created resources with terraform destroy. The expected monthly cost with this setup with the basic-xxs instance size and development database is around $12 per month, but the starter subscription tier of DOCR is free of charge.

You can read more about the pricing models from the DigitalOcean docs: App platform, DOCR pricing.

Conclusion

I think that DigitalOcean is an excellent lightweight alternative for smaller projects. It does have all the expected features for general app development, and it can be extended with the add-ons found in the marketplace. I have not yet used it for long-term projects since my side projects don't last long enough, but so far, I've been happy with it from the tinkerer's perspective. I would rather not spend my leisure coding time dealing with infrastructure.

Once again, thanks for reading, and I hope you found this helpful. Here's the accompanying GitHub repository.

Show me the Javascript!

Toni — Fri, 27 Oct 2023 05:32:48 +0000

When I first started learning Clojurescript, I often thought about how the code I'm writing translates to Javascript. This was because I had spent a few years prior primarily working on Typescript; hence, it was the reference I had at the time.

I figured there must be an easy way to see the translation result, and I asked around what's the default way to do this in the REPL. As far as I know, I might as well have been the first one asking about this since there was no quick answer available (usually there is).

Since Clojurescript is compiled into Javascript, I thought the most straightforward way to do this would be to use the plain JS interop.

Plain JS Interop

The toString() method of Function instances returns a string representing the source code of this function.

MDN Docs

Knowing that toString method prints out the source code for functions. We can view the compiled source by simply calling the .toString. In Javascript, it'd be as simple as:

>> function piece_of_code(){ 
>> console.log("hello")
>> };
undefined
>> piece_of_code.toString()
'function piece_of_code(){ 
  console.log("hello")
}'

The same approach works in the CLJS context, and we can make the result read better by just printing out the value.

(.toString piece-of-code)
"function cljs$user$piece_of_code(){\nreturn console.log(\"hello\");\n}"

(defn print-src [f]
  (println (str f)))

(print-src piece-of-code)
;; => 
;; function app$core$piece_of_code(){
;; return console.log("hello");
;; }

CLJS Compiler Option

But after some digging, I found the fantastic video: "Mike Fikes explains the ClojureScript Compiler" and learned that a dynamic CLJS compiler variable *print-fn-bodies* does the same thing without the extra work. Later, I discovered that Mike also has a blog post explaining how to do this.

(set! *print-fn-bodies* true)

(do piece-of-code)
;; => #object[app$core$piece_of_code "function app$core$piece_of_code(){
;; return console.log("hello");
;; }"]

With this approach, it is enough to set the variable, and you're good to go.

Conclusion

I think it is good to know how to do this when you're working with CLJS. You might not need to drop into the JS level that often, but knowing how to do it quickly without using external tools makes all the difference in the dev flow, and even if you don't need to, it's good to know how to do it just for curiosity's sake.

Ultimately, the code we are dealing with is just JavaScript, and we can use the same functions we'd be using on the JS development. On top of that, if you're working in a browser context, it's also possible to view and interact with the compiled code in the browser console.

Thanks for reading. I hope you found this helpful. I recommend taking a look at Mike's blog to learn more about Clojurescript.

Clojure and Cross Origin Resource Sharing (CORS)

Toni — Sat, 21 Oct 2023 07:48:51 +0000

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8000/. (Reason: CORS header Access-Control-Allow-Origin missing). Status code: 200.

If you're dealing with the above error with your Clojure web app you've come to the right place. This is a common problem when fetching data from a different origin (read URL) for single-page web applications (SPA). A common scenario is that an SPA is served from AWS S3 and it is fetching data from different servers.

How to Fix "The Same Origin Policy disallows reading the remote resource"

A common fix is to add a backend middleware to deal with the CORS headers such as ring-cors and this is what this post is about. Alternatively, you can serve the web app from the API's origin if possible or write your logic (i.e. middleware) to handle the CORS requests or if you're dealing with a third-party API there is likely an option to configure your allowed web origins.

Configure CORS Middleware

First, add the latest version of ring-cors to your dependencies.

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        ring/ring-core {:mvn/version "1.6.3"}
        ring/ring-jetty-adapter {:mvn/version "1.6.3"}
        ring-cors/ring-cors {:mvn/version "0.1.13"}}}

Require the dependency and wrap the application handler with ring.middleware.cors/wrap.cors.

(ns tvaisanen.cors
  (:require [ring.middleware.cors :refer [wrap-cors]]
            [ring.adapter.jetty :as jetty]))

(defn app [_request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "OK"})

(defonce server (atom nil))

(defn start! []
  (reset! server
          (jetty/run-jetty
           (wrap-cors #'app
                      :access-control-allow-origin #"https://tvaisanen.com"
                      :access-control-allow-methods [:get :put :post :delete])
           {:port 8000 :join? false})))

(comment
  (.stop @server)
  (start!))

Let's run the server and validate that it is working as expected.

Verify the Configuration

Verify the configuration by making an HTTP request from the browser console and inspecting the headers from the network tab.

Send the Request

Type this in your browser's dev tools console (from the origin configured for CORS).

fetch("http://localhost:8000").then(console.log)

Request Headers

You should see something similar to this in your request headers when you inspect the network tab.

GET / HTTP/1.1
Host: localhost:8000
Origin: https://tvaisanen.com
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site

Response Headers

And the response headers are similar to this.

HTTP/1.1 200 OK
Date: Thu, 19 Oct 2023 05:53:16 GMT
Content-Type: text/html
Access-Control-Allow-Methods: DELETE, GET, POST, PUT
Access-Control-Allow-Origin: https://tvaisanen.com
Transfer-Encoding: chunked
Server: Jetty(9.2.21.v20170120)

What is important to notice here is that the response header Access-Control-Allow-Origin has the value from the requests Origin header. Having this with the Access-Control-Allow-Methods header tells the browser that it is okay to use the data they are asking for.

Let's double-check that this is indeed what is happening from the command line.

CORS Headers Included

When the origin header is passed and the value matches the one that is configured on the server side we get the expected Access-Control-Headers.

 http localhost:8000 'origin:https://tvaisanen.com'

HTTP/1.1 200 OK
Access-Control-Allow-Methods: DELETE, GET, POST, PUT
Access-Control-Allow-Origin: https://tvaisanen.com
Content-Type: text/html
Date: Thu, 19 Oct 2023 06:02:10 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked

OK

CORS Headers Missing

If the value of the origin header does not match the configured value the access control headers should not be in the response.

 http localhost:8000 'origin:https://not-tvaisanen.com'

HTTP/1.1 200 OK
Content-Type: text/html
Date: Thu, 19 Oct 2023 06:02:18 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked

OK

Looks like everything is working as expected!

There's more to cross-origin resource sharing but this should be enough to get you over the initial problem. I recommend reading related MDN Docs to learn more about the topic.

Thanks again for reading, I hope you found this useful.

Study Smarter With AI Tools

Toni — Wed, 04 Oct 2023 17:32:17 +0000

The remote culture has its advantages. Especially when it comes to studying and working at the same time. It's super valuable to have access to the lecture recordings later for you to be able to do the study part outside office hours.

As a resourceful engineer, this also opens up new possibilities. If you think about it, do we need to listen to the two-hour lecture when we can just transcribe the lecture into text and browse through it quickly to see if there's any additional useful information besides the source materials?

The answer to this question is of course not.

What I did, I fired up a Google Colab Notebook and installed Open AI's whisper.

!pip install git+https://github.com/openai/whisper.git

I extracted the audio from the lecture videos with ffmpeg and uploaded the audio files to Drive and then loaded these files into the Notebook.

from google.colab import drive
drive.mount('/content/drive')

Import Whisper and load the model. See the available configurations in the docs. The next step is to just iterate over the files to transcribe and enjoy the results via text search.

import whisper
import os

model = whisper.load_model("medium")
path = "drive/MyDrive/MY_DRIVE_PATH"
files = os.listdir(path)
files
# ['audio1025532515.m4a', 'audio1358476021.m4a']

for f in files:
  result = model.transcribe(f"{path}/{f}")

  with open(f"{f}.txt", "w") as f:
    f.write(result['text'])

And that's it.

print(result["text"])
# Huomentapiv. Aloitellaan tmn pivn luentoa....

Now I have access to the transcriptions in my Drive!

os.listdir(path)

# ['audio1025532515.m4a',
# 'audio1358476021.m4a',
# 'audio1025532515.m4a.txt',
# 'audio1358476021.m4a.txt',]

I thought that it would be nice to share as I found this useful, I hope that you do too! Thanks for reading.

Data Validation in Clojure

Toni — Sun, 01 Oct 2023 18:03:24 +0000

Malli is a data-driven schema Clojure library for defining types and validating data. It can be used for example to:

type check data when received from untrusted sources i.e. validate that HTTP request bodies before writing to the database
define domain entities as code and generate UML diagrams
generate clj-kondo types for static type checking in editors.

Malli uses hiccup-inspired vector syntax for defining types. There's also a map syntax variant but the use case for that is to be used as an internal library representation. The vector syntax looks like this.

  (def Todo
    [:map
     [:id :int]
     [:author :string]
     [:created inst?]
     [:status [:enum :todo :doing :done]]])

Here we have a "Todo" schema that represents a map with properties: id, author, created, and status. Each child of the map vector defines a property, meaning the property :id is a type of :int and :created is a type of instant and so on. All of these properties can be validated separately.

(malli.core/validate :int 1)
;; => true
(malli.core/validate :int "1")
;; => false
(malli.core/validate inst? (java.time.Instant/now))
;; => true
(malli.core/validate [:enum :todo :doing :done] :done)
;; => true

Or as a whole.

(malli.core/validate Todo
                    {:id 1 
                     :author "Toni" 
                     :created #inst "2023-09-30"
                     :status :todo})
;; => true

In the case that the data is not valid we probably want to know the details on why so. For this, we can use malli.core/explain and malli.error/humanize.

(def invalid-todo
    {:id 1
     :author "Toni" 
     :created "not-an-instant"
     :status :todo})

(malli.core/validate Todo invalid-todo)
;; => false
(:errors (malli.core/explain Todo invalid-todo))
;; =>
({:path [:created],
  :in [:created],
  :schema inst?,
  :value "not-an-instant"})

(malli.error/humanize
  (malli.core/explain Todo invalid-todo))
;; => 
{:created ["should be an inst"]}

The error messages can be customized and internationalized to fit your needs. Malli has a lot more features to offer, so I encourage you to go through the documentation to learn more.

Thank you for reading, I hope you found this useful. Please let me know if there's some specific Malli feature that you would like to know more about.

Typescript Like Intellisense for Clojure Functions With Malli

Toni — Sun, 10 Sep 2023 07:09:41 +0000

To some, it might come as a surprise that you can type Clojure functions and get static type-checking in your editor to warn you about invalid arguments, etc. This is not a feature of the language itself, it is possible via amazing open-source projects CLJ Kondo and Malli.

Similarly to typing Javascript code in Typescript, we can use separate type declaration files to instruct the TS compiler and linter. In Clojure, we can generate clj-kondo type configurations for our library or import these from library authors and enhance the developer experience.

In Malli, typing is similar to TS.

(ns core
  (:require [malli.core :as m]
            [malli.dev :as dev]))

(defn add [x y]
  (+ x y))

(m/=> add [:=>
           [:cat :int :int]
           :int])

;; start the instrumentation ~ type checking
(dev/start!)

Which generates the following type definitions in the clj-kondo cache that is picked up by the editor integration to provide type hints in your editor.

{:linters
 {:unresolved-symbol {:exclude [(malli.core/=>)]},
  :type-mismatch
  {:namespaces
   {core
    {add {:arities {2 {:args [:int :int], :ret :int}}}}}}}}

The type definition is fairly close to the TS ones

(m/=> add [:=> ;; type is a function
           [:cat :int :int] ;; there's two arguments of type int
           :int ;; the function returns an int
          ])


type add = (Number, Number) => Number;

When we have the instrumentation turned on

(add "2" 1) ;; evaluating this would result in following

-- Schema Error ------------------------------------------------

Invalid function arguments:

  ["2" 1]

Function Var:

  core/add

Input Schema:

  [:cat :int :int]

Errors:

  {:in [0], 
   :message "should be an integer", 
   :path [0], 
   :schema :int, 
   :value "2"}

More information:

  https://cljdoc.org/d/metosin/malli/CURRENT/doc/function-schemas

This can also be seen in the editor even before evaluating the code.

This week I was working on a small bug on how Malli generates CLJ Kondo type definitions and I thought it'd be a good moment to share some of the basics I picked up on while doing it. I haven't missed the types from TS in a long time but now I know how to configure these when needed.

Thanks for reading, I hope you found this useful.

Clojure Ring Hot Reload Server on Code Changes

Toni — Sun, 03 Sep 2023 12:34:56 +0000

Many Javascript developers use nodemon or a backend framework that provides the feature out of the box to reload their web server when code files change to have the latest code running instantly. This makes the development feedback cycle much faster than the alternative of manually restarting the server each time changes are made.

In Clojure, we often re-evaluate our function in the REPL and our server will pick up the changes. There are a few little tricks to remember when doing this and a couple of utilities that we can use to make the workflow faster for ourselves.

Here's an example to demonstrate this.

(ns acme.reload
  (:require [ring.middleware.params]
            [ring.adapter.jetty :as jetty])
  (:gen-class))

(defn app [_request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "response OK"})

(defonce server (atom nil))

(defn start! []
  (reset! server
          (jetty/run-jetty app
           {:port 8080 :join? false})))

(comment
  (.stop @server)
  (start!))

Now when the server is started with the function start! we can make an HTTP GET request and get the hardcoded value "response OK".

 http :8080
HTTP/1.1 200 OK
Content-Type: text/html
Date: Sun, 03 Sep 2023 12:10:24 GMT
Server: Jetty(9.2.21.v20170120)
Transfer-Encoding: chunked

response OK

Now, if we alter the response string and re-evaluate the app in reply

(defn app [_request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "response still OK"})
;; => #'acme.reload/app

We will still get the same value "response OK" because the response handler is passed to the run-jetty function as a value.

The first simple step we can do to improve the workflow is to use the reader macro #' to pass the value as a reference. Doing using this the value is resolved by reference and not as the value the reference presents at evaluation time.

When used it will attempt to return the referenced var. This is useful when you want to talk about the reference/declaration instead of the value it represents.

https://clojure.org/guides/weird_characters#_var_quote

The only thing we need to change from the code is to add #' before the app.

(defn start! []
  (reset! server
          (jetty/run-jetty #'app
           {:port 8080 :join? false})))

By doing this Jetty will pick up the changes whenever we re-evaluate the app function. This is super helpful when working dynamically with the REPL and it is a definite productivity boost.

But if you want to have the same effect when saving source code files you might want to reach out to the wrap-reload middleware that is part of ring-devel that can listen to changes on source files and reload the code based on this.

(ns acme.reload
  (:require [ring.middleware.params]
            [ring.middleware.reload :refer [wrap-reload]]
            [ring.adapter.jetty :as jetty])
  (:gen-class))

(defn app [_request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body "OK"})

(defonce server (atom nil))

(defn start! []
  (reset! server
          (jetty/run-jetty
           (wrap-reload #'app)
           {:port 8080 :join? false})))

(comment
  (.stop @server)
  (start!))

By default, wrap-reload listens to changes in the src folder but it takes dirs as an option if other folders are needed. To summarize, the development workflow can be streamlined by using the var quote #' to point to vars by reference which makes the latest REPL evaluated code available right away and wrap-reload middleware to hot reload the server on file changes.

As usual, thanks for reading and I hope you found this useful.

Clojure Ring Hot Reloading HTML in the Browser

Toni — Tue, 29 Aug 2023 16:40:47 +0000

In some cases when you're developing a full-stack app you want to be able to refresh your website when static assets change automatically. For example HTML templates or CSS files.

There's a nice little utility tool ring-refresh to achieve just that.

Here's an example app that uses Selmer templates to render HTML. During development we want the browser to update to have a faster feedback loop in the browser.

Here's a small example of how this works!

Add the dependencies.

{:paths ["src" "resources"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}
        ring/ring-core {:mvn/version "1.6.3"}
        ring/ring-jetty-adapter {:mvn/version "1.6.3"}
        selmer/selmer {:mvn/version "1.12.59"}
        metosin/reitit {:mvn/version "0.7.0-alpha5"}
        ring-refresh/ring-refresh {:mvn/version "0.1.3"}}}

Create a template file resources/templates/layout.html. And remember to add the <head> tag because the ring-refresh will depend on that having in place already. If that's missing the reload will not work because the middleware uses a regex to recognize where to inject the hot reload script. This took me a moment to realize so just a heads up.

<!DOCTYPE html>
<html>
    <head>
    </head>
    <body>
        <div>hello {{name}}</div>
        <div>
            <ul>
                {% for item in items %}
                <li><strong>{{item|capitalize}}</strong></li>
                {% endfor %}
            </ul>
        </div>
    </body>
</html>

And here's the app. Notice the selmer.parser/ache-off! call if you are using extended templates.

When rendering files Selmer will cache the compiled template. A recompile will be triggered if the last modified timestamp of the file changes. Note that changes in files referenced by the template will not trigger a recompile. This means that if your template extends or includes other templates you must touch the file that's being rendered for changes to take effect.

Alternatively you can turn caching on and off using (selmer.parser/cache-on!) and (selmer.parser/cache-off!) respectively.

https://github.com/yogthos/Selmer#important

(ns acme.app
  (:require [selmer.parser :as selmer]
            [ring.middleware.refresh :as refresh]
            [ring.middleware.params]
            [ring.adapter.jetty :as jetty])
  (:gen-class))

(selmer.parser/cache-off!)

(defn app [_request]
  {:status 200
   :headers {"Content-Type" "text/html"}
   :body (selmer/render-file "templates/layout.html"
                             {:name "World"
                              :items ["one" "two" "three" "four"]})})

(defonce server (atom nil))

(defn start! []
  (reset! server
          (jetty/run-jetty (refresh/wrap-refresh #'app)
                           {:port 8080 :join? false})))

(comment
  (.stop @server)
  (start!))

Now when you update your templates the browser should refresh the page automatically and save you a bunch of time and nerves. I hope you found this useful!

HTML Form Fields: Readonly vs. Disabled

Toni — Wed, 16 Aug 2023 17:20:15 +0000

I've been tinkering with the form element once again as web developers often do and lately, I've been using the FormData to parse the values and validate the collected form data with Malli. I often default to using disabled="true" when sketching the forms and I want to block the user from altering a static value, but this week, it hit me that read-only and disabled HTML attributes have different implications.

Here's a simple example of what I mean.

(defnc app []
  (d/form
   {:id "form"}
   (d/input {:name (str :static)
             :default-value "non-mutable"})
   (d/fieldset
    {:name "demo"}
    (d/div
     (d/label "one")
     (d/input {:name (str [:data :value-1])
               :default-value "value-one"}))
    (d/div
     (d/label "two")
     (d/input {:name (str [:data :value-2])
               :default-value "value-two"})))))

And here's a naive form parser.

  (defn naive-form-parser [^js/HTMLFormElement form]
    (reduce
     (fn [form-data [k v]]
       (let [path (edn/read-string k)]
         (if (sequential? path)
           (assoc-in form-data path v)
           (assoc form-data path v))))
     {}
     (for [entry (.entries (js/FormData. form))]
       entry)))

  (naive-form-parser form)
;; => 
{:static "non-mutable", 
 :data {:value-1 "value-one", 
        :value-2 "value-two"}}

Now if we set the static field as disabled

(d/input {:name "static" 
          :disabled true
          :default-value "non-mutable"})
;; ...

(naive-form-parser form)
;; => 
{:data {:value-1 "value-one", 
        :value-2 "value-two"}}

The disabled=true makes the input field "invisible" to the FormData, but when using the readonly=true it will be visible.

(d/input {:name "static" 
          :read-only true
          :default-value "non-mutable"})
;; ...

(naive-form-parser form)
;; => 
{:static "non-mutable", 
 :data {:value-1 "value-one", 
        :value-2 "value-two"}}

And of course, this is pretty clear if you read through the docs:

The Boolean disabled attribute, when present, makes the element not mutable, focusable, or even submitted with the form. The user can neither edit nor focus on the control, nor its form control descendants.

https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/disabled

These details can be easy to forget when working with React and other libraries. There's nothing new in this post. I just thought that this was curious since I don't remember paying attention before to this small detail.

If you were wondering the code examples are in Helix.

Clojure Ring Logging

Toni — Sat, 12 Aug 2023 11:39:30 +0000

A quick tutorial on how to set up request logging for a Clojure web app backend when using Ring and Jetty adapter.

First, create a new project with clj-new.

clojure -Tclj-new app :name acme/app

And then add the Ring dependencies to the deps.edn file.

< :deps {org.clojure/clojure {:mvn/version "1.11.1"}}
---
> :deps {org.clojure/clojure {:mvn/version "1.11.1"}
> ring/ring-core {:mvn/version "1.6.3"}
> ring/ring-jetty-adapter {:mvn/version "1.6.3"}
> ring-logger/ring-logger {:mvn/version "1.1.1"}}

Next, set up the clojure tools logging with log4j2 logging backend.

Update the deps.edn file by adding clj-log4j2 dependency and set:jvm-opts

5c5,6
< ring-logger/ring-logger {:mvn/version "1.1.1"}}
---
> ring-logger/ring-logger {:mvn/version "1.1.1"}
> clj-log4j2/clj-log4j2 {:mvn/version "0.4.0"}}
7c8,9
< {:run-m {:main-opts ["-m" "acme.app"]}
---
> {:run-m {:main-opts ["-m" "acme.app"]
> :jvm-opts ["-Dclojure.tools.logging.factory=clojure.tools.logging.impl/log4j2-factory"]}

Create a file resources/log4j2.properties and save the file with the following content. Read more about the configuration options from the here.

status = warn
monitorInterval = 5

appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %date %level:%logger: %message%n%throwable

rootLogger.level = debug
rootLogger.appenderRef.stdout.ref = STDOUT

Next wrap the ring handler with the wrap-with-logger middleware to log the requests. Configuration instructions can be found from the library github page.

(ns acme.app
  (:require [clojure.tools.logging :as logging]
            [ring.middleware.params]
            [ring.adapter.jetty :as jetty]
            [ring.logger :as logger])
  (:gen-class))

(def port 3000)

(defn ring-handler [_request]
  (logging/info "Info message")
  (logging/debug "Debug message")
  (logging/warn "Warn message")
  {:status 200
   :body "OK"})

(defonce server (atom nil))

(def app
  (-> #'ring-handler
      logger/wrap-with-logger))

(defn start! []
  (logging/info "Listening on port: " port)
  (reset! server
          (jetty/run-jetty #'app {:port port})))

(defn -main []
  (start!))

Now you are ready to run the server with clj -M:run-m and you should see something like this in your console when the API is called with a POST request.

 clj -M:run-m
2023-08-12 14:23:50.358:INFO::main: Logging initialized @1171ms
2023-08-12 14:23:50,440 INFO:acme.app: Server starting with arguments: {}
2023-08-12 14:23:50.452:INFO:oejs.Server:main: jetty-9.2.21.v20170120
2023-08-12 14:23:50.472:INFO:oejs.ServerConnector:main: Started ServerConnector@4602f874{HTTP/1.1}{0.0.0.0:3000}
2023-08-12 14:23:50.472:INFO:oejs.Server:main: Started @1284ms
2023-08-12 14:23:51,716 INFO:ring.logger: {:request-method :post, :uri "/", :server-name "localhost", :ring.logger/type :starting}
2023-08-12 14:23:51,717 INFO:acme.app: Info message
2023-08-12 14:23:51,717 WARN:acme.app: Warn message
2023-08-12 14:23:51,717 INFO:ring.logger: {:request-method :post, :uri "/", :server-name "localhost", :ring.logger/type :finish, :status 200, :ring.logger/ms 1}

That's about it for the minimal setup. Read more about advanced configuration from the sources.

Thanks for reading, hope you found this helpful.