DEV Community: Kari Marttila

Good Enough Developer

Kari Marttila — Wed, 27 Oct 2021 00:00:00 +0000

Infra code and Clojure/script full-stack app code.

Introduction

In my previous blog post Real Full-Stack Developer, I was pondering the concept of a full-stack developer. I also wrote a blog post on the Metosin site regarding how developers co-operate together as a superorganism. After spending time with these thoughts, I was also thinking about what makes a good developer. On the same day, I listened to an interesting Finnish podcast about Impostor syndrome (Huijarisyndrooma on kettumainen ilmiö) All these things boiled in my head, and I got an idea: I must write a short blog post with the title “Good Enough Developer.”

What Makes a Great Developer?

In my personal opinion, anyone can be a great developer. You don’t have to be a virtuous programmer. What is essential is to have an open mind, a lot of curiosity, and be a relentless learner. And to have the courage to enter new areas in which you don’t have much experience - the work itself will give you experience.

In addition to those qualities, if you are a superb programmer by God’s mercy, that’s a plus. I have seen a couple of really, really good programmers. It is an absolute joy to work with a virtuous programmer since you learn fast with that kind of mentorship.

Everything is Interpretation

Working with a really, really good programmer can also be a bit intimidating experience. E.g., in my previous project, I worked with one of the best Clojure programmers on this planet (read more in my earlier blog post My First Project at Metosin!). That’s the moment when you can choose between two different narratives. In the sad narrative, you tell yourself: “I’ll never be as good a programmer as this guy. All hope is lost. I might as well quit and start a new career flipping burgers.” Or you can choose a more positive narrative and tell yourself: “Gosh, this is just great! One of the best clojurians on the planet is reviewing my every PR - what a great learning opportunity this is!” Remember: Everything is interpretation - choose a narrative that is easier to live with.

Impostor Syndrome

When working in a company full of great programmers, you might feel that you are not as good as the other programmers, and they soon see through you. You will be exposed. An average programmer. That’s the Impostor syndrome talking, don’t give it any power over you. Once you choose that negative narrative, you will teach your brain that you lack something and are not worth enough. That’s bullshit. If you are working as a programmer, most probably someone already is happy with your work. Relax. Create a more positive narrative for yourself.

Good Enough

You can tell yourself: “It’s pretty darn sure I’m not the most talented programmer in this company. But that’s ok. I’m good enough. With the help of these great programmers, I can be pretty productive, bring food to the table and enjoy my work. And I can learn every day! And if I really want to grow as a programmer, isn’t it good to be among better programmers you can learn from than with some programmers who have nothing to teach you?”

Being good enough is a comforting feeling. In most jobs, you don’t have to be the best of the best of the best - usually good enough, is good enough. I think I’m a pretty good example of this. I don’t consider myself a great clojurian. But with my experiences creating infra code and also be able to implement full-stack apps using Clojure/script (with occasional help from my colleagues in our #tech-clojure Slack channel) - I’m pretty productive member of our company. I enjoy my work enormously, and I can also give back to our community with my IaC knowledge. Now that I think about it, I kind of like being a generalist. I’m not a superhero of any specialty, but I’m good enough in IaC and various programming languages so that I can be a Good Enough Real Full-Stack Developer.

Conclusions

Be merciful to yourself. Don’t create negative narratives about your competence and your career. Be positive. Most often good enough, is good enough. And you can learn every day to be a better programmer.

The writer is working at Metosin using Clojure in cloud projects. If you are interested to start a cloud or Clojure project in Finland or you are interested in getting cloud or Clojure training in Finland you can contact me by sending an email to my Metosin email address or contact me via LinkedIn.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

Real Full-Stack Developer

Kari Marttila — Tue, 26 Oct 2021 00:00:00 +0000

(First published in karimarttila.fi)

Infra code and Clojure/script full-stack app code.

Introduction

In my previous blog posts (AWS IoT First Reflections and AWS IoT Storage Considerations), I described my first AWS IoT platform project and its infrastructure building process. Now I have moved on to implementing the full-stack Clojure/script app to visualize the IoT data on that platform. While developing the full-stack app, I realized that I need to think about what full-stack app development really is and who is a full-stack developer.

Traditional Definition

The w3schools provides a concise definition for a full-stack developer: A full stack web developer is a person who can develop both client and server software.

I have been in IT business for some 25 years. I remember the Java application server era when we created HTML UIs using JSP, Java Server Faces, and other mechanisms that were popular at that time. Later on, the so-called single-page application paradigm got popular, and developers started using Javascript to implement the HTML UIs. In many projects, the backend was implemented still using Java. This caused in bigger projects the development to divide into two different teams - the backend team (using e.g. Java) and the frontend team (using Javascript). Later on, when Nodejs got popular, developers started to implement the backend also using Javascript. Now you could build a modern single-page application using just one programming language, and those developers began to call themselves full-stack developers.

Enter the Cloud Era

Now that we have entered the cloud era in which most applications are running in the cloud, a new competence is required: the ability to create Infrastructure as Code. In many bigger projects, there is a DevOps team that is responsible for the cloud infra (actually a bit of a contradictory term since the whole point of DevOps is that there is no dedicated Ops team). This is because most application developers do not feel comfortable creating IaC solutions - they just want to be application developers (backend, frontend, or both).

Real Full-Stack Developer

In my latest project, I’m responsible for creating an IoT platform that can ingest a large number of IoT events and store them in two storages: all events in S3 based data lake and the latest events in RDS. I’m also responsible for creating a full-stack application (using Clojure and Clojurescript). It’s pretty easy to develop a full-stack app using Clojure/script since you can use the same language in both backend and frontend sides, and Reagent makes building a modern React-based single-page application easy.

I’m the only developer responsible for building the whole system (thankfully, a competent UI designer creates the UI layout design since I have no artistic eye). It is really enjoyable to work on a small green-field project like this - you get to build the cloud infrastructure and the applications running in the infra yourself. When implementing the infrastructure, the backend, and the frontend, you can create a coherent solution and optimize the overall development process. I would say that this is real full-stack development.

Conclusions

Perhaps we should define the concept of full-stack development and a full-stack developer in this new cloud era. You are a full-stack developer if you can create the cloud infrastructure as code and then implement the backend and frontend for your application running in that infra.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

Start Learning Cloud Skills!

Kari Marttila — Thu, 24 Jun 2021 00:00:00 +0000

A screenshot of one of my personal cloud projects.

Introduction

In my previous corporation an Azure guy once told me: “It is not possible to be an expert in more than one cloud. If you want to be an expert in Azure you have to focus on Azure.” A few days passed and an AWS guy told me: “If you want to be an expert in AWS you have to focus on AWS.”

Is It True?

Is it really true? If you want to be an expert in one cloud, do you really have to ditch other clouds away and focus solely on one cloud? No, it isn’t true. You can happily be an expert in so many clouds you want. I have done IaC in Azure, GCP and AWS projects. There is no law that says you can’t do so. I also have done certifications in all of these clouds (see in www.karimarttila.fi/about the status of my current cloud certs).

The Argument

I guess those guys just were a bit troubled talking to a guy who does multi cloud work. It is very human to try to come up with an argument to rationalize your choices. Their choice was to stick with one cloud - and one of the arguments for that choice was that “it is not possible to be an expert in more than one cloud.”

On the Other Hand

On the other hand those guys were right, sort of. If you focus only on one cloud and dedicate all your learning resources to study that cloud - naturally you will be a better expert in that cloud compared to studying two or three clouds.

But this argument does not fly that far. I have two academic degrees, M.A. in psychology and M.Sc. in Software Engineering. It is perfectly possible to be interested in various fields and technologies and study them all. To study those academic degrees took me a lot more efforts than studying three clouds. The thing that those dedicated cloud guys didn’t get was that I work as a software architect and programmer. I don’t have to remember by heart every little detail of any specific cloud. It’s entirely sufficient to understand the basic concepts well and the most important services. Every cloud provides VPCs, subnets, firewalls, VMs, container registries, Kubernetes as a service, databases etc. If you have learned one cloud it is easier to learn the same basic concepts and how to use them in new clouds, just like with programming languages. No one says that you cannot learn more than one programming language and therefore you have to stick with Javascript (if you don’t believe me, read my story Five Languages - Five Stories).

Why to Study Cloud Technologies?

I’m old enough to remember the era of the managed services divisions in big IT corporations. At the beginning of the project the software architect had to calculate the needed capacity for various servers since when you sent the order to the managed services division and they started to order those physical servers it took weeks to have them at your data center with all middleware and firewalls installed and configured. In my first cloud project some years ago I was just astonished when I realized that I can create a whole virtual infrastructure with VPC, subnets, firewalls, frontend servers, backend servers, databases, queues, alarms etc using infrastructure code, and without any help from some managed services division. And that I could deploy a new copy of that environment in minutes, up and running, everything configured just as defined in the infrastructure code. Ever since that insight I never looked back. I decided that I will work only in cloud projects in which I can create the cloud infrastructure as code and the applications running in that cloud infra.

Public clouds with virtual infrastructures are the new paradigm. If you are an application developer I strongly recommend you to start learning cloud technologies as well.

Do You Have to Certify?

No, you don’t. You can learn everything you need to know about public clouds without ever going to one certification exam. The reason I like to take these certification tests is that the requirements for the basic cloud certifications provide really good list for the most important concepts and services you need to understand anyway for being a successful cloud expert in the cloud projects (and I’m talking here the “basic” developer/engineer/architect certs and not the specialty certs). And once you have learned those requirements why not to take the certification exam as well, put the cert in your LinkedIn profile and start getting calls from headhunters.

How to Start Learning Cloud Skills?

Everyone has his or her own learning methods. Mostly just follow your common sense and start learning the basics whatever methods suites you. There are a lot of material on the Internet. I have liked a lot about the courses in Pluralsight and in Coursera and do the Qwiklabs or other labs related to those courses. I have also attended some traditional courses related to some specifics like Security on AWS.

If you know nothing of public clouds, just choose one of the “Big Three” - i.e. AWS, Azure, or GCP. It is not that important which one you choose - they are all good public clouds. Once you master the basics in that cloud it is easier to learn the basics regarding the other two clouds as well.

Your next step is to get a cloud account - you cannot learn cloud skills just by reading books - you absolutely need a cloud account. All Big Three clouds provide “free tier services” or a certain amount of free credits if you order a personal account (at least they used to do that when I learned the basics - nowadays I mostly use my company’s cloud accounts for my personal projects as well and let the capitalists pay my learning bills).

I worked in my previous corporation as a Cloud Mentor. One of my duties was to teach cloud skills to our developers. I always emphasized that there are three things you need to learn:

Basic information regarding the cloud and its basic services (e.g. virtual network, basic storage options, basic computing options).
How to use the cloud provider’s portal efficiently.
How to create infrastructure as code.

You can learn the basic concepts and services, and how to use the cloud provider’s portal following various video courses eg. in Pluralsight or Coursera, and do the labs in dedicated sites (like Qwiklabs) or using your personal cloud account (or do like I do: use your company’s cloud account (with permission, of course) and let the capitalist pay your learning bills). But what those courses don’t teach that much is how to create infrastructure as code (IaC) using eg. Terraform or Pulumi or cloud provider’s native IaC tool like CloudFormation, ARM, or Deployment Manager. It is utterly important to learn how to create infrastructure as code since in real projects that is how the cloud solution is done professionally (and not by clicking resources with a mouse in the cloud portal).

So, what are you waiting for? Start learning cloud skills! (And once you think you are an expert, read: Cloud Infrastructure Golden Rules - my personal manifest how to create cloud infrastructures.)

The writer is working at Metosin using Clojure in cloud projects. If you are interested to start a Clojure project in Finland or you are interested to get Clojure training in Finland you can contact me by sending email to my Metosin email address or contact me via LinkedIn.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

Comparing Clojure IDEs - Emacs/Cider vs IDEA/Cursive

Kari Marttila — Wed, 27 Jan 2021 00:00:00 +0000

IntelliJ/Cursive and Emacs/Cider editors side by side in a family portrait.

Introduction

Recently I edited a blog post in which I interviewed Metosinians regarding their favorite Clojure editors. It was quite interesting to see that there is a diverse group of editors in use. In that blog post, I told myself that I have configured my Emacs/Cider setup to be as close to IDEA/Cursive regarding look-and-feel. Someone in Reddit asked about this and I got an idea that I look at my latest Cursive and Dygma related configurations and check if there is some need for fine-tuning regarding my Emacs setup to make both Cursive and Emacs as similar regarding their look-and-feel as possible - and write a new blog post about this experience. There are a couple of earlier blog posts in which I have touched this topic a bit:

Versions and Repository

I experimented with Cursive and Emacs editors using my latest World Statistics exercise - you can find e.g. the deps.edn (e.g. emacs profile) and Justfile (e.g. commands backend-debug-reveal-kari-port for starting the repl for Cursive, and backend-debug-kari-emacs for starting the repl for Emacs). If you are interested in reading about the World Statistics exercise itself, I have another blog post about that: World Statistics Exercise - I just updated the above mentioned deps.edn and Justfile files for this new blog post.

Versions used in this blog post:

IntelliJ IDEA & Cursive: IntelliJ IDEA 2020.3 Ultimate, Cursive 1.10.0-2020.3, and nrepl 0.8.2.
Emacs & Cider: Emacs 26.3, emacs-cider 1.0.0, and cider-nrepl 0.25.5.

My workstation is Ubuntu 20.

Emacs and Cider

Emacs is an old editor - the first versions were created in the 1970’s, the GNU Emacs development started in the 1980’s. I started using Emacs back in my studies at the Helsinki University of Technology in the 1990’s - and have been using Emacs ever since. I really like Emacs, though I’m not by any standard an Emacs or eLisp guru. Emacs is implemented using Lisp (Emacs Lisp) which makes it a nice editor for writing Lisp code, e.g. Clojure. Having said that I must emphasize that Emacs is a general-purpose editor, not specifically meant for Lisp programming - you can find a so-called Emacs major mode basically for any programming language. The best way to describe Emacs is that it is an extensible, customizable editor - you can customize it any way you like using the eLisp language. And since eLisp is a Lisp you can add new editing commands while editing. There are also a lot of Emacs packages someone has written for you using the eLisp language - the best way to extend Emacs is first to search a melpa package, and if you can’t find one that suits your needs, only then write one for yourself.

Cider is an Emacs package that extends Emacs into a full-blown Clojure/script development environment. Using Cider you can do the same REPL wizardry as with any Clojure REPL. The trick actually is that you run a nREPL server and then connect Emacs via Cider to this server - Cider is the nREPL client that communicates with the nREPL server. This way you can send any form inside your Clojure code for evaluation to the nREPL and it sends back the results to Emacs.

IntelliJ and Cursive

IntelliJ IDEA just turned 20 years old, but I haven’t been using IDEA that long. I have been programming with Java and Python for some 20 years, but I started programming Java / Python with Emacs (with Java and Python major modes), and a few years after that started using Eclipse for Java (and continued using Emacs for Python). I used Eclipse quite a few years but at some point, I started using IntelliJ IDEA for both Java and Python programming - kind of nice to use the same editor with the same look-and-feel for both programming languages, and I have stuck with IntelliJ IDEA ever since, though I still use Emacs for various other programming / editing tasks. (Lately, I have also started using Visual Studio Code, but that’s another story.)

IntelliJ IDEA is a really good IDE, I really love it. It’s not so bloated as Eclipse, and I really like the layout of the IDE tools more in IDEA than in Eclipse (which I always found difficult to use). Cursive is an IDEA plugin and provides a really enjoyable Clojure programming environment. So, nowadays I can use IDEA for all of my favorite programming languages.

At this point, I must admit that Clojure is my favorite programming language, no question about it. I used to use Python as a quick scripting language, but with Babashka you can write shell scripts also using Clojure without the long start-up time of the JVM. Java is a bit yesterday - I strongly recommend using Clojure in the JVM, and if Clojure is not an option, then Kotlin which is a kind of “Java done right”. Btw, you can read more about my experiences regarding different programming languages in my blog post Five Languages - Five Stories and Kotlin - Much More Than Just a Better Java - I really should update the “Five stories” blog post into “Six stories” blog post one day.

What Is a Clojure REPL?

The REPL is the secret weapon of the Lisp world - a way to interact with the Lisp program under development. Other programmers might say that their favorite language also has a “repl” - it’s nothing compared to a real Lisp REPL - you really need a homoiconic language to implement a real powerful REPL, and Lisp (and Clojure) is a homoiconic language.

A REPL is an acronym for Read–eval–print loop. When programming Lisp (and Clojure) a seasoned programmer always keeps a REPL open, and evaluates various forms (the whole namespace, top level form, or some S-expression inside a top level form) in the source code. There are good introductions for the Clojure REPL - I strongly recommend reading them and learn how to use the REPL.

All right! I guess that’s enough for the introduction. Let’s get into the business.

Starting the REPLs for the Editors

For Clojure newbies I explain that there are a couple of ways to use the REPL - either start the repl as part of your IDE, or start an external REPL and connect to it from your editor - I’m using this second way.

I have in my deps.edn the Clojure, nrepl and cider-nrepl versions defined in the dedicated aliases:

{:paths ["resources"]
  :deps {org.clojure/clojure {:mvn/version "1.10.1"}}
  :aliases {
...
    :backend {:extra-paths ["src/clj"]
              :extra-deps {metosin/ring-http-response {:mvn/version "0.9.1"}
...
                          nrepl/nrepl {:mvn/version "0.8.2"}
...
    ;; Emacs Cider specific.
    :emacs {:extra-deps {cider/cider-nrepl {:mvn/version "0.25.5"}}}}

The backend alias was the actual alias for backend development - I created the emacs alias for this blog post.

The Justfile provides the commands for starting the repls with the aliases: backend-debug-reveal-kari-port for starting the repl for Cursive, and backend-debug-kari-emacs for starting the repl for Emacs:

# For Cursive
@backend-debug-reveal-kari-port:
    # clj -J-Dvlaaad.reveal.prefs="{:theme :light}" -M:dev:test:common:backend:reveal:kari -m nrepl.cmdline --middleware '[com.gfredericks.debug-repl/wrap-debug-repl vlaaad.reveal.nrepl/middleware]' -p 44444 -i -C
    clj -M:dev:test:common:backend:reveal:kari -m nrepl.cmdline --middleware '[com.gfredericks.debug-repl/wrap-debug-repl vlaaad.reveal.nrepl/middleware]' -p 44444 -i -C

# Start backend repl with my toolbox for Emacs.
@backend-debug-kari-emacs:
    PROFILE=emacs clj -M:dev:test:common:backend:reveal:kari:emacs -m nrepl.cmdline --middleware '[com.gfredericks.debug-repl/wrap-debug-repl vlaaad.reveal.nrepl/middleware cider.nrepl/cider-middleware]' -p 55555 -i -C

I created both Just recipes for this blog post - the idea is that I added an explicit port for both repls so that I can be sure that Cursive and Emacs are connecting to the respective repls (Cursive port 44444, and Emacs port 55555). The first command has another version with a ligth theme that I tried but didn’t like it that much. In both REPLs I start the nrepl (-m nrepl.cmdline) and then add some middleware ('[com.gfredericks.debug-repl/wrap-debug-repl vlaaad.reveal.nrepl/middleware]' , and cider.nrepl/cider-middleware for Emacs). The first one is a debug helper and the next is a REPL output tool, more about that later.

Connecting to the REPLs from the Editors

In IDEA / Cursive create a Run/Debug Configuration as in the picture below:

In Emacs / Cider give command: cider-connect and give the localhost and port you used when starting the repl previously (55555).

The REPL Output in the Editors

The picture below shows the Cursive REPL output window. It is important to mention here that Clojurians do not write in the REPL. You write in the editor and send the forms for evaluation to the REPL, typically using some hotkey (more about that later). So, in the picture below I have first reset my Integrant state (using a hotkey, of course). Then I have evaluated the forms (e.g. (set! *print-length* 100)) one by one with a dedicated hotkey.

I actually have three monitors at my table, and I keep my Cursive REPL output window in another monitor than the editor itself. Recently I have been experimenting with the Reveal REPL tool and I keep both output windows side by side when experimenting stuff in the Reveal:

Let’s next see the same thing using Emacs:

In Emacs, you see 4 buffers. The top buffer shows a Clojure file (one of my scratch files in which I do some experiments). Below that are side by side the cider-repl buffer which is connected to the REPL I described in the previous chapter, and a message buffer next to it (REPL output echoed). At the bottom is the command buffer which also echoes the REPL output. Emacs also nicely echoes the REPL output also in the editor buffer:

=> {:country_name "Finland", :country_code :FIN, :series-name "Hospital beds (per 1,000 people)", :series-code :SH.MED.BEDS.ZS, :year 2002, :value 7.4, :country-id 246}

You can start the same way the Reveal REPL output tool, as previously (it’s a REPL middleware and not connected to the editors) - I have the Reveal window next to Emacs.

Now I have explained how to start the REPLs and how to connect to the REPLs using the editors. Let’s next dive into the look-and-feel.

Look-And-Feel - the Look

The look is basically the theme of the editor - how colors are used, font ligature, etc.

IDEA/Cursive.

When I started Clojure programming with IDEA/Cursive I wanted to find a light theme that would be nice to my eyes. I have never liked the dark themes that much. I actually thought that there was a Leuven theme in IntelliJ but now that I checked it I noticed that I have created a custom “KariLeuven” theme - possibly used some existing Leuven-like theme as a basis, can’t remember anymore. Anyway, the color theme is rather simple: a light theme with just some coloring (def, defn and language macros like comment using blue, keywords using violet, and strings using green). The picture below shows my theme customizations.

Emacs.

I took the initial setup for my Clojure Emacs configuration from flyingmachine’s Emacs Cursive setup. Then I fine-tuned it a bit, e.g. using the Leuven theme which I initially got from fniessen, and fine-tuned it a bit further.

Look-And-Feel - the Feel

The Feel part is how one navigates in the editor and manipulates S-expressions. Since Clojure is a Lisp and therefore a homoiconic language all expressions are so-called S-expressions - S-expressions can be constructed from other S-expressions. That’s why Lisps have a lot of parentheses - but this also makes the language really powerful (macros etc) and nice to edit. There are two major schools related to how to edit Lisp code: the older paredit style and the newer parinfer style. Using paredit you have certain commands that you use to manipulate the S-expressions, e.g. move this S-expression inside the next S-expression, etc. When using parinfer you don’t have to remember any special commands but you achieve the same results by indenting Lisp code.

Here are a couple of web pages that illustrate the difference nicely:

I use paredit myself. I tried parinfer but it felt a bit odd. I have the same hotkeys for slurping and barfing for both IDEA/Cursive and Emacs and it is therefore pretty effortless for me to edit Lisp code using paredit. When interviewing my colleagues at Metosin most of the programmers used paredit. But if you are a newcomer to the Lisp / Clojure land I suggest using parinfer - it is easier to start editing the code when you don’t have to learn any special commands.

Hotkeys for Slurping and Barfing

To understand my slurping and barfing hotkeys the reader needs to read my previous blog post Dygma Raise Keyboard Reflections Part 1 first. In that blog post, I explain how I have configured the CapsLock key to function as AltGr key in order to use it to get the various parentheses without twisting my right thumb. Then the reader needs to understand the two layers I have configured in my Dygma Raise. I really recommend Dygma Raise - the best keyboard I have ever used - a perfect keyboard for a programmer.

The following two pictures show my favorite hotkeys in IDEA/Cursive :

The most used REPL hotkeys are the Integrant reset = Alt-J and Send Form Before Caret to REPL = Alt-L.

Then the paredit manipulation hotkeys:

Then the same settings in Emacs :


;; override the default keybindings in paredit
(eval-after-load 'paredit
  '(progn
     (define-key paredit-mode-map (kbd "C-M-j") nil)
     (define-key paredit-mode-map (kbd "C-M-l") nil)
     (define-key paredit-mode-map (kbd "C-M-j") 'paredit-backward-slurp-sexp)
     (define-key paredit-mode-map (kbd "M-<right>") 'paredit-forward-slurp-sexp)
     (define-key paredit-mode-map (kbd "C-M-l") 'paredit-backward-barf-sexp)
     (define-key paredit-mode-map (kbd "M-<left>") 'paredit-forward-barf-sexp)
     (define-key paredit-mode-map (kbd "C-<right>") 'right-word)
     (define-key paredit-mode-map (kbd "C-<left>") 'left-word)
     ))
...
(eval-after-load 'cider
  '(progn
...
     (define-key cider-mode-map (kbd "M-l") 'cider-eval-last-sexp)
     (define-key cider-mode-map (kbd "M-ö") 'cider-eval-defun-at-point)
     (define-key cider-mode-map (kbd "M-n") 'cider-repl-set-ns)
     (define-key cider-mode-map (kbd "M-m") 'cider-load-buffer)
     (define-key cider-mode-map (kbd "M-{") 'cider-format-buffer)
     (define-key cider-mode-map (kbd "M-å") 'cider-test-run-ns-tests)
     (define-key cider-mode-map (kbd "M-ä") 'cider-test-run-test)

As you can see it is pretty simple to configure the same REPL hotkeys and the same paredit hotkeys for both IDEA/Cursive and Emacs/Cider. When reading this you have to remember that I’m not using arrow keys but my arrow keys are defined as CapsLock + i/j/k/l, so when I’m forward barfing I actually have my left little finger in CapsLock and my left thumb in one of the Dygma thumb keys (which is Alt) and hit J with my right index finger. This may sound very complicated, but actually, it isn’t - I have considered various layouts and the current layout is a kind of evolutionary result of my keyboard layout experiments. I really like the system since it resembles a bit playing the classical guitar: I hit the “control” (ctrl, shift, alt, CapsLock) key combinations using my left hand, and the navigation (arrow), manipulation (barfing/slurping), and evaluation (REPL) keys with my right hand. All these key combinations are in my muscle memory and I don’t consciously think about them - I just edit code. Even though this system suits me nicely I wouldn’t recommend it to someone else - you have to experiment and find your own keyboard, keyboard layout, and finally the hotkeys for your favorite programming language in that layout.

Conclusions

Both IDEA/Cursive and Emacs/Cider are excellent editors and Clojure integrated development environments. If you want to switch from one to another you can quite easily configure both editors to have pretty much the same look-and-feel.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

World Statistics Exercise

Kari Marttila — Tue, 19 Jan 2021 00:00:00 +0000

World Statistics App.

Introduction

After implementing my web-store exercise using six languages (Clojure, Python, Javascript, Java, Go and Kotlin) and with 5 data stores (CSV, DynamoDB, Azure Tables, PostgreSQL, and Datomic, read more about in my blog posts, e.g. Five Languages - Five Stories ) I was pretty much fed up with that exercise. I wanted to do something totally different. But what? I spent quite a lot of time figuring out what to do. I was dreaming to do something with Finnish Open data that could benefit the general public. I was browsing the Finnish Open data but couldn’t figure out anything useful that would contain a lot of data points. Then I decided to take a look at the World Bank Open data. Just out of curiosity I downloaded various metrics between the years 2002-2017, some 150.000 data points altogether. Then I just started to experiment with the data using the Clojure REPL - a great tool for studying and experimenting with data. Little by little a new exercise was formed in my head: Implement a full-stack app using Clojure and ClojureScript to filter and enrich the data and visualize the data sets using a frontend. This blog post depicts the story of this exercise.

The application source can be found in my Github clojure repo in directory statistics/worldstat. The application is live in Heroku (as long as the Heroku free tier dyno hours have not run out). NOTE: If there are still dyno hours and no-one has used the app for some time you may need to wait for a while for Heroku to start the app. Since the app is running in the free tier there might be other aspects as well that may impact using the app (max connections etc.)

Technology Overview

I used Clojure, and some Clojure libraries (like reitit) to implement the backend. The backend reads the data from CSV files that I downloaded from the Worldbank site, and manipulates the data so that we have the data and meta-data nicely as Clojure data structures stored as part of the Integrant state. I used kixi as the statistics library.

I used ClojureScript, Reagent, and Re-Frame to implement the frontend. For frontend development I used shadow-cljs. For CSS I used the excellent and light CSS library Bulma. For data visualization I used Vega Lite and as a Clojure wrapper for Vega I used Metasoarous’ Oz library. The world-110m.json world map is from Vega.

Backend - Creating the Data

In main.clj I read the Integrant configuration: the data files (the CSV files that I downloaded from the Worldbank, and the toposon file) and inject them to init.clj:

(defmethod ig/init-key :backend/data [_ {:keys [data-files topojson-file]}]
  (init/get-data data-files topojson-file))

In init.clj I parse the data from the CSV files and convert it to edn. In that file I do some processing and finally return the data structure back to main.clj which stores the data as part of the Integrant state:

(defn get-data [data-files topojson-file]
  (log/debug "ENTER get-data")
  (let [topojson-country-codes (read-topojson-country-codes topojson-file)
        ids (into {} (map (juxt :acr :id) topojson-country-codes))
        data-points (get-data-points data-files ids)
        {:keys [countries series years]} (get-metadata data-points)]
    {:raw-points data-points
     :points (remove-non-country-values data-points non-country-codes)
     :countries countries
     :series series
     :years years
     :country-codes (country-codes-to-names countries)
     :series-codes (series-codes-to-names series)
     :non-country-codes non-country-codes
     :country-ids ids
     :topojson-country-codes topojson-country-codes
     }))

An important part of this processing is to store the topojson country ids as part of the data points so that we can later use them as hooks to actual countries in the topojson map.

I downloaded some 150.000 data points for this exercise (:raw-points). I removed various data points (:points) that were associated to continents, income groups, etc. Finally, I had some 130.000 data points that were associated to actual countries found in the topojson map. It’s easy to study the data using the Clojure REPL:

(count (:raw-points (user/data)))
=> 153152
(count (:points (user/data)))
=> 129824

… but more about that in the next chapter.

An individual data point is a Clojure map. Let’s get one data point from Finland and check it in more detail:

(first (transduce (ws-filter/filter-by :country_code :FIN) conj (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS))))
=>
{:country_name "Finland",
 :country_code :FIN,
 :series-name "Hospital beds (per 1,000 people)",
 :series-code :SH.MED.BEDS.ZS,
 :year 2002,
 :value 7.4,
 :country-id 246}

Backend - Using Clojure REPL to Experiment with the Data

Using a real REPL is such a joy with Clojure. Let’s use it to experiment with the data:

(set! *print-length* 100)
=> 100 ; So that we don't accidentally list 150.000 data points in repl...
;; Require some namespaces.
(require '[worldstat.backend.data.world :as ws-world])
=> nil
(require '[worldstat.common.data.filter :as ws-filter])
=> nil
(require '[kixi.stats.core :as kixi])
=> nil
;; Check how many points there are...
(count (:raw-points (user/data)))
=> 153152
(count (:points (user/data)))
=> 129824
;; What keys do we have in our data?
(keys (user/data))
=>
(:country-codes
 :raw-points
 :series
 :non-country-codes
 :country-ids
 :points
 :series-codes
 :countries
 :topojson-country-codes
 :years)
;; Let's sort countries...
(sort-by :country_name (:countries (user/data)))
=>
({:country_name "Afghanistan", :country_code :AFG, :country-id 4}
 {:country_name "Albania", :country_code :ALB, :country-id 8}
 {:country_name "Algeria", :country_code :DZA, :country-id 12}
 {:country_name "American Samoa", :country_code :ASM, :country-id 16}
 ...)
;; Get hospital beds, filter data points belonging to Finland, and get first data point. 
(first (transduce (ws-filter/filter-by :country_code :FIN) conj (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS))))
=>
{:country_name "Finland",
 :country_code :FIN,
 :series-name "Hospital beds (per 1,000 people)",
 :series-code :SH.MED.BEDS.ZS,
 :year 2002,
 :value 7.4,
 :country-id 246}
;; Let's check some basic statistics, mean and standard deviation for certain data. 
(transduce (map :value) kixi/mean (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS)))
=> 3.668379451410944
(transduce (map :value) kixi/standard-deviation (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS)))
=> 2.601440577329428

Repl is truly such a powerful tool in Lisp. I always keep a repl scratch file in all my Clojure projects and I do experimentation like above in those scratch files - it’s nice to have those experimentations later on as a historical record of how I implemented functions.

Backend - API

I used the excellent Metosin reitit library for routing: routes.clj:

    ["/data" {:swagger {:tags ["data"]}}
     ["/metric/:metric" {:get {:summary "metric"
                               :parameters {:path [:map [:metric string?]]}
                               :responses {200 {:description "data"}}
                               :handler (fn [req]
                                          (let [metric (get-in req [:parameters :path :metric])]
                                            (-> (world/get-world-data env (keyword metric))
                                                r/ok)))}}]
     ["/countries" {:get {:summary "countries"
                          :responses {200 {:description "data"}}
                          :handler (fn [req]
                                 (-> (world/get-countries env)
                                     r/ok))}}]
     ["/years" {:get {:summary "years"
                      :responses {200 {:description "data"}}
                      :handler (fn [req]
                                 (-> (world/get-years env)
                                     r/ok))}}]
     ["/metric-names" {:get {:summary "metric-names"
                             :responses {200 {:description "data"}}
                             :handler (fn [req]
                                        (-> (world/get-metric-names env)
                                            r/ok))}}]]]])

So, you can query the actual data points by giving a metric (:series-code) and the backend returns the data points for that metric. Basically, I just implemented a minimum API that is needed by the frontend to do the visualization.

Reitit library provides swagger documentation out of the box, so you can experiment with the API easily also with the browser:

API Swagger documentation.

Next, let’s move to the frontend.

Frontend - Reagent and Re-frame

I used ClojureScript, Reagent, and Re-Frame to implement the frontend. For frontend development I used shadow-cljs. Reagent provides a nice interface to React, and Re-frame is ClojureScript framework, that uses Reagent and provides simple state management to build single page applications using Clojurescript. All this tooling works pretty seamlessly with the Clojure backend repl so that when I refresh the Integrant state (with one hot key, of course) the frontend also refreshes. One can get a Clojurescript repl and connect to the frontend application running inside the browser, if you need it. Usually you don’t, since there are nice tools and tricks to examine the frontend state.

Clojurescript frontend tools.

On the right side of the browser, you see the standard Chrome developer tools window. There you can examine the html structure, see console log, etc, nothing new for any frontend developer.

At the bottom is the Metosin reagent-dev-tools. You can subscribe to all re-frame application data there and browse the data. I also implemented a simple debug panel in which I show some most important application data that I used when implementing the frontend. Btw. I really recommend using the Metosin reagent-dev-tools. E.g. I’m in a customer project in which we are implementing a rather complex Clojure full stack application and there is really complex business logic - we have various fixtures in the reagent-dev-tools which makes it a breeze to initialize the frontend and the whole application to a certain state in the middle of the business logic.

All this tooling works really nicely. I’m an old backend programmer myself, but using these tools it was quite effortless to implement this full stack application. You don’t have to be a Javascript + React guru to implement a frontend like the one in this exercise.

CSS - Bulma

I used Bulma as a CSS framework. Bulma is really great - I barely added any custom css in this application. Let’s see part of the frontend using the hiccup notation:

[:div.row
 [:div.columns
  [:div.column.is-9
   [oz/vega-lite (ws-data/world-schema points selected-year selected-metric-name min max) (ws-util/vega-debug)]]
  [:div.column.auto
   (if country-stats
     [:div.rows
      [:div.row
       [:p.is-size-3 selected-country-name]]
      [:div.row
       [:p.is-size-5 "Min: " (ws-util/one-decimal (:min country-stats))]
       [:p.is-size-5 "Max: " (ws-util/one-decimal (:max country-stats))]
       [:p.is-size-5 "Mean: " (ws-util/one-decimal (:mean country-stats))]
       [:p.is-size-5 "Std.dev: " (ws-util/one-decimal (:standard-deviation country-stats))]]
      [:div.row
       [oz/vega-lite (ws-data/country-year-diagram points selected-metric-name selected-country-code min max) (ws-util/vega-debug)]]]
     [:p.is-size-4 "Click a country!"])]]]
[:div.row
 [:p "(Missing data for country is shown with color gray)"]]
[:div.row
 [:p "Read more about this app in my blog "
  [:a {:href "https://www.karimarttila.fi/clojure/2021/01/19/world-statistics-exercise.html"} "www.karimarttila.fi"]]]

You can use Bulma styles seamlessly with hiccup, e.g. :div.columns tells the browser to render the div using Bulma columns etc. Check more in the Bulma documentation.

The dropdown box for selecting the metric and the slider for selecting the year to visualize in the map are also Bulma components. You can customize the Bulma components quite nicely using Clojurescript, example showing the year slider:

(defn slider [id initial step min max type]
  (let [slider-value (r/atom initial)]
    [:div.slider-content
     [:input.slider.is-fullwidth {:id id :step step :min min :max max :value @slider-value :type type
                                  :on-change (fn [event]
                                               (.preventDefault event)
                                               (let [new-value (.-value (.getElementById js/document id))]
                                                 (reset! slider-value new-value)
                                                 (re-frame/dispatch [::ws-state/select-year (js/parseInt new-value)])))}]]))

So, basically I just take the current value of the slider and store it in the re-frame database: (re-frame/dispatch [::ws-state/select-year (js/parseInt new-value)]) .

Frontend - Creating the Map

For data visualization, I used Vega Lite, and as a Clojure wrapper for Vega, I used Metasoarous’ Oz library. The world-110m.json world map is from Vega.

I think I spent most of the time developing this application by tuning the world map. But it was really fun to experiment with how to visualize data and how to interact with the map. The result of this experiment can be seen in file data.cljs, I highlight here the most important part of the map configuration:

(defn world-schema [points year metric-name min max]
  {:schema "https://vega.github.io/schema/vega/v5.json"
   :autosize "none"
   :width 900
   :height 500
   :scales [{:name "color",
             :type "quantize",
...             
   :marks [{:type "shape"
            :from {:data "graticule"}
            :encode {:update {:strokeWidth {:value 1}
...
           {:type "shape"
            :from {:data "world"}
            :encode {:enter {:tooltip {:signal "datum.value === null ? {'country': datum.country_name, 'value': 'Missing data'} : {'country': datum.country_name, 'value': format(datum.value, '.2f')}"}
                             :href {:field "url" :type "nominal"}}
                     ;; If missing data show as grey.
                     :update {:fill [{:test "datum.value === null" :value "gray"}
                                     {:scale "color" "field" "value"}]}
                     :hover {:fill {:value "red"}}}
            :transform [{:type "geoshape"
...
   :data [{:name "countries"
           :values (transduce (ws-filter/filter-by-year year) conj points)}
          {:name "world"
           :url "data/world-110m.json"
           :format {:type "topojson" :feature "countries"}
           :transform [{:type "lookup"
                        :from "countries"
                        :key "country-id"
                        :fields ["id"]
                        :values ["value" "country_name" "country_code"]}
                       {:type "formula"
                        :from "countries"
                        :expr "'#/country/' + datum.country_code" :as "url"}]}
          {:name "graticule"
           :transform [{:type "graticule"}]}]})

So, the Vega interface provides quite extensive ways to inject data to the Vega component and how the component should transform, interpret and visualize the data.

Let’s spend some time with this configuration since it is the actual beef of this app.

First I inject to the map configuration the data (points, year, metric-name, min and max of the points): (defn world-schema [points year metric-name min max]. Then I transform the data points using a join (lookup) between the topojson map data and my data points using the id of the topojson data and country-id of my data points as joining the data. In this join I inject the corresponding country_name and country_code to the new data set so that I can use the country_name to provide the country name when the user hovers the mouse in certain country. I use the country_code for providing a hook when the user mouse clicks a certain country: :expr "'#/country/' + datum.country_code" :as "url"}]} and :href {:field "url" :type "nominal"} . That’s basically what you have to do - and the Vega library does the rest. Then I just show the map component in the application home page, main.cljs (injecting the re-frame subscribed data):

          {selected-metric-code :code selected-metric-name :name} @(re-frame/subscribe [::ws-state/current-metric])
          selected-year @(re-frame/subscribe [::ws-state/current-year])
          {selected-country-code :code selected-country-name :name} @(re-frame/subscribe [::ws-state/current-country])
          {:keys [points min max mean standard-deviation]} @(re-frame/subscribe [::ws-state/world-data selected-metric-code])
          metric-names @(re-frame/subscribe [::ws-state/metric-names])
          _ (if-not points (re-frame/dispatch [::ws-state/get-world-data selected-metric-code]))
          country-stats (if selected-country-code (ws-data/get-stats points selected-country-code))]
...
          [:div.column.is-9
           [oz/vega-lite (ws-data/world-schema points selected-year selected-metric-name min max) (ws-util/vega-debug)]]

Using Vega Editor

Since the map component is a bit complicated and for a Vega newcomer the transform operations are a bit hairy Vega provides excellent development tools. See the three dots icon here:

Going to Vega Editor.

Just click it and choose the Open in Vega Editor and the browser opens this for you:

Vega Editor.

You can experiment with the graphics first using the Vega editor, see the data, and check how various tunings in parameters look like in the map - and once you are happy just copy-paste the configuration back to your Clojurescript source file.

Frontend - Navigation etc.

There is very little navigation in the application. I used Reitit also to do the frontend navigation:

(def routes-dev
  ["/"
   [""
    {:name ::ws-state/home
     :view home-page
     :link-text "Home"
     :controllers
     [{:start (fn [& params] (ws-log/clog (str "Entering home page, params: " params)))
       :stop (fn [& params] (ws-log/clog (str "Leaving home page, params: " params)))}]}]
   ["country/:country-code"
    {:name ::ws-state/country
     :parameters {:path {:country-code string?}}
     :view home-page
     :link-text "country"
     :controllers
     [{:start (fn [& params] (ws-log/clog (str "Entering country, params: " params)))
       :stop (fn [& params] (ws-log/clog (str "Leaving country, params: " params)))}]}]])

Basically, the only navigation is provided when the user clicks a country - we show the year statistics of the selected metric in the line diagram. I actually spend some time with this functionality. I couldn’t figure out how to register the mouse button click event in the vega map - but I found in the documentation a way to inject a href as part of the map, so I used it: :expr "'#/country/' + datum.country_code" :as "url"}]} and :href {:field "url" :type "nominal"} - then in the frontend routing I just provide a route for this uri: ["country/:country-code", and in state.cljs:

(re-frame/reg-event-db
  ::navigated
  (fn [db [_ new-match]]
    (let [old-match (:current-route db)
          new-path (:path new-match)
          country-code (get-in new-match [:parameters :path :country-code])
          country-name (get-country-name country-code)
          controllers (rfc/apply-controllers (:controllers old-match) new-match)]
      (ws-log/clog (str "::state/navigated, new-path: " new-path))
      (ws-log/clog (str "::state/navigated, country-code: " country-code))
      (cond-> (assoc db :current-route (assoc new-match :controllers controllers))
              country-code (assoc :current-country {:code country-code :name country-name})))))

So, here we just get the country-code from the uri, find the matching country-name for it and store them into the re-frame database for further use.

The full stack app is now ready. Let’s next wrap the app into a deployment unit.

Docker Container and Heroku Deployment

I considered deploying the app to AWS but since I’m pretty stingy I didn’t want to spend my personal money on that. Then I realized that I can use Heroku Free Tier to deploy the app to the cloud (at least for the time the Heroku free dyno hours run out, or the World Bank wants to start sponsoring the app :-) ).

First I needed to create a docker container. Instructions for creating and running the Docker container (basically just testing that it works, Heroku creates the image using the heroku cli, more about it later):

just uberjar
docker build -f Dockerfile . -t worldstat:latest
docker run -e PORT=9999 -it -p9999:9999 worldstat:latest

The Dockerfile is pretty minimalistic. I just create a non-root user to run the app (Heroku requires this), I copy the app and the data to the image, and provide the entrypoint to the application: run-app.sh. Notice the pretty harsh memory limits in the app (Heroku gives just max 512MB memory in the free tier).

NOTE: The application is running in the Heroku free tier and Heroku removes the application if it is not used for a certain period of time - in this case just wait patiently a few seconds and you should see the app unless the dyno hours are completely spent.

The Heroku deployment process for Docker images is pretty nicely documented here. Let’s give short instructions if you want to try it yourself:

First build the uberjar: just uberjar
Then basically just follow the instructions in the Heroku documentation.
Create a Heroku account
heroku create
heroku container:login
heroku container:push web -a <YOUR-HEROKU-APP-NAME>
heroku container:release web -a <YOUR-HEROKU-APP-NAME>

I then changed the application name to some more descriptive: km-worldstat .

Conclusions

This exercise was really interesting. I learned quite a lot about how to experiment data using REPL in the backend. I also learned quite a lot how to create nice visualizations using the Vega library. I also learned that using Clojure and Clojurescript you can create quite powerful full stack applications without the need being a Javascript / React guru.

The writer is working at Metosin using Clojure in cloud projects. If you are interested in starting a Clojure project or getting Clojure training in Finland you can contact me by sending an email to my Metosin email address or contact me via LinkedIn.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

Malli Clojure Library

Kari Marttila — Mon, 07 Dec 2020 00:00:00 +0000

Using malli library to validate POST body data in the REST API.

Introduction

At Metosin we use our own libraries quite a lot in our projects - therefore a good reason to learn to use these libraries. For learning to use the malli library I experimented with it using the Clojure REPL, and I also added malli validation to my previous re-frame exercise that can be found in my Clojure repo, in directory re-frame.

I did most of the experimentation in my REPL scratch file (not included in the repo). I then wrote the observations regarding this learning project in this blog post mostly for myself - verbalizing what one has discovered is a good learning method.

What is Malli?

malli is a Clojure/Script library for data validation and specification. It provides tools for validation, coercion, and other aspects related to the question of whether the data in hand is valid for your purposes. malli also integrates nicely with the reitit routing library providing nice tools to validate and coerce data coming to the REST API.

If you are interested to learn more about malli I list below some resources I used myself while experimenting with malli:

My colleague Miikka Koskinen has written a couple of excellent blog posts regarding malli and other data validation libraries:

Why Data Validation?

Clojure is a dynamic language and there is no static type checking by the Clojure compiler. Some developers see this as a downside but it actually makes Clojure both simple and also very expressive and powerful regarding how to manipulate data. One could say that in Clojure we don’t have types - we have something better: data. You can do a lot more with data than with types. And if you are interested in whether your data conforms to a certain data shape - you have validation.

Malli, Spec, or Schema?

There are three major data validation libraries in the Clojure ecosystem:

schema: older data validation library which was the de facto schema library before spec.
spec: the data validation library that comes with Clojure 1.9 or higher and is also used to describe the Clojure core language and Clojure library apis.
malli: the data validation library provided by Metosin.

I tried to verbalize to myself what is the purpose and strengths of these libraries, their sweet spots in projects, and how they differ from each other. I wrote about this blog post in the Metosin Slack, and one of my colleagues hinted that our colleague Miikka Koskinen had already written a blog post that does a very good job describing these libraries, I really recommend reading it: Schema, Spec, and Malli. So, I’m not going to describe the differences between those data validation libraries - Miikka has already done a much better job describing them than I ever could have done.

Experimenting with Malli Using the Clojure REPL

It is quite nice and effortless to experiment with the Malli library using a clojure REPL: just add metosin/malli {:mvn/version "0.2.1"} (0.2.1 is the latest version as of writing this blog post - check for newer versions in malli github repo ) to your deps.edn file and then require malli and start experimenting with it:

(ns malli
  (:require [malli.core :as m]))

(m/validate
  [:map {:closed true} [:ping string?]]
  {:ping "hello"}) ; => true

(m/validate
  [:map {:closed true} [:ping string?]]
  {:wrong-key "hello"}) ; => false

(m/validate
  [:map {:closed true} [:ping string?]]
  {:ping "hello" :extra-key "hello"}) ; => false

A common workflow in the Clojure land is to use a scratch REPL file to experiment with the data validation and once you are confident your validation works as it should you just move the validation code to the production code base.

Malli with Reitit

As an exercise I implemented malli validations to the REST api of the SimpleServer. In this chapter, I provide a few examples. The REST api can be found in the server.clj namespace.

Let’s first use the ping api as an example.

   ["/api"
    {:swagger {:tags ["api"]}}
    ; For development purposes. Try curl http://localhost:6161/api/ping
    ["/ping" {:get {:summary "ping get"
                    ; Don't allow any query parameters.
                    :parameters {:query [:map]}
                    :responses {200 {:description "Ping success"}}
                    :handler (constantly (make-response {:ret :ok, :reply "pong"}))}
              :post {:summary "ping post"
                     :responses {200 {:description "Ping success"}}
                     ;; reitit adds mt/strip-extra-keys-transformer - probably changes in reitit 1.0,
                     ;; and therefore {:closed true} is not used with reitit < 1.0.
                     :parameters {:body [:map {:closed true} [:ping string?]]}
                     :handler (fn [req]
                                (let [body (get-in req [:parameters :body])
                                      myreq (:ping body)]
                                  (-> {:ret :ok :request myreq :reply "pong"}
                                      (make-response))))}}]
...

For ping GET I define that we don’t allow any query parameters, the query map is empty: :parameters {:query [:map]}. For ping POST we define the only allowed body parameter :ping: :parameters {:body [:map {:closed true} [:ping string?]]}. malli is open by default (allows extra keys in maps), but you can configure your schema as closed using {:closed true}. The comment block mentions that at the moment reitit adds mt/strip-extra-keys-transformer and therefore we don’t have any extra keys when we hit the validation - I left the {:closed true} here just as an example.

The server_test.clj provides tests to validate that our malli validation works. First the GET ping:

(deftest ping-get-test
  (log/debug "ENTER ping-get-test")
  (testing "GET: /api/ping"
    (let [ret (ss-tc/-call-api :get "ping" nil nil)]
      (is (= (ret :status) 200))
      (is (= (ret :body) {:reply "pong" :ret "ok"})))))

(deftest failed-ping-get-extra-query-params-test
  (log/debug "ENTER failed-ping-get-extra-query-params-test")
  (testing "GET: /api/ping"
    (let [ret (ss-tc/-call-api :get "ping?a=1" nil nil)]
      (is (= (ret :status) 400))
      (is (= (ret :body) {:coercion "malli"
                          :humanized {:a ["disallowed key"]}
                          :in ["request"
                               "query-params"]
                          :type "reitit.coercion/request-coercion"})))))

When calling just GET ping api we must succeed. When adding any query parameters ("ping?a=1" in the example) we must fail. malli also provides nice humanized error messages regarding the validations: :humanized {:a ["disallowed key"]}. You get the humanized error messages by providing :humanized in the :error-keys set when you create the coercion to be used with reitit:

    (re-ring/ring-handler
      (re-ring/router routes {
                              :data {:muuntaja mu-core/instance
                                     :coercion (reitit.coercion.malli/create
                                                 {;; set of keys to include in error messages
                                                  :error-keys #{:type :coercion :in :humanized ...

Then the POST ping tests:

(deftest ping-post-test
  (log/debug "ENTER ping-post-test")
  (testing "POST: /api/ping"
    (let [ret (ss-tc/-call-api :post "ping" nil {:ping "hello"})]
      (is (= (ret :status) 200))
      (is (= (ret :body) {:reply "pong" :request "hello" :ret "ok"})))))

(deftest failed-ping-post-missing-key-test
  (log/debug "ENTER failed-ping-post-missing-key-test")
  (testing "POST: /api/ping"
    (let [ret (ss-tc/-call-api :post "ping" nil {:wrong-key "hello"})]
      (is (= (ret :status) 400))
      (is (= (ret :body) {:coercion "malli"
                          :humanized {:ping ["missing required key"]}
                          :in ["request"
                               "body-params"]
                          :type "reitit.coercion/request-coercion"})))))

The POST ping must succeed if we provide the required :ping key in the body map. If the required key is missing we get a humanized error: :humanized {:ping ["missing required key"]}.

Let’s provide another example, POST signin. The meaning of this API is to provide the user a method to sign-in to the web-store. The API requires the user to send his first-name, last-name, email, and password:

    ["/signin" {:post {:summary "Sign-in to get an account"
                       :responses {200 {:description "Sign-in success"}}
                       :parameters {:body [:map
                                           [:first-name string?]
                                           [:last-name string?]
                                           [:email string?]
                                           [:password string?]]}
                       :handler (fn [req]
                                  (let [body (get-in req [:parameters :body])
                                        {:keys [first-name last-name password email]} body]
                                    (-signin env first-name last-name password email)))}}]
...

In real-world projects, if you have a big structured POST body, you should define the validation map as a def making the routes part more readable. This way you can also compose validations from other validations. Let’s imagine instead of the previous flat representation we could have a more structured POST body, like {:name {:first-name "mikko" :last-name "mikkonen"} :password "salainen" :email "mikko.mikkonen@foo.com"}. We could have defined the validation schema separately as:

(def name-schema [:map {:closed true} [:first-name string?] [:last-name string?]])
(def sign-in-schema [:map [:name name-schema] [:email string?] [:password string?]])
...
;; and then in the routes:
    ["/signin" {:post {:summary "Sign-in to get an account"
                       :responses {200 {:description "Sign-in success"}}
                       :parameters {:body sign-in-schema}
...

Finally, let’s experiment with our name-schema and sign-in-schema a bit:

(ns malli
  (:require [malli.core :as m]
            [malli.error :as me]))

(-> name-schema
    (m/explain {:first-name "mikko" :last-name "mikkonen"})
    (me/humanize)) ; => nil (ok)

(-> name-schema
    (m/explain {:first-name "mikko" :middle-name "pekka" :last-name "mikkonen"})
    (me/humanize)) ; => {:middle-name ["disallowed key"]}

(-> sign-in-schema
    (m/explain {:name {:first-name "mikko" :last-name "mikkonen"} :password "salainen" :email "mikko.mikkonen@foo.com"})
    (me/humanize)) ; => nil (ok)

(-> sign-in-schema
    (m/explain {:name {:first-name "mikko" :last-name "mikkonen"} :id 1 :password "salainen" :email "mikko.mikkonen@foo.com"})
    (me/humanize)) ; => nil (also ok)

I.e. Our name-schema is closed - no middle-name allowed. But we have left the sign-in-schema open - therefore the user of the schema can add extra keys (as :id in the example above). If you want to close the whole schema malli.util provides a function for this: closed-schema:

(ns malli
  (:require [malli.core :as m]
            [malli.error :as me]
            [malli.util :as mu]
            [clojure.data :as data]))

(data/diff (m/schema sign-in-schema) (mu/closed-schema sign-in-schema))
;; =>
;[[:map 
; [:name [:map {:closed true} [:first-name string?] [:last-name string?]]] 
; [:email string?] 
; [:password string?]]
; [:map
; {:closed true}
; [:name [:map {:closed true} [:first-name string?] [:last-name string?]]]
; [:email string?]
; [:password string?]]
; nil]

Conclusions

I’m sure the spec library is great for spec’ing the Clojure itself. But for providing a schema for REST interfaces I would use malli. spec uses macros for specifying schemas - malli uses pure data: vectors and maps. In my opinion, this makes malli nicer to work with schemas and also providing various technical advantages, e.g. you can manipulate malli schemas using any standard library functions that operate on vectors and maps.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

GCP Kubernetes Exercise

Kari Marttila — Sat, 28 Nov 2020 00:00:00 +0000

The workloads in the GKE cluster.

Introduction

I have earlier used Terraform to create Kubernetes in AWS EKS (Amazon Elastic Kubernetes Service) and Azure AKS (Azure Kubernetes Service) and therefore I wanted to create Kubernetes also in the GCP (Google Cloud Platform) to have some perspective on how different the Kubernetes infrastructure is to create in these three major clouds.

My earlier AWS EKS and Azure AKS blog posts are here:

This GCP GKE Kubernetes exercise can be found in my gcp repo in directory simpleserver-kube. I might, later on, continue with this exercise - creating a Helm chart for the Clojure simple server to be deployed to this GKE cluster.

Initialization Scripts

You can create the solution in many ways. You could have a GCP Project made for you and you have to use that specific project. I didn’t have those kinds of restrictions when making this exercise. In this exercise, I used Admin project / Infra project pattern. I.e. I create an Admin project just for hosting the GCP Service Account - the Service Account is used by Terraform to create the actual Infra project and all resources belonging to that infra project. Let’s walk through these steps (one-time initialization - later on, you just use Terraform to develop the solution).

In the init directory you can find a few scripts that I used to automate the admin project infrastructure. I gathered all relevant information I needed into the env-vars-template.sh file - one should copy-paste this template e.g. to ~/.gcp/my-kube-env-vars.sh and provide the values for the environment variables. Not all environment variables are needed, I created a bunch of them, some are for administrational purposes and used to label resources (optional), but some are needed by GCP (billing information) and to create the resources (admin / infra project id…).

Then you are ready to create the Admin project and related resources: create-admin-project.sh. This script just uses the environment variables and creates the admin project, then sets certain configuration values and creates a gcloud configuration for the admin project. Finally, we link the billing account to this admin project and enable container.googleapis.com so that the Service Account that belongs to this admin project and is used by Terraform can, later on, create GKE cluster.

We are going to store Terraform state in the GCP Cloud Storage Bucket therefore we need to create it: create-admin-bucket.sh.

Finally, we are ready to create the last admin project resource: the Service Account that is used by Terraform to create the resources in the infra project side: create-service-account.sh. The script also binds certain roles to that Service Account - e.g. the role needed to create the infra project.

The last thing to create is the infra project gcloud configurations. Since we already know the infra project id we are going to use let’s create the infra configuration right now: create-infra-configuration.sh. This way the gcloud infra configuration is ready when we start creating the infra resources - we can use this gcloud configuration to examine the resources with the gcloud cli.

Terraform Solution

I have often used a kind of “Mother Module” Terraform pattern, e.g. env-def.tf in one of my previous exercises. But this autumn I and my colleague Kimmo Koskinen created an AWS Terraform solution to be used internally in Metosin cloud training and also in our AWS projects and - my colleague Kimmo Koskinen suggested an “Independent Terraform States by Module” solution which is quite nice. It provides a nice way to create independent Terraform states per module and you can create and develop the modules independently as the name suggests. We wrote a blog post regarding this work: Terraform ECS Example. So, I used this “Independent Terraform States by Module” solution in this GCP GKE exercise. There are basically three independent Terraform modules in the terraform directory:

All modules comprise a setup.tf file that includes the Terraform google provider and the state configuration. All modules comprise also main.tf, variables.tf, and outputs.tf files - respectively giving configurations for the main resources, variables used, and outputs.

The project and vpc modules just create the infra project and the vpc used in this project and are more or less trivialities. Let’s spend some time with the kube module instead.

The GKE Terraform configuration is ridiculously small, just 60 lines. First, we create the cluster itself and then the nodes used by the cluster. The simplicity and easiness of the GKE Terraform solution was a pleasant surprise. And more surprises ahead: it took just some 60 seconds for Terraform to create the GKE cluster. I remember that in our previous project creating AWS EKS using Pulumi took quite a while.

Connecting and Testing the GKE Cluster

Use gcloud to get the credentials for the new GKE cluster:

 gcloud container clusters get-credentials YOUR-CLUSTER-NAME --region YOUR-REGION

Then you can use kubectl cli tool to examine the GKE cluster and its resources. Another nice tool is Lens.

Just to make sure the cluster works properly I deployed some dummy service to it:

kubectl create deployment hello-server --image=gcr.io/google-samples/hello-app:1.0
kubectl get pods --all-namespaces
kubectl expose deployment hello-server --type LoadBalancer --port 80 --target-port 8080
kubectl get service hello-server

You’ll get the external IP for the service and you can curl it:

λ> curl http://EXTERNAL-IP
Hello, world!
Version: 1.0.0

Resource Naming

I like to name my cloud resources so that all resources have a prefix providing information regarding the project and the environment. We used this same strategy with my colleague in the project I previously mentioned. I used this same strategy in this exercise. Example:

locals {
  workspace_name = terraform.workspace
  module_name = "kube"
  res_prefix = "${var.PREFIX}-${local.workspace_name}"
...

resource "google_container_cluster" "kube-cluster" {
  name = "${local.res_prefix}-${local.module_name}-cluster"

So, if the prefix is the project name (e.g. projx) and the Terraform workspace is e.g. dev all resources have a resource prefix projx-dev. E.g. the gke cluster name will be: projx-dev-kube-cluster. You can have many environments in the same GCP account using this pattern, e.g. dev, qa, test etc. - all environments have a dedicated Terraform state. Just to make it explicit - you should always keep your production environment in a dedicated production account.

Patterns for Creating Environments

Since it is so easy and quick to create new environments using GKE I wouldn’t simulate environments using Kubernetes namespaces in the same GKE cluster. You can do this but interacting with the other resources in the same environment would be complex (inside the Kubernetes cluster you can have a dynamic number of virtual environments but outside the Kubernetes cluster you typically have one environment, the one the cluster itself belongs to). Instead, I would just create exact copies of the environments (by parameterizing the instance sizes, of course). Read more about this pattern in my earlier blog post Cloud Infrastructure Golden Rules .

Conclusions

It was a real positive surprise how easy it is to set up a Kubernetes cluster using GCP GKE service and Terraform. Considering my experiences using Kubernetes with AWS and GCP I would recommend using the simplest solution for running containers in each cloud: with AWS, use ECS; with GCP, use GKE.

The writer is working at Metosin using Clojure in cloud projects. If you are interested to start a Clojure project in Finland or you are interested in getting Clojure training in Finland you can contact me by sending an email to my Metosin email address or contact me via LinkedIn.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

Clojure Datomic Exercise

Kari Marttila — Sat, 14 Nov 2020 00:00:00 +0000

Clojure Datomic Exercise in IntelliJ IDEA / Cursive IDE.

Introduction

In my previous Integrant exercise, I had converted my earlier SimpleServer exercise to use the Integrant state management library. In that Integrant exercise, there were three datastores: in-memory datastore that read the initial data from CSV files, AWS DynamoDB datastore, and PostgreSQL datastore. I implemented the domain layer using Clojure Protocols so that in the application it was easy to switch the datastore by changing the value in one Integrant configuration (:backend/active-db) - and reset the application state by using Integrant reset.

My next exercise was to implement a frontend to that application using re-frame, read more about that exercise in my blog article Clojure Re-Frame Exercise. Since this new Datomic exercise is basically the same exercise as the previous re-frame exercise except the new Datomic datastore added, I created this exercise into the same re-frame application directory.

So, this Datomic exercise can be found in my Clojure repo in directory re-frame.

What is Datomic?

Datomic is an immutable transactional hosted database that uses Datalog language for queries. Being hosted means that Datomic uses another data store service, e.g. DynamoDB, for persistence.

If you are interested to learn more about Datomic I list below some Datomic resources I used myself while implementing this exercise:

I strongly recommend to read Learn Datalog Today and do all exercises - a great resource to learn the Datalog language. If you are interested in watching the Day of Datomic videos there are newer videos from 2018 in Youtube. Stuart Halloway is an excellent presenter and it was a joy to watch those video presentations.

Choosing Datomic Version

Choosing the Datomic version for development was a bit confusing: should I choose Datomic dev-local, Datomic Free, or Datomic Starter? Finally, I decided to use Datomic Starter.

I used the following instructions to set up the local Datomic Starter instance:

I must say that choosing the right Datomic version was a bit confusing - not to talk about the documentation. There really should be one dedicated “start here” page which could provide easy to follow instructions for a complete Datomic newcomer.

Once you have unzipped the Datomic server installation package, configure the dev-transactor-template.properties as instructed in the Datomic documentation and then run the bin/maven-install to install the library jar. Add the dependency to your project’s deps.edn file.

Datomic Tooling

I created some Just recipes for Datomic development:

# Start local Datomic database.
@datomic:
    cd datomic && ./start-datomic.sh

# Reset Datomic SimpleServer databases.
@datomic-ss-reset:
    cd datomic && ./reset-simpleserver-db.sh

# Start Datomic Peer Server. NOTE: NOT NEEDED IN THIS EXERCISE!
@datomic-peer-server:
    cd datomic && ./start-peer-server.sh

# Start Datomic Console.
@datomic-console:
    cd datomic && ./start-console.sh

Short descriptions of the Just recipes:

datomic: Starts the Datomic database (transactor etc.).
datomic-ss-reset: Deletes and creates simpleserver and simpleserver_test databases (for clean development start).
datomic-peer-server: Starts the Datomic Peer Server. Not needed in this exercise, I just tested that it works.
datomic-console: Starts the Datomic Console server, open Chrome: http://localhost:8080/browse

The datomic-ss-reset is a bit of a hack, calls reset-simpleserver-db.sh:

#!/bin/bash
THIS_DIR=$(pwd)
TRANSACTOR_DIR=/mnt/ssd2/local/datomic-pro-1.0.6202
pushd $THIS_DIR
cd $TRANSACTOR_DIR
pwd
sleep 2
bin/repl < $THIS_DIR/reset-simpleserver-db.clj
popd

So, it starts the repl which is provided by the Datomic server. For some reason, there is no deps.edn for this repl but a cryptic shell script which populates the jar files, the reason I start it like this. I pipe the required function calls to this repl, see reset-simpleserver-db.clj:

(require '[datomic.api :as d])
(println " **********************************")
(println "Starting to reset Simpleserver databases...")
(def db-uri "datomic:dev://localhost:4334/simpleserver")
(def test-db-uri "datomic:dev://localhost:4334/simpleserver_test")
(println "Listing databases before reset...")
(d/get-database-names "datomic:dev://localhost:4334/*")
(println "Deleting databases...")
(d/delete-database db-uri)
(d/delete-database test-db-uri)
(println "Listing databases after deletes...")
(d/get-database-names "datomic:dev://localhost:4334/*")
(println "Creating databases...")
(d/create-database db-uri)
(d/create-database test-db-uri)
(println "Listing databases after creations...")
(d/get-database-names "datomic:dev://localhost:4334/*")
(def exercise-dir "/a/prs/github/clojure/webstore-demo/re-frame-demo")
(def ss-schema (read-string (slurp (str exercise-dir "/datomic/simpleserver-schema.edn"))))
(def ss-conn (d/connect db-uri))
(def test-ss-conn (d/connect test-db-uri))
@(d/transact ss-conn ss-schema)
@(d/transact test-ss-conn ss-schema)
...
(def product-groups (load-product-groups))
(def product-group-datoms (get-product-group-datoms product-groups))

@(d/transact ss-conn product-group-datoms)
...

An easy way to create the development and test databases and transact the schema to the databases, and then transact the development data to the database.

Now that we have some real data in the Datomic database let’s look at the data using the Datomic Console we started earlier with the Just recipe (open Chrome and navigate to http://localhost:8080/browse):

Datomic Console.

I ran the same query I can also run quite easily with the Clojure REPL:

  (d/q '[:find ?title ?a_or_d ?year
         :in $ ?pg-id ?y
         :where
         [?e :domain.product/title ?title]
         [?e :domain.product/year ?year]
         [?e :domain.product/a_or_d ?a_or_d]
         [?e :domain.product/pg-id ?pg-id]
          [(>= ?year ?y)]]
       (d/db ss-conn) 2 2000)

I.e. Find all products that belong to the product group 2 and are created after year 2000 (including): print the title, author/director and year. In REPL you get:

#{["A History of Violence" "Cronenberg, David" 2005]
  ["Mulholland Dr." "Lynch, David" 2001]
  ["Paha maa" "Aku Louhimies" 2005]
  ["Avatar" "Cameron, James" 2009]
  ["Hyvä poika" "Zaida Bergroth" 2011]
...

I didn’t use the Datomic Console at all during the exercise. It was a lot easier to develop and test the queries interactively using the Clojure REPL.

Experimenting with the Datomic Sample

There are quite a lot of samples with the Datomic Starter installation package. Unfortunately, there was no standard deps.edn file with the samples. Therefore I added nrepl to the run script to be able to start a nrepl server and then connect to that server using IntelliJ IDEA / Cursive to make it easier to experiment e.g. with the Seattle sample that was provided with the installation package.

I also threw the Seattle sample data to the Datomic dev server instead of the in-memory server.

Setting Up the Connection to the Datomic Database

The connection to the Datomic database is created in the Integrant state management of the application:

Integrant configuration in config.edn:

 :backend/datomic {:active-db #ig/ref :backend/active-db
                   :uri "datomic:dev://localhost:4334/simpleserver"}

And the state set up in core.clj:

(ns simpleserver.core
  (:require
...
    [datomic.api :as d]
...

(defmethod ig/init-key :backend/datomic [_ {:keys [active-db uri]}]
  (log/debug "ENTER ig/init-key :backend/datomic")
  (if (= active-db :datomic)
    {:conn (d/connect uri)}))

Using the Datomic Api

Using the Datomic api was really easy and experimenting with the api using the Clojure REPL was a real joy. Some examples.

Transact the schema to the Datomic database (reset-simpleserver-db.clj):

(def db-uri "datomic:dev://localhost:4334/simpleserver")
(d/create-database db-uri)
(def exercise-dir "/a/prs/github/clojure/webstore-demo/re-frame-demo")
(def ss-schema (read-string (slurp (str exercise-dir "/datomic/simpleserver-schema.edn"))))
(def ss-conn (d/connect db-uri))
@(d/transact ss-conn ss-schema)

The schema consists of the same datoms that you use also in ordinary queries (simpleserver-schema.edn):

[
 ; DOMAIN
 ; Product group
 {:db/ident :domain.product-group/id
  :db/valueType :db.type/long
  :db/cardinality :db.cardinality/one
  :db/unique :db.unique/identity
  :db/doc "The id of the product group"}

 {:db/ident :domain.product-group/name
  :db/valueType :db.type/string
  :db/cardinality :db.cardinality/one
  :db/doc "The name of the product group"}

 ; Product
 {:db/ident :domain.product/id
  :db/valueType :db.type/long
  :db/cardinality :db.cardinality/one
  :db/unique :db.unique/identity
  :db/doc "The id of the product"}
...

Transact data to the Datomic database (simpleserver-schema.edn):

...
(defn get-product-group-datoms [product-groups]
    (mapv (fn [product-group]
            (let [id (:id product-group)
                  name (:name product-group)]
              {:domain.product-group/id id
               :domain.product-group/name name}))
          product-groups))

(def product-groups (load-product-groups))
(def product-group-datoms (get-product-group-datoms product-groups))

@(d/transact ss-conn product-group-datoms)
...

Query datoms from the Datomic database (user-datomic.clj):

(credentials-ok?
    [_ _ email password]
    (log/debug (str "ENTER credentials-ok?"))
    (let [found (d/q '[:find ?email ?hashed-password
                       :in $ ?email ?hashed-password
                       :where
                       [?e :user.user/email ?email]
                       [?e :user.user/hashed-password ?hashed-password]]
                     (d/db conn) email (str (hash password)))]
      (if found
        (= found #{[email (str (hash password))]})
        false)))

Development Flow

The development flow using the Clojure REPL and Integrant was really nice. Usually, I experimented the queries in my scratch file (see more about this technique in my blog post Clojure Power Tools Part 1). Once I was confident the query worked I moved the code snippet from the scratch file to the production source file and reset Integrant state (with IntelliJ IDEA hotkey, of course), and ran the unit tests to see that the code worked also as part of the application.

Conclusions

It was really fun to do this exercise. The Day of Datomic video presentations were really good, also other material regarding Datomic and the Datalog query language. The exercise itself was quite effortless to implement using Datomic.

I would definitely use Datomic with Clojure in real projects, too. The idea of an immutable database is a real nice fit for many use cases.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

Clojure Power Tools Part 2

Kari Marttila — Thu, 29 Oct 2020 00:00:00 +0000

Introduction

This is the second part of my Clojure Power Tools series (I’m a bit interested myself how many blog posts I will write to this series). If you haven’t read the first part I recommend you to read it first: Clojure Power Tools Part 1. In this second blog article, I list a couple of new power tool tricks for debugging. I first introduce a poor man’s debug repl, and then the real debug repl.

Poor Man’s Debug Repl

I was refactoring some Clojure tests for the second version of a complicated application. There were quite a few structural changes in the application logic, and the domain was rather complicated comprising a lot of various recursive data structures. In some tests, I was quite puzzled about what kind of state there was in several places of the application. The tests were also rather complicated and I wanted a simple way to record certain bindings in several places of the application when I was running a certain area of the test - and only that area. I quickly implemented a poor man’s debug recorder:

(ns recorder)

(def value (atom {}))
(def counter (atom 0))
(def flag (atom false))

(defn next-id []
  (swap! counter inc))

(defn assoc-value-with-id! [k v]
  (let [id (next-id)
        new-k (keyword (str (name k) "-" id))]
    (swap! value assoc new-k v)))

(defn add-value-if-recording! [k v]
  (when @flag
    (assoc-value-with-id! k v)))

(defn start! []
  (reset! value {})
  (reset! counter 0)
  (reset! flag true))

(defn stop! []
  (reset! flag false))

The idea is to provide a way to add stuff to the value map with unique keys - but only when I’m recording.

(I’m using my own exercises as an example here, the data here is ridiculously simple - no need for a recorder here, but I guess you get my point if you change the data to several hundreds of lines of recursive maps and vectors and complex business logic…)

So, let’s add the start/stop recording to the test we are interested:

(deftest product-groups-test
  (log/debug "ENTER product-groups-test")
  (testing "GET: /api/product-groups"
    (let [_ (re/start!)
          login-ret (ss-tc/-call-api :post "login" nil {:email "test-kari.karttinen@foo.com" :password "Kari"})
          _ (log/debug (str "Got login-ret: " login-ret))
          json-web-token (get-in login-ret [:body :json-web-token])
          params (-create-basic-authentication json-web-token)
          get-ret (ss-tc/-call-api :get "/product-groups" params nil)
          status (:status get-ret)
          body (:body get-ret)
          right-body {:ret "ok", :product-groups {:1 "Test-Books", :2 "Test-Movies"}}
          _ (re/stop!)]
      (is (= (not (nil? json-web-token)) true))
      (is (= status 200))
      (is (= body right-body)))))

See the (re/start!) and (re/stop!) commands in the test - we are recording only during that time.

Then I can add stuff to my recorder where-ever I want:

(defn -valid-token?
  "Parses the token from the http authorization header and asks session ns to validate the token."
  [env req]
  (log/debug "ENTER -valid-token?")
  (let [basic (get-in req [:headers "authorization"])
        _ (re/add-value-if-recording! :valid-token-basic basic)
        basic-str (and basic (last (re-find #"^Basic (.*)$" basic)))
...

I.e. (re/add-value-if-recording! :valid-token-basic basic).

Run the tests.

After the tests I can examine the recorder in my scratch file:

  (require '[recorder])
  (keys @recorder/value)
  @recorder/value
  (require '[portal.api :as portal-api])
  (portal.api/clear)
  (portal.api/open)
  (portal.api/tap)
  (tap> @recorder/value)

You can also use the excellent portal visualization tool to examine your complicated data you recorded (well, in this example, not so complicated):

Real Debug-Repl

The example above was a simple trick but a real Clojurian stops the world if he is interested to see what is happening in a particular moment of time instead of just recording it. We are going to use Gary Fredericks’s debug-repl library for it. You need the dependency - I have these tool dependencies in my ~/.clojure/deps.edn file:

{
 :aliases {
           :kari {:extra-paths ["scratch"]
                  :extra-deps {hashp/hashp {:mvn/version "0.1.1"}
                               com.gfredericks/debug-repl {:mvn/version "0.0.11"}
                               djblue/portal {:mvn/version "0.6.1"}
                               }
                  }
           }
 }

Then we need to be able to start the backend REPL with this debug repl middleware, I have this in my Justfile as a Just recipe which I use to start my backend repl with various options:

# Start backend repl with my toolbox and with debug-repl capability.
@backend-debug-kari:
    clj -M:dev:test:common:backend:kari -m nrepl.cmdline -m com.gfredericks.debug-repl/wrap-debug-repl -i -C

Then you can add a debug-repl breakpoint in your code, run your code, stop the world and examine the snapshot context of your world just before the breakpoint. Let’s see this in action:

I have added _ (break! "Yihaa!") breakpoint in one of the tests. Then I run the test and the world stops at the breakpoint (Hijacking repl for breakpoint: Yihaa! output in the REPL output window). Then I have moved the cursor in various let-bindings: json-web-token, params and get-ret and evaluated the forms: you can see the evaluated values in the REPL output window on the right. If I try to evaluate status I get an error “Unable to resolve symbol: status in this context” - of course, because it’s outside the context.

Conclusions

There are a lot of other Clojure tricks and tools - maybe I’ll write “Clojure Power Tools Part 3” blog post in the future.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

Clojure Power Tools Part 1

Kari Marttila — Mon, 26 Oct 2020 00:00:00 +0000

Introduction

I have been working at Metosin for some months now and I’m learning new Clojure tricks almost every day. I list here some of my favorite Clojure tricks that I have learned either from various Clojure presentations or at Metosin. I know that for a newcomer Clojure might be a bit weird language and newcomers usually don’t know simple productivity tricks. Hopefully, this presentation makes it a bit easier for newcomers to start being productive with Clojure.

Start Doing It

If you are a complete newcomer in the Clojure land I have one recommendation: Just start doing it. Choose an editor, install the required tools, and start learning. The most important thing is that you start learning with a Clojure REPL. And with an editor that has good integration with the Clojure REPL. There are several options:

I believe those are the most used editors with good REPL integrations, at least according to the latest State of Clojure.

If you are a complete newcomer a good idea is to choose an editor that you already know. Install the Repl plugin to that editor and learn to use it. Especially learn how to send the forms (Clojure expressions) from the editor to the REPL for evaluation.

Once you have a working development environment start learning Clojure. A good resource is Brave Clojure. And remember to do the exercises and experiments with your editor and send the forms to REPL for evaluation using your hotkey!

Use a REPL Scratch File

I believe that newcomers often write their experiments in the REPL editor, at least I did. Don’t do it. Instead, create a scratch file and write your experiments there. I watched some Stuart Halloway presentation in which he told that he has a dedicated Clojure directory in which he has written all his Clojure experimentations for several years - must be pretty nice to be able to grep when looking for something you have written years ago. I have another habit. I have in my ~/.clojure/deps.edn file:

{
 :aliases {
           :kari {:extra-paths ["scratch"]
                  :extra-deps {hashp/hashp {:mvn/version "0.1.1"}
                               com.gfredericks/debug-repl {:mvn/version "0.0.11"}
                               djblue/portal {:mvn/version "0.6.1"}
                               }
                  }
           }
 }

Focus on row :kari {:extra-paths ["scratch"]. This line adds directory scratch into the Clojure path in any project in which I add my personal profile kari, e.g.:

clj -M:dev:test:common:backend:kari -m nrepl.cmdline -i -C

I.e. All projects have various aliases that you need to use when starting the REPL. I just add my personal alias kari at the end and then I’m able to create scratch.clj file in that scratch directory and write all project specific Clojure experimentation there. I think this is also a nice way - I have all my project related Clojure experimentation in one place. A screenshot from my IntelliJ IDEA showing the scratch directory and two scratch files: one for backend experimentation (scratch.clj - content showing in the editor window) and one for frontend experimentation (scratch-cljs.cljs).

Rich Comments

Clojurians use so-called rich comments. These are typically small code snippets at the end of the file surrounded in comment form. This way the Clojure code inside the rich comment is valid Clojure code that is read by the Clojure reader but it is not evaluated. Therefore you can put all kinds of namespace specific experimentation or code examples in the rich comment block. Example from one of my exercises: domain_postgres.clj:

(comment
  (simpleserver.test-config/go)
  (simpleserver.test-config/halt)
  (simpleserver.test-config/test-env)
  (let [db (get-in (simpleserver.test-config/test-env) [:service :domain :db])]
    (sql-get-products db {:pg-id (str 2)}))
  (let [db (get-in (simpleserver.test-config/test-env) [:service :domain :db])]
    (sql-get-product db {:pg-id (str 2) :p-id (str 4)}))
  )

It’s pretty easy later on to remember stuff in those rich comments. E.g. in the above-mentioned example I have some functions to start/halt the Integrant test state and some functions in which I test some SQL operations in the domain using the database connection from the test state.

Use Defs Inside Functions When Debugging

This is a classic Clojure trick. If you are wondering what is happening inside some function - just take a snapshot of the most important entities in that function. Let’s look at the same file, domain_postgres.clj:

(if-let [kv (sql-get-product db {:pg-id (str pg-id) :p-id (str p-id)})]

I wonder what kind of data structure is returned from the function and bound to kv? Let’s see:

    (if-let [kv (sql-get-product db {:pg-id (str pg-id) :p-id (str p-id)})]
      (let [_ (def todo-kv kv)]
        [(:id kv)

The magic is in the let I added: (let [_ (def todo-kv kv)]: we just create a var (todo-kv) and bind the value of kv to it.

Then let’s run the tests so that this function gets called and after the test let’s examine todo-kv:

todo-kv
=>
{:id "4",
 :pg_id "2",
 :title "Test Once Upon a Time in the West",
 :price 14.40M,
 :a_or_d "Leone",
 :year 1968,
 :country "Italy-USA",
 :g_or_l "Western"}

Use Hashp for Printing Values

Sometimes you want to preserve the data structure and examine it in your scratch file - then the def trick described in the previous chapter is all you need. But sometimes you just want to see the value. You could use e.g. clojure.pprint to print the value of the var but there is another nice power tool for it: hashp. E.g. in your rich comment evaluate: (require '[hashp.core]) and then add #p before the function you are interested in, e.g.:

(if-let [kv #p (sql-get-product db {:pg-id (str pg-id) :p-id (str p-id)})]

Evaluate the namespace (or reset Integrant state or something similar) and run the tests and you will see an output like:

Testing simpleserver.service.domain.domain-test
#p[simpleserver.service.domain.domain-postgres.PostgresR/fn:37] (sql-get-product db {:p-id (str p-id), :pg-id (str pg-id)}) => 
{:a_or_d "Leone",
 :country "Italy-USA",
 :g_or_l "Western",
 :id "4",
 :pg_id "2",
 :price 14.40M,
 :title "Test Once Upon a Time in the West",
 :year 1968}

Use Panel to Visualize Data

This is pretty cool and I learned this only this week from one Metosin Clojure guru (aah, I need to remember to write a chapter: “Find a Clojure Shop”).

If you have a complex domain in which you have a lot of data with a lot of children and those children having children and so on - it is a bit daunting to try to visualize this in the REPL output window or using the REPL to navigate in the data. portal to the rescue!

Write the following forms in your scratch file or in the rich comment of your namespace - I’ll use the same domain_postgres.clj file as an example.

  (require '[portal.api :as portal-api])
  (portal.api/open)
  (portal.api/tap)
  (tap> (let [db (get-in (simpleserver.test-config/test-env) [:service :domain :db])]
    (sql-get-products db {:pg-id (str 2)})))

So, the (portal.api/open) opens the visualization window, (portal.api/tap) adds portal as a tap, and then using tap> you can send data to the visualization window. See example:

This is a very simple example. But imagine that there are hundreds of lines of data, vectors, maps, more vectors and maps inside them, etc. A visualization tool like Portal is a must-have tool. There are other similar visualization tools - the Clojure itself has one nowadays: clojure.inspector

Find a Clojure Shop

If you really want to learn Clojure I strongly recommend you to find a job in which you are able to spend 8 hours every workday with Clojure - with other seasoned Clojurians. I really can’t overemphasize how important this is. I learned the basic stuff of Clojure alone, just reading books, googling stuff, doing exercises. I had a long career in the corporate world and one day I realized that I have paid my mortgage and my kids are adults and I can basically do whatever I like with my life - and I asked myself: What do you want really want to do? I realized that I want to do Clojure & cloud projects. I contacted the most prominent Clojure shops in Finland and got a job at the best of them, Metosin. Damn, it’s been a jolly ride with Metosin. The company is full of world-class Clojurians, just look at some extremely popular Metosin libraries like reitit. And I got a chance to work on the same project with guys who build these libraries. I should be paying them instead of the company paying me. What incredible luck. In a few months, I have already learned so much from these guys. And learning more and more every day.

So. If you really want to learn Clojure, it is really, really important to find a great Clojure shop where you can learn from the best.

Conclusions

There are a lot of other Clojure tricks and tools - maybe I’ll write “Clojure Power Tools Part 2” blog post in the future.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

Clojure Re-Frame Exercise

Kari Marttila — Thu, 15 Oct 2020 00:00:00 +0000

Clojure Re-Frame Exercise in IntelliJ IDEA / Cursive IDE.

Introduction

I created a simple frontend for my SimpleServer exercise a couple of years ago: SimpleFrontend. In that exercise I used Reagent library which is a minimalistic React wrapper for ClojureScript - I have documented the exercise in Become a Full Stack Developer with Clojure and ClojureScript! blog post. At my company, Metosin, we also use quite a lot another ClojureScript framework that provides some additional utilities like state management - Re-Frame. In this blog post I describe my recent Clojure exercise when I re-implemented the SimpleFrontend using Re-Frame.

The exercise can be found in my Clojure repo in directory re-frame.

What is ClojureScript?

I wrote this frontend exercise using ClojureScript. ClojureScript is a Clojure dialect that transpiles to Javascript. So, if you are a Clojurian, you love functional paradigm, immutable data structures, and clean code - you can use your favorite functional language both in the backend (Clojure) and frontend (ClojureScript).

And what is Clojure? Possibly the most beautiful language in the world. Once you learn it, you never go back.

What are Reagent and Re-Frame?

Reagent library is a minimalistic React wrapper for ClojureScript. With Reagent you can easily create React components using Hiccup which provides a very “Clojurian” way to represent HTML code. Reagent also introduces its own version of the Clojure Atom - any component that dereferences a Reagent atom will be automatically re-rendered when there are changes in that atom.

Re-Frame provides a framework that uses Reagent internally. The Re-Frame framework adds various capabilities e.g. an application database into which you can store the application state and an easy way to subscribe to the elements in the application database and get automatic re-rendering when those elements change in the application database. I’m not going to explain the Re-Frame framework in more detail - the Re-Frame website provides excellent documentation for the framework.

Tooling

I did most of the tooling as I have learned to use them in the Metosin frontend projects (by the way, if you are a Clojurian and you want to learn from the best - join Metosin).

Shadow-cljs. Shadow-cljs is a build tool for ClojureScript. Shadow-cljs is really nice to use - you get to see your changes both in the ClojureScript code and Sass code in real-time in the browser. You can also get a REPL which runs your ClojureScript code in the browser (see the example in the picture above - I have a scratch file in which I have written some ClojureScript code and I send the forms for evaluation to the REPL running in the browser - you can e.g. examine the re-frame app db using the REPL). Shadow-cljs configuration, see: shadow-cljs.edn.
Justfile. I use Just to provide commandline interface to setup the project fixture before development, e.g. just postgres: start the PostgreSQL development database, just backend-kari: start backend repl with my own Clojure config, just frontend-kari: start frontend etc. Justfile configuration, see: Justfile.
Npm. Shadow-cljs integrates nicely with Npm. See package.json for a list of npm packages I’m using in this exercise.
Sass. I’m using Sass which is an extension language for CSS. The Sass in this exercise is pretty minimalistic - the purpose of this exercise was not to create a beautiful frontend but to learn to use the tooling and re-frame. See example in main.scss.
Deps.edn. In the deps.edn file you can find the frontend alias which gives the dependencies for the frontend in this exercise.
Metosin Reagent-dev-tools. I used quite a lot the excellent Metosin Reagent-dev-tools (as you can see in the picture below). I tried the re-frame-10x tool as well, but I liked more the Metosin Reagent-dev-tool’s visual layout (and I’m a company man - you eat your own dog food). The Metosin Reagent-dev-tool was one of the most important development and debugging tools during this exercise. Once you configure the tool to show the Re-frame application db state you get a nice tree view to the app-db:

(defn ^:export init []
  (js/console.log "ENTER init")
  (re-frame/dispatch-sync [::initialize-db])
  (dev-tools/start! {:state-atom re-frame.db/app-db})
  (dev-setup)
  (start))

Routing

For routing, I used Reitit both in the backend and in the frontend.

(def routes-dev
  ["/"
   [""
    {:name ::sf-state/home
     :view home-page
     :link-text "Home"
     :controllers
     [{:start (fn [& params] (js/console.log "Entering home page"))
       :stop (fn [& params] (js/console.log "Leaving home page"))}]}]
   ["signin"
    {:name ::sf-state/signin
     :view sf-signin/signin-page
     :link-text "Sign-In"
     :controllers
...

Since I’m a company man I’m using Metosin libraries, of course. I examined Metosin Reitit + re-frame example and based my work on it. The example provides a simple solution for frontend routing based on the Reitit library

Using Re-Frame

I created a dedicated ClojureScript namespace for every view (see directory simplefrontend ). Using Hiccup, Reagent and Re-frame you can quickly create React application even though you know almost nothing of React itself. Example products.cljs:


(defn products-page
  "Products view."
  [match] ; NOTE: This is the current-route given as parameter to the view. You can get the pgid also from :path-params.
  (let [_ (sf-util/clog "ENTER products-page, match" match)
        {:keys [path]} (:parameters match)
        {:keys [pgid]} path
        pgid (str pgid)
        _ (sf-util/clog "path" path)
        _ (sf-util/clog "pgid" pgid)]
    (fn []
      (let [products-data @(re-frame/subscribe [::products-data pgid])
            product-group-name @(re-frame/subscribe [::product-group-name pgid])
            _ (if-not products-data (re-frame/dispatch [::get-products pgid]))]
        [:div
         [:h3 "Products - " product-group-name]
         [:div.sf-pg-container
          (products-table products-data)]
         [:div
          [:button.sf-basic-button
           {:on-click (fn [e]
                        (.preventDefault e)
                        (re-frame/dispatch [::sf-state/navigate ::sf-state/home]))}
           "Go to home"]
          ]
         (sf-util/debug-panel {:products-data products-data})]))))

In the page, you can subscribe to re-frame subscriptions (example: products-data @(re-frame/subscribe [::products-data pgid]) ). Whenever that data change the React component gets re-rendered. The paradigm is really easy to understand and work with.

You can trigger co-effects to populate data in your app-db, example:

(re-frame/reg-event-fx
  ::get-products
  (fn [{:keys [db]} [_ pg-id]]
    (sf-util/clog "get-product, pg-id" pg-id)
    (sf-http/http-get db (str "/api/products/" pg-id) nil ::ret-ok ::ret-failed)))

In this code snippet we are sending a get request to the backend to get the products for a given product group id. Then the ::ret-ok handler gets triggered when we get the response from the server:

(re-frame/reg-event-db
  ::ret-ok
  (fn [db [_ res-body]]
    (sf-util/clog "reg-event-db ok: " res-body)
    (let [pgid (:pg-id res-body)]
      (-> db
          (assoc-in [:products :response] {:ret :ok :res-body res-body})
          (assoc-in [:products :data pgid] (:products res-body))))))

As you can see we are storing the data in the re-frame app-db. The actual data needed is stored in map in hierarchy [:products :data pgid] but I’m storing the whole response there also just for development, learning and debugging purposes.

Ok, now we have the data in the re-frame application database. Then in the view you just make a subscription for that data: products-data @(re-frame/subscribe [::products-data pgid]). The actual subscription registration is:

(re-frame/reg-sub
  ::products-data
  (fn [db params]
    (sf-util/clog "::products-data, params" params)
    (let [pgid (second params)
          data (get-in db [:products :data])
          _ (sf-util/clog "products-data" data)]
      (get-in data [pgid]))))

So, we just fetch the data from the re-frame application database and return it to anyone who subscribes to that data. When the data changes in the re-frame application database the React component that uses that data gets re-rendered.

Development Flow

ClojureScript REPL. I’m an old backend programmer and I haven’t done that much frontend development, not to speak of Single Page Application development. But using ClojureScript, Hiccup, Reagent, Re-Frame, and Shadow-cljs made the journey quite effortless. In the picture at the beginning of this article, you can see a screenshot of my IntelliJ IDEA / Cursive setup. When I’m implementing some Clojure application I always keep a Clojure REPL running (as all Lisp programmers - the REPL is an integral part of Lisp programming). In this exercise I kept two REPLs running: the backend REPL (running in the JVM, state managed by Integrant, see my other blog post about that: Clojure Integrant Exercise ) and the frontend REPL (running in the browser). I guess most of the Javascript/Typescript programmers can’t understand how powerful a Lisp REPL can be in frontend development, you just have to use it yourself to understand it fully. If you look at the scratch file and the REPL output window you can see that you can use any Javascript commands (console, alert…) and also examine and modify the running state of the frontend application.

Metosin Reagent-dev-tools. I already mentioned the Metosin Reagent-dev-tools in the Tooling chapter. Reagent-dev-tools was an integral part of the development workflow. You could see in real-time the current state of the application in the re-frame application database using the tool tree view. An excellent tool for development and debugging, I highly recommend it.

Chrome DevTools. I’m not that fluent with the Chrome DevTools but I used it quite a lot. The tool is really nice: you can print in the console log various debugging information, you can inspect elements and debug various CSS related issues, you can inspect the http requets and responses etc. But I guess all frontend developers already know this stuff.

The following picture shows the Metosin Reagent-dev-tools (bottom part of the picture) and the Chrome DevTools (right side).

Browser view and development tools.

Because it was so easy I also implemented a very simple Debug panel for my own debugging purposes:

(defn debug-panel
  "Debug panel - you can use this panel in any view to show some page specific debug data."
  [data]
  (let [debug @(re-frame/subscribe [::sf-state/debug])]
    #_(js/console.log (str "ENTER debug-panel, debug: " debug))
    (if debug
      [:div.sf-debug-panel
       [:hr.sf-debug-panel.hr]
       [:h3.sf-debug-panel.header "DEBUG-PANEL"]
       [:pre.sf-debug-panel.body (with-out-str (clojure.pprint/pprint data))]])))

Then you can use this debug-panel in any view adding it as a React component e.g. to the bottom of the page:

(defn login-page
  "Login view."
  []
...
         (sf-util/debug-panel {:login-data login-data
                               :ret ret
                               :msg msg
                               :r-body r-body})]))))

… and show there whatever data is important for debugging purposes in that view.

The following picture shows debug panel in action in that page:

Custom Debug Panel.

Styles

I’m using raw CSS with Sass. I considered using some React UI library like Ant Design or Material UI but then decided to use raw CSS/Sass instead - the rationale being that this is a re-frame exercise and I didn’t want to spend time in this exercise learning some UI library. I might do another exercise later on in which I convert this frontend to use e.g. Ant Design which has a nice ClojureScript wrapper (syn-antd).

A Sass example giving the style for the custom debug panel:

.sf-debug-panel {
  .hr {
  margin-top: 100px;
  }
  .header {
    font-size: 28px;
  }
  .body {
    font-size: 12px;
  }
}

Issues

Some issues I need to fix later.

index.html in uri. I’d like to get rid of the index.html in uri, e.g. http://localhost:6161/index.html#/products/1 should be http://localhost:6161/#/products/1.

edn data in http requests. I tried passing edn instead of json in http requests but I had some issues with it (e.g. the response body came just fine as edn, but re-frame error handler got triggered even though the reply was 200).

Live Reloading

Live reloading worked really well. If you make any changes to any cljs or sass file, you could see the changed React components re-rendering in that view in realtime. Using the tooling and libraries mentioned in this exercise was a really nice experience.

Conclusions

It was an interesting journey to do this exercise. As an old backend developer frontends have always been my Achilles’ heel. When implementing this frontend exercise I needed to learn about the ClojureScript tooling, React (just a bit), Reagent and Re-Frame and the overall workflow related to the frontend development - very beneficial for my future frontend projects. If you are an old backend programmer like me and you would like to have an easy environment to start working with Single Page Application frontends - ClojureScript with Reagent and Re-frame is a good choice.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

Dygma Raise Keyboard Reflections Part 1

Kari Marttila — Mon, 28 Sep 2020 00:00:00 +0000

My Dygma Raise Keyboard.

Introduction

I have been using Microsoft Natural Keyboard 4000 for several years and it has served my programming and writing needs pretty well with my Ubuntu laptop .Xkeymap configurations (more about that later). But last spring I followed Koodiklinikka Slack’s “nappaimistot” (“keyboards” in English) channel in order to read experiences regarding mechanical keyboards. There I read some experiences regarding the Dygma Raise Keyboard. I ordered the Dygma Raise keyboard while the company was manufacturing the second batch of the keyboard. I ordered the keyboard in May and I got the keyboard a couple of days ago. Now that I have spent a couple of days configuring the keyboard and using it I thought that it might be beneficial for other programmers to write a bit about my experiences regarding the Dygma Raise - because I fell in love with it immediately.

Build Quality

The build quality of the Dygma Raise Keyboard seems to be really good. The chassis is very rigid and sturdy. All keys feel very consistent and there is an overall feeling of premium quality when using the keyboard.

Ergonomics

You can split the keyboard into two halves and place the halves on your table as you wish. I keep some space between the halves and also have a small angle between the halves so that my shoulders and wrists are in a pretty natural position on my table. I also have a large elbow support on my table. All these together provide pretty good ergonomics for my programming. Not to talk about the ergonomics regarding the key presses but more about that in the following chapters. Dygma provides in its website a more detailed description regarding the Dygma Raise Ergonomics.

Led Lights

I haven’t found any real use for the various colors - I do programming. I type with all my ten fingers - I took a typing lesson in the Finnish Elementary school about 40 years ago and have been typing with 10 fingers ever since - so, I don’t look at my keyboard at all when typing. But the colors are kind of nice anyway. Maybe gamers use these colors for some real tasks or they just “look cool”, I don’t know. Anyway, I don’t mind the colors. I did use the Bezecor to categorize certain buttons with consistent colors (see the two pictures below). And it is also kind of nice that the Shift to 1 button also changes the color in the frame of the halves - a visual aid that we are now in the next layer.

Layout

One of the most important reasons to buy Dygma Raise was that I could order it with the Nordic layout. There are other split keyboards in the market but some of them provided just ANSI layout, and the rest didn’t provide the kind of programmability of the keyboard like Dygma Raise. The keyboard layout itself is really important to me. There are certain special characters in the Nordic languages (e.g. Ä and Ö) and I want those keys to be in their right places so that I can write Finnish text fluently both using the laptop keyboard and my external keyboard and I don’t need to remember that those letters are in different positions in different keyboards. There are many different ANSI and ISO layouts available for the Dygma Raise - consult the Dygma Raise website for the options.

Switches

Dygma Raise is a mechanical keyboard and you can order the kind of mechanical switches you like. Dygma Raise is my first mechanical keyboard for recent years so I ordered rather standard switches not to have too extreme experience to start with: Cherry MX Brown. Dygma Raise provides a good introduction to the mechanical switches - I really recommend reading it before placing the order for your Dygma Raise. All Dygma Raise switches are swappable so you can change the switches later on if you think that you want to experiment with different switches. I kind of like the “clicky” touch of the Cherry MX Brown switches but I might next try a bit “lighter” touch (but with a tactile feeling with a clear click sound).

My Linux Keymap

Before understanding how I configured Dygma Raise I need to tell the reader a bit about my Linux key mapping. For a programmer, the CapsLock key is a totally useless key. In the Nordic layout the various parentheses ( {, [, [, }) are in the first row behind 7, 8, 9 and 0 keys when you press Alt-Gr key at the same time. The problem is that this Alt-Gr key is in a really awkward position in the last row behind your right thumb - practically impossible to press this Alt-Gr and 7, 8, 9 and 0 keys without getting a Carpal Tunnel Syndrome in your wrists after a few years. Therefore many Nordic programmers tend to configure these special parentheses in new positions. My solution was to keep the special character keys where they are but to configure CapsLock key as Alt-Gr key since hitting the CapsLock key with my left little finger and at the same time hitting 7, 8, 9 and 0 keys with my right hand fingers to produce {, [, [, } was relatively easy. Since I could now configure keys with CapsLock for special functions, I also configured I, J, K and L keys to be arrow keys with CapsLock, D to be delete with CapsLock (as Ctrl+D is delete in Emacs) etc.

A screenshot regarding some of these configurations (.Xkeymap file):

Configuring Dygma Raise: The Bazecor Software

Ok, let’s go back to Dygma Raise and how my Linux Keymap works now with Dygma Raise. It turned out that Linux Keymap and Dygma Raise is a match made in heaven.

The configurability of the Dygma Raise keyboard is just superb. This is something that I’m just beginning to realize after configuring the keyboard for a couple of days. You can use your imagination as much as you like and it seems that Dygma does not restrict you in any way.

Every key is programmable. And there are ten layers and in all of those layers, you can program every key or configure the key to be “transparent” - meaning it will function as the key in the previous layer. I have configured two layers using a rule of thumb that I want all special characters to be as near the base row (ASDF & JKLÖ) where I keep my fingers while resting.

One very neat feature is also that there are eight “thumb keys” instead of one big spacebar. You really should give some thought on how to use these thumb keys since they can be really powerful. Let’s start with the thumb keys. Below you can see my Layer 0 configuration in the Bazecor application:

I have played classical guitar some 20 years and now that I look at that picture I realize something regarding the hotkeys I tend to use. Like playing the guitar one has different functions for the left hand and for the right hand: left hand fingers press the strings in various positions on the guitar neck, and right hand fingers pick the strings. The same way I tend to use my keyboard: I press some combination of the Shift, Ctrl, Alt and CapsLock (remember: my CapsLock is the Alt-Gr key) with my left hand fingers, and then I click some key on the right side of the keyboard with my right hand fingers. Therefore I configured my new Dygma Raise so that I have all relevant modifiers in the lefthand side four thumb keys: Alt, Shift, Ctrl and Shift-to-1. Some examples of how I use these combinations may shed some light to the importance of these new left hand side thumb keys. I program Clojure and I use either IntelliJ IDEA with excellent Cursive plugin or Emacs editor with Cider plugin (mostly IDEA/Cursive, though). In Clojure you program quite a lot with REPL and to make the Clojure programming workflow fluent one needs good hotkeys e.g. to reload stuff into the REPL, send namespace to the REPL for evaluation, evaluate certain S-expression in the Clojure code or edit the code, e.g. kill S-expression, slurp/barf, etc. I’m not going to explain the meaning of those manipulations but let’s just say that I do a lot of them when programming Clojure and there needs to be fast hotkeys for them. Some examples of those hotkeys:

Kill text from cursor position till the end of line (Emacs kill): Ctrl+K. Easy, now I don’t stretch my left hand little finger to the Ctrl key: I have Ctrl as a thumb key on both sides.
Kill S-expression in Clojure code: Shift+Ctrl+K. Adding Shift since it is logical because it´s kind of kill like the previous one. Now very easy with Dygma Raise: Just press with left hand thumb the lower thumb keys (Left Shift and Left Ctrl - at the same time) and with right hand middle finger press K. This used to be a bit awkward before Dygma: I had to press the left side Shift and Ctrl keys with my left hand little finger…
Send Clojure form for evaluation to REPL: Alt+L: Now very easy: Left Alt with left hand thumb and L with right hand ring finger.
Move cursor to right: CapsLock+L (remember: CapsLock is Alt-Gr and I have configured I, J, K and L as arrow keys).
Because move the cursor to right is CapsLock+L it is logical that Slurping right is CapsLock+Alt+L: easy, left hand: just press Alt with thumb and CapsLock with little finger.
Send text to Slack: Ctrl+Enter: Now very easy: left hand thumb and right hand thumb in the lower thumb keys.

And so on, and so on… I could write a whole series of blog posts regarding these hotkeys - I have a bunch of them for various IDEs, editors, tools, and terminals, but I guess you got the point. I now realize that I play my keyboard a bit like the guitar: both hands tend to have certain functions: I use modifiers with my left hand and give the actual commands with my right hand. (With some exceptions, like CapsLock+D for delete, but that´s because it is logical to do so, since Ctrl+D is delete in emacs and CapsLock and D are so close to each other that it is easy to do so…).

Ok, let’s move on. What’s that Orange Shift to 1 thumb button in the left upper row? It’s the beauty of Dygma Raise: You can shift to the next layer when pressing this key (you can also configure some key so that you move to the next layer instead of activating it only when this key is pressed, but it is better for my personal workflow to have it as “shift”). Let’s see what´s there in the Layer 1:

In this layer I have some Media controllers on the left side just to make playing Spotify easier and also to control volume in meetings: I.e. I press the Shift to 1 with my left hand side thumb and then some media buttons also with my left hand side. This is a bit awkward since I have to use the same hand but as a guitarist, I have pretty flexible fingers and I don’t use these buttons that often. But then the actual beef: Why do I have the numbers on the right hand side in two rows?. The actual reason is not the numbers per se but what’s behind the numbers. The most important row in this layer is the resting position row J, K, L, Ö => 8, 9, 0, +. The reason is that I have mapped the various parentheses behind these numbers, for example: { => Alt-Gr+7, and because Alt-Gr is Caps-Lock it is Caps-Lock+7, and because in Layer 0 I can shift to the next Layer 1 with the left hand side button key, and the number 7 is not in the upper row in this layer but in the resting row, it is very easy to get { => Shift to 1+CapsLock with my left hand fingers (thumb and little finger, I use this very often so it is very easy for me) + H (layer 0) (which is 7 in layer 1). Therefore:

{ => Shift to 1+CapsLock+H
[ => Shift to 1+CapsLock+J
] => Shift to 1+CapsLock+K
} => Shift to 1+CapsLock+L
\ => Shift to 1+CapsLock+Ö
( => Shift to 1+Shift+J
) => Shift to 1+Shift+K

Every programmer knows that various parentheses are very much used in any programming language, and Clojure is no exception. On the contrary, you use parentheses very much in Clojure programming: every S-expression needs to be in parenthesis, all literal vectors are inside brackets, all literal maps are inside curly braces, etc. Now with Dygma, I can write all of these parentheses in my right hand resting position.

How about the upper number row with smaller numbers. The special characters:

! => Shift to 1+Shift+U
" => Shift to 1+Shift+I
# => Shift to 1+Shift+O
% => Shift to 1+Shift+Å
& => Shift to 1+Shift+~
@ => Shift to 1+CapsLock+I

So, just the same characters behind those numbers in the Nordic keyboard layout but now just added to the right hand side near the resting row for easier access.

And Esc is now Shift to 1+CapsLock+Y, what a relief. Since e.g. in IntelliJ IDEA when going to the embedded terminal I haven’t found any other way to return to the editor than clicking Esc twice and Esc used to be so far away… not anymore with Dygma.

When I read this article for proof-reading purposes I also realized that I tend to prefer combinations with near the resting positions (like the Shift to 1+CapsLock+Y I told earlier) over just one keypress but farther away (like Esc in the upper left side of the keyboard). This is personal, of course. Maybe the reason is that I write with all 10 fingers and it is faster to write when you don’t have to stretch your fingers far away.

BTW. If you are wondering why I don’t have e.g. Delete and Backspace keys configured, I have but not using Bazecor. I have a lot of keys mapped with my custom CapsLock key using Linux .Xkeymap configuration, e.g:

D: Delete
F: Backspace
I, J, K and L: arrow keys.
U: Page up
O: Page down
H: Home
Ö: End

… and it is pretty easy to hit CapsLock with my left little finger and one of those keys with my right hand fingers (except D and F using left hand, of course). These key mappings work the same way if I use my laptop keyboard or Dygma Raise - which is a good thing: they can be in my muscle memory for any keyboard I use with my Ubuntu (in fact, I have these mappings in all my Ubuntu machines at home).

What Next?

Who knows? Maybe I realize that there is a way to make my new Dygma Keyboard even more ergonomic with some new astounding realization how to configure it using the Bazecor.

One thing that I have been thinking about with the split keyboard is to use a trackpad instead of the mouse - I could use my thumbs with a trackpad positioned between the keyboard halves now that I have space for it there - thanks to Dygma Raise split keyboard.

Update 2020-10-03

I have now used my brand new Dygma Raise about a week. I have forced myself to use the new key mappings I configured and I have done some fine-tuning. This chapter is an update to the original blog post.

Forcing oneself to use the new key mappings. Learning the new key mappings will take some time. After one week I’m still stumbling to find the numbers in the new place in layer 1. The parenthesis I learned pretty quickly - probably because in programming one uses the parenthesis a lot more than numbers. I realized that one just have to force oneself to use the new key mappings even though it would be faster to use the old keys father away. But I believe that in a few weeks the new key mappings will go to the muscle memory and then it is a lot faster to use the new keys near the home row.

Some tweaking with layer 0. I did some modifications to layer 0 regarding the thumb buttons. The new key mapping is below.

Changes:

I changed the right lower thumb buttons as in the left side. I realized that it is easier to remember and use them this way. Now Shift and Ctrl keys are mirror images just like in their standard positions.
I moved Enter key next to Space key. It is a bit awkward to twist the right hand thumb to this key but I sacrifice a bit in ergonomics to have the Shift and Ctrl keys in more logical positions.
Then a bit later I realized that I can utilize the double action key functionality: I configured Enter in the left side but with double action: when hit once it is Enter but when hold down it is Shift to layer 1 as previous.

Some tweaking with layer 1. I did some modifications to layer 1. The new key mapping is below.

Changes:

I moved the Esc key below the home row. The reason was to have a more logical layout for the numbers.
I realized that the keys below the right home row are empty and I can utilize that row. So:
Tab added next to Esc.
I moved \ key next to tab. \ key provides the ``character that I need in Slack to provide code snippets without formatting. - The weird two dots key provides the precious tilde key (~) that a programmer needs in terminal when referring to the user home directory. This key is in standard position really far a way in second upper row near Enterkey. -' key actually provides the asterisk (*) when used with Shift. The asterisk key is used in terminal to represent any number of characters - also used quite often. This key is in standard position next to the Enterkey and therefore a bit far away. - I addedEnter also here in the second layer in the lower right thumb. After some experimentation I realized that it is actually easier to hit Shift to Layer 1 and Enter in this new position than the Enter key in layer 0 (I hit Enter quite often and I began to feel uncomfortable to twist my thumb to the new Enter key next to Space key). This is actually not important anymore when I realized to utilize the double action key (see previous layer). Well, maybe sometimes when adding a lot of numbers it might be easier to have the Enterkey also in this layer. - I also addedCapsLock and Left Shift` keys to this layer since I’m planning to start learning to hit these keys as modifiers for the various parentheses (possibly, let’s see). - I also added the mouse buttons in this layer since I realized that sometimes it is nice to paste something that I painted earlier using the mouse.

Cheat sheet. I created a cheat sheet for myself to learn the new key mappings with Dygma Raise faster. On the left side note are the thumb buttons, on the right side note are layer 0 key mappings.

Ordered new switches. I experimented with the test switches Dygma kindly added to the box. First I thought that I would order Kailh Speed Silver switches (Linear, 40 cN, Pretravel 1.1 mm, Travel 3.5 mm) since I feel that the Cherry MX Brown (Tactile, 55 cN, Pretravel 2 mm, Travel 4 mm) switches that I ordered with my Dygma are a bit too heavy and Kailh Speed Silver switches were really easy to press. I also tried the Kailh Speed Copper (tactile, 50 cN, Pretravel 1,1 mm, Travel 3,5 mm) - it was a nice switch also, a bit more consistent feeling that with Cherry MX Brown. But then I surprised myself. I tried Kailh Speed Bronze (clicky, 60 cN, Pretravel 1.1 mm, Travel 3.5 mm) - and I immediately fell in love with the feeling and the “click” which is really sharp and provides a really nice haptic and auditory feedback regarding a key press without being too heavy. I ordered Kailh Speed Bronze switches and also Kailh Low Profile Choc White switches for the lower thumb buttons.

Conclusions

If you are a programmer and you are looking for a top-quality mechanical keyboard with absolute configurability - look no further: you want Dygma Raise. With Dygma Raise your imagination is your limit on how you can configure your new keyboard.

Kari Marttila

Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/