DEV Community: Robert Johnson

A Developer's Guide to Blogging

Robert Johnson — Sat, 26 Aug 2023 14:58:52 +0000

So you're a developer looking to start a blog? In this guide I'm going to take you through running your own blog site, from domain names & site creation to SEO & syndication.

(Originally featured at HackerNoon)

Table of Contents

Section 0 (optional) - Start writing on blog platforms
- Dev.to
- Hashnode
- HackerNoon
Section 1 - Buy a domain
- What domain name to choose?
- Where to buy a domain name?
- Domain costs
Section 2 - Create a simple site using a Static Site Generator
- What is a Static Site Generator?
- Why use an SSG?
- Creating a basic site with Hugo
Section 3 - Get your blog hosted
Section 4 - Analytics
Section 5 - Basic SEO
- Letting Google know you exist
- Checking for issues
- Canonicalization & Canonical URLs
- Private GitHub repo
- Backlinks & domain authority
Section 6 - Sharing your work
- Use Reddit
- Post on specific subreddit(s)
- Check the rules for each subreddit
- Be part of the community
Section 7 - Republishing on blog platforms
- Canonicalization for republished articles
  - Canonicals for dev.to
  - Canonicals on Hashnode
  - Canonicalization on HackerNoon
  - You can't canonicalize on FreeCodeCamp!
- Publicity done for you
- Republishing drawbacks
Appendix
- Why blog?
  - Improves your writing & communication skills
  - Gives you a way to prove you know what you're talking about
  - Helps you market yourself & creates opportunities
- A quick comparison of SSGs
  - Jekyll
  - Eleventy
  - Gatsby
  - Hugo

Section 0 (optional) - Start writing on blog platforms

If you want to warm up a bit before going through the effort of setting up your own site, the easiest & simplest thing to do is to just get writing and contribute your work to existing blogging platforms.

Ultimately the hardest thing about blogging is the actual writing - at least, that's usually the case for us developers! So it doesn't hurt to get a feel for it first to see if you enjoy it. Also, as we'll see later, these are still useful to make use of even if/when you set up your own separate blog site.

Dev.to

You've probably heard of dev.to already - it's still the biggest developer blog platform and it's where I recommend you start. You can easily start writing using their simple markdown editor, and it has a large readership so you'll get a good number of eyes on your work straight away.

It has great built-in analytics, which can help you see how many people are reading your posts, and even how they found them - e.g. Reddit or Twitter. This helps you see what's working best when you share your work.

Its design contains a lot of social media style elements, with likes & reactions to posts, as well as message-board style discussion threads. (This could be a pro or a con, depending on your tastes.)

If in any doubt, start here!

Hashnode

Hashnode is a newer dev blog site. In my opinion it has a much more professional feel than dev.to. It feels much more blog-oriented than dev.to, and less like a social media site; it gives you a separate blog subdomain, giving your blog a bit of its own separate identity within the site. If you want to you can even hook it up to your own private domain name if you wanted.

Sadly though it's a lot less popular than dev.to, and in my experience I hardly get any traffic through it. If you like its cleaner style it could be worth a look, but expect to do more legwork to get your posts noticed.

HackerNoon

HackerNoon is very different to dev.to & Hashnode in that any article you submit there has to go through a human editor who works with you to ensure your article is at its best before it is published. However, they may choose not to publish your article at all.

This has its tradeoffs; on the one hand this is a great learning experience, but on the other it limits your freedom to simply post when and what you want. Therefore I would recommend submitting work to HackerNoon for the learning experience it will give you, but consider keeping the primary home for your blog elsewhere.

Section 1 - Buy a domain

Sadly, creating your own blog involves one of the most dreaded developer tasks - naming things! Though not very difficult from a purely technical point of view, it's worth looking into and thinking about up-front; think it over while you're getting your site set up.

What domain name to choose?

There are two main choices here:

Name your blog after yourself
Give your blog its own identity with a separate name

I've come across varying opinions on each; for my own blog I've simply named it after myself. My thinking is that if you're using a blog to help get your name out there, why not use your name in your blog? (If this logic is good enough for Will Smith, it's good enough for me! 😅)

Whatever you choose remember that your domain is a bit of a pain to change, so it's worth taking the time to find one you're confident with. However you may have to rethink or adapt your domain if the one you'd like is already taken, so check if it's available before you get your heart too set on it.

Where to buy a domain name?

You'll probably have heard of many domain registrars already, such as BlueHost, Hostinger, GoDaddy, and Namecheap.

However, a key one to check out is Google Domains, because this is the only registrar which sells .dev domains. My advice is that .com is still the best if you can get one, so if there's one available that you like then go for it, but if not then .dev could be a great alternative.

Domain costs

Note that domains cost varying amounts, and moreover you have to pay up every year to keep them. Domains can be as cheap as $10 per year, so unless you're buying a domain for business purposes, think hard before shelling out for an expensive one.

Section 2 - Create a simple site using a Static Site Generator

What is a Static Site Generator?

A static site generator - or SSG - will template & generate the content for a website such that it can be served up as static files held on a web server. After setting up the main structure of the site - such as your home page, posts index, 'about' page, etc. - you can generate complete pages simply by adding markdown files.

Why use an SSG?

In short, SSGs

give you fast sites
let you publish using a GitOps workflow
are secure (no database to hack into!)
let you take advantage of your tech skills, rather than being limited to what platforms can provide

As developers we use something pretty much every day which does something very similar: GitHub. In our code repos we'll write readmes & other documentation in markdown, and GitHub nicely formats it for viewing on a webpage.

An SSG can let you follow much the same GitOps workflow for your own blog. When I got started writing on dev.to I managed my drafts in a Git repo. This was fine up until wanted to publish, at which point I needed to transfer the contents into their web editor. This was fine, but when I found out that SSGs can let you use the same workflow as for updating a readme in GitHub, that was enough for me to want to find out more.

A common way to set up a blog is to use something like WordPress; this has plenty of advantages for sure, but it stores all your content in a database. A statically-generated site doesn't need one, which for one thing means it will be much faster. Speed isn't everything of course, but Google cares about how fast your site loads, so why not take advantage of the speed that an SSG-generated site can give you?

Creating a basic site with Hugo

For creating a static site I recommend Hugo. In short this is because it is popular, well-supported, fast, and allows you to get up and running quickly with premade templates.

(For details on alternative SSGs, see the Comparing SSGs section in the appendix.)

The quick start page of Hugo's official docs gives a great description of getting a basic Hugo site set up. Follow the steps it provides, but when you reach its 'create a site' commands, I recommend changing them instead to the following:

hugo new site quickstart
cd quickstart
git init
echo "/public/" >> .gitignore
echo "/resources/_gen/" >> .gitignore
echo ".hugo_build.lock" >> .gitignore
git clone https://github.com/leafee98/hugo-theme-flat themes/flat
rm -rf themes/flat/.git/ themes/flat/.github/
echo "theme = flat" >> hugo.toml
hugo server

This has the following differences to the official guide:

Sets up a .gitignore file
Uses the excellent 'Flat' theme instead of the Ananke theme
Sets up the theme as code within your own repo rather than as a git submodule; this makes it a lot easier to tweak the theme to your own taste later

When creating your GitHub repo, make it private - this is for SEO reasons, which we'll explore later.

Once you've followed the guide then you'll have a site ready to host! At this point there will most likely be aspects to the site that you'll want to change. However, you don't need to let that stop you from getting hosted. The main thing that you want to avoid changing later is your URLs; everything else can be changed later on.

Section 3 - Get your blog hosted

As the Hosting and Deployment section of the Hugo docs describes, since you've got a static site it can be hosted virtually anywhere, and almost certainly for free too. Of course there are plenty of free/cheap hosts for WordPress-based sites as well, but any particular host will only give you so much bandwidth; in general, a static site gives your more bandwidth for your buck, even if you haven't actually handed over any money.

Hugo's hosting guide lists plenty of possibilities, but personally I can agree with Bryce Wray's recommendation to go with CloudFlare Pages; its free tier is possibly the fastest out there, and it's easy to use. Simply follow their guide from 'setting up a GitHub repository' onwards. At this point your site will be online! But you'll have an ugly domain like my-blog-xyz.pages.dev. Simply follow CloudFlare's guide on setting up a custom domain to get your site live on the domain which you bought earlier.

Section 4 - Analytics

Vanity of vanities, all is vanity!

— Ecclesiastes 1:2

Now strictly speaking this is an optional step, but at this point it's worth setting up analytics for your site. Perhaps I'm just vain, but for me a lot of the fun of blogging is being able to see that people actually see and care about what you've written.

One limitation of setting up a static site is that you can't host analytics yourself since that would require some sort of database. But that's not too much of a problem anyway since there are plenty of good third-party analytics providers available. The biggest one which you'll likely have heard of is Google Analytics.

Setting up analytics with a third-party provider generally involves

Setting up an account
Adding a javascript snippet or link to your pages.

In the Hugo template I recommended, you would simply add something like the following to your head.html template file:

{{/* Include analytics, but only in production */}}
{{- if hugo.IsProduction | or (eq site.Params.env "production") }}
<script defer data-domain="yourdomain.com" src="/link/to/script.js"></script>
{{- end }}

If you want to set up GA for your site, follow Google's official docs.

The analytics provider I've gone with is Plausible. Sadly it's not free - about $9 a month - but it's easy to use, lightweight (the script is less than 1kb), and respects privacy, so it's worth a look IMO.

Section 5 - Basic SEO

If you build it, they will come

— Field of Dreams

There is some truth to the above quote; there's not too much you need to do to get your blog on Google, and thankfully the days of keyword-stuffing shenanigans are over; what ultimately matters is writing good content, which is good news for independent bloggers such as us.

That said, there are a few things you can do to make sure that your site is up to scratch for search engines.

Letting Google know you exist

Google's web crawlers will find your site eventually, but it certainly helps to give Google a head start and get your site set up on Google search console. You'll want to do the following:

Sign up
Register your domain address (this will be greatly simplified if your bought your domain from Google Domains)
Submit the URL for your site's sitemap.xml on the sitemap tab. Thankfully, Hugo will have generated one for you at https://yourdomain.com/sitemap.xml.

Once this is done then Google will at some point over the next few days start a crawl of your website. Note that there's a bit of a lag of a few days to the search console though. The most reliable way of seeing what's indexed on your site is to google using a site: query, e.g. site:yourdomain.dom.

Another useful benefit to getting set up on the search console is that under the 'Pages' tab you can see any reported issues about why Google can't or won't index your pages.

Checking for issues

Hugo and the template we're using should cover most of the bases for good SEO practices. But still, it doesn't hurt to check your site on Google's PageSpeed Insights tool. (This will become more important if/when you start making your site more to your own liking by tweaking the template files.) This will give you a good overview of the performance, accessibility and any SEO issues (such as missing meta tags) for the page.

Canonicalization & Canonical URLs

One SEO concept that is key to understanding as a blogger is canonicalization & canonical URLs. Basically the idea is that the same content can be accessed via different URLs, but you don't want Google to split the page ranking for a single page across multiple URLs. Therefore you can have a page declare what URL search engines should consider as being the URL for the page.

Remember that CloudFlare also generates an "ugly" domain like my-blog-xyz.pages.dev alongside your custom domain. Most (if not all) hosting providers don't allow you to turn off this basic domain, but so long as you have canonical URLs set up on your pages this won't be a problem - Google will only list under your custom domain, not the "ugly" one.

One of the reasons that I recommend the Flat theme for Hugo is that (unlike the recommended default of Ananke) it already includes a canonical link. However if you want to use a different theme you can add this simply to your header template like so:

<link rel="canonical" href="{{ .Permalink }}" />

You can check that this is set up correctly for any given page by checking it in the PageSpeed Insights tool mentioned above; you'll see whether a page has a canonical URL as a checklist item under the "SEO" section.

We'll explore canonicalization further in the section on republishing, further below.

Private GitHub repo

You'll recall earlier that you made your GitHub repo private. This is because (at time of writing) you can't add canonical URLs or noindex tags to the markdown documents, and so this serves as another subtle code duplication issue. A private repo works around this issue.

Backlinks & domain authority

You've likely heard that one of the factors in Google ranking a site is how many other sites link to it; this is known as domain authority, and such links from one site to another in this context are known as backlinks.

I'm not going to talk about how to "farm" backlinks to make your blog rank higher on Google; for one thing Google has got wise to such schemes, and besides one of the benefits of creating your own site is that you can help make the web a better place rather than further filling it with such douchebaggery.

The main takeaway instead is simply that it'll take time before you can rank well on Google; you won't have much domain authority to start with, but that will grow over time. As ever, your focus should simply be on writing the best posts you can.

Section 6 - Sharing your work

Advertising is only evil when it advertises evil things.

— David Ogilvy

As much as I would love to say that Google being able to find your blog is enough, realistically it really helps to do a bit of self-publicity and share your posts online.

Use Reddit

The best way to do this is by post links to your articles on Reddit. Posting on social media doesn't hurt, but the key advantage with Reddit is that its voting system means that good posts (as I'm sure yours will be) can stay on a subreddit's front pages for a while, rather than immediately getting swept along down the timeline.

Post on specific subreddit(s)

Post to a specific subreddit such as the programming language or technology which you're writing about. The bigger the subreddit the more people will potentially see your post, but the catch is that big subreddits like /r/programming can often have a lot of people tactically downvoting good posts simply to bury them, in order to help keep their own posts on top.

Instead, you'll have more luck by sticking to the most specific Reddits that apply to your post; such communities are more likely to care and read your post, and they're more receptive to self-publicity.

Check the rules for each subreddit

If you're unfamiliar with a subreddit always check its community rules before posting. Each subreddit will have different rules, in particular different tolerances to linking to your own articles. This can vary between total bans on self-publicity, an expectation that you don't self-post too much, or no restrictions at all.

Another key rule to watch out for is undesired post topics with possibly a redirection towards a more appropriate subreddit instead. For example /r/programming's rules ask you to redirect technical questions towards r/learnprogramming instead, and similarly job listings towards /r/forhire.

Be part of the community

Regardless of the rules for the subreddit you're submitting to, it's always a good idea to be a part of the community rather than only using Reddit as a tool for your own ends. Post links from other blogs you enjoy, engage in discussions and just have fun.

Read through the "Reddiquette" once in a while to check that you're taking part in a responsible way.

Section 7 - Republishing on blog platforms

As rewarding as it is to have your own separate blog it's still useful to republish posts on dev blogging platforms such as dev.to. You might wonder why we'd come back around full circle after taking the time to set up your own blog.

Ultimately all that matters is getting your thoughts read and appreciated by other developers, and republishing helps you do that simply by adding another way for people to come across your work. Another key benefit though is that it helps you build up domain authority for your own blog through backlinks - but in order for your site to be credited with those backlinks, you need to republish in the right way.

Canonicalization for republished articles

Remember the concept of canonical URLs that we discussed earlier? When reposting your articles elsewhere, ideally you only want to post on sites that let you indicate that the canonical version of your article is the one on your blog. That way you get the exposure benefits of republishing but any resulting backlinks build up the domain authority of your own blog.

Canonicals for dev.to

As detailed in dev.to's editor guide, you can add a canonical URL simply by adding a canonical_url property to a post's "front matter" properties.

Canonicals on Hashnode

You can set a canonical URL on a Hashnode article under the "Are you republishing?" section on the "Article settings" view for an article. (See their docs for further details.)

Canonicalization on HackerNoon

As covered in HackerNoon's docs, you can set a canonical URL in the "First Seen At" in your post's settings.

You can't canonicalize on FreeCodeCamp!

As noted by this Hashnode article, think twice before republishing your work on FreeCodeCamp since they don't let you set a canonical URL. Indeed, this dev.to post details the negative impact this has made on certain bloggers because of this.

Publicity done for you

Another benefit of republishing is that some of the legwork for sharing your work on Reddit & other social media will often be done for you. Bots and bloggers who curate articles for particular communities will often find your work through particular tags on blog platforms, and share it on social media.

Republishing drawbacks

Republishing isn't without some costs, however:

It can be a bit of work to repost on these sites, especially in contrast to the nice GitOps flow when publishing to your own site. (Some automated & semi-automated solutions are available to help things here though.)
It's possible for republished work to outrank your originals on search engines. Ideally canonical URLs will mean that the republished work is unindexed altogether, but this isn't guaranteed (especially for Bing, in my experience)

Therefore it's worth considering republishing some time after you publish the original (say, one or two weeks, but it's up to you). This has the following benefits:

Reduced duplication of work making corrections & additions across your original & republications; inevitably, despite your proofreading efforts, you or your readers will spot issues or things you missed - better to have to do this while the article only exists on your own blog
Search engines have time to index your site first

Appendix

Why blog?

Improves your writing & communication skills

While thankfully Agile means that software development rarely involves producing reams & reams of documents like it may have done in previous eras, I think that the pendulum has swung a bit too far the other way and so good documentation which helps to introduce newcomers to a system or repo is sadly neglected. And as you progress in your career as a developer you'll have more and more need to distil your thinking and explain complicated concepts & systems in a clear & concise way.

Blogging gives you a great way to practice these skills. In particular, having an audience is a great motivating factor!

Gives you a way to prove you know what you're talking about

It's one thing to have a list of technologies you know listed on your resume; it's quite another to prove that you really understand them by being able to link to articles you've written that clearly demonstrate that you know your craft inside & out.

Helps you market yourself & creates opportunities

Jeff Atwood met Joel Spolsky in part through his blog, Coding Horror, and they then went on to create a little thing called Stack Overflow.

You can't guarantee such things will happen simply through creating a blog, but it certainly doesn't hurt either. And especially in niche technology areas, you might find your name might stick out to someone through them having read an article of yours in the past.

A quick comparison of SSGs

Here are my thoughts on a few popular SSGs, and how I came to settle on Hugo. See jamstack.org for a much bigger list.

Jekyll

Jekyll used to be very popular but it is less so now. It powers GitHub pages and is very blog-oriented. I initially considered it, but was put off by its strict URL structure which insists that blog pages have a date component embedded within the URL, which is something I wanted to avoid for my own blog.

This can be avoided by making writing posts as part of a more abstract "collection" type, but then that would lose many of the benefits of working within its own blog posts abstraction.

Eleventy

Eleventy is an SSG that has been gaining a lot of popularity. It is powered by node.js, and is very flexible; you can customise the templating engine and markdown renderer, even allowing you to use different options for different pages within the same site.

The main drawback I found with this is that it doesn't come with any built-in templates, so you can't easily just generate a blog straight out of the box. Also of note is that runs on node, making its installation more complicated; not a deal-breaker by any means, but not as convenient has Hugo.

Gatsby

Gatsby is also becoming very popular lately. However, it generates react-based single-page apps rather than static sites. This is a valid option, I wanted the simplicity of pure HTML & CSS when creating my blog. Moreover, I've seen that this writer has had a poor experience using it, recommending Hugo instead.

Hugo

Lastly, we get to Hugo, the SSG I use for my own blog. It is written in Go meaning that using it means installing a single binary. This has advantages from a hosting point of view; you can just specify the Hugo version on my provider, and you can be sure that it will behave just the same way as it would locally.

It is blazing fast. While this makes little difference when your blog contains just a few posts, it's good to know that the build time will remain reliably fast as your site grows.

It is the most popular SSG (judging by GitHub stars), meaning that it is well-used & documented, and well-supported by pretty much any static site hosting platform.

Above all it has templates to get you started. Though you will likely want to create your own templates at some point (or at least modify them to your own taste) - it at least gives you something to get started out of the box.

Its main downside is that it uses Go's templating language; it's not terrible by any means, but it's a little clunky compared to some, and if you're keen on another template language then you'd be out of luck here. If this is a problem for you then Eleventy might be worth a look.

Emacs Lisp Cheat Sheet for Clojure Developers

Robert Johnson — Fri, 04 Aug 2023 10:55:30 +0000

If you're a Clojure dev and you're looking to learn Elisp, you have a great head start since you know a lisp dialect already. Here's a concise guide which focuses on the main differences between the two languages.

Table of Contents

Numeric types
Character types
Conditional logic
Equality
Functions
Let forms
Comments
Namespaces and access modifiers
Variables
Symbols, values & lambdas
Threading macros & higher-order functions
False friends
- concat
- string
Aliases
- Suggested aliases
Going further
Acknowledgements

Numeric types

Elisp has integers and floats. These can be mixed in arithmetic functions how you'd usually expect:

(+ 1 1) ; => 2
(+ 1 1.0) ; => 2.0
(+ 1.0 1.0) ; => 2.0

It doesn't have Clojure's ratio type though, so integer division works similarly to most other languages:

(/ 3 2) ; => 1

Integers can be expressed in multiple ways; the printed representation of REPL output looks odd at first, but it's simply showing you the alternate representations of the integer (I've omitted this elsewhere for brevity):

16 ; => 16 (#o20, #x10, ?\C-p)
;; Shows decimal, octal, hex and character value
;; (see 'character types' section below)

Floats can be expressed in scientific notation:

1e2 ; => 100.0

Integers can be arbitrarily large;

(+ 1 10000000000000000000)
; => 10000000000000000001

Character types

Elisp has strings and characters. Strings are essentially arrays of characters. They are represented and escaped in much the same way as Clojure:

"This is a simple string"
"A newline character: \n"

"Escaped line breaks\
are ignored"
;; => "Escaped line breaks are ignored"

Character literals a displayed using a ? prefix:

(string ?f ?o ?o) ; => "foo"

Characters are in fact simply integers:

?A ; => 65 (#o101, #x41, ?A)
(string 102 111 111) ; => "foo"
(+ ?f ?o) ; => 213 (#o325, #xd5)

Though not all integers are valid characters:

(string 12356789) ; => error!

You can format strings much the same as in Clojure using the format function:

(format "I have %s bananas" 3)
;; => "I have 3 bananas"

You can also display messages (also with optional formatting elements) in the minibuffer using the message function.

(message "Hello, world!")
(message "Show a message to %s",
         "Alice")

Such messages are also sent to the *Messages* buffer, making message helpful for debugging.

Conditional logic

nil and the empty list '() are falsy; everything else is truthy:

(if nil "truthy" "falsey")
;; => "falsey"
(if '() "truthy" "falsey")
;; => "falsey"

(if "a" "truthy" "falsey")
;; => "truthy"
(if "" "truthy" "falsey")
;; => "truthy"
(if 1 "truthy" "falsey")
;; => "truthy"
(if 0 "truthy" "falsey")
;; => "truthy"

Elisp doesn't have a separate Boolean type with true and false values; instead, the symbol t is used to represent true and nil is used to represent false. Predicate functions conventionally have a p suffix (rather than ? as in Clojure):

(integerp 1) ; => t
(integerp 1.5) ; => nil

(Elisp will accept symbol names containing ?, so you can name your own predicates in the Clojure style if you want.)

Equality

= is specifically for checking comparisons on number types (integers & floats):

(= 1 1) ; => t
(= 2 2.0) ; => t
(= 3.0 3.0) ; => t

(= 1 2) ; => nil

(= 1 "a") ; => error!

string= is for comparing strings and characters. It is case sensitive:

(string= "a" "a") ; => t
(string= "a" 'a) ; => t

(string= "a" "b") ; => nil
(string= "a" "A") ; => nil

(string= 1 1) ; => error!

equal does value-based comparisons. It works for numbers & string types too, but it doesn't compare across ints & floats and strings & symbols like = & string= do. It too is case sensitive.

(equal 1 1) ; => t
(equal "a" "a") ; => t

(equal "a" "A") ; => nil
(equal 1 1.0) ; => nil

eq Performs reference-based equality (like Clojure's identical fn):

(eq "a" "a") ; => nil

(See Comparison Functions on the Emacs Wiki for further info.)

Functions

Functions are defined using defun. The arguments are enclosed in a parens (rather than in brackets):

(defun foo (x y)
  (+ x y))

Docstrings are a bit different in Elisp though:

they go after the argument list
line breaks are preserved, but so is any indentation-related whitespace; therefore the convention is to simply remove indentation for docstrings for all but the first line.

(defun foo (x y)
  "This is a docstring, continuing
onto the next line"
  (+ x y))

Variadic functions (AKA functions with "rest args") can be created using
&rest:

(defun bar (x &rest r)
  (format "First arg: %s; rest: %s" x r))

(bar 1 2 3)
;; => "First arg: 1; rest: (2 3)"

You can't create true multi-arity functions in Elisp, but you can get close enough by using optional arguments:

(defun greet (person1 &optional person2)
  (if person2
      (message "Hello, %s and %s!"
               person1 person2)
    (message "Hello, %s!" person1)))

(greet "Alice")
;; => "Hello, Alice!"

(greet "Alice" "Bob")
;; => "Hello, Alice and Bob!"

Let forms

For let bindings, use let*. The bindings are placed in an (unquoted) list, and each binding pair has to be placed in its own list too:

(let* ((x 1)
       (y (+ 1 x)))
  (* 3 y)) ; => 6

let also exists, but it doesn't let you use earlier bindings within later ones:

(let ((x 1)
      (y (+ 1 x)))
  (* 3 y)) ; => error!

(let ((x 1)
      (y 2))
  (* 3 y)) ; => 6

Comments

Line comments still work just as you expect - e.g. ;; this is a comment
There isn't a comment reader macro ( #_ )
There isn't a built-in "Rich comment" block macro - BUT thankfully you can create one with a macro:

(defmacro comment (&rest body)
  nil)

(comment
 (blow-up-the-world))

Namespaces and access modifiers

Emacs doesn't have namespaces, and so doesn't have access modifiers either.

By convention, packages prefix all their functions & variables with a prefix, packagename-; e.g. all functions in the org package are prefixed with org-.

You do still have to require packages in order to ensure they are loaded before using them. E.g. (require 'org) would ensure that the code for the org package is loaded. Depending on your emacs config / distribution, many packages may already be required for you.

Functions intended as private are by convention signified with an extra - in the prefix; e.g. org--get-local-tags is a private function in the org package.

Variables

Variables can be created & updated using setq (short for "set quoted"):

(setq my-variable 2)
(+ 1 my-variable) ; => 3

defvar will create a variable, but won't update it if it's already been set; you can basically think of it as Clojure's defonce:

(defvar cheese "brie")
(message "cheese = %s" cheese)
;; => "cheese = brie"

(defvar cheese "stilton")
(message "cheese = %s" cheese)
;; => "cheese = brie"

(setq cheese "stilton")
(message "cheese = %s" cheese)
;; => "cheese = stilton"

Symbols, values & lambdas

One of the more confusing parts of Elisp is that its symbols point to multiple storage locations; a single symbol can refer to both a function and a variable:

(setq baz 123)
(defun baz (x y)
  (- x y))

(message "baz as value: %s" baz)
;; => "baz as value: 123"

(baz 10 3)
;; => 7

This means that most of the time when passing around functions you have to quote them:

;; Not quoting + here - will error
(apply + '(1 2 3))
;; => error: (void-variable +)

;; Quoting + here - OK
(apply '+ '(1 2 3))
;; => 6

We can create anonymous functions using lambda - but because lambdas are stored in variables rather than defuns, you can't call them directly:

(let* ((times-2 (lambda (x) (* 2 x))))
  (times-2 3))
;; => error: (void-function times-2)

Instead, you need to use funcall:

(let* ((inc (lambda (x) (+ x 1)))
       (dec (lambda (x) (- x 1))))
  (message "result for inc: %s"
           (funcall inc 10))
  (message "result for dec: %s"
           (funcall dec 10)))
;; =>
;; result for inc: 11
;; result for dec: 9

Threading macros & higher-order functions

A lot of Clojure-style goodness can be provided by using the excellent dash package:

;; (if not installed already)
(package-install 'dash)

(require 'dash)

;; Threading macros:
(-> 10
    (+ 1)
    (* 2)) ; => 22

;; Higher order functions:
(let* ((inc (lambda (x)
              (+ x 1))))
  (-map inc '(1 2 3)))
;; => (2 3 4)

(let* ((evenp (lambda (n)
                (= 0 (% n 2)))))
  (-filter evenp '(1 2 3 4)))

Note that Emacs often has its own built-in versions for these - e.g. thread-first & thread-last for threading, mapcar as an equivalent to Clojure's map - but as a Clojurist you'll probably find what you're looking for quicker by using dash's functions instead.

False friends

`concat`

Elisp's concat not only concatenates but also converts everything received into a string:

(concat "foo" "bar")
;; => "foobar"

(concat '(65 66) '(67 68))
;; => "ABCD"

`string`

Elisp's string function - unlike Clojure's str - only works on characters:

(string ?f ?o ?o) ; => "foo"

(string "foo") ; => error!
(string "f") ; => error!
(string 1.0) ; => error!

It can operate on integers, but it converts them to the character corresponding to its charcode (assuming it has one):

(string 65) ; => "B"

(string 123456789) ; => error!

Aliases

You can create aliases using defalias:

(defun foo (x y)
  (+ x y))
(defalias 'bar #'foo)

(foo 3 1) ; => 4
(bar 3 1) ; => 4

NB we linked to the function cell of foo using #' above, so we can redef foo, and bar will pick up the change too:

(defun foo (x y)
  (- x y))
(foo 3 1) ; => 2
(bar 3 1) ; => 2

Suggested aliases

Here are some Clojure-style aliases for their Emacs builtin equivalents:

(defalias 'do 'progn)
(defalias 'when-not 'unless)

Going further

If you'd like to put some Elisp into practice, you could also check out my intro to extending Emacs using Elisp. The Awesome Elisp GitHub repo is also a great resource.

Acknowledgements

Thanks to ambirdsall, hvis & Philip Kaludercic for suggestions. See Reddit for discussion.

Everyday editor extensions in Emacs

Robert Johnson — Thu, 01 Jun 2023 23:37:53 +0000

A good number of years ago, back when I had first started using Emacs, I came across an article describing the joy of customising your development workflow using Emacs:

...I could now create empty lines above with a single keystroke. It took just a couple of minutes, I didn't have to install any plugins, or restart the editor - this is normal everyday business for an emacs user.

I was curious about experiencing this for myself but it was only relatively recently that I took the plunge into Emacs scripting; I was more than happy with the fine dev experience that Spacemacs had given me for Emacs, and I had always assumed that it would be too hard and boring to get to grips with Elisp. But I don't know why I held off for so long - you can get a lot of bang for your back with just a little Elisp, and what's more it's pretty fun too!

In this post I want to do a few things:

Encourage any Emacs users that haven't used Elisp yet to give it a try
Highlight some key useful Elisp functions
Provide some examples & inspiration for creating your own extensions (whether you're an Elisp newbie or not)

I'm not going to explain Elisp completely from scratch, since this post would be far too long otherwise; take a look at the 'Learn X in Y minutes' page for Elisp if you need a primer (or if you know Clojure, my cheat sheet for Clojure devs).

TL;DR
Web bookmarks
Parameterising by prompt
Parameterising by editor contents
Fetching web data into your editor
Running shell scripts
Wrapping up

TL;DR

There's loads of easy, everyday extensions that you can write; they're quick to write, and what's more these are the sorts of extensions that you can find yourself coming back to again and again. Every so often I write some more complex Elisp, but 90% of the time it's just some simple scripting that can really smooth over some bumps in my workflow.

In my experience, such extensions involve some or all of the following:

Grabbing some text from the context of your editor
Performing some simple text transformation
Sending that text somewhere - e.g. your browser, a shell or a web API
Displaying results in an Emacs buffer

This shouldn't be too surprising; the main advantage of extending your editor over writing a shell script is that you don't need to leave your editor - or at least, you don't need to copy & paste information to and from it by hand.

The main bits of Elisp I'll show you for this are:

How to create interactive functions
Functions for capturing input, such as
- read-string for prompting
- the thingatpt package for grabbing data from your buffer
The format function for formatting strings
Functions for sending formatted strings around, such as
- browse-url for opening a link in your default web browser
- the plz package for simple web requests
- shell-command-to-string for running (you guessed it!) shell commands
- Some basic buffer-handling functions: generate-new-buffer, set-buffer, insert & switch-to-buffer-other-window

Web bookmarks

One of the simplest things we can do is to create interactive functions that serve as web bookmarks. So simple, the word "extension" seems a bit too fancy to describe these - and yet I find them so useful to make. The advantage of adding them to your Emacs setup is that

you get to take advantage of fuzzy-searching your bookmarks (using Helm or similar)
you don't even have to leave your editor to search for them

This is as simple as adding the following to your init.el:

(defun open-github-spacemacs ()
  (interactive)
  (browse-url "https://github.com/syl20bnr/spacemacs"))

To eval the function (i.e make it available to run), simply hover your cursor somewhere within it then execute M-x eval-defun (i.e. hold the alt & x keys together, release, then type eval-defun, then press enter.)

Once that's done, we can execute M-x open-github-spacemacs to take us to the Spacemacs GitHub page. But then, using Helm, you don't even need to type all that - something like M-x op git sp RET would be enough.

The interactive function requires a little explanation - basically, if your function contains a call to interactive, then emacs will make it available to be ran interactively (i.e. via M-x) as opposed to only runnable from other Elisp code.

Parameterising by prompt

So, to start getting a little more interesting - let's start looking at parameterised bookmarks. For example, taking us to a GitHub issue with a particular ID:

(defun open-github-spacemacs-issue ()
  (interactive)
  (browse-url
   (format "https://github.com/syl20bnr/spacemacs/issues/%s"
           (read-string "Issue ID: "))))

After executing M-x open-github-spacemacs-issue, Emacs will prompt us to enter an issue ID - e.g. 1234; once we press enter, Emacs will open a new tab with that issue ID (assuming it exists).

If you prefer more of a top-to-bottom approach, you can make use of let*:

(defun open-github-spacemacs-issue ()
  (interactive)
  (let* ((issue (read-string "Issue ID: "))
         (url (format "https://github.com/syl20bnr/spacemacs/issues/%s"
                      issue)))
    (browse-url url)))

Parameterising by editor contents

Getting yet more interesting (and useful): we can parameterise a bit like the above, but instead of being prompted we can make use of the text our cursor happens to be on. We get this functionality from the super-helpful thingatpt ("thing at point") package:

(require 'thingatpt)

(defun open-github-spacemacs-issue-from-pt ()
  (interactive)
  (browse-url
   (format "https://github.com/syl20bnr/spacemacs/issues/%s"
           (int-to-string (thing-at-point 'number)))))

Eval both forms, then enter 1234 into a buffer somewhere, and then hover your cursor somewhere over it; execute M-x open-github-spacemacs-issue-from-pt, then you'll get taken to the corresponding issue in GitHub.

Now we're cooking on gas! This is where we can really start to take advantage of the fact that we're running commands from our editor, rather than from within a shell script.

Where this can be especially useful is when you have some sort of global ID that is used to correlate requests or transactions across multiple systems, especially when those systems have web APIs you can query for said ID. I often find myself opening a log file in Emacs, then simply by hovering over an UUID I can take myself to many different diagnostic systems that care about it.

Speaking of UUIDs, you'll need to use thing-at-point slightly differently in order to grab one from your buffer:

(defun open-widget-page ()
  (interactive)
  (browse-url
   (format "https://postman-echo.com/get?my-id=%s"
           (thing-at-point 'uuid))))

Note that if the "thing at point" isn't a valid UUID, then (thing-at-point 'uuid) will be nil.

It's a good idea to combine both the prompting & "thing at point" techniques; that way if your cursor isn't pointing to a valid "thing", you can still enter it in:

(defun open-widget-page ()
  (interactive)
  (browse-url
   (format "https://postman-echo.com/get?my-id=%s"
           (or (thing-at-point 'uuid)
               (read-string "Enter ID: ")))))

As per the documentation for the thingatpt package, the types of "thing" you can specify include:
symbol, list, sexp, defun, filename, url, email, uuid, word, sentence, whitespace, line, number, and page.

If you happen to be working with UUIDs, then grabbing them in your editor in this way is particularly handy since it's a lot easier than selecting them using your mouse; and in most GUIs, double-clicking any part of the UUID generally doesn't select all of it.

(For the rest of this guide I'll use (thing-at-point 'word) since it's easier to experiment with.)

Fetching web data into your editor

So far so good, but all this switching to and from our browser window is starting to get a little old... Let's improve things by grabbing the data into an editor window:

;; only need to run this the first time
(package-install 'plz')

(require 'plz)

(defun show-widget-data ()
  (interactive)
  (let* ((id (or (thing-at-point 'word)
                 (read-string "Enter ID: ")))
         (url (format "https://postman-echo.com/get?my-id=%s" id))
         ;; create the name we'll use for the buffer we'll create.
         ;; The asterisks aren't required; they're merely there as
         ;; convention to help indicate that it's a generated buffer
         ;; not associated with a file
         (buffer-name (format "*widget-%s*" id))
         ;; create a new buffer with the above name
         (buf (generate-new-buffer buffer-name)))
    ;; Tell emacs to set the current buffer - i.e. the
    ;; buffer that buffer-related operations will work on -
    ;; to the buffer we've just created
    (set-buffer buf)
    ;; Download the data from the URL and write it into the
    ;; buffer. We're doing this synchronously; async
    ;; callbacks are possible, but this simple approach is
    ;; more than enough most of the time
    (insert (plz 'get url :as 'string))
    ;; An optional step - we've downloading JSON in this example,
    ;; so we may as well automatically enable the JSON mode
    (json-mode)
    ;; Display the buffer in another window
    (switch-to-buffer-other-window buf)))

Similar to before, we can run this interactively via M-x show-widget-data when hovering over a suitable item of data with our cursor; this time though the data will be opened in a new buffer.

The main functions of note in the above are plz for fetching the data, and the various buffer-handling functions: generate-new-buffer, set-buffer, insert & switch-to-buffer-other-window. Working with Emacs buffers in Elisp can feel a bit arcane at first, but thankfully it's fine once you get started. See the elisp docs on buffers if you want to go deeper.

Running shell scripts

We can still take advantage of the context of our editor when running shell commands; we can use parameterised input as we've been doing, run a shell command, then display the results in another buffer:

(defun run-widget ()
  (interactive)
  (let* ((id (or (thing-at-point 'word)
                 (read-string "Enter ID: ")))
         (buffer-name (format "*widget-%s*" id))
         (buf (generate-new-buffer buffer-name)))
    (set-buffer buf)
    (insert (shell-command-to-string (format "cowsay %s" id)))
    (switch-to-buffer-other-window buf)))

This is especially useful if you're new to elisp; you can create a script in your favourite language to do the heavy lifting, and just use the simple glue code above to make use of it seamlessly within emacs. For example, for downloading data it can be easier to handle things like authentication tokens and connection pooling in a more modern scripting language. (As a Clojure dev, I love Babashka for this!)

Wrapping up

Hopefully this has shown you how to get started in the wonderful world of Emacs scripting, and given you plenty of inspiration for how you can make your day-to-day development worklow better with just a little bit of Elisp. And what's more, you'll have experienced just how quickly you can try out these sorts of changes - all without even restarting Emacs!

If you want to go deeper into Elisp, I highly recommend checking out awesome Elisp, which lists many helpful tutorials and libraries (which is where I came across the plz library I used earlier).

As always, please feel free to ask any questions in the comments.

Thin (ish) Clojure jars for better Docker containers

Robert Johnson — Mon, 22 May 2023 21:53:32 +0000

Recently I've been creating "thin" jars for my Java applications, in order to avoid the waste caused by putting uberjars in containers. Unlike an uberjar, a "thin" jar contains only our application code, with all the dependencies split out and referenced in the uberjar's manifest file. This lets us store the app code in a separate docker layer, allowing us to save all the storage & network IO that we'd otherwise waste re-transferring all those app dependencies.

Despite this, it's still common practice to deploy Clojure apps as uberjars, so I thought I'd try implementing the same dependency/app-jar separation for my Clojure apps. Wondering why such an option isn't already a built-in function for tools.build, I thought it would be as simple as recreating Maven's copy-dependencies step in Clojure.

The bad news: we can't get all the way there; the good news: we can get pretty close - at the very least, a concrete improvement over containerized uberjars.

The Problem

Given that both Java & Clojure apps ultimately get complied down to JVM bytecode, in theory we should be able to apply to Clojure apps the same tricks we can use for Java. There's an unexpected catch though: Clojure's complier is non-deterministic in its output, i.e. it can output different class files across different runs for the same input.

Any differences in this output don't affect the behaviour of the compiled program, but just the difference of even a single byte here invalidates any layer caching. Therefore we could create a "thin" jar and split it into a different layer of our docker image, but we wouldn't actually benefit because each new build would require transfer & storage of the same amount of data as an uberjar, losing the very benefits we are hoping to achieve.

A (partial) Solution

All is not lost, however; we can still split out any jars containing already-compiled bytecode, creating a slimmer uberjar containing only bytecode generated from our Clojure application code and any Clojure libraries it uses.

In the build.clj file below, we

Work out which libraries contain already-compiled code (as opposed to only .clj source files)
Copy the precompiled library jars to a separate lib directory
Compile any remaining Clojure dependencies & our app code, and put just those into an uberjar
Reference the jars in the lib directory in the uberjar's manifest file

(ns build
  (:require [clojure.tools.build.api :as b]
            [clojure.string :as str]
            [clojure.java.io :as io])
  (:import java.util.zip.ZipFile))

(defn- contains-class? [absolute-path]
  (with-open [zip-file (ZipFile. absolute-path)]
    (->> (iterator-seq (.entries zip-file))
         (remove #(.isDirectory %))
         (map #(.getName %))
         (filter #(str/ends-with? % ".class"))
         first)))

(defn copy-dependencies [{:keys [libs] :as _basis}]
  (let [paths (mapcat (comp :paths val) libs)]
    (doseq [src paths]
      (let [target (str "target/lib/" (.getName (io/file src)))]
        (b/copy-file {:src src :target target})))))

(defn- split-basis [{:keys [libs] :as basis}]
  (let [grouped (group-by
                 (fn [[_sym {:keys [paths]} :as _lib]]
                   (boolean (contains-class? (first paths))))
                 libs)]
    [(assoc basis :libs (into {} (get grouped true)))
     (assoc basis :libs (into {} (get grouped false)))]))

(defn jar
  ([] (jar {}))
  ([opts]
   (b/delete {:path "target"})
   (let [[precompiled-basis uncompiled-basis] (split-basis (b/create-basis opts))]
     (println "Copying source...")
     (b/copy-dir {:src-dirs ["src" "resources"]
                  :target-dir "target/classes"})
     (println "Copying precompiled dependency jars...")
     (copy-dependencies precompiled-basis)
     (println (str "Compiling " main "..."))
     (b/compile-clj {:basis uncompiled-basis
                     :class-dir "target/classes"
                     :src-dirs ["src"]
                     :ns-compile [my-app.core]})
     (println "Building jar")
     (b/uber {:basis uncompiled-basis
              :class-dir "target/classes"
              :main my-app.core
              :manifest {"Class-Path" (->> (.listFiles (io/file "target/lib"))
                                           (map #(str "lib/" (.getName %)))
                                           (str/join " "))}
              :uber-file "target/my-app.jar"}))))

This gives us a smaller app jar, alongside its jars that contain compiled bytecode. (NB this includes Clojure core!)

.
├── my-app.jar
└── lib
    ├── asm-9.2.jar
    ├── avro-1.11.0.jar
    ├── checker-qual-3.8.0.jar
    ├── clojure-1.11.1.jar
    ├── commons-compress-1.21.jar
    ...
    └── zstd-jni-1.5.2-1.jar

With this in place, we just need to make sure our Dockerfile copies over the libs in a separate layer to the uberjar, like so:

COPY --from=app-build /app/target/lib /app/lib
COPY --from=app-build /app/target/my-app.jar /app/my-app.jar

As a result, each time our CI server builds a new Docker image we go from something like this:

5f62416f87c9: Pushing [==================================================>]  31.95MB
1668bc9c2910: Layer already exists
e6857d4762fd: Layer already exists
3af14c9a24c9: Layer already exists

to something like this:

acd5c6ddc038: Pushing [==================================================>]  4.705MB
8bf418842124: Layer already exists
1668bc9c2910: Layer already exists
e6857d4762fd: Layer already exists
3af14c9a24c9: Layer already exists

i.e. each new build results in less network IO and image repo storage; the cost is an extra image layer, and a slightly more involved build.clj. YMMV of course, dependending on your app's dependencies - but Clojure apps tend to depend upon a large stack of Java dependencies, and so savings like the above wouldn't be uncommon.

What about deploying Clojure sources?

Of course, an alternative to all this for which we could definitely achieve a truly thin application image layer would be to not bother with AOT compilation and deploy Clojure sources. The tradeoff here is between build & infrastructure costs against application startup time. This is a tradeoff potentially worth making - but if you're already deploying uberjars, then you're probably doing so because you want to avoid that compilation at application start.

Conclusion

Compared to containerizing an uberjar, we get:

Faster app upgrade times
Reduced image registry storage and costs
No compromise to app start startup time

While we sadly can't achieve the same image size efficiency gains as compared to a Java application, there are definitely still gains to be had - and they come at just the cost of some extra lines of code in our build.clj file.

How to deploy thin jars for better Docker caching

Robert Johnson — Fri, 12 May 2023 22:30:09 +0000

There's plenty of advice around on how to create small Docker containers, but what is often missed is the value of making your main application layer as small as possible. In this article I'm going to show you how to how to do this for a Java application, and why it matters.

I'm assuming that you're already familiar with Docker layer caching; if not, then the official docs will give you a good overview.

TL;DR

Instead of a single uberjar creation build step, split it into two steps:
1. Copy dependencies to a lib folder (e.g. via Maven's copy-dependencies plugin)
2. Create a jar, ensuring that the lib folder is added to its manifest
The resulting docker image is much the same size - but Docker can now cache the dependencies, meaning that each subsequent build requires only a few extra kilobytes to store & transfer, rather than tens of MBs
As an added benefit to splitting out the libraries, it makes it easier to trim down our dependency jars, such as removing useless alternate native libraries from JNI jars

The downsides of uberjars for Docker

For a long time the standard way to deploy Java applications has been to deploy the humble uberjar, which contains both our compiled code and all our application's dependencies. Packaging Java apps in this made sense back when deploying an application meant transferring files to and from servers directly, but it's not so useful in this age of containerized apps.

The reason for this is that for every single change to your application's source, all those dependencies bundled inside the uberjar end up inside the same docker layer as your app - and so each subsequent build effectively duplicates all of those third-party dependencies within your Docker repository. This is especially wasteful when we consider that your application code is what is going to change the vast majority of the time.

Suppose we have a nicely slimmed-down dockerfile, using an Alpine base image,
and even creating a tiny "custiom" JRE using jlink. However, suppose we
have a lot of dependencies - 77MB in this example.
The key part of our Dockerfile will look like

WORKDIR /app
CMD ["java", "-jar", "/app/my-app.jar"]
COPY --from=app-build /app/target/my-app-1.0-SNAPSHOT-jar-with-dependencies.jar /app/my-app.jar

And the docker layers created will look something like

$ docker history my-app
IMAGE          CREATED              CREATED BY                                      SIZE      COMMENT
548d9d7dc060   28 seconds ago       COPY /app/target/my-app-1.0-SNAPSHOT-jar-wit…   77.7MB    buildkit.dockerfile.v0
<missing>      About a minute ago   CMD ["java" "-jar" "/app/my-app.jar"]           0B        buildkit.dockerfile.v0
<missing>      About a minute ago   WORKDIR /app                                    0B        buildkit.dockerfile.v0
<missing>      About a minute ago   RUN /bin/sh -c apk add --no-cache libstdc++ …   2.55MB    buildkit.dockerfile.v0
<missing>      About a minute ago   COPY /javaruntime /opt/java/openjdk # buildk…   37.1MB    buildkit.dockerfile.v0
<missing>      About a minute ago   ENV PATH=/opt/java/openjdk/bin:/usr/local/sb…   0B        buildkit.dockerfile.v0
<missing>      About a minute ago   ENV JAVA_HOME=/opt/java/openjdk                 0B        buildkit.dockerfile.v0
<missing>      3 months ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      3 months ago         /bin/sh -c #(nop) ADD file:40887ab7c06977737…   7.05MB

The total image size is 128mb, and after pushing the first version, this takes up 96MB in my container registry:

On each subsequent build & push of the new image, we're pushing up a whole layer including all our dependencies all over again:

The push refers to repository [registry.digitalocean.com/my-repo/my-app]
f78ba99c16ec: Pushing  77.74MB/77.74MB
76d6a32df414: Layer already exists
a7b85e634de7: Layer already exists
0eea560269c8: Layer already exists
7cd52847ad77: Layer already exists

So after just 5 pushes, we've already used up 386MB (!)

Splitting apart our dependencies from our application code

Using Maven, we simply replace any uberjar creating step(s) with the following 2 steps:

to copy the dependencies to target/lib
create the jar containing our compiled code only - but set up to look for its dependencies in ./lib :

<plugins>
    <!-- (other plugins...) -->

    <!-- Copy runtime dependencies to lib folder -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <executions>
            <execution>
                <id>copy-dependencies</id>
                <phase>prepare-package</phase>
                <goals>
                    <goal>copy-dependencies</goal>
                </goals>
                <configuration>
                    <outputDirectory>${project.build.directory}/lib</outputDirectory>
                    <includeScope>runtime</includeScope>
                </configuration>
            </execution>
        </executions>
    </plugin>

    <!-- Creates the jar and configures classpath to use lib folder -->
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-jar-plugin</artifactId>
        <configuration>
            <archive>
                <manifest>
                    <addClasspath>true</addClasspath>
                    <classpathPrefix>lib/</classpathPrefix>
                    <mainClass>path.to.MainClass</mainClass>
                </manifest>
            </archive>
        </configuration>
    </plugin>
</plugins>

All we need to do to our Dockerfile is add an extra build step to copy the libs:

WORKDIR /app
CMD ["java", "-jar", "/app/my-app.jar"]

COPY --from=app-build /app/target/lib /app/lib
COPY --from=app-build /app/target/my-app-1.0-SNAPSHOT.jar /app/my-app.jar

(NB It's important here that the lib directory copy happens before the jar copy; if not, then application updates would invalidate the layer cache for the lib directory, thus losing the benefit!)

Benefits of the split

Upon pushing the first version of this build, there's not much difference to what we had before - the overall image size stored in the repository is almost the same. The benefit comes when we push subsequent new builds where we've changed only our source code; after 5 builds (where only our app source has changed), the total size in our image repo is only a few kb larger:

During each push, we only have to push a few more kb each time; now that the libs are in their own layer, which remains unchanged, the image repository is able to re-use the dependencies pushed in previous builds:

b54d889308d2: Pushing   16.9kB
8a9ec9b0baaa: Layer already exists
76d6a32df414: Layer already exists
a7b85e634de7: Layer already exists
0eea560269c8: Layer already exists
7cd52847ad77: Layer already exists

We can now push hundreds of new image builds without getting anywhere near to the limit of the free tier of this image repository, whereas before we'd hit the limit after not even 10 builds!

How the thin jar works

All we're doing with the above build configuration is creating a jar with just our compiled code in it, but telling it where to find the dependencies. If you're not familiar with the structure of a jar file, it's basically a zip file that contains

a load of compiled java classes (.class files)
a "manifest" file, there to tell java things like where your main class is and where any external dependencies are, when you try to run it via java -jar

Uberjars simplify things by putting all the dependency compiled code into the jar along with your own app. But as described in the official docs for jar manifests, all we need to do to split out the dependencies is to specify them in a Class-Path\ entry in our manifest. With the split in place, our manifest file will look something like:

Manifest-Version: 1.0
Created-By: Maven JAR Plugin 3.3.0
Build-Jdk-Spec: 20
Class-Path: lib/kafka-clients-3.4.0.jar lib/zstd-jni-1.5.2-1.jar lib/lz4
-java-1.8.0.jar lib/slf4j-api-2.0.6.jar lib/slf4j-simple-2.0.6.jar
Main-Class: my.app.Main

Notice that annoyingly we have to manually specify each jar file explicitly in the manifest; thankfully, the maven-jar-plugin does this work for us (as should other build tools).

Bonus - Slimming down JNI dependencies

An added bonus to splitting out our library jars is that it makes it easier to further trim down the contents of JNI (Java Native Interface) dependencies. JNI dependencies often contain native libraries for multiple platforms - more waste! With an extra command in our Dockerfile, we can remove these unwanted native libraries from within those jars to trim down the image size even further.

The example application I'm working with is a Kafka Streams app, and so has a transitive dependency on rocksdbjni. This is a whopping 53MB! Within our Dockerfile we can remove the unwanted libraries from it using zip --delete (recall that jar files are just zip files), like so:

RUN zip --delete ./target/lib/rocksdbjni-7.1.2.jar \
    librocksdbjni-linux32-musl.so \
    librocksdbjni-linux32.so \
    librocksdbjni-linux64.so \
    librocksdbjni-linux-ppc64le.so \
    librocksdbjni-linux-ppc64le-musl.so \
    librocksdbjni-linux-aarch64.so \
    librocksdbjni-linux-aarch64-musl.so \
    librocksdbjni-linux-s390x.so \
    librocksdbjni-linux-s390x-musl.so \
    librocksdbjni-win64.dll \
    librocksdbjni-osx-arm64.jnilib \
    librocksdbjni-osx-x86_64.jnilib

This trimmed-down rocksdbjni jar is only 4MB - nice!

Conclusion

We get a lot of bang for our buck here: storage & network costs converted from megabytes to kilobytes with just a few tweaks to our app & container builds. This might sound rather academic, but as we've seen here it can make the difference between hitting your repository storage limit VS having headroom for hundreds of builds.

Give it a try, and let me know what you think!

Hi, I'm Robert Johnson

Robert Johnson — Mon, 17 Apr 2017 09:16:06 +0000

I have been coding for 6 years.

You can find me on GitHub as robert-m-johnson

I live in Poole, Dorset, UK working for Siemens.

In my day job I use Java but in my spare time I like playing with Clojure.

Nice to meet you.