DEV Community: Daniel Fitzpatrick

What should I know about software engineering?

Daniel Fitzpatrick — Tue, 28 Nov 2023 03:25:12 +0000

Today, I received the following text from a close acquaintance with no engineering background.

Hey Daniel. I've really been considering a job in coding. I've always enjoyed working with computers. Can you tell me if you enjoy coding and what do you think I should know?

Suddenly, I realized there was much to share about software engineering as an occupation, too much for a text message.

Do you think like a programmer?

All programmers have some shared characteristics. Donald Knuth, possibly the most decorated professor of computer science, said this during an interview.¹

the psychological profiling [of a programmer] is mostly the ability to shift levels of abstraction, from low level to high level. To see something in the small and to see something in the large.

He explains:

When you're writing a program, you're saying, "Add one to the counter," but you know why you're adding one to the counter. You can step back and see a picture of the way a process is moving. Computer scientists see things simultaneously at the low level and the high level.

Then, he identifies two more shared characteristics.

The other main characteristic of the computer scientist is dealing with nonuniformity of discreet, nonuniform things. We have ten cases instead of one. We don't have one law of thermodynamics, or something. We have case one, case two, case three -- each is different. We build models out of nonuniform parts.

We have this aesthetic of cost, how much work it takes to do things. If your mental model is APL, you optimize in different ways than if your mental model is a RISC machine.

Make programming fun again

When you search the internet for "make programming fun again," you'll find many helpful articles and videos covering that topic. In fact, a driving factor shared by many new programming languages is the creator's desire to "make programming fun again."

Programmers must share this characteristic also: In the beginning, it was fun.

One of your early goals should be determining if you share all these characteristics. If you're dealing with adult life, how do you find time for that!?

Make it a hobby first

When you find time, you should make programming a hobby.

These are the best ways I know to do that. Try all of them for the best results.

Work on coding problems

Advent of Code and Project Euler are a couple I have enjoyed before. These teach you to enjoy problem-solving.

Invent a passion project

For a beginner, this is where you can have the most fun. Your passion project can be anything, but to maximize enjoyment, you should try to induce 'flow' state. To maximize your odds, try solving a problem at the very edge of your ability.

Contribute to an open-source project

Choosing an opportunity that is well within your ability would be best before attempting this. It gives you a sense of satisfaction and has numerous other benefits if you turn your hobby into a career.

Temper your expectations

You think like a programmer, and coding is fun. Now it's time to make money. Maybe you're gifted and can launch a killer app straight away. But if not, brace yourself.

In a software engineering job, much of the work differs from what you enjoy. You must be able to read and write code, but odds are you only relish one of those things. Often, you have to review code, help manage infrastructure (commonly called Operations/Ops), perform quality assurance, and manage releases. You'll find business processes that turn simple tasks into 8-hour ordeals, and you'll often need to wait for other people to finish tasks with unaccommodating deadlines.

Lots of technical components can affect your enjoyment of the job as well. These things run the gamut from your editing environment to your business' deployment platform.

Nothing is special about those things: every white-collar office job shares similar grievances. Software engineering is not your "get out of jail free" card.

But this next part might be unique. If you live in south-central Kentucky (like me) or many cities outside of California, your hobby and job is a black hole where socializing goes to die.

Don't go into detail about work kicking your butt: nobody understands what you do.
Don't go into detail about your accomplishments: nobody understands what you do.

Sidebar: You probably won't become a game developer

Lots of beginning coders want to work on games. Those who succeed have immense energy and are willing to live in their car for peanuts to get started. That isn't most people. It probably isn't you. You'll probably work on something relatively boring.

Learn to love boring

You may have heard other developers talking about this. "Boring" means nobody is panicking & nothing is breaking. You can only afford this luxury by staying behind the curve and making releases at a reasonable pace.

Here be dragons

Software engineers "have it pretty good", but even perfect work environments can yield psychological and physical problems. Many developers deal with imposter syndrome. Jamis Buck offers a cautionary tale about burnout, another common issue with developers. RSI, carpal tunnel, and "emacs pinky" are all possible dangers.

You may transform into a keyboard enthusiast who wastes hundreds of dollars on keyboards and has lost touch with friends and family. 😱 (this is a joke)

Will you enjoy it?

That's hard to say. Some people regret turning their hobby into a job, owing to the age-old anecdote², but I see no other way to be a developer. Others say they're blessed because they would do it for free, owing to another age-old colloquialism³. The truth is likely somewhere in the middle.

https://www.drdobbs.com/an-interview-with-donald-knuth/184409858 ↩
"don't make your hobby your job." ↩
"Choose a job you love, and you will never have to work a day in your life." ↩

Having fun with money

Daniel Fitzpatrick — Tue, 20 Jun 2023 15:05:45 +0000

I reached a significant financial milestone this month and am focusing on new goals. Like solving a performance problem, you should always measure first.

After poking around at different account aggregation tools, I settled on pocketsmith. Their simple REST API was a main draw:

Docs
OpenAPI

That task alone wasn't interesting enough, so I wrote a small Clojure library to talk to the API. Ideally, it would leverage something like martian, but my attempt was unsuccessful. That could be a task for the future.

crinklywrappr / pocketsmith-api

small wrapper around the pocketsmith api for clojure developers

com.github.crinklywrappr/pocketsmith-api

Clojure library for interacting with the Pocketsmith REST API. Small, opinionated, and hand-crafted.

Coordinates

com.github.crinklywrappr/pocketsmith-api {:mvn/version "1.0.37"}

Usage

First, require the library and define your user key. You can generate a key on the dashboard from your pocketsmith account.

(require '[crinklywrappr.pocketsmith-api.core :as ps])
(def mykey "xxx")

Then, begin querying.

(def myuser (ps/authorized-user mykey :convert? true :minify? true))
;=> {:name "John Doe",
     :login "my-username"
     :email "my-address@email.com",
     :week-start-day 0,
     :id 1111111,
     :base-currency-code #object[org.joda.money.CurrencyUnit 0x4d1b5405 "USD"],
     :time-zone #object[org.joda.time.tz.CachedDateTimeZone 0x468ce331 "America/Chicago"]}

(def mycategories (into [] (ps/categories mykey myuser :flatten? true :normalize? true :convert? true :minify? true)))
;=> [{:id XXX, :title "Charity", :parents []}
     {:id XXX, :title "Taxes", :parents []}
     {:id XXX, :title "Bank",

…

View on GitHub

pocketsmith-api uses Joda Time & Joda Money and has 99% code coverage. Because it deals with money, I wanted to ensure it read amounts accurately in various currencies.

That said, Joda Money has some limitations. I have listed some on the README.

Let's build something with it!

Acorns...?

Acorns is a savings tool that, in the past, worked by sending users monthly reports of how much they could save if they rounded up all their charges to the nearest dollar and transferred the difference to a savings account.

They may do more now.

I always thought that idea was excellent but I wanted to avoid paying for it. With Pocketsmith, we can accomplish that easily.

Here is the code.

Call (acorns) to get the desired amount.

Project setup

I prefer deps-new these days.

clojure -Tnew lib :name <project-name>

Add crinklywrappr/pocketsmith-api to your deps.edn. Clojure will pull in all of the other dependencies you need transitively.

com.github.crinklywrappr/pocketsmith-api {:mvn/version "1.0.37"}

Sanity checks

I like to be extra careful when dealing with money. Please verify some elementary assertions before borrowing this code for any purpose. I will demonstrate some tests on my account and list additional applicable checks.

The Clojure repl is unmatched for this.

Am I looking at the correct accounts? ✓

I only want to consider credit card accounts.

(into [] (r/map :name (cards (user))))
;=>
["Amex card" "Food card" "Amazon card" "Gas card" "Points card"]

Am I requesting the correct transactions? ✓

Specifically, we only want to look at last month's debit transactions. The current date is June 19, 2023.

(last-month-query (user))
;=>
{:start_date "2023-05-01", :end_date "2023-05-31", :type "debit", :per_page 100}

Am I receiving the correct transactions? ✓

Easy. We can sort the transactions by date and inspect the first and last elements.

(let [me (user)]
  (->> (last-month-charges me (cards me))
       (sort-by :date clj-time.core/before?)
       ((juxt first last))
       (mapv #(select-keys % [:date :payee]))))
;=>
[{:date
  #object[org.joda.time.LocalDateTime 0x5df45749 "2023-05-01T00:00:00.000"],
  :payee "SPOTIFY NEW YORK NY P22C7B5A33 XXXXXX1161"}
 {:date
  #object[org.joda.time.LocalDateTime 0x140ae942 "2023-05-31T00:00:00.000"],
  :payee "DOORDASH*LUNCH BOX CSAN FRANCIS NT_NZXE7CMB +XXXXXXX9470"}]

Does the math make sense? ✓

We can do this by taking 10 of the transactions and eyeballing the numbers.

(require '[clojurewerkz.money.format :as mf])
(let [me (user)]
  (clojure.pprint/print-table
   (->> (last-month-charges me (cards me))
        (map
         (fn [x]
           {:rounded (mf/format (round-up x))
            :amount (mf/format (:amount x))
            :difference (mf/format (difference x))}))
        (take 10))))

| :rounded | :amount | :difference |
|----------+---------+-------------|
|  $-55.00 | $-54.80 |       $0.20 |
|  $-26.00 | $-25.44 |       $0.56 |
|  $-29.00 | $-28.02 |       $0.98 |
|  $-23.00 | $-22.19 |       $0.81 |
|   $-3.00 |  $-2.99 |       $0.01 |
|  $-38.00 | $-37.94 |       $0.06 |
|  $-50.00 | $-49.81 |       $0.19 |
|  $-16.00 | $-15.44 |       $0.56 |
|  $-24.00 | $-23.90 |       $0.10 |
|   $-1.00 |  $-0.64 |       $0.36 |

$54.80 rounds up to $55.00, and the difference is $0.20, and so forth.

Additional checks

There are other assertions you may want to test, and I encourage you to do so.

Are all the differences under $1.00?
Does difference respond with $0.00 when given an even dollar charge?
Is the total amount less than the number of charges?
Are all the charges truly debits (negative dollar amounts)?

Conclusion

This project was a fun distraction while gearing up for my next financial goals. Hopefully, this selfish exercise will help someone else. 😉

By the way, I need to squirrel away $40.63 from last month.

The hard way or the easy way?

Daniel Fitzpatrick — Mon, 13 Feb 2023 16:59:08 +0000

This story is about doing the wrong thing™ for selfish reasons, although there is an adequate amount of how-to, so it qualifies as a tech article.

The background

Where I last worked, the backend developers agreed that we had a problem with HTTP response schemas. We agreed that we needed them. We agreed that they were a nuisance.

Have you ever seen a Malli invalid schema report?

Nobody wanted to deal with these after each commit, so something had to be done.

Doing something!

Removing the schemas was not an option because they provided too much benefit to the front-end developers.

Likewise, if we stopped maintaining the HTTP response schemas, a front-end developer would eventually discover our negligence.

We had to be clever.

How we did it

A significant feature of reitit is that routes are composed of simple vectors and maps. That means you can pre-process that data structure however you want before building the router.

That means we can jerry-rig our router to include, for instance, different middleware for use only in our tests.

I warned you. 😈

We'll start by copying our response schemas to a new key-value pair called test-schemas.

["my/route"
 {:get
  {:middleware []
   :handler my/handler
   :test-schemas {200 schemas/ResponseSchema}}}]

test-schemas expresses our validation schema to the middleware.

What middleware?

Let's make a new middleware function in our testing layer.

(def schema-warnings
  "aggregates a vector of {:request ..., :explanation ...} maps"
  (atom []))

(defn build-test-middleware
  [schemas]
  (fn test-middleware
    ([handler]
     (test-middleware handler {}))
    ([handler _]
     (fn [request]
       (let [response (handler request)
             status (:status response)
             body (:body response)]
         (if-let [schema (get schemas status)]
           (if (not (malli/validate schema body))
             (do
               (swap! schema-warnings conj
                      {:request (:original-request request)
                       :explanation (malli/explain schema body)})
               response)
             response)
           response))))))

build-test-middleware returns a standard ring middleware function. Whenever there is a request, it captures the response and compares it to the schema corresponding to the HTTP response status using malli/validate. If the response is invalid, we save it to an atom for later review.

How do we wedge the testing middleware into place?

(defn test-routes [routes]
  (letfn [(add-test-middleware
            [[request-method
              {:keys [test-schemas] :as details}]]
            (if (some? test-schemas)
              [request-method
               (-> details
                   (update :middleware (comp #(conj % (build-test-middleware test-schemas)) #(or % [])))
                   (dissoc :test-schemas))]
              [request-method details]))]
    (mapv
     (fn [[url request-methods]]
       [url
        (->>
         request-methods
         (mapv add-test-middleware)
         (into {}))])
     routes)))

test-routes modifies routes by adding the test-middleware when a test-schemas key-value pair exists i.e.

(test-routes
 [["my/route"
   {:get
    {:middleware []
     :handler (fn [& args] {:status 200 :body {}})
     :test-schemas {200 [:map [:foo number?]]}}}]])

;; result
[["my/route"
  {:get
   {:middleware
    [#function[my-ns/build-test-middleware/test-middleware--14034]],
    :handler #function[my-ns/fn--14169]}}]]

Let's see if it works!

((reitit.ring/ring-handler
  (reitit.ring/router
   (test-routes
    [["my/route"
      {:get
       {:middleware []
        :handler (fn [& args] {:status 200 :body {}})
        :test-schemas {200 [:map [:foo number?]]}}}]])))
 {:request-method :get :uri "my/route"})

;; result
{:status 200, :body {}}

We were able to make a request. However, you can see that the response body doesn't match the schema. Did we record a problem?

Let's check.

@schema-warnings

;; result
[{:request nil,
  :explanation
  {:schema [:map [:foo number?]],
   :value {},
   :errors
   ({:path [:foo],
     :in [:foo],
     :schema [:map [:foo number?]],
     :value nil,
     :type :malli.core/missing-key,
     :message nil})}}]

There wasn't any request to save because the original-request key-value pair was missing from the request. Adding that is easy.

(defn run-request [router request]
  (router (assoc request :original-request request)))

Let's try again.

(run-request
 (reitit.ring/ring-handler
  (reitit.ring/router
   (test-routes
    [["my/route"
      {:get
       {:middleware []
        :handler (fn [& args] {:status 200 :body {}})
        :test-schemas {200 [:map [:foo number?]]}}}]])))
 {:request-method :get :uri "my/route"})

;; result
{:status 200, :body {}}

@schema-warnings

;; result
[{:request {:request-method :get, :uri "my/route"},
  :explanation
  {:schema [:map [:foo number?]],
   :value {},
   :errors
   ({:path [:foo],
     :in [:foo],
     :schema [:map [:foo number?]],
     :value nil,
     :type :malli.core/missing-key,
     :message nil})}}]

Now we have the request paired with the invalid schema report.

Outcome

Despite being thoroughly the wrong thing to do, this generally led to a happier backend team. But what about the front-end developers? Did they ever call out our negligence? Did it cause problems in production?

Surprisingly no. We continued to handle problems with the response schema inside feature branches. Mostly. 😈

Build a unique résumé

Daniel Fitzpatrick — Mon, 23 May 2022 15:23:52 +0000

Well-documented resources can help make your résumé stand out even if you aren't artistic. Today, I'll walk you through how I used existing resources to build an exciting résumé without design knowledge.

Latex is essential, IMHO. Other tools simplify résumé creation by degrees, but Latex will help you avoid the cookie-cutter stereotype. I pay for Overleaf, but anyone willing to learn Latex can do this for free.

An additional benefit to Latex is using it to generate PDFs. It's the primary publication format on Overleaf.

DISCLAIMER

Although this post is mostly about résumé presentation, a small portion is about content.

I am not an expert recruiter and am not sharing advice to help you pass ATS software. I am sharing approaches I have learned to promote myself to people, and I am still learning!

Please be nice in the comments.

TL;DR

Scan for the 💡 to pick out helpful suggestions, like the suggestion below.

💡 A benefit PDFs have over some alternate formats is that it's more difficult for a recruiter to change your perfectly-crafted résumé.

Choosing a template

It starts with a good template. I chose this one because

It claims to be easy to alter
It places the sidebar on the right

💡 A lot of résumés place their sidebar on the left, but that is sub-optimal for content placement. A left sidebar will cause more lines to overflow in your primary content area, which means you can shove more content onto a page with the sidebar on the right.

Transferring content

I have a lot of content from my old résumé, so I'm going to transfer that first, minus some contact information. I won't go into all the details, but it's probably worth pointing out how I changed the header, images, and bar graph.

The header is full of clickable contact info.

\name{Daniel}{Fitzpatrick}
\address{Bowling Green, Kentucky}
\github{crinklywrappr}
\homepage{dev.to/crinklywrappr}

I uploaded a personal photo, edited it with Gimp, and exported it to PDF.¹

\photo[circle,edge,fill,left]{2.5cm}{pics/headshot.pdf}

The template measures experience with various tools in hours, but I want to instead express a vague competency with each tool using values 1-10. Also, in my old résumé, I used a heart to express an affinity for a particular tool. 😉

\skillset[label/0\hfill 10]{SQL/.8,Java/.5,Perl/.6,C\#/.5,♥ Clojure(script)/.9}

💡 The major categories should be placed on the page from most relevant to least relevant, top to bottom. In my case, this means Professional Experience should go at the top.

Cleaning up

What's that unsightly comma doing there?

The honors section I re-purposed for Research has some incorrect assumptions. You can see the problem here, in ReCeiVe.cls.

You can also see an alignment issue near the top. That is because I omitted the position field.

We can fix that by adjusting the experience section position. After figuring out a rough measurement, I added this line to the top of cvSections/professionalExperience.tex.

\vspace{-7.5mm}

Nice.

Theme it!

💡 You don't have to be an artist to pick a good color palette. Many tasteful palettes exist already because we've been theming our desktops and editors for ages. I encourage you to visit r/unixporn for inspiration.

I have chosen the Nord theme for my résumé. The docs are essential.

💡 A dark style can be tempting. However, some people will print your résumé, which makes light styles a better choice in most cases.

I begin by copying the palette into ReCeiVe.cls in the Color configurations section.

\definecolor{nord0}{HTML}{2E3440}
\definecolor{nord1}{HTML}{3B4252}
\definecolor{nord2}{HTML}{434C5E}
\definecolor{nord3}{HTML}{4C566A}
\definecolor{nord4}{HTML}{D8DEE9}
\definecolor{nord5}{HTML}{E5E9F0}
\definecolor{nord6}{HTML}{ECEFF4}
\definecolor{nord7}{HTML}{8FBCBB}
\definecolor{nord8}{HTML}{88C0D0}
\definecolor{nord9}{HTML}{81A1C1}
\definecolor{nord10}{HTML}{5E81AC}
\definecolor{nord11}{HTML}{BF616A}
\definecolor{nord12}{HTML}{D08770}
\definecolor{nord13}{HTML}{EBCB8B}
\definecolor{nord14}{HTML}{A3BE8C}
\definecolor{nord15}{HTML}{B48EAD}

You can tweak the color of some elements immediately.

Nord0 is intended to be used for text.

Nord7 is perfect for subheadings.

Nord9 and Nord10 are suitable for the job title beneath the Business subheading. After playing around, I decided on Nord10.

Finally, Nord1 is perfect for the lighttext class of elements. Just look at the tagline.

In Latex, it all looks like this bit of templating.

\colorlet{darktext}{nord7}
\colorlet{text}{nord0}
\colorlet{graytext}{nord10}
\colorlet{lighttext}{nord1}

Here, you can see how this breaks down.

That styling looks good, but I have a nagging feeling that the location should be Nord10 to match the date range. That's easy to change in ReCeiVe.cls under "Section styles".

\newcommand*{\entrylocationstyle}[1]{{\fontsize{9pt}{1em}\bodyfontlight\slshape\color{graytext} #1}}

Now it looks better.

Next up, we have another minor issue with our template. We can set the header color, but the background color is hardcoded to black. So, we need to change that in a couple of places in ReCeiVe.cls.

First, we define a new color, highlightbox, so named because our title color is highlight. I default it to black to preserve the original style.

\colorlet{highlightbox}{black}

Then, we replace \colorbox{black} with \colorbox{highlightbox} everywhere it occurs. Finally, we can set the colors for our new header.

Nord8 must be our header color.

And Nord doesn't leave us any questions about the new background color either. It's obviously Nord6.

Here's the template code.

\colorlet{highlightbox}{nord6}
\colorlet{highlight}{nord8}

And here is the result.

The main body looks promising. But, unfortunately, our sidebar looks like trash. So, let's examine that next.

Fixing the sidebar

Here is where I think the major problems lie.

The sidebar color does not belong to the Nord color scheme.
The bar graph colors do not belong to the Nord color scheme.
The bar graph labels appear to be styled with the same color as the main headers. They would look better if they matched the font and color of the job title from the previous section.
The sidebar corner radius is too large.

Let's take this one by one.

I will take the same approach as highlightbox to fix the sidebar color. Start by defining a new color in ReCeiVe.cls.

\colorlet{sidebarcolor}{lightgray}

You want to set the fill to that color.

Last, I set the sidebarcolor near the top of my résumé.

\colorlet{sidebarcolor}{nord6}

That is a significant improvement to our sidebar, but fixing the background color has caused the graph labels to stick out even more. So let's fix them next.

That is pretty easy to do. Just find the For elements of sidebar section of ReCeiVe.cls and

Set the color to graytext.
Set the font to scshape.

Nice! However, like with the last change, the improved labels make the bar graph stick out worse than before. So let's tackle the bar graph next.

Once again, I will take the same approach as highlightbox and sidebarcolor. Start by defining some colors in ReCeiVe.cls.

\colorlet{skillsetfillbg}{white}
\colorlet{skillsetfillfg}{gray}

Then use them in the skills definition.

Finally, set the colors at the top of your résumé. But what colors should we use?

I feel like the goal of expressing "a vague competency with each tool" is better served by matching the background color with the sidebar color. The Nord8 description suits the bar graph color, perhaps surprisingly.

# near the top of the resume
\colorlet{skillsetfillbg}{nord6}
\colorlet{skillsetfillfg}{nord8}

Finally, I want to ensure that the ♥ symbol has a suitable color, so I chose a reddish hue from Nord, Nord11.

\skillset[label/0\hfill 10]{SQL/.8,Java/.5,Perl/.6,C\#/.5,{\color{nord11} ♥} Clojure(script)/.9}

Our histogram looks fantastic!

Last, let's tackle these feathery corners. I will use the same approach as before. First, define a value sidebarRadius in ReCeiVe.cls.

\newlength{\sidebarRadius}
\setlength{\sidebarRadius}{0.5cm}

Then, apply that to the sidebar definition.

Finally, please give it a new definition near the top of your résumé.

\setlength{\sidebarRadius}{0.15cm}

Last, I am going to remove the ring around my portrait. Find the \photo definition near the top of your résumé, and change edge to noedge.

That sidebar is looking good now!

Improve "skimmability."

I have links in the main body of my résumé. In the future, they may be in the sidebar, too. Can you find them in this image?

Congratulations if you picked this yellow highlighted area.

💡 Quickly identifiable links will encourage hiring managers to explore beyond your résumé. It gives you depth.

We can do better. Latex allows you to customize bullet points, so let's use the "chain link" Unicode character. Other characters could work, too. You can also remove the bullet point and move the symbol to the right - dealer's choice.

I'm going to edit these three bullets in cvSections/professionalExperience.tex.

\begin{cvitems}
\item[{\fontspec{Symbola}\symbol{"1F517}}] \href{https://dev.to/crinklywrappr/using-ec2-to-enhance-k6-load-tests-57nj}{Using EC2 to enhance K6 load tests}
\item[{\fontspec{Symbola}\symbol{"1F517}}] \href{https://dev.to/crinklywrappr/saving-the-union-with-honeysql-4apm}{Solving a DRY problem with UNIONs using HoneySQL}
\item[{\fontspec{Symbola}\symbol{"1F517}}] \href{https://dev.to/crinklywrappr/recording-hikaricp-metrics-in-clojure-1baf}{Recording HikariCP metrics in Clojure}
\end{cvitems}

💡 It's good to maintain this practice in new sections, even if they're stylistically different.

What to emphasize?

💡 You probably already know that it's best to emphasize significant contributions and accomplishments at prior institutions.

That approach can be more difficult for developers than it sounds. Take this line from my résumé, for example.

When I think about that accomplishment, the customer is of little interest to me. Instead, I recall the niche technology and strange development environments: coding on a 5-gallon plastic bucket in the middle of a grocery store self-checkout area.

💡 An interview is largely about determining the boundaries of your comfort zone.

I think some developers have a blind spot about that. Your comfort zone includes technology but also people. Think about the social aspect. Are you comfortable talking with a customer, for instance?

Identifying comfort zone boundaries can be framed as a contribution. Again, from my résumé.

💡 You should absolutely tout and link to significant open-source contributions. PRs, reviews, projects, blogs. Whatever you think highlights the qualities you want to promote.

I think some developers view their open-source contributions as orthogonal, irrelevant, and perhaps even detrimental to their image during a review. That view couldn't be further from the truth. I have met more than one hiring manager who elected to do a codebase walkthrough instead of a technical assessment.

My final résumé has these sections.

Final result

Here it is. Does it look compelling? Please give me your thoughts!

💡 Self-portraits add polish, but recruiters may suggest they harm résumés because hiring managers can subconsciously accept or reject you based on your appearance. Choose wisely. ↩

[Ann] Rivendell updated

Daniel Fitzpatrick — Mon, 02 May 2022 15:33:57 +0000

About

Rivendell adds useful functionality to elvish to take your shell to the next level!

🌈 functional bits: high-level functions which encapsulate common design patterns.
💤 lazy iterators: transducer-inspired iterators which allow you to represent infinite sequences.
📈 visual aids: sparklines & histograms.
🔧 plus a toolbelt of common utility functions which operate on strings, lists, and maps.

How to install

It's a 3-step process.

Add these lines to your ~/.config/elvish/rc.elv

epm:install &silent-if-installed=$true github.com/crinklywrappr/rivendell
epm:upgrade github.com/crinklywrappr/rivendell

Request modules à la carte.

use github.com/crinklywrappr/rivendell/test t
use github.com/crinklywrappr/rivendell/base b
use github.com/crinklywrappr/rivendell/fun f
use github.com/crinklywrappr/rivendell/lazy l
use github.com/crinklywrappr/rivendell/rune r
use github.com/crinklywrappr/rivendell/algo a
use github.com/crinklywrappr/rivendell/vis v

Do cool stuff!


# lazily graphing population data from the 2021 census

var file = (a:lines NST-EST2021-alldata.csv)

var f = {|line| s:split , $line | f:listify}

var popkeys = ($f (l:first $file))

l:drop 6 $file ^
| l:each (f:comp $f (f:partial $f:zipmap~ $popkeys) (f:juxt (f:k NAME) (f:k POPESTIMATE2021)) $f:listify~) ^
| l:take 20 ^
| l:blast ^
| v:barky &min=0

        Alabama ████████
         Alaska █
        Arizona ███████████
       Arkansas ████
     California ███████████████████████████████████████████████████████████████
       Colorado █████████
    Connecticut █████
       Delaware █
District of Co… █
        Florida ███████████████████████████████████
        Georgia █████████████████
         Hawaii ██
          Idaho ███
       Illinois ████████████████████
        Indiana ███████████
           Iowa █████
         Kansas ████
       Kentucky ███████
      Louisiana ███████
          Maine ██

Documentation in the terminal

Command-line users are recommended to use glow.

Run the following command to browse the docs.

glow ~/.local/share/elvish/lib/github.com/crinklywrappr/rivendell/

Flashcards in the terminal

Daniel Fitzpatrick — Mon, 25 Apr 2022 08:05:06 +0000

Several weeks ago, I wrote a webdriver app in Clojure to pull directory information from congregate websites. The general idea was this:

I have trouble remembering names and faces. What if I had a flashcard system to help me memorize them?

However, I got bored before I could finish.

Rivendell update

I've been hard at work this week updating Rivendell for Elvish v0.17. I recently finished testing and documenting the fun module, which houses most of the functional pieces. You can view the docs here.

I have another blog post about how I generated that documentation if you find that interesting.

Generating docs from tests

Daniel Fitzpatrick ・ Apr 4 ・ 3 min read

#testing #bash

With the progress made on Rivendell, I think now would be an excellent time to finish the flashcard app.

Setup instructions

Since Rivendell is currently under development, I am using Elvish's rsync functionality.

Clone rivendell and switch to the update-for-0.17 branch. Then place this file in ~/.local/share/elvish/lib/dev/epm-domain.cfg

{
    "method": "rsync",
    "location": <path to rivendell parent directory>,
    "levels": "1"
}

Then, add this to your ~/.config/elvish/rc.elv.

use epm
epm:install &silent-if-installed=$true dev/rivendell
epm:upgrade dev/rivendell
use dev/rivendell/fun

Last, you'll need a terminal that understands the Terminal Graphics Protocol. I'll be using Kitty.

Needed functions

The functions we need from Rivendell are shuffle and destruct. Next, we need a way to display images. If you want to replicate my environment, I have this function defined in my rc.elv for displaying images.

fn icat {|@a| kitty +kitten icat $@a }

Next, we need a way to score guesses. For that, let's define a simple Levenshtein function. Low scores are better.

use dev/rivendell/base b
use dev/rivendell/fun f
use math

fn levenshtein {|w1 w2|
  var cell-value = {|same-char prev-row cur-row col-idx|
    math:min (b:inc $prev-row[$col-idx]) ^
             (b:inc (b:end $cur-row)) ^
             (+ (if $same-char { put 0 } else { put 1 }) ^
                $prev-row[(b:dec $col-idx)])
  }

  var row-idx = 1
  var max-rows = (b:inc (count $w2))
  var @prev-row = (range (b:inc (count $w1)))

  while (!= $row-idx $max-rows) {
    var ch2 = $w2[(b:dec $row-idx)]
    set row-idx = (b:inc $row-idx)
    set prev-row = (f:reduce {|cur-row i|
        var same-char = (eq $w1[(b:dec $i)] $ch2)
        b:append $cur-row ($cell-value $same-char $prev-row $cur-row $i)
    } [$row-idx] (range 1 (count $prev-row)))
  }

  b:end $prev-row
}

That code takes advantage of Elvish's module aliasing feature and also demonstrates a few more simple Rivendell functions, reduce being the most interesting.

I placed this code in a file called levenshtein.elv and imported it with use ./levenshtein l.

Next, we need a function to tie these together. It's not fancy. I just defined this in the terminal.

fn guess {|name image|
  icat $image; echo
  var guess = (read-line)
  var score = (l:levenshtein $name $guess)
  put $name $score
}

We have all the components now. So let's pull our directory info and start scoring guesses.

Running the flashcard "app"

Is this thing categorically different from an app? Probably.

After you've built the congr project from the first post, execute with:

var @directory = (java -jar target/uberjar/congr-0.1.0-SNAPSHOT-standalone.jar site username password)

That last command may take several seconds.

Next, shuffle it.

var @shuffled = (each {|x| echo $x | from-json} $directory | fun:shuffle)

Last, start guessing.

each (fun:destruct $guess~) $shuffled

For privacy reasons I won't show actual names and photos from the directory, but you can view a screenshot with mocked information from knowyourmeme.com.

Conclusion

I always envisioned a stupid simple terminal "app" for this project, so it appears I succeeded. Furthermore, I appreciate that Clojure and Elvish worked together to make the experience genuinely hacker-ish.

Happy hacking!

Using scittle to solve wordle

Daniel Fitzpatrick — Mon, 18 Apr 2022 16:08:21 +0000

A co-worker wrote a web app to solve Wordle puzzles, and I thought it would be fun to port it to Scittle. It works, but not super well (as you shall see). Use his app instead.

High-level overview (tl;dr)

I used the following tools for static site development

Babashka's task runner for site building.
Selmer for templating.
Hiccup for html.
Scittle with reagent for frontend logic.

Everything worked well, but Scittle seemed laggy. The lag might be attributable to a mistake that triggers too many redraws of some components. Readers are welcome to submit a PR to fix it.

Project
App

Rather than steal Vincent's algorithm, I wrote my solver from scratch. I cannot say the same for the window-dressing: I have blatantly stolen his CSS. 😄

The algorithm

The algorithm is a fancy filter/remove predicate at bottom.

(defn filter-words [words]
  (remove unfit-word? words))

An excellent first step would be determining which letters are possible at a given index. A letter is allowable if it's

green at the given index
yellow, but not at the given index
missing from the blacklist

These possibilities are mutually exclusive.

(defn get-possible-letters
  "Returns a set of allowable letters for a given index"
  [index]
  (if-let [letter (get @greenlist index)]
    #{letter}
    (set/difference alphas @blacklist (get @yellowlist index))))

I have modeled the individual lists to facilitate this function.

(def blacklist (r/atom #{}))

(def yellowlist (r/atom [#{} #{} #{} #{} #{}]))

(def greenlist (r/atom [nil nil nil nil nil]))

unfit-word? can now be written:

(defn unfit-letter? [[index letter]]
  (nil? ((get-possible-letters index) letter)))

(defn unfit-word? [indexed-yellows word]
  (some unfit-letter? (map-indexed vector word)))

This code represents most of the work required, but there's an important piece missing. If a letter is in the yellow list, then it must be part of the word. But, unfortunately, we haven't guaranteed that.

If we could transform a word into a set containing only the letters that aren't in a given set of indices, then it would be possible to perform this check.

Imagine that we have the word "truth," and both t's are yellow. In our model that looks like eg [#{t} #{} #{} #{t} #{}]. The word "about" fits the criteria. Let's work this backward.

;; remove the indices specified by the yellowlist and see if 't' is in the resultset
(#{\b \o \t} \t) ;=> \t
(#{\b \o \t} \x) ;=> nil

;; how do we get #{\b \o \t}?
;; there are many ways but let's try this one
(disj (set (replace-idx {0 nil 3 nil} (vec "about"))) nil)

;; `replace-idx` doesn't exist in scittle.
;; We could write it but let's try this instead
(reduce-kv
 (fn [a k v]
   (if (#{0 3} k)
     a (conj a v)))
 #{} (vec "about"))

;; how do we go from [#{t} #{} #{} #{t} #{}] to #{0 3}?

I will define a function called index-yellow-letters.

(defn index-yellow-letters []
  (reduce-kv
   (fn [a k v]
     (reduce
      (fn [ax bx]
        (update ax bx conj k))
      a v))
   {} @yellowlist))

This get's pretty close to what we want.

(reset! yellowlist [#{t} #{} #{} #{t} #{}])
(index-yellow-letters) ;=> {\t (0 3)}

Next, let's define a function called unfit-subword?, where 'subword' refers to a set of letters, e.g. #{\b \o \t} in the previous example. This function will encapsulate the rest of the logic we worked through earlier.

(defn unfit-subword? [word [letter ix]]
  (nil?
   (reduce-kv
    (fn [a k v]
      (if ((set ix) k)
        a (conj a v)))
    #{} (vec word))
   letter))

Finally, redefine unfit-word? & filter-words to take this new logic into account.

(defn unfit-word? [indexed-yellows word]
  (or (some unfit-letter? (map-indexed vector word))
      (some (partial unfit-subword? word) indexed-yellows)))

(defn filter-words [words]
  (remove (partial unfit-word? (index-yellow-letters)) words))

The good

Using Selmer & Hiccup for static site construction (and Babashka's task runner for running it) worked so marvelously that I want to use them to write a fully featured static site generator.

Shout-out to miniserve. I didn't need it for this project because I wanted to generate a single file. If I had generated multiple output files, Miniserve would have been very useful for testing. 😄

The bad

If I want to write a "general use" static site generator, I will likely need to add many tags. yogthos/selmver#278 for reference.

The ugly

Scittle is super cool but under-performing in its current state. You probably noticed some lag when toggling colors.

That might be my fault, though. I chose to model the state like this:

(def blacklist (r/atom #{}))

(def yellowlist (r/atom [#{} #{} #{} #{} #{}]))

(def greenlist (r/atom [nil nil nil nil nil]))

As you can imagine, a color toggle alters all three of these "ratoms." This behavior means that, unless there is some debouncing under the covers, color toggling triggers more redraws than necessary. I will happily accept a PR if you think this is the problem.

Using EC2 to enhance k6 load tests

Daniel Fitzpatrick — Mon, 11 Apr 2022 08:17:58 +0000

k6 is the best load test tool available, but it's pricey. I wanted to start unlimited VUs (virtual users) without paying a lot of money and found a neat way to do it by spawning many short-lived EC2 instances!

Too many templates

I used a simple user-data script with heredocs. Make sure you have an AMI with Docker installed. You will also need S3 access.

Start by setting up AWS credentials.



# load_test.user-data
echo "setting up aws-cli"
mkdir -p /home/ec2-user/.aws
cat <<EOF > /home/ec2-user/.aws/config
[default]
output=json
region=us-west-1
EOF

cat <<EOF > /home/ec2-user/.aws/credentials
[default]
aws_access_key_id={{ aws-creds.access-key-id }}
aws_secret_access_key={{ aws-creds.secret-access-key }}
EOF

Next, you can write out the load-test and make it executable:



# load_test.user-data
echo "writing load test: {{load-test}}"
mkdir -p /home/ec2-user/load_test
chmod a+rw /home/ec2-user/load_test
cat <<EOF > /home/ec2-user/load_test/{{load-test}}
{{load-test-content|safe}}
EOF

We pull the k6 docker image and execute it with a single command.



# load_test.user-data
echo "running load test"
sudo -u ec2-user docker run --rm \
  --ulimit nofile=65536:65536 \
  --network host \
  -v /home/ec2-user/load_test/:/mnt/load_test:rw \
  -i loadimpact/k6 \
  run --summary-export=/mnt/load_test/out.json \
  /mnt/load_test/{{load-test}} > /dev/null

Finally, upload the test results to s3.



# load_test.user-data
echo "uploading results"
sudo -u ec2-user aws s3api put-object \
  --bucket load-test-results \
  --key {{uuid}}/{{instance-name}} \
  --body /home/ec2-user/load_test/out.json > /dev/null

IMPORTANT: You should set a trap function to call systemctl halt to shutdown the EC2 instances. Leaving EC2 instances running would be counter-productive to our goal of saving money.

You may have noticed that I have used templates in the user-data. We can use Selmer to supply all of the important stuff.



(defn build-user-data
  [aws-creds uuid instance-name load-test load-test-template aws-creds]
  (selmer/render-file
   "load_test.user-data"
   {:aws-creds aws-creds
    :uuid uuid
    :instance-name instance-name
    :load-test (.getName (io/file (io/resource load-test)))
    :load-test-content (selmer/render-file load-test load-test-template)}))

aws-creds should be a map with keys :access-key-id and :secret-access-key. These are your S3 credentials.

You may have noticed that load-test-content is itself a template. That is so we can point the load test to an arbitrary address and host our web app from anywhere. Here is an example load-test.



import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
    stages: [
        { duration: '30s', target: 5000 }, // ramp up
        { duration: '30s', target: 5000 }, // sustained load
        { duration: '30s', target: 0 }, // cool down
    ],
};

export default function () {
    let res = http.get('{{url-prefix}}/ping')

    check(res, {'status is 200': (r) => r && r.status === 200});

    sleep(1);
}

load-test-template should be a map with a single key :url-prefix. You may find more exciting uses for using templates in your load tests. 😉

Orchestration

We can use the excellent Cognitect AWS library to kick off our tests.



(defn launch-load-test-instance
  [ec2 aws-creds uuid instance-name load-test load-test-template]
  (aws/invoke ec2
              {:op :RunInstances
               :request {:ImageId "my-ami"
                         :InstanceType "my-size"
                         :MinCount 1 :MaxCount 1
                         :InstanceInitiatedShutdownBehavior "terminate"
                         :UserData (-> (build-user-data aws-creds uuid instance-name load-test load-test-template)
                                       byte-streams/to-byte-array
                                       base64/encode
                                       byte-streams/to-string)
                         :TagSpecifications [{:ResourceType "instance"
                                              :Tags [{:Key "Name" :Value instance-name}
                                                     {:Key "load-test-id" :Value uuid}]}]
                         :SecurityGroupIds [...]
                         :KeyName "my-company"}}))

(defn launch-load-test-instances
  [ec2 aws-creds uuid instance-prefix load-test load-test-template num-instances]
  (doall
   (map
    #(launch-load-test-instance ec2 aws-creds uuid
                      (str instance-prefix %)
                      load-test load-test-template)
    (range num-instances))))

There are some gaps in the above code that you will need to fill.

I feel like a mad scientist after writing that code. Wreak havoc, my minions!

Gathering test results

In the previous code, you may have noticed that we supplied a uuid to each ec2 instance and used it in the user-data script as a shared bucket for all of our k6 results. I hope you kept track of it. 😈



(def s3-bucket "load-test-results")

(defn get-k6-summary [s3 key]
  (->> {:op :GetObject
        :request {:Bucket s3-bucket
                  :Key key}}
       (aws/invoke s3)
       :Body
       io/reader
       json/decode-stream))

(defn get-k6-summaries [s3 uuid]
  (map
   (comp (partial get-k6-summary s3) :Key)
   (->> {:op :ListObjects
         :request {:Bucket s3-bucket
                   :Prefix uuid}}
        (aws/invoke s3)
        :Contents)))

Calling get-k6-summaries will return a sequence of all the generated results. Those individual results may be helpful for some analysis, but you should aggregate as many metrics as can be combined.



(defn metric-type [m]
  (condp = (set (keys m))
    #{"count" "rate"} :counter
    #{"min" "max" "p(90)" "p(95)" "med" "avg"} :trend
    #{"fails" "value" "passes"} :rate
    #{"min" "max" "value"} :gauge
    :unknown))

(defn merge-summaries
  "summaries is a vector of k6 output summaries converted from JSON to EDN.
   get-k6-summaries returns this structure precisely. This function works with the various types
   of k6 metrics (https://k6.io/docs/using-k6/metrics) and does the following:
    - min values are the minimum in all the maps
    - max values are the maximum in all the maps
    - avg values are the average in all the maps
    - count is the sum in all the maps
    - rate is the average in all the maps
    - fails is the sum in all the maps
    - passes is the sum in all the maps
   p, med, & 'value' values are not included in the merged summary, but they can still be viewed
   for individual machines by inspecting the output from get-k6-summary"
  [summaries]
  (letfn [(merge-metrics [v1 v2]
            (condp = (metric-type v2)
              :counter {"count" (+ (get v1 "count") (get v2 "count"))
                        "rate" (double (/ (+ (get v1 "rate") (get v2 "rate")) 2))}
              :trend {"min" (min (get v1 "min") (get v2 "min"))
                      "max" (max (get v1 "max") (get v2 "max"))
                      "avg" (double (/ (+ (get v1 "avg") (get v2 "avg")) 2))}
              :rate {"fails" (+ (get v1 "fails") (get v2 "fails"))
                     "passes" (+ (get v1 "passes") (get v2 "passes"))}
              :gauge {"min" (min (get v1 "min") (get v2 "min"))
                      "max" (max (get v1 "max") (get v2 "max"))}
              {}))
          (merge-checks [v1 v2]
            {"passes" (+ (get v1 "passes") (get v2 "passes"))
             "fails" (+ (get v1 "fails") (get v2 "fails"))})
          (max-vus [m]
            (get-in m ["metrics" "vus_max" "max"]))]
    {:metrics (apply
               merge-with
               merge-metrics
               (map #(get % "metrics")
                    summaries))
     :checks (apply
              merge-with
              merge-checks
              (map
               #(get-in % ["root_group" "checks"])
               summaries))
     :max-vus (->> summaries
                   (map max-vus)
                   (apply +))}))

Call merge-summaries to get aggregate results.

Tradeoffs

A paid k6 plan will provide you with a friendlier interface and better result-sets. But is $25k/year worth that amount of money? I don't know. That's up to each user.

If you're willing to manage some ec2 instances yourself and work with less information in aggregate, then perhaps this approach is better.

One problem I didn't go into any detail about is managing failed EC2 instances. Certain failure states require an external TerminateInstances operation: management of 'dangling' ec2 instances is now your problem.

k6 cloud will also distribute traffic across regions. I think it will also simulate clients with slower bandwidth. Could you provide all of this if you need it?

I believe you could get close to k6's paid feature-set, but only with a lot of effort. Hopefully, I have given you a starting point.

Don't forget to have fun!

Generating docs from tests

Daniel Fitzpatrick — Mon, 04 Apr 2022 17:57:57 +0000

This post is about Rivendell, my general-purpose library for the elvish shell.

I haven't kept up maintenance of Rivendell because elvish is changing rapidly. So, I took the first steps toward fixing that last week.

Testing seemed an excellent place to start because it would tell me which parts are currently broken and would make maintenance more manageable in the future. Also, I want to document it: a single image on the README isn't helpful to new users.

What if I could tackle both of these objectives at once by generating the docs from the tests?

That appears to be an uncommon practice. A google search reveals a smattering of tools offering this feature¹, but I've never done professional documentation using this method. So sound off if you are quietly generating docs from tests.

It's a practical approach for Rivendell, however. Tests and documentation appear to be mutually complementary, and the test runner I had previously written is suited to this.

Documentation shows users how to use the software, and tests incidentally cover use-cases by testing invariants.
Tests should make the documentation more comprehensive with edge-case coverage, something I tend to overlook when writing docs.
The documentation should help explain the tests. Detailed explanations are another thing I tend to overlook when writing tests.

The result

I think it looks pretty good.

Elvish continues to impress. Armed with documentation-generating tests, I now feel prepared to forge ahead by updating the rest of the project.

How does it work?

Reading the documentation for test.elv is like reading the script for a particular Christopher Nolan movie. But doing that won't reveal the stage crew behind the camera.

At the highest level, I have split testing into three parts.

Assertions
Test runner
Reporters

Assertions take a predicate and return a function that takes a test.² They run the test and check the result against the predicate, then produce a map.

The test runner chains these together, one after the other, and spits out the result. At the terminal, this looks like unreadable garbage.

Last, a few different reporters turn that garbage into readable text. One shows statistics, another describes failing tests, and another gives general pass/fail information about every test. Choose the one most suitable.

although not shown, the error reporter can show more info

There is another reporter which generates markdown. This process works because I designed the test runner to take a structured list.³ Most strings are emitted verbatim, and you can include whatever markdown syntax you want. It transforms specially-located text into headers and subheaders.

I am re-writing the code using org-mode, so head over to the project if I have glossed over something and suggest improvements. (=

lread/test-doc-blocks is my favorite, and is used in honeysql. Although it takes a documentation-first approach, whereas I will take a test-first approach. ↩
There are plans for different kinds of assertions. ↩
The doc fills in the details about the structure required. ↩

Reloaded workflow with nbb & expressjs

Daniel Fitzpatrick — Mon, 28 Mar 2022 16:19:28 +0000

nbb (whatever the 'n' represents 😄 ) is a fascinating project. It brings the power of babashka to nodejs. I spent this week playing with it and want to share what I've found. It's pretty cool!

tl;dr

You can view the code for this blog post here. Supply this as a dependency using clj -Spath and nbb --claspath.

Hello, world

A strong learning experience for this project consists of a "hello world" web server, command-line argument parsing, and state management to simulate a database.

Along the way, I should learn something about dependency management and tooling.

Tooling

I almost can't believe it, but nbb can start a nrepl server. It's a bit fussy (not all clojure-mode commands operate correctly in Emacs, for example), but it works.

To launch a nrepl server, run nbb nrepl-server.

Then, in Spacemacs open a .cljs file. Then SPC m i (sesman-start) and connect to localhost: with cider-connect-clj. This operation will connect you to the nrepl-server with a sweet 2-dimensional buffer.

There are a handful of things that don't currently work (like cider-switch-to-repl-buffer)¹, but you can switch to it with SPC b b (list-buffers).

So far, nbb's nrepl-server has blown me away with its polish at this early stage of development.

Parsing command-line arguments with `yargs`.

I began with yargs, and while it functioned, yargs was not ideal.

yargs complects arguments with commands/options.

The following code illustrates how you cannot describe commands and options without first providing user arguments.

(-> argv # argv should be unecessary
    yargs
    (.command ...)
    (.options ...)

yargs kills the process after handling --help

This behavior is not ideal because it makes testing difficult at the repl. I should be able to craft help instructions without starting a new process.

Luckily, borkdude packaged tools.cli with v0.3.0 of nbb. Of course, if you need to use subcommands, yargs might still be a better option, but I'm going with tools.cli for now.

Parsing command-line arguments with `tools.cli`.

tools.cli works the same as in Clojure. Feel free to skip this section if you are already familiar with tools.cli.

The application's entry-point is a 'main' function to which command-line arguments are passed as varargs. nbb also stuffs arguments into a seq called *command-line-args*.

First, create a hello_world.cljs file, then paste the following code.

(ns hello-world
  (:require [clojure.tools.cli :as cli]))

(def default-port 3000)

(def cli-options
  [["-p" "--port PORT" "Port number"
    :default default-port
    :parse-fn js/Number
    :validate [#(< 1024 % 0x10000) "Must be a number between 1024 and 65536"]]
   ["-h" "--help"]])

(defn handle-args [args] (println args))

(defn main
  [& args]
  (handle-args
   (cli/parse-opts
    args cli-options)))

Try this at the repl to see how tools.cli works.

hello-world> (main)
{:options {:port 3000}, :arguments [], :summary   -p, --port PORT  3000  Port number
  -h, --help, :errors nil}

hello-world> (main "--port" "9093")
{:options {:port 9093}, :arguments [], :summary   -p, --port PORT  3000  Port number
  -h, --help, :errors nil}

hello-world> (main "--help")
{:options {:port 3000, :help true}, :arguments [], :summary   -p, --port PORT  3000  Port number
  -h, --help, :errors nil}

hello-world> (main "--port" "foobar")
{:options {:port 3000}, :arguments [], :summary   -p, --port PORT  3000  Port number
  -h, --help, :errors [Failed to validate "--port foobar": Must be a number between 1024 and 65536]}

cli/parse-opts generates a map containing all the components we need for handling command-line arguments.²

:options: The parameters the application will use
:summary: A formatted string we can print for help docs
:errors: Any validation errors. You can see our custom error message here.

Let's change the definition of handle-args to do something useful.

(defn start-app [{:keys [port]}]
  (println "starting server on port" port))

(defn print-help [summary]
  (println "hello world server")
  (println summary))

(defn print-errors
  [{:keys [errors summary]}]
  (doseq [e errors]
    (println e))
  (print-help summary))

(defn handle-args
  [{:keys [options summary errors] :as args}]
  (cond
    (seq errors) (print-errors args)
    (:help options) (print-help summary)
    :else (start-app options)))

Feel free to run the same things from the repl again. You should see formatted text no matter what you pass in.

Running from the terminal

This next task admittedly gave me some trouble, but three discoveries helped immensely.³

A --main <ns>/<fn> parameter can be supplied to nbb the command line.
You shouldn't pass the script as an argument. Instead, ensure it's in the classpath with --classpath <dir1:dir2:...>.
nbb automatically includes the current directory in the classpath.

#2 is particularly notable because you could add all your scripts to a central directory, include that directory by default in your shell init, and run your scripts without specifying their name or filesystem location.

Feel free to do that, but the rest of this article will assume you're executing from the directory where you saved hello_world.cljs.

$ nbb --main hello-world/main --help
hello world server
  -p, --port PORT  3000  Port number
  -h, --help

$ nbb --main hello-world/main
starting server on port 3000

$ nbb --main hello-world/main --port 9093
starting server on port 9093

$ nbb --main hello-world/main --port foobar
Failed to validate "--port foobar": Must be a number between 1024 and 65536

expressjs

The installation process for expressjs is mundane if you are familiar with nodejs. First, run npm install express to get expressjs. Then, alter the namespace form to make it available to our project.

(ns hello-world
  (:require [clojure.tools.cli :as cli]
            ["express$default" :as express]))

You can start a server with the following code, but don't do that just yet. We need to take a brief detour.⁴

(.listen
  (doto (express)
        (.get "/" (fn [_ res] 
                    (.send "hello, world"))))
  default-port)

The reloaded workflow

If you are not familiar with the Clojure ecosystem, there is an idea made trendy by Stuart Sierra called "the reloaded workflow." Most large Clojure applications use it, and there are many libraries from which to select.

The basic idea is that it provides a way to quickly stop and start stateful resources without halting the main process. It's a necessity for a killer repl experience.⁵

After reviewing the options, I settled on weavejester/integrant because it's small - only one dependency and two source files in total.

Integrant isn't suitable for nbb in its current state, so I eliminated a couple of features, and now it works fine. View the GitHub project @crinklywrappr/integrant.

The shortlist of cut features:

EDN configuration
spec validation

It's npm for node dependencies and clj for Clojure dependencies.

$ classpath="$(clj -A:nbb -Spath -Sdeps '{:aliases {:nbb {:replace-deps {com.github.crinklywrappr/integrant {:git/tag "v1.0.3" :git/sha "8462388"}}}}}')"

$ nbb --classpath $classpath nrepl-server

Using Integrant with expressjs

First, let's define our handler.

(defn hello-world [count]
  (fn [_ res]
    (swap! count inc)
    (.send res (str "Hello, World! (count: " @count ")"))))

We will use count to simulate a database. We will count how many requests users have made to the server and restart the count at 0 whenever we start the server.⁶

The best place to start with Integrant is with a config map.

(ns hello-world
  (:require [integrant.core :as ig]
            ["express$default" :as express]
            [clojure.tools.cli :as cli]))

(def config
  {:express/server {:port default-port :app (ig/ref :express/app)}
   :express/app {:handler hello-world :count (ig/ref ::count)}
   ::count {:start 0}})

This config map is as simple as it looks. Each key-value pair refers to the configuration of a future stateful component. You specify dependencies with the (ig/ref <qualified-key>) function.

Next, we tell Integrant how to start everything up. This process is accomplished semi-declaratively with the ig/init-key multimethod. The first parameter is the key corresponding to the component, and the second parameter is a map of that component's config, replaced with all initialized dependencies.

(defmethod ig/init-key :express/app [_ {:keys [handler count]}]
  (doto (express)
    (.get "/" (handler count))))

(defmethod ig/init-key :express/server [_ {:keys [port app]}]
  (.listen app port))

(defmethod ig/init-key ::count [_ {:keys [start]}]
  (atom start))

Only the server needs to be closed. We can specify how to do that with the ig/halt-key! multimethod. Again, we are only interested in the second parameter, which is the server object. This function should be idempotent.

(defmethod ig/halt-key! :express/server [_ server]
  (when (and (some? server) (.-listening server))
    (.close server)))

Feel free to test this at the repl.

hello-world> (def system (ig/init config))

; now visit localhost:3000/ and refresh a few times

hello-world> (ig/halt! system)

If you found this section confusing, Let me encourage you to inspect system or peruse the 'canonical' Integrant README. Doing so will be very enlightening if you feel that I have glossed over some details.

Putting it all together

We will define a couple of start/stop functions to simplify the process of bringing the system up and down.

(def system (atom nil))

(defn start
  "system is an atom"
  ([] (start config))
  ([config] (start config system))
  ([config system] (reset! system (ig/init config))))

(defn stop
  "system is an atom"
  ([] (stop system))
  ([system]
   (when (map? @system)
     (swap! system ig/halt!))))

Finally, re-define start-app to call start with the (possibly) user-modified config.

(defn start-app [{:keys [port]}]
  (-> config
      (assoc-in [:express/server :port] port)
      start))

Congratulations! You now have a script suitable for command-line consumption and repl development.

hello-world> (start) ; or eg (start-app {:port 9093})
hello-world> (stop)

$ nbb --classpath $classpath --main hello-world/main --port 9093

Going one step further

You may notice that ctrl+c is required to stop the server from the command line. That's fine, but what if expressjs doesn't clean up after itself properly?

Maybe it does already: I'm no expert. But what if you switch to a different server that doesn't? It might be good to hook our stop function up to SIGINT.

(defn exit
  [& _]
  (stop)
  (.exit js/process 0))

(.on js/process "SIGINT" exit)

Happy hacking!

Closing thoughts about `nbb`

During this process, the only 'bug' I encountered was that I couldn't specify the request handler using partial, e.g. (partial hello-world count). To make it work, I returned a closure from hello-world. I'm not sure if this is an nbb problem or an expressjs problem.

I love nbb. Maybe even more than bb 😉. The biggest issue is the ergonomics around specifying Clojure dependencies and that it can't currently read jars. But I am hopeful both of those aspects will improve.

I don't think that will stop me from using it.

Emacs thinks it's a Clojure repl, but it's connected to an nbb server - we've confused it a bit. ↩
arguments isn't essential for us right now, but if you run (main "foobar"), you can see it in action. ↩
I later discovered the new clj build tool also does this. ↩
Most expressjs "hello, world" tutorials would stop here. ↩
In my experience, "Clojure" will automatically restart altered components on eval (and their dependents). I'm not sure which tool provides this feature (Cider, nrepl, something else...), and I didn't tempt fate to determine if that works with this approach. 😁 ↩
Using an actual database like SQLite would be a good learning step to do next. ↩

Zig (notes)

Daniel Fitzpatrick — Mon, 21 Mar 2022 16:16:10 +0000

NOTE: The numbered bullet points are written correctly in the markdown, but dev.to's markdown renderer overwrites them. Sorry about the inconvenience.

This is a strange entry.

I was fascinated by Andrew Kelley’s interview on the Corecursive podcast and wanted to learn more about Zig. There isn’t much material (yet) because it’s a young project, and short of reading the beefy language doc, I decided to go with the primer at ziglearn.org.

This blog post isn’t well organized or edited - it is a humble dump of notes, thoughts, and questions I had while reading the material. It is (mostly) divided into chapters that follow the source material.

This part is important. Andrew stated in the interview that Rust users like to attack Zig. I write Clojure and have no idea what the politics in either camp are like. Many of the bullet points in these notes are geared toward discovering where the footguns are hidden, but that shouldn’t be mistaken as an agenda or part of some campaign to throw shade at Zig.

I am honest and vulnerable with what are likely many stupid questions and conclusions.

Basics

Are type assignment bugs discovered at runtime or compile time? I’m especially interested in branching expressions in the rvalue of type assignments.
Where does allocation take place? Some general guidelines would be helpful for reasoning about some code.
Why does this code need a return statement?

test "returning an error" {
    failingFunction() catch |err| {
        try expect(err == error.Oops);
        return;
    };
}

Is there some way to query this?

Error unions returned from a function can have their error sets inferred by not having an explicit error set. This inferred error set contains all possible errors which the function may return.

fn createFile() !void {
    return error.AccessDenied;
}

Are ignored error unions caught at compile time or runtime?
Why can’t cases fall through to other branches? Philosophical or technical reasoning?
Any plans for something like D’s immutable type qualifier? While occasionally useful, const is too nuanced to not get wrong once in a while.

Type Qualifiers

Exhaustive switch - does it also check for exclusivity?
For the above, do these checks happen at compile time?

NOTE TO SELF: functions which take or return arrays, slices, slices known at compile time, and pointers may have different semantics. Pass by value/reference may muddy the waters even further. Need to investigate.

I am sensing a pattern that pointers are going to be behind most footguns. Are pointers prevalent or idiomatic in zig (outside of c interop?)

Interesting how enum member functions are both static and virtual. I can’t imagine this ever being useful but I can imagine wasting time to look this up when debugging.
I’m guessing that Union types and structs can’t be used inside of arrays unless it’s an array of pointers. Union types can be used inside of structs though.
Tagged unions essentially give us Rust-style enums. Cool.
Can I get a pointer to fn parameter and modify the parameters that way? Hopefully not…
How does the conditional work in the following code if there are no truthy values? eventuallyNullSequence() returns ?u32.

test "while null capture" {
    var sum: u32 = 0;
    while (eventuallyNullSequence()) |value| {
        sum += value;
    }
    try expect(sum == 6); // 3 + 2 + 1
}

If optionals don’t take extra memory and they use a pointer to memory address 0 then I’m guessing optionals are always heap allocated?
Are labels common in idiomatic zig? I find them difficult to read.
This is super cool and useful in conjunction with Union types and numeric widening const b: if (a < 10) f32 else i32 = 5; Can it be used even if the condition is unknown at compile time?
Something about this feels too much like the C preprocessor and “what color is your function”. Am I okay with it? Unsure.

fn Matrix(
    comptime T: type,
    comptime width: comptime_int,
    comptime height: comptime_int,
) type {
    return [height][width]T;
}

The amount of reflection, even at compile time, is very cool.
~~Shouldn’t this be try expect(values.len == 10);~~ Nevermind - PEMDAS mistake on my part.

test "tuple" {
    const values = .{
        @as(u32, 1234),
        @as(f64, 12.34),
        true,
        "hi",
    } ++ .{false} ** 2;
    try expect(values[0] == 1234);
    try expect(values[4] == false);
    inline for (values) |v, i| {
        if (i != 2) continue;
        try expect(v);
    }
    try expect(values.len == 6);
    try expect(values.@"3"[0] == 'h');
}

Sentinel terminated arrays and slices appear to exist primarily for c interop.
Vector types are super cool.

Standard patterns

createFile allocates resources but makeDir doesn’t. Why the inconsistency?
Many deallocations appear to be missing from this code. In a normal function, would they be automatically deallocated when they go out of scope or does this result in a memory leak or illegal behavior?

test "make dir" {
    try std.fs.cwd().makeDir("test-tmp");
    const dir = try std.fs.cwd().openDir(
        "test-tmp",
        .{ .iterate = true },
    );
    defer {
        std.fs.cwd().deleteTree("test-tmp") catch unreachable;
    }

    _ = try dir.createFile("x", .{});
    _ = try dir.createFile("y", .{});
    _ = try dir.createFile("z", .{});

    var file_count: usize = 0;
    var iter = dir.iterate();
    while (try iter.next()) |entry| {
        if (entry.kind == .File) file_count += 1;
    }

    try expect(file_count == 3);
}

The formatting examples show a format fn defined in a struct. Is this common or idiomatic? I mean… is there anything like Java’s toString or Rust’s debug trait that’s ubiquitous. EDIT: json stringify looks promising.
The first example in the json subsection parses a map without an allocator which appears to invalidate this statement. which is true?

The json parser requires an allocator for javascript’s string, array, and map types. This memory may be freed using [std.json.parseFree](https://ziglang.org/documentation/master/std/#std;json.parseFree).

Iterator return types seem well thought-out.

This chapter is missing a lot.

Any plans for atomic operations on shared mutable state (eg atoms in Clojure)?

Working with C

I didn’t read through this section but I wonder if you could integrate Guile scheme with no issues. That sounds like an effective heat check.

Async

I skimmed this section and my initial thoughts are

Coroutines are awesome but ridiculously hard to develop
I’m getting some “what color is your function?” vibes.
I’ll bet this stack manip upsets the guile scheme integration idea.
Maybe futures/promises would have been sufficient.

Closing thoughts

Overall I still have too many questions and feel a bit uneasy about using it.
Somewhere I read that Zig avoids truthy/falsy values but that appears to be false.
I suspect most of the footguns hide in the ability to dereference any value passed to a function.
There are probably a fair number of footguns w/ arrays & slices passed to and returned from functions.
const is encouraged but too nuanced to not screw up once in a while.
While impressive, coroutines might not be the right approach to async.
I’m surprised it didn’t go over closures and higher-level functions.
While looking around for benchmarks, a hn post directed me to these 2 tickets. One of Zig’s biggest promises has asterisks. Quick gut check for anyone reading: what’s your prediction for these 2 tickets?

https://github.com/ziglang/zig/issues/1966

https://github.com/ziglang/zig/issues/2301

Saving the UNION with HoneySQL

Daniel Fitzpatrick — Mon, 14 Mar 2022 20:25:43 +0000

SQL might be an unparalleled success story of narcissistic design, but it's a robust apparatus. Consequently, you've probably brushed against some rough edges.

The problem

Have you ever noticed that UNION breaks DRY principles?

SELECT x.foo, x.bar, '' as baz FROM x
UNION
SELECT '' as foo, '' as bar, y.baz FROM y;

The code above requires more references to foo, bar, and baz than necessary. Adding additional queries to the union is an exercise in masochism because this unnecessary code duplication increases exponentially.

If each query introduces just a single column, and you have 5 of them, that's 20 times need to repeat yourself! We can use HoneySQL to put this unruly child in timeout. ¹

TL;DR

If you just want the code and don't want to follow along, I have uploaded a gist which includes a solution for the problem mentioned at the end of this post.

Spread some honey...sql

Translating the previous query into HoneySQL makes a good beginning.

(require '[honeysql.helpers :as hh]
         '[honeysql.core :as hsql])

{:union
 [(-> (hh/select :foo :bar ["" :baz])
      (hh/from :x))
  (-> (hh/select ["" :foo] ["" :bar] :baz)
      (hh/from :y))]}

;=>
{:union
 [{:select (:foo :bar ["" :baz]), :from (:x)}
  {:select (["" :foo] ["" :bar] :baz), :from (:y)}]}

Where is the SQL? You can view it from the repl with (hsql/format *1).

HoneySQL displayed its first big win in the previous output. Did you spot it? It emits regular Clojure data structures, which we can easily manipulate with the standard library. It also allows us to describe our desired output succinctly.

{:union [{:select [:foo]}
         {:select [:bar]}]}

;=>
{:union [{:select [:foo [nil :bar]]}
         {:select [[nil :foo] :bar]}]}

As in the example above, we will supply nil as the value for "back-filled" columns.

Columns in HoneySQL will commonly look like one of three patterns.

:column
:table.column
[:processing-stuff :column]

A function that parses out the column name we need for back-filling would look like this.

(defn get-column-name [col]
  (cond
    (vector? col) (second col)
    (and (keyword? col)
         (str/includes? (name col) ".")) (-> col name
                                             (str/split #"\.")
                                             second keyword)
    (keyword? col) col))

A quick test proves that this works.

(get-column-name :baz)
;=> :baz
(get-column-name :bar.baz)
;=>:baz
(get-column-name [:foo.bar :baz])
;=>:baz

Next, we should contrive some way to get an ordered collection of all our column names. That will simplify everything else. Ironically this will require another set theory UNION, but we want to preserve element order.

(defn ordered-union
  [cmp coll1 coll2]
  (loop [xs [] coll1 coll1 coll2 coll2]
    (cond
      (and (empty? coll1) (empty? coll2)) xs
      (empty? coll1) (vec (concat xs coll2))
      (empty? coll2) (vec (concat xs coll1))
      (cmp (first coll1) (first coll2)) (recur (conj xs (first coll2))
                                               (rest coll1) (rest coll2))
      :else (recur (conj xs (first coll1)) (rest coll1) coll2))))

Has it been a while since you last saw loop/recur?

The function above takes a comparator cmp and two collections. It assumes that coll1 "comes before" coll2 - an assumption made visible in the last branch. The first three branches are obvious exit conditions. The fourth branch ensures that collisions are not duplicated in the resultset, thus enforcing the set theory aspect of our function.

We can employ the usual tricks to support more than two inputs. In this case, it's just a reduction over the inputs. Because ordered-union is generally helpful outside of the scope of this article, feel free to add a [cmp coll1 coll2 & colls] arity and repurpose.

We will apply this trick to get our complete ordered list of columns.

(defn get-all-columns-in-union [m]
  (reduce
   (fn [a b]
     (ordered-union = a (map get-column-name b)))
   [] (map :select (:union m))))

We now have something worth testing.

(get-all-columns-in-union
 {:union [{:select [["first" :key] :a :b :c]}
          {:select [["second" :key] :d :e :f]}
          {:select [["third" :key] :g :h :i]}]})
;=> [:key :a :b :c :d :e :f :g :h :i]

A crucial insight is that we can use ordered-union to rewrite the SELECT clauses. Check this out.

(defn format-nil-column [column] [nil column])

(ordered-union
 #(= (get-column-name %1) (get-column-name %2))
 (mapv format-nil-column [:key :a :b :c :d :e :f :g :h :i])
 [["first" :key] :a :b :c])
;=> [["first" :key] :a :b :c [nil :d] [nil :e] [nil :f] [nil :g] [nil :h] [nil :i]]

(ordered-union
 #(= (get-column-name %1) (get-column-name %2))
 (mapv format-nil-column [:key :a :b :c :d :e :f :g :h :i])
 [["second" :key] :d :e :f])
;=> [["second" :key] [nil :a] [nil :b] [nil :c] :d :e :f [nil :g] [nil :h] [nil :i]]

(ordered-union
 #(= (get-column-name %1) (get-column-name %2))
 (mapv format-nil-column [:key :a :b :c :d :e :f :g :h :i])
 [["third" :key] :g :h :i])
;=> [["third" :key] [nil :a] [nil :b] [nil :c] [nil :d] [nil :e] [nil :f] :g :h :i]

I plugged our previous three SELECT statements into ordered-union, and it returned what we wanted in each case.

Put it all together

We are now in the red zone. So, finally, let's wrap things up.

(defn add-columns [nil-columns provided-columns]
  (ordered-union
   #(= (get-column-name %1) (get-column-name %2))
   nil-columns provided-columns))

(defn normalize-union
  ([m]
   (let [cols (get-all-columns-in-union m)]
     (update m :union (partial normalize-union cols))))
  ([columns queries]
   (let [nil-columns (mapv format-nil-column columns)]
     (mapv #(update % :select (partial add-columns nil-columns)) queries))))

To test

(normalize-union
 {:union [{:select [["first" :key] :a :b :c]}
          {:select [["second" :key] :d :e :f]}
          {:select [["third" :key] :g :h :i]}]})
{:union
 [{:select
   [["first" :key]
    :a
    :b
    :c
    [nil :d]
    [nil :e]
    [nil :f]
    [nil :g]
    [nil :h]
    [nil :i]]}
  {:select
   [["second" :key]
    [nil :a]
    [nil :b]
    [nil :c]
    :d
    :e
    :f
    [nil :g]
    [nil :h]
    [nil :i]]}
  {:select
   [["third" :key]
    [nil :a]
    [nil :b]
    [nil :c]
    [nil :d]
    [nil :e]
    [nil :f]
    :g
    :h
    :i]}]}

The devil in the details

It's time for me to confess a problem that I have been hesitant to mention.

Types.

Postgres will treat nil as type TEXT for a UNION with more than two queries, and it forces us to care about this because column foo must have a matching type across every row in the resultset.

I will offer some clues about resolving this but won't go into an in-depth solution. The solution is less work than you probably think. It would be best to start by allowing normalize-union to accept an optional type-map. It should look like this.

{:thing1 :integer
 :thing2 :uuid
 :thing3 :timestamp}

If you pass that map into format-nil-column, then you can leverage honeysql.core/call to force nil to be the type you want. Something like this should do the trick.

(honeysql.core/call :cast nil (get type-map column))

Do it right, and you might even be able to provide limited support for wildcards. I hope this has been a fun trip down the dark side of SQL.

I was reluctant to embrace HoneySQL when someone first introduced me, but its core feature has won me over with time: eschewing cleverness. For example, other ORM users talk about "fighting the query generator" to win their desired performance target - a story foreign to my user experience. ↩

DEV Community: Daniel Fitzpatrick

What should I know about software engineering?

Do you think like a programmer?

Make programming fun again

Make it a hobby first

Work on coding problems

Invent a passion project

Contribute to an open-source project

Temper your expectations

Sidebar: You probably won't become a game developer

Learn to love boring

Here be dragons

Will you enjoy it?

Having fun with money

crinklywrappr / pocketsmith-api

small wrapper around the pocketsmith api for clojure developers

com.github.crinklywrappr/pocketsmith-api

Coordinates

Usage

Acorns...?

Project setup

Sanity checks

Am I looking at the correct accounts? ✓

Am I requesting the correct transactions? ✓

Am I receiving the correct transactions? ✓

Does the math make sense? ✓

Additional checks

Conclusion

The hard way or the easy way?

The background

Doing something!

How we did it

Outcome

Build a unique résumé

DISCLAIMER

TL;DR

Choosing a template

Transferring content

Cleaning up

Theme it!

Fixing the sidebar

Improve "skimmability."

What to emphasize?

Final result

[Ann] Rivendell updated

About

How to install

Documentation in the terminal

Flashcards in the terminal

Rivendell update

Generating docs from tests

Daniel Fitzpatrick ・ Apr 4 ・ 3 min read

Setup instructions

Needed functions

Running the flashcard "app"

Conclusion

Using scittle to solve wordle

High-level overview (tl;dr)

The algorithm

The good

The bad

The ugly

Using EC2 to enhance k6 load tests

Too many templates

Orchestration

Gathering test results

Tradeoffs

Generating docs from tests

The result

How does it work?

Reloaded workflow with nbb & expressjs

tl;dr

Hello, world

Tooling

Parsing command-line arguments with yargs.

Parsing command-line arguments with tools.cli.

Running from the terminal

expressjs

The reloaded workflow

Using Integrant with expressjs

Parsing command-line arguments with `yargs`.

Parsing command-line arguments with `tools.cli`.

Closing thoughts about `nbb`