DEV Community: Brett Martin

Advent of Code (AoC) Day One

Brett Martin — Wed, 01 Dec 2021 22:09:35 +0000

Intro

Hello all! This years Advent of Code has begun! If you are unfamiliar with this, every day from December 1st through the 25th, a new programming puzzle with two parts is posted. The prompts are language agnostic so you can use any language you wish. I enjoy using languages that I am learning with - it gets me away from tutorial learning and into problem solving!

This is the seventh year puzzles, if you want to check out previous years take a look at: 2015, 2016, 2017, 2018, 2019, 2020

For more info check about the page and creator check out the about page.

Also, check out the subreddit r/adventofcode to find solutions for previous years, see cool visualizations of problems being solved, and to see the mega thread where people post their solutions for the day if you need inspirations / are stuck.

This series will be my solutions to AoC, and for a TLDR I will post the full solution towards the start of each post, but then I will go into way too much explanation about the problem, my thoughts on how to solve it, and how I solved it.

Day One Solution

module Solutions.DayOne
  ( d1p1,
    d1p2,
  )
where

d1p1 :: [Int] -> IO ()
d1p1 depths = do
  print $ countLarger depths 0

d1p2 :: [Int] -> IO ()
d1p2 depths = do
  let depthGroups = mkDepthGroups depths
  print $ countLarger depthGroups 0

countLarger :: [Int] -> Int -> Int
countLarger depths@(x : y : _) count
  | x < y = countLarger (tail depths) (count + 1)
  | otherwise = countLarger (tail depths) count
countLarger _ count = count

mkDepthGroups :: [Int] -> [Int]
mkDepthGroups depths@(x : y : z : _) = x + y + z : mkDepthGroups (tail depths)
mkDepthGroups _ = []

Above is the full solution to day one, parts one and two: d1p1 is day one part one, d2p2 is day two part two. My future posts will follow a similar format.

The thing that isn't visible in the solution is some of the helper functions that I write to make to make my life easier. Basically, a lot of these puzzles require the programmer to read / parse text files, so most of my helpers focus around making that quicker for me.

In this case, I was given a text file that had 2000 lines of integers, one integer per line. Here is the truncated file, the first 10 lines:

To avoid putting a bunch of helper functions as the month goes on (I will put some this time to give an idea), I am making the choice to only show the code required to solve the problem. I will mention the steps I took to prep the data in the explanation

Explanation

Scenario

We are going deep into the ocean in a submarine. As soon as we are submerged, our helpful submarine starts to perform a sonar sweep of the ocean floor! From this, we get a report of the sonar sweep (our input), where each line is a measurement of the ocean floor depth, and the scan goes out from the submarine: the first line is below the submarine, the second a little further from the submarine, the third even further ... continuing until the furthest from the submarine being the last line of the report.

We are then given the following sample report:

Tip: Even though you can get the full puzzle input before solving, try writing a solution that solves the smaller sample given in the scenario. If you run into issues, its much easier to desk check the code when you know the expected solution!

Part One Objective

After given this information, we are told our objective: figure out how quickly the depth increases

We are also told how to determine this info: count the number of times a depth measurement increases from the previous measurement

So given our small input above, we can manually count how many times this occurs. For example 199 -> 200 is an increase, so that is 1. 200->208, 208->210 are also increases, so we are at 3. 210->200 is a decrease so we still have a count of 3, but then 200->207 is another increase so we now have a count of 4. If you do this for all of the remaining depths we end up with a total of 7 increases, which would be the answer to part one if this was your input!

Solving Part One

So, now that we know what the objective is as well as the solution expected with the sample input, we can write out the steps for how to solve. This is how I think of the flow, assuming the input is in a .txt file:

Load contents of text file as a String.
Split the contents of the file into a list, with each element of the list representing one line.
Since we are doing a mathematical comparisons of each line, convert that list of strings into a list of ints.
Go through the list comparing each element with the one succeeding it to get a total count of increases.

Steps 1-3 are done with helper functions in this case:

readFileByLines :: FilePath -> IO [String]
readFileByLines f = do
  fileExists <- doesFileExist f

  if fileExists
    then do
      content <- readFile f
      pure $ lines content
    else error $ "File Does Not Exist: " ++ f ++ " does not exist!"

readFileOfInts :: FilePath -> IO [Int]
readFileOfInts f = do
  contents <- readFileByLines f
  pure $ map read contents

Basically I just write something like readFileOfInts "input/day-1.txt" and I get my list of ints, as shown in the function signature. That takes care of the first 3 steps!

Step 4 is the core of the problem, and from the solution posted at the start, it is:

d1p1 :: [Int] -> IO ()
d1p1 depths = do
  print $ countLarger depths 0

d1p1 uses the list of ints (depths) and passes it to another function called countLarger that takes the depths and an initial count, which is 0.

The function signature for countLarger is [Int] -> Int -> Int and it is the core of the logic:

countLarger :: [Int] -> Int -> Int
countLarger depths@(x : y : _) count
  | x < y = countLarger (tail depths) (count + 1)
  | otherwise = countLarger (tail depths) count
countLarger _ count = count

I want to look at the two definitions provided for countLarger to illustrate their purpose.

`countLarger` First Definition

countLarger depths@(x : y : _) count
  | x < y = countLarger (tail depths) (count + 1)
  | otherwise = countLarger (tail depths) count

The first definition of countLarger makes use of pattern matching to extract out the first two elements of the list (x : y : _) as well as give a name to the entire list depths@(x: y: _) to refer to it as a whole later. This pattern matching ensures that we have at least 2 elements to compare, x and y. Once this pattern matches, we go into the guard to check if x < y. If it is, then we see that there was an increase and we recursively call countLarger again with all of the depths except for the first one, and an increased count by 1.

Otherwise, if x < y is not true then we hit the otherwise guard and recursively call countLarger with all of the depths except for the first one and no change in count.

The way we get the the depths without the first element is by using the built in function tail. If you have a list [1,2,3,4] and you invoke tail [1,2,3,4] you are returned the list [2,3,4]

Basically, as long as we had at least two elements we will always end up recursively calling countLarger with the tail of depths (all depths except x). The only difference is if we increase our count or not!

`countLarger` Second Definition

countLarger _ count = count

This second definition is what we call our base case. Ever recursive function needs at least one base case if you want it to terminate!

This can be read as if you didn't pattern match the first definition requiring at least two elements, we are done comparing so return the provided count.

`countLarger` examples

I am going to over explain the execution of countLarger. If you are comfortable with recursion you can skip this.

countLarger [] 0
0

If we call countLarger like above, we will fail to match the first definition, and end up matching the second definition. The second definition just returns the value provided for count, which is 0 when we invoke the function, so the result is 0

countLarger [10] 15
15

With this, we call countLarger with a list with a single element which means that it will fail to match the first definition again. Since we invoked the function with a count of 15, we end up with the result of 15 since the second definition just returns the value provided for count!

countLarger [10, 11] 0
countLarger [11] 1
1

If we pass at least 2 elements then we will match the first definition. In this case, since 10 < 11 we enter the first guard and call countLarger again with the tail and an increased count.

When we call countLarger the second time, it will fail to match the first definition and we end up at the base case, giving us the final result of 1 which was the last value provided for count.

countLarger [199, 200, 208, 210, 200, 207, 240, 269, 260, 263] 0
countLarger [200, 208, 210, 200, 207, 240, 269, 260, 263] 1
countLarger [208, 210, 200, 207, 240, 269, 260, 263] 2
countLarger [210, 200, 207, 240, 269, 260, 263] 3
countLarger [200, 207, 240, 269, 260, 263] 3
countLarger [207, 240, 269, 260, 263] 4
countLarger [240, 269, 260, 263] 5
countLarger [269, 260, 263] 6
countLarger [260, 263] 6
countLarger [263] 7
7

Above is the "full" example of how countLarger is being recursively called with the sample input. Each call has a list that is smaller by 1 element, and the count is incremented if x < y, until we hit the base case and return the final count.

Phew, that was a lot, hopefully it makes some sense though! If not, I can try and make a better diagram to illustrate the recursion.

Wrapping up Part One

So back to the function that is called for part one:

d1p1 :: [Int] -> IO ()
d1p1 depths = do
  print $ countLarger depths 0

We can see that we just call countLarger with a list of Int and set our initial count to 0. When it resolves, it will give us the total count and it is printed to the terminal!

Part Two Objective

Part two usually throws a wrench into things: it adds an extra requirement, or asks to interpret the data in a different way, or any number of other things.

In this case, we are asked to interpret the data differently:
Looking at every measurement is causing too much noise! What if we considered creating a three measurement sliding window, where we sum the measurements in each window and then compare the windows the same way we compared each measurement!

New goal: count the number of times the sum of measurements in this sliding window increases from the previous sum.

Solving Part Two

In this case, we are lucky that our solution to part one still applies to the solution to part two. We still want to compare two measurements, and update a count in the same way.

The only difference is we want to find the sums of windows and compare those measurements. If you look at the function that kicks off part two, the majority of the function should look very familiar!

d1p2 :: [Int] -> IO ()
d1p2 depths = do
  let depthGroups = mkDepthGroups depths
  print $ countLarger depthGroups 0

The only change is the addition of let depthGroups = mkDepthGroups depths that does something to the depths that we were provided. Lets take a look at its implementation:

mkDepthGroups :: [Int] -> [Int]
mkDepthGroups depths@(x : y : z : _) = x + y + z : mkDepthGroups (tail depths)
mkDepthGroups _ = []

First thing to note is the signature of [Int] -> [Int] which tells us that we are taking an int list and returning one as well.

We can also see that the definition of this function is also recursive, the first definition calls mkDepthGroups.

From that we can see the base case mkDepthGroups _ = [] which can be read as if we fail to pattern match on a list with three or more elements (x : y: z : _) then we return an empty list

The more interesting definition is:

mkDepthGroups depths@(x : y : z : _) = x + y + z : mkDepthGroups (tail depths)

This definition matches on three elements (x, y, z) and names the list as depths. We then sum those three elements, and start constructing a new list with the cons operator (:) with the list returned from mkDepthGroups (tail depths) which makes use of tail again! As soon as our list of depths has less than 3 elements to sum, we terminate with an empty list.

So we ultimately end up with a new list of ints that represent the sums of the sliding windows. This list is what we pass to our original countLarger function and we end up with our result!

Wrapping up Part Two

d1p2 :: [Int] -> IO ()
d1p2 depths = do
  let depthGroups = mkDepthGroups depths
  print $ countLarger depthGroups 0

Looking at the function one more time, hopefully you can see the flow. We take the original depths and pass it to mkDepthGroups to create a new list that represents our sliding window group sums. We then hand our new list off to countLarger to do our original operation.

I didn't go into as much detail into the recursion in the mkDepthGroups function, but I challenge you to map out all of the intermediate values / calls that would occur if you passed in the sample input list. To get you started:

mkDepthGroups [199, 200, 208, 210, 200, 207, 240, 269, 260, 263]
    -> 607 : mkDepthGroups [200, 208, 210, 200, 207, 240, 269, 260, 263]
        -> 607 : 618 : mkDepthGroups [208, 210, 200, 207, 240, 269, 260, 263]
            -> ...

Conclusion

I will be honest, this post is the epitome of over explaining. The core of this is pretty simple, I just wanted to talk through every part of the problem and solution. For that reason, please don't let the length of this post scare you from trying to solve these problems! Try doing this same one with a different language if Haskell is unfamiliar.

If you enjoyed the over explanation let me know and I will try to do some more as the event goes on.

JavaScript Bites: Closure

Brett Martin — Tue, 16 Nov 2021 00:18:54 +0000

TLDR: Closure is the concept of storing a function and its environment together. When you create a function, it stores the functions local environment and its outer environment together. If you are ever confused about what value will be present, understand what value existed when the function scope was created!

Formal Definition

If you were to look up what a closure is, Wikipedia's definition has this to say in the first two lines:

In programming languages, a closure, also lexical closure or function closure, is a technique for implementing lexically scoped name binding in a language with first-class functions. Operationally, a closure is a record storing a function together with an environment.

That is a bit of a dense definition, but its not a complex as it seems at first glance! This article aims to explain what this means, bit by bit, so you can use closures with confidence.

Scoping

I first want to touch on what scoping means in JavaScript. Before ES6, JavaScript only had Global Scope and Function Scope. You have probably seen how variables are accessible based on what scope they were declared in. Here is an annotated example:

// Variable declared at the global scope.
var globalVariable = 'Neat';

function func() {
  // Variable declared in function scope.
  var funcVar = 12;

  console.log(globalVariable);
}

console.log(funcVar);
func();

If you were to execute the above code, you would get a ReferenceError: funcVar is not defined error. If you remove the console.log(funcVar); line, the output would be Neat. The reason for this is that scopes can only reference variable declared in their own scope (local) and any outer scopes relative to the current scope. In this case, the scope in func() can access the outer scope (global) to get the value of globalVariable, however the global scope does not have access to the scope created for func() so it cannot access the funcVar variable. One more example to show how inner scopes can access values in outer scopes.

var globalVar = 'Hello';

function func() {
  var innerVar = 'World';

  function innerFunc() {
    var name = 'innerFunc';

    console.log(`${globalVar} ${innerVar}, from ${name}`);
  } 
  innerFunc();
}

func();

Executing the above will show Hello World, from innerFunc in the console. We can see that innerFunc() has access to its local scope, the scope of func() and the global scope.

Closure

The example above is actually a closure! It represents the second part of the Wikipedia definition, Operationally, a closure is a record storing a function together with an environment. In this case, the function is innerFunc() and the environment that is being stored is the local scope along with all of the outer scopes present at the time of function creation.

Thats it! If you have been writing functions, you have been creating closures this whole time!

Whats the Big Deal

The reason this can be a confusing topic is that closures can enable a handful of different patterns and ideas in JavaScript, even if they don't seem related at all. So here are some quick examples of things that are possible because of closures:

Access Data Through Interface

Say you wanted to create a simple counter with a variable representing the current count, and four functions: add, subtract, reset, show.

let count = 0;

const add = () => {
  count = count + 1;
};

const subtract = () => {
  count = count - 1;
};

const reset = () => {
  count = 0;
};

const show = () => {
  console.log('Count: ', count);
};

If you were to use these functions to add and show, like

add();
add();
add();
add();
show();

you would get Count: 4. The issue is that if I were to throw in count = 0; right before the show() it would show Count: 0! We are operating on a variable that any scope can access and modify, since it is global, and that is dangerous. Something can accidentally mess with count and cause a headache of a bug. This can be written in a different way:

const mkCounter = () => {
  let count = 0;

  const add = () => {
    count = count + 1;
  };

  const subtract = () => {
    count = count - 1;
  };

  const reset = () => {
    count = 0;
  };

  const show = () => {
    console.log('Count: ', count);
  };

  return {
    add,
    subtract,
    reset,
    show
  };
};

This code is very similar, but you can see that we have declared it inside of a new function called mkCounter that defined the count variable locally to its scope. At the end, we return an object that exposes the four functions but not the count variable, however since all of these functions are defined inside the mkCounter scope, the closing environment for all of them contain count! Here is how it would be used:

const counter1 = mkCounter();
const counter2 = mkCounter();

counter1.add();
counter1.add();
counter1.add();
counter1.subtract();

counter2.subtract();
counter2.subtract();

counter1.show();
counter2.show();
console.log(counter1.count);

which will give the output of:

Count: 2
Count: -2
undefined

Awesome, so not only can we not access the count as shown by the last line, each counter has its own count in their own environment to work with!

Partial Application

Edit: Updated this section thanks to @zaferberkun and @peerreynders in the comments!

Another closure example that I use all the time is partial application. A simple example could be formatting a log with some data that you don't want to set every time you invoke the function:

function logger(route, message, showDate) {
  const header = showDate ? `${new Date().toISOString()} | ${route}` : route;
  console.log(`${header} | ${message}`);
}

function mkLogger(route, showDate = false) {
  // Implement "partial application" with the values
  // in the closure
  return (message) => logger(route, message, showDate);
}

Then you can use the function like:

const docLogger = mkLogger('DOCS', true);

docLogger('This is my log message');
docLogger('Another log message');

with the output of:

2021-11-15T23:55:26.672Z | DOCS | This is my log message 
2021-11-15T23:55:26.672Z | DOCS | Another log message

This is nice because you can initialize things like the route and if you want to display the date when the program starts, then pass the simple docLogger function to other parts of the application that need to use it instead of calling something like logger('DOCS', 'This is my log message', false) every time you want to use it.

Other Uses

I just wanted to mention some other use cases that you can explore as well: Memoization, Singleton, Event Listeners.

Conclusion

Hopefully the concept of closure isn't too complex any more! If you do have any questions please let me know and I will do my best to address them and refine the article for clarity.

How do you like to learn a new language?

Brett Martin — Fri, 12 Nov 2021 17:07:00 +0000

I have noticed there are a couple of different approaches to teaching a new language through online courses:

"Traditional" - each section covers a language feature with some small exercises at the end, similar in flow to college courses.
Small Focused Projects - learn a language feature with a small, focused project that highlights that feature. Each project is small, and distinct.
Large project - Learn the language by building a large project that incorporates and highlights features throughout. At the end, you will have something closer to a "real world" application

What do you prefer and why? If I missed a type that you prefer let me know in the comments!

JavaScript Bites: Ternary Operator

Brett Martin — Thu, 11 Nov 2021 23:08:17 +0000

If you have ever been looking at JavaScript code and wondered what the heck the ? and : is doing, look no further!

These two characters (?:) when used together is collectively called the ternary conditional operator, or ternary for short. It is used to express a basic conditional operation, in the form:

predicate ? truthy : falsey

The operand that comes before the ? is the predicate, or the expression that you are basing your decision logic on. The expression that follows the ? will be what is executed if the predicate is true. The expression after the : is what is executed if the predicate is false

Example

Lets imagine you have an object that represents a user, and it potentially has a key username that would store that users username. If we wanted to determine what to display in the UI, we could decide to show the username if it exists, or Anonymous if it does not. We can quickly create a variable called username that will hold the correct value:

const username = user.username ? user.username : 'Anonymous';

This would be opposed to writing the same functionality with a traditional if statement:

let username;

if (user.username) {
    username = user.username;
} else {
    username = 'Anonymous';
}

Of course, the two expressions after the ? can be anything that evaluates to a value:

const someVal = checkSomething() ? doCoolThingSinceTrue() : doOtherLogicSinceFalse();

Suggestions

This part is opinionated, but I try to avoid using multiple ternary operators chained together. Nothing stops you from doing:

const res = check() ? (checkTwo ? 'true checkTwo' : 'false checkTwo') : 'false check';

but in my opinion this causes more confusion than clarity and it would be better to write it in a more traditional if/else structure.

I usually prefer ternary if it is a quick 'inline' computation that I am passing as a parameter, or like in the original example, I am picking one of two expressions to execute at runtime and I need the result.

Conclusion

Hopefully that helps clear up what this operator is all about and how it can be used. If you have any questions, feel free to ask them below!

Tilld Devlog #1 - Building a Better Knowledge Source

Brett Martin — Tue, 09 Nov 2021 05:26:47 +0000

Originally published @ catstach.io

Hello, and welcome to the my first ever Devlog! I wanted to attempt to build a new web application in public, mostly so I can look back on the experience one day, but also to possibly connect with other developers and builders that I wouldn't meet if I worked in the shadows!

My plan for this first post is to explain the application and what it attempts to solve, and then talk about the first steps I have taken in the first week of development. I will also try to do some small tweets about what I am doing throughout the week and if you are interested in seeing those, give me a follow @BoudroBam!

The Problem

When trying to decide on what to build, I wanted to make sure that I was addressing a problem. It didn't have to be a big problem - in fact I spent a lot of time thinking of smaller problems to try and attempt to solve, but I was usually greeted with a handful of Software as a Service (SaaS) solutions that already existed! However, I didn't let that deter me and I finally settled on a problem to attempt to solve...

Finding information within a company is a massive pain!

I believe this is true for most industries, but it is especially true for software jobs. Not only do you have to know about the systems you are developing and interacting with, you also need to know about the domain of the business, the business rules, which teams own what part of the system, and the list goes on and on.

When you are part of a company for a long enough time you may learn a lot of the things that I pointed out above through conversations and past experiences, but there is usually still some gap in knowledge. If you are new to a company those points are even worse as you drown in a flood of domain knowledge and acronym soup.

I have found that if I was lucky my company had some sort of wiki or knowledge base where people published documentation. However, I still find that with such a system in place, the issue still exists. In fact, I feel that having a wiki gives false security that there is correct, and useful documentation.

The Goals

I decided to title this section "The Goals" because ill be honest, I don't really have a solution! This will be a hard problem to solve, and I imagine there will be many mistakes, learning opportunities, and pivots as I start building. Instead I want to state some goals that will help define what a system that wants to solve this problem should be able to do. It should:

Assist with clarifying documentation.
Surface the subject matter experts (SMEs) of topics.
Ensure knowledge is up to date.
Enable users discovery.
Identify knowledge gaps.

I believe a system that does all of these things would massivly improve the correctness and usefulness of information sharing. The goal that is the most daunting, but also the most important in terms of changing how information is compiled, is identifying knowledge gaps. If the system can automatically detect, then ask for someone to fill the gap, that would be huge!

Of course, these goals may change as time goes on. I may add new ones, or rework existing ones to capture new ideas, all based on user feedback.

First Milestone - Clarifying Documentation Through Key Word Detection

With all of the goals laid out, I need to pick something to start driving toward. While I can't tackle all of a singular goal in one go, I can at least start making milestones that I think will address them! So the first goal area I am going to work on is assiting with clarifying documentation, and I think a good place to start would be to help highlight missing definitions.

The idea is that as pages of documentation are added and modified, the system can start to pick out keywords and request that maintainers of the space add definitions to a glossary. I want to start here because it appears to be a pretty simple process and will allow me to get the basic infrastructure of the app set up. I also think it would be a good launching point for starting to add layers of complexity to the process - first ill start with a simple detection and alert system, but then I can move on to suggesting definitions for industry terms, searching for "synonyms" if its a term used internally by the company, etc.

So, that is what I will be working towards first!

Some Wireframes

That has been a lot of words, so I figured now is a decent time for pictures! I have drafted up some wireframes to get just enough of a UI to drive the system. Most are pretty basic and boring, focusing on creating flows for creating Organizations and Workspaces. What I do want to show is just the initial "dashboard", and the two concepts that will help me drive towards the first milestone, the inbox and glossary.

This is the main view when you are in an organization. Its a pretty common layout but its a starting point, and there isn't really a point in reinventing the wheel at this point!

I wanted to add an inbox as the feedback loop for how the system will indicate to the user when something actionable is detected. In this case, when something should have an entry created in the glossary. You can see in the mock up some other ideas I have for use of the inbox such as identifying SMEs, calling out gaps in documentation, etc.

Having an easily accessible glossary is and important first step in achieving usefulness and clarity. At this point in time it will be a basic list of term to definition mappings, but I imagine this will become more content rich as time goes on.

Initial System Design

Now that you have seen some of the visual ideas for the initial UI, I thought it would be good to show the first pass at an overall system design.
I went back and forth in my mind deciding on certain parts of the tech stack, seeing if there was any reason to use some languages and frameworks I wouldn't usually use on my own projects such as Rust with Actix Web or Haskell with Servant for the API but I ended up deciding to keep it simple and use Node with Express for the server since I was going to be using JavaScript anyways for the React front end!

That API server will then be responsible for the common CRUD operations for the things we see in all apps such as Users, and domain specific things like Organizations and Workspaces. These will be written to a PostgreSQL database, since a lot of this info makes sense in a relational database. I decided to go with a second database, a NoSQL database, increasing complexity a bit. This is where the documents will be stored but this may end up being a mistake that I try and roll back in the future. My thinking is that because im not really sure what kind of metadata ill be associating with documents to power this app, it will be easier to work quickly and try new things.

Finally the server will also be responsible for sending messages to a RabbitMQ queue for processors to consume. Again, you can see that there is a plan to create a lot of processors to work off of messages being generated on document CRUD operations. The first one is the keyword processor but ill need to process images to extract data, for example, and I will want to start building a graph of how things in the organization are connected, and who knows what else in the future!

The processors themselves will be written in whatever language makes the most sense, since it is highly decoupled they don't need to interact. That being said, I am starting with Python for the first few because it was the best option when it comes to machine learning and data libraries, and I am keeping them all in the same codebase to allow for code reuse for the common tasks of working with the queues and database.

Current Progress

So, I am not sure the best way of showing progress from a coding and development standpoint. If you have any suggestions of what you would like to see let me know in the comments!

General Notes

As of this writing I started working on the shell of the backend. I wrote a docker-compose file to spin up containers to run the PostgreSQL DB, MongoDB, and RabbitMQ services for local development, and I created two new projects on Gitlab: api and data-processors.

Data Processors

The data processor has been setup to take a command line argument to determine which processor to run on launch, so something like python data-processors.py keyword will start the processor and kick off the logic to pull from the queue and take actions to determine keywords (currently it does nothing besides echo any messages it gets, no fun keyword extraction yet!). It pulls from the message queue using the pika. Unfortunately there isn't too much that is interesting going on in here yet to show code wise.

The next thing to do will be to connect to the MongoDB instance to enable read/write operations on the documents.

API

Similar to the data processor, the first thing I did besides just the basic "Hello World" setup of an express server was create the functionality to write to the queue. I set it up to be a singleton connection to avoid opening and closing all the time, and I can show the first pass of that code here real quick:

const amqp = require('amqplib');

/**
 * Function to create a singleton connection to the RabbitMQ host.
 * Returns a function `get` to return the connection to the caller, and
 * `release` to close the connection.
 * 
 * `release` should only really be invoked if the server is closing and is added
 * for completness.
 */
const conn = (() => {
  let connection = null;

  const connect = async () => {
    try {
      return await amqp.connect(process.env.RABBITMQ_HOST);
    } catch (e) {
      console.log('Failed to connect to RabbitMQ Host: ', e);
    }
  }

  const get = async () => {
    if (connection === null) {
      connection = await connect();
    }

    return connection;
  }

  const release = async () => {
    if (connection !== null) {
      await connection.close();
    }
  }

  return {
    get,
    release
  };
})();

/**
 * A channel is just a multiplexed connection on the open TCP connection.  This allows us to have distinct channels for different logical
 * operations without maintaining extra TCP connections.
 */
const createChannel = async () => {
  const connection = await conn.get();
  const channel = await connection.createChannel();

  return channel;
};

/**
 * We make use of an exchange under the hood since most messages will be intended for more than
 * one consumer, and a standard queue is a 1-to-1 producer to consumer ratio.
 */
const createExchange = async (exhange) => {
  const channel = await createChannel();
  await channel.assertExchange(exhange, 'topic', { durable: false });

  return channel;
}

/**
 * The only function that should be used outside of this file.
 * 
 * Given an exchange name and a topic, return two functions.  One to publish to the exchange, and one 
 * to close the channel connection when done.
 * 
 * TODO - Write up some documentation to link to where we record all the exchanges / keys in use.
 */
const mkQueue = async (exchange) => {
  const channel = await createExchange(exchange);

  const publish = (key, msg) => {
    channel.publish(exchange, key, Buffer.from(msg));
  };

  const close = async () => {
    await channel.close();
  }

  return [publish, close];
}

module.exports = mkQueue

This allows me to make use of just the exported function like this:

const { rabbitmq }  = require('./queue');

const [publish, close] = rabbitmq('docs');

publish('#', 'This is my message');

await close();

So anywhere that I want to write to the queue (any API endpoint that needs to produce a message), I can pass a handle to the publish function to and boom!

The next thing I did was setup my connection the PostgreSQL database using the node-postgres library, and setup migrations to define my schema with node-pg-migrate. I created some simple create functions for the various parts of the database like this one for creating an organization:

const createOrganization = query => {
  return async (name, description = '', industry = '') => {
    const { rows } = await query('INSERT INTO organizations (name, description, industry) VALUES ($1, $2, $3) RETURNING *', [name, description, industry]);
    const { id } = rows[0];

    return id;
  };
};

I might end up moving away from the migration library, but it seemed like it would work for my current use case.

As you can see from the code above, its not really production ready. I need to set up error handling & recovery when executing these operations and do some logging, but ill probably do that at a later time when I am getting close to releasing the first milestone.

Next Steps

This upcomming week I hope to achieve the following:

Establish the connection to MongoDB for both the API and the data processor.
Build the /api/docs route for all the verbs (GET, POST, PUT, DELETE)
Persist docs to the MongoDB instance, send a message to the queue, and see the processor successfully retrieve the doc from MongoDB.
Determine which data science package to use for keyword extraction
- Implement the keyword extraction!
Stretch: setting up the rest of the API routes for CRUD operations would be good as well.

What Are You Building?

If you made it this far, thank you so much for reading! I am hoping to release these either on a weekly or bi-weekly basis, so make sure to follow if you want to see the progress!

I would love to hear about what you are building as well! Is it a library, a program to solve a problem that has been nagging you, the next Google? Let me know in the comments!

Why You Should Write Pure Functions

Brett Martin — Sun, 31 Oct 2021 05:01:05 +0000

Originally posted @ CatStache.io - Check it out for more posts and project updates!

Pure functions are a cornerstone of functional programming, but even if you are writing code that isn't purely functional its a great idea to prefer them!

Defining Pure Function

The two properties of a pure function:

Given the same set of arguments, the function will always produce the same result.
Invoking the function produces no side effects.

A side effect can be thought of as any observable effect besides returning a value to the invoker.

A simple example of a pure function:

const add = (a, b) => a + b;

For any input into this function, it will always produce the same value. That is to say, invoking the function like add(5,2) will always produce 7. It is also possible to see that nothing else, such as modifying state or interacting with other systems, so this function is pure!

Technically, if we were to rewrite the previous function to call console.log to output some info, that would make the function impure because it is having an observable effect that is not just returning the function.

Another example of an impure function would be Math.random() as it modifies the internal state of the Math object (breaking point 2) and you get different results each time the function is invoked (breaking point 1).

Side Effects Cause Complexity

Functions that are pure are easier to reason about - you can create a mapping of inputs to outputs, and that mapping will always hold true. It doesn't depend on external state or effects to produce a result!

Lets look at a function that might be written to determine the number of days since the UNIX epoch (January 1, 1970 00:00:00 UTC) to now (don't use this, and prefer a library if you are working with time, this is just an example 😉)

const daysSinceUnixEpoch = () => {
  const currentDate = new Date();
  const epochDate = new Date('1/1/1970');

  return Math.floor((currentDate - epochDate) /  (24 * 60 * 60 * 1000));
}

This function will produce the value 18930, and every time I run it it will produce that value. Well, it will produce that every time I run that today. Depending on when you read this, if you were to copy this function and invoke it, I have no idea what value it will produce! This makes it difficult to reason about, because I need to know the external state, namely the current day, to try and figure out what value should be produced. This function would also be incredibly difficult to test, and any test that might be written would be very brittle. We can see that the issue is that we are making use of an impure value produced by new Date() to determine the current date. We could refactor this to make a function that is pure and testable by doing the following:

const daysSinceUnixEpoch = (dateString) => {
  const currentDate = new Date(dateString);
  const epochDate = new Date('1/1/1970');
  return Math.floor((currentDate - epochDate) /  (24 * 60 * 60 * 1000));
}

A simple swap to require a date string for computing the difference makes this a pure function since we will always get the same result for a given input, and we are not make use of any effectful code. Now, if I were to call this with daysSinceUnixEpoch('10/31/2021') I get the same result, but now if you were to call it you should also get 18930, neat!

Side Effects Are Unavoidable

Now, while pure functions are awesome, we can't really build an app that does anything of note without side effects. If the user can't see output, or interact with the app in any way, they probably won't have much reason to stick around! Therefore, the idea of preferring pure functions isn't to get rid of side effect, but to reduce the surface area where effectful code is executed and extract pure functionality into reusable and testable functions.

Let's look at another example of some code that might be written server side with the Express web framework. A common thing that is done server side is ensuring that the data sent in a request contains all the expected values. Imagine writing a handler for a POST request to an endpoint /api/comment that expected a request body with keys for postId, userId, comment to indicate who posted the comment, what post the comment was on, and what the comment was. Lets take a first stab at this:

router.post('/api/comment', async (req, res) => {
  const {postId, userId, comment} = req.body

  try {
    if (postId !== null && userId !== null && comment != null) {
      const res = await Comment.create({postId, userId, comment})
      return res.send(res)
    } else {
      return res.status(400).json({message: 'Expected keys for postId, userId, and comment'})
    }
  } catch (e) {
    return res.status(500).json({error: e})
  }
})

This would work, we see that we pull the keys out of the request body, then we check that they all exists. If they do we do something to create the comment, otherwise we send back a 400 with the message saying we expected certain keys. If we want to test that our logic for rejecting the request based on the payload is correct we would need to do a lot of mocking and faking a request with different payloads. Thats a huge pain! What if we instead extracted the pure code from this effectful function?

const expectedReqBody = (body, keys) => {
  return keys.every(key => key in body)
}

router.post('/api/comment', async (req, res) => {
  const expectedKeys = ['postId', 'userId', 'comment']

  if(!expectedReqBody(req.body, expectedKeys)) {
    return res.status(400).json({message: `Body of request needs to contain the following keys: ${expectedKeys}`})
  }

  const {postId, userId, comment} = req.body

  try {
    const res = await Comment.create({postId, userId, comment})
    return res.send(res)
  } catch (e) {
    return res.status(500).json({error: e})
  }
})

Now, we have extracted out the pure functionality of checking if values exist. If we are given an array of expected keys and the request body we can ensure they all exist. Now we can test the functionality by testing the pure function expectedReqBody and feel safe when we are using this function as part of validation. As a bonus, if you wanted to validate the body on other requests you have an already tested solution!

Extra Bonuses

I have previously written briefly about function composition and this works really well with pure functions! If you compose a handful of pure functions it is really easy to reason about what will happen throughout the 'data pipeline'. If you have effectful code sprinkled in, it can cause a massive headache!

Pure functions can also be memoized! If you have functionality that takes a lot of CPU power to compute, but is pure, you can cache the results! I can write a bit about memoization but some libraries to use include ramda's memoizeWith and lodash's memoize

Conclusion

Thanks for taking the time to read about pure functions! I will leave you with a tldr bullet point list on the topic:

Pure functions always map the same input to output, and contain no side effects.
We can reason about and test pure functions easily, and pure functions are easier to reuse and compose with.
Side effect add extra complexity, but they are unavoidable if we want to write meaningful apps.
Writing pure functions allows us to reduce the surface area of effectful code.

Higher Order Functions & Functional Composition

Brett Martin — Sun, 24 Oct 2021 05:45:40 +0000

Originally published @ CatStache.io

Have you ever wondered what a Higher Order Function (HOF) is, and how it might help you when writing code? Well look no further!

In this post I will show some examples of where HOFs are used in daily JavaScript, how you can write your own, and as a bonus a small section on composing functions.

What are Higher Order Functions (HOF)?

Simply put, a higher order function is a function that does either, or both of the following:

Take one or more functions as arguments.
Returns a function.

Here is a simple example of some code that you may write that makes use of a HOF:

const nums = [1,2,3,4,5,6];

const isEven = number => {
  return number % 2 === 0;
};

// evens will be an array with elements [2,4,6]
const evens = nums.filter(isEven);

Here we make use of the array filter method which takes as an argument a function that takes a single argument (an element of the array) and returns true if the element should be kept, and false if the element should be discarded.

You might also find the above code written with an anonymous function instead, which would look something like this:

const nums = [1,2,3,4,5,6];
const evens = nums.filter(n => n % 2 === 0);

While the above code demonstrates filter, you will find many examples of passing in functions as arguments to other methods for arrays such as map, reduce, every, etc. This pattern will start to pop up all over the place in JavaScript code - you probably were already making use of this concept without knowing its name!

Extending our isEven function

One thing that is nice about HOFs is that they allow use to write some code to combine functions in new and interesting ways. Imagine our code from above is growing and we need to also be able to get a list of odd numbers. We could easily write an isOdd function that would be very similar to isEven, but we also know that in this case it would just be the exact opposite of isEven. Unfortunately we can't just write const evens = nums.filter(!isEven), even though that is basically what we want. What we can do is create a new function called not that takes a function as an argument, and returns a new function that will invert the value of the passed in function:

const nums = [1,2,3,4,5,6];

const isEven = n => {
  return n % 2 === 0;
};

const not = f => {
  return x => !f(x);
};

const isOdd = not(isEven);

// odds will be an array with elements [1,3,5]
const odds = nums.filter(isOdd);

Awesome! Our function not satisfies both requirements for being a HOF because it takes a function as an argument and it returns a new function as its result, which we bind to isOdd. You can make use of HOFs to build more complex functionality by reusing logic of smaller functions, sweet!

Cleaner Code With Function Composition

While function composition isn't strictly related to the topic of HOF, it is something you may want to use for writing clean functional code. I would suggest you reach for a library such as lodash/fp or ramdajs to grab their compose function. Here is the documentation for ramdajs's compose.

While writing not as a HOF, I did add some extra machinery by taking a function and returning a function to work in my example. The simplest, standalone version of not would be written as const not = v => !v;. That is, it just takes a boolean value and returns the inverse. I can write the not function this way if I make use of compose. Here is an example of doing just that:

import R from 'ramda';

const nums = [1,2,3,4,5,6];

const isEven = n => n % 2 === 0;
const not = v => !v;
const isOdd = R.compose(not, isEven);

const odds = nums.filter(isOdd);

In my opinion, this is the cleanest version of the code so far, and the best part is R.compose takes as many functions as necessary! This allows us to start writing functional code that is composable and easy to test and reason about!

Conclusion

In the end, Higher Order Functions are prevalent in JavaScript - if you stumbled across this while trying to learn there is a chance you were already making use of this concept without knowing it! If you end up leaning towards writing more functional JavaScript, HOFs will be a concept you make use of heavily, and when paired with composition the code will become easy to test, combine, and reason about!

How Do You Name Projects?

Brett Martin — Wed, 20 Oct 2021 21:22:20 +0000

A commonly quoted joke about programming:

There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors

And that is talking more about naming variable and functions - not to mention naming a project!

I am curious what everyones strategy is when it comes to naming a new library, or website, or app?

My personal blog was just a silly observation of my cat, who I wanted to make the mascot so I called it CatStache!

JavaScript Array .map() vs .forEach()

Brett Martin — Sun, 17 Oct 2021 21:49:13 +0000

originally published @ CatStach.io

When working with arrays of data, it can be a bit confusing when you should reach for map or forEach, so in this post I hope to help clear that up!

tldr: use map if you want to make transformations to the data, use forEach if you want to perform side effects.

Lets imagine we have an array of store items that we would like to work with:

const storeItems = [
  { name: 'Sunglasses', price: 15.00, memberDiscount: true},
  { name: 'Flip flops', price: 50.00, memberDiscount: true},
  { name: 'Sunscreen', price: 5.99, memberDiscount: false}
];

Mapping

Say we wanted to use this data, but we wanted to modify the items to have all their names uppercase:

const uppercaseItems = storeItems.map(item => {
  return {...item, name: item.name.toUpperCase()};
});
/*
[
  { name: 'SUNGLASSES', price: 15.00, memberDiscount: true},
  { name: 'FLIP FLOPS', price: 50.00, memberDiscount: true},
  { name: 'SUNSCREEN', price: 5.99, memberDiscount: false}
]
*/

The above function will return a new array with modified names, and it WONT update the storeItems array so that will continue to hold the initial values.

For another example, what if we wanted to modify the price based on if there is a member discount, and the person currently viewing the data is a member. We could write the following function:


const applyMemberDiscounts = (items, isMember) => {
  return items.map(item => {
    if (!item.memberDiscount || !isMember) return item;

    const price = item.price - (item.price * 0.20);
    return {...item, price};
  });
};

const discounted = applyMemberDiscounts(storeItems, true);
/*
[
  { name: 'Sunglasses', price: 12.00, memberDiscount: true},
  { name: 'Flip flops', price: 40.00, memberDiscount: true},
  { name: 'Sunscreen', price: 5.99, memberDiscount: false}
]
*/

For a final example, I'll talk about how I use map when rendering React components. If I had some component, <StoreItem> that took all the values and made them look pretty, I would display them by building all the components and storing them in a new array, then later in the JSX I can just pass the array as children.

const comps = storeItems.map(item => <StoreItem {...item} />)

...

// In component return
return (
  <div>
    {comps}
  </div>
)

You can probably start to imagine how you would use this in your code now - basically if you want to transform your data, and keep a copy of the old data, use map!

Using forEach

The cases where you would want to use forEach would fall under "side effect" scenarios. I pretty much only use forEach where I don't care about the result of the function that is being applied to the array elements. If you have some logging function (or in the simplest example console.log), the result of applying that function to the value is generally uninteresting, so forEach works well:

storeItems.forEach(item => console.log(`${item.name}: $${item.price}`));

Or, if you have some sort of action dispatching system where you dont care about the result of dispatching, you could take some array of actions and use forEach to dispatch them all.

const actions = [firstAction, secondAction]
actions.forEach(action => dispatch(action));

If you find yourself writing code like

storeItems.forEach(item => item.name = item.name.toUpperCase());

I would highly suggest you swap to using map instead. While the above is code that will compile and work, it is transforming the original dataset, and that can cause bugs that are difficult to track down!

Conclusion

Both map and forEach have their place when working with arrays of data. If you are wanting to make transformation to the data, reach for map. If you are looking to do effectful computations based on the data without modifying it, reach for forEach!

I hope this helped in identifying the right tool for the job at hand!

ES2020: Nullish Coalescing with Babel Plugins

Brett Martin — Sun, 05 Jan 2020 05:51:45 +0000

Photo by Vincent Botta on Unsplash

Also published @ CatStache.io

Moving into 2020, a lot of developers have been focusing on what ECMAScript 2020 (ES2020) has to offer! This is a short tutorial on how to take advantage of the the Nullish Coalescing operator ?? which is specified in this GitHub Repository: Nullish Coalescing for JavaScript

Getting and Adding The Plugin

If you already have babel configured in your workflow, it is as easy as installing the proposal plugin for babel (if you don't, check out part 1 of this series!)

npm i --save-dev @babel/plugin-proposal-nullish-coalescing-operator

and then updating your babel.rc plugin with the plugin you just installed!

{
  ...,
  "plugins": [..., "@babel/plugin-proposal-nullish-coalescing-operator"]
}

That is all that you should need to be able to use the new operator ?? and compile to compatible code for the browser!

Babel will parse, transform , and then print the code back out as the final output. By default, the transform phase doesn't do anything! This is where the plugins come into play - generally you can get away with just using @babel/preset-env to use the latest and greatest JavaScript. *However, when you want to use the true bleeding edge like Nullish Coalescing you will need to seek out a specific plugin to use, like in this tutorial.

Simple `??` Example

The basics of nullish coalescing is allowing for sane defaults when a value is either null or undefined. The old way people would check for existence and then going to a default was using the or operator: let displayname = username || "User Profile"; where this would check for the existence of username and if not found default to the string "User Profile".

The problem with this old way was that something like an empty string "" is coerced into being falsey in this expression! The ?? operator only utilize the default for null or undefined. Here is a quick example - this code:

const logUsername = username => {
    let displayname = username ?? "User Profile";
    console.log(displayname);
}

const oldLogUsername = username => {
    let displayname = username || "User Profile";
    console.log(displayname);
}

console.log("New logging of username: ");
logUsername(null);
logUsername("");
logUsername("bamartindev");

console.log("\nOld logging of username: ");
oldLogUsername(null);
oldLogUsername("");
oldLogUsername("bamartindev");

will compile to:

"use strict";

const logUsername = username => {
  let displayname = username !== null && username !== void 0 ? username : "User Profile";
  console.log(displayname);
};

const oldLogUsername = username => {
  let displayname = username || "User Profile";
  console.log(displayname);
};

console.log("New logging of username: ");
logUsername(null);
logUsername("");
logUsername("bamartindev");
console.log("\nOld logging of username: ");
oldLogUsername(null);
oldLogUsername("");
oldLogUsername("bamartindev");

and the following will be output:

New logging of username:
User Profile

bamartindev

Old logging of username:
User Profile
User Profile
bamartindev

Notice that this is the replacement for ??:

let displayname = username !== null && username !== void 0 ? username : "User Profile";

So, in theory we could also write logUsername(void 0); and get our output of "User Profile", but I am not sure when that might happen 😉

The End

This concludes this small tutorial on adding a plugin to your babel configuration and and why you might need to do that in the future!

If you see any issues with this post, please let me know so I can make edits and give you a shout out!

The First Two Weeks: A Compiler Writing Journey

Brett Martin — Sat, 04 Jan 2020 06:04:32 +0000

Photo by Nick Fewings on Unsplash

Welcome back to the first update in my compiler & interpreter journey! I want to spend some time this post to share some things I have learned about so far, as well as what I plan to tackle in the next two weeks! The first few sections will focus on Compiler / Interpreter information that I have been gathering and starting to digest. The later section will focus on my experience so far with Standard ML, the language that I will be utilizing with the book "Modern Compiler Implementation in ML"

Here is a look at the sections that I want to cover:

What Are Compilers And Interpreters?
What Are The Stages of Compilation?
Lets Check Out The "Super Tiny Compiler"!
What I Have Learned About Standard ML
Useful References And Links

What Are Compilers And Interpreters?

A compiler is a program that translates code written in one language (the source language) into another language (the target language). The target language is usually something like assembly or machine code when the compiler is creating an executable, but it can also be another high level programming language like JavaScript. The Rust compiler is an example of the former, while Babel is an example of the latter.

A key characteristic is that a compiler does its work ahead of time. You can use the compiler to generate its output and then wait to execute it at a later time. A side effect of this, is that an executable generated by the compiler can exist even if the source code is lost.

An interpreter, on the other hand, directly executes the instructions written in the source language without converting it to some target language before. It requires the source code every time it is executed!

Some examples of compiled & interpreted languages:

Compiled	Interpreted
C, C++, Rust, Standard ML, and Java	Lua, Python, and JavaScript

Now, an interesting thing is that any language can be compiled or interpreted. In fact, a lot of programming languages provide both capabilities to developers to improve the development experience. For example, if I want to write some Elixir code it will be compiled to bytecode to be run on the Erlang VM, but if I write an Elixir script or use the interactive mode REPL (Read-Evaluate-Print-Loop) it will behave more like an interpreted language.

As I will show in the next section, compilation (and interpretation) have multiple stages to it. The "front end" stages of parsing and lexing, and creating an abstract representation of the source will be very similar for compilers and interpreters. However, the "back end" stages will start to diverge as an interpreter is more concerned with immediate execution while a compiler is concerned with creating an executable.

What Are The Stages of Compilation?

Compilation is broken down into two major stages, and in those stages are smaller stages that will be of focus as I work through implementing a compiler. I hope to speak to each part in my own words as I go through them!

The two major stages are the front end, and the back end. I know, super descriptive. Similar to any other piece of software, a compiler should be designed with modularity in mind. The front end is the stage of the compiler that takes the source code written in the programming language and turns it into an intermediate representation (IR). This IR is some data representation of the program, that is independent of the source programming language itself. The back end takes the IR and does optimizations and also generates the output of the compiler into some other target language.

This is a very broad description, but it is the first division that can be seen in compiler architecture. A slightly more in depth look can be seen in a lovely picture of a mountain in Crafting Interpreters, in the "A Map of the Territory" chapter. This shows some sub-steps of the front end and back end, like scanning, parsing, analysis, and code generation. The book that I am following has twelve stages listed out! Those stages are:

Lex
Parse
Semantic Actions
Semantic Analysis
Frame Layout
Translate
Canonicalize
Instruction Selection
Control Flow Analysis
Dataflow Analysis
Register Allocation
Code Emission

If I am being honest I know what maybe two of those stages entail (lex and parse). The rest I feel like the profit gnomes in south park.

I am excited to be able to someday intelligently speak to all of these stages in the future, and I think that will be the focus of most future posts. This post is a little all over the place as I am getting my bearings and looking at various resources as I learn Standard ML. Speaking of all over the place...

Lets Check Out The "Super Tiny Compiler"!

Now that we have a bit of understanding of what a compiler is, I think it would be fun to look at a very simple compiler, The Super Tiny Compiler! This compiler is written in JavaScript and very well annotated, so I won't dive too deep into it because I encourage you to take a look at the authors great work!

Essentially this compiler takes some Lisp like function calls, and compiles them into C like function calls! Essentially taking something like (add 2 2) and converting it to add(2, 2).

What I want to take a look at is the entry point for this compiler, this function that loosely follows the lex, parse, translate, and code emission steps above:

function compiler(input) {
  let tokens = tokenizer(input);
  let ast    = parser(tokens);
  let newAst = transformer(ast);
  let output = codeGenerator(newAst);

  // and simply return the output!
  return output;
}

This takes us through the major steps: tokenization, parsing, transforming, and code generation. The first step is what finds the individual parts of the input, taking something like (add 2 2) and building a list like

const tokens = [{type: 'paren', value: '('}, {type: 'string', value: 'add'},
                {type: 'number', value: '2'}, {type: 'number', value: '2'},
                {type: 'paren', value: ')'}];

to represent the input program, with added metadata about what each token is. The next step takes care of making sure that those tokens make sense with the semantics of the language. Luckily in the case of this compiler, there are no keywords, but there are expectations of matching parens!

After creating a correct program as defined by its semantics, the transformation of the abstract syntax can take place to change its representation to a C like function call.

Finally, after the transformation is applied, the code is generated in the C like manner! Again, this is a highly simplified overview of what is in the compiler, but I encourage you to take a look at the source - pull it down and tinker with it. See if you can add something new to it. What about transforming from C -> Lisp instead?

As an aside, I think that is an interesting property of compilers and languages - they can be as simple or complicated as needed. That means that I might detour here and there as I am learning to write small, simple languages that focus on implementing the new techniques! 🤓

What I Have Learned About Standard ML

This is where I spent the majority of my last two week. Standard ML (Standard Meta Language aka SML) is a modular functional programming language that is supposedly very well suited to compiler implementation. I have been posting to the GitHub repository standard-ml-learning all of the code that I have been writing as a result of learning - feel free to check it out!

I have a couple of directories in this project, one for following the text "Programming in Standard ML", and the other is a workspace for me to do practice problems from an online course I found CS 312.

One thing I have noticed is that while the text has been useful, its easy to think that I am learning while following along with examples. Its another thing to know that I am learning by tackling small coding challenges and other more free form problems and getting the correct solution!

I am lucky to have some experience with functional programming - I try and use a functional style when it makes sense at work with JavaScript and I have dabbled in Haskell and Elixir as well. The idea of coding in a declarative is a little less harsh for me coming in with that background. Even though that is the case, I still love tripping up over little things as captured in my notes:

Ok, I have typed var and let way too many times when trying to write sml - I have to remember its val!

And even that note to myself is misleading, because there IS a keyword let as well!

One thing that I liked was the concept of sharp notation, which is the following:

   val person = ("Jim", "Bob", "Software Developer", #"A", 45)
   val fullName = (#1 person) ^ " " ^ (#2 person)
   val jobClass = #4 person

The sharp notation is a way of accessing values in an n-tuple. As you can see, it isn't the most clear what it is trying to do, but I can imagine that it could be useful in an anonymous function for mapping or something like that.

Another thing that I always love with languages like this is the ease of pattern matching.

   val person = ("Jim", "Bob", "Software Developer", #"A", 45)
   val (firstName, lastName, _, jobClass, age) = person

This will bind the individual values of the tuple to the variables firstName, lastName, jobClass, and age. Note the use of _ in the pattern matching, this ignores the field "Software Developer" and doesn't bind it to anything!

Another powerful concept is the cons operator which is written as :: and is used during list processing. Can you guess what this function does?

fun mystery [] = 1
  | mystery (x::xs) = x * mystery xs

Yeah, I cheated a bit by throwing a bunch of new syntax at you, but it takes a list of integers, like mystery [1,2,3,4,5] and returns 120 - thats all of the elements multiplied together! So, the new keyword introduced is fun, which is a function declaration. Then pattern matching is employed in a new way! The first pattern checks for the value [] which is an empty list. If there is an empty list we return 1. So calling mystery [] would result in 1. The next part of the pattern matching uses the cons operator to destructure the list provided. By doing this, we are grabbing the head and tail of the list - x is the head, xs is the tail. If the list is [1,2,3,4,5], then x = 1 and xs = [2,3,4,5]! Now, given that information hopefully the function body of x * mystery xs makes sense. We are recursively calling the mystery function with a reduced version of the initial input: 1 * mystery [2,3,4,5]. This will continue to happen until we reach the base case of [] and return 1, then the full evaluation will occur of 1 * 2 * 3 * 4 * 5 * 1, which returns 120. Awesome!

There have been a lot more topics that I have learned as well from recursion, to higher order functions, to exception handling. I don't think I am the right person to teach these things, as I have linked to the primary source of my learning, so instead I will conclude by sharing some of the code that I wrote that wasn't guided by the book - some challenges that I tackled on this site as well as some problem set code I wrote for the CS 312 course:

Daily Challenge #148 - Disemvowel Trolls

val vowels = [#"a", #"e", #"i", #"o", #"u"]
fun member_of (item, list) = List.exists (fn x => x = item) list
fun disemvowel s = implode (List.filter (fn x => not(member_of(Char.toLower x, vowels))) (explode s))

Daily Challenge #149 - Fun with Lamps

fun gen_alt (starting, next, len) = List.tabulate(len, fn x => if x mod 2 = 0 then starting else next)

fun diff ([], []) = 0
  | diff (x::xs, y::ys) = (if x = y then 0 else 1) + diff(xs, ys)
  | diff (_, _) = ~1 (* List lengths don't match for some reason *)

fun lamps [] = 0
  | lamps i = Int.min(diff(i, gen_alt(0, 1, length i)), diff(i, gen_alt(1, 0, length i)))

Answers to some parts of Problem Set 1

exception NumberFormatException

fun parseInt (s: string) : int = 
    let
        val SOME x = Int.fromString s
    in
        x
    end
    handle Bind => raise NumberFormatException


datatype tree = Node of tree list

val tt = Node([Node([Node([Node([])])]), Node([Node([]), Node([Node([])])]), Node([Node([])])])

fun treeSize (Node([])) = 1
  | treeSize (Node(x::[])) = 1 + treeSize x
  | treeSize (Node(x::xs)) = (treeSize x) + (treeSize (Node(xs)))

val correctSize = treeSize tt = 10

fun rev [] = []
  | rev (hd::tl) = rev tl @ [hd]

fun isWhitespace c = c = #" "
fun reverseWords words = 
    String.concatWith " " (rev (String.tokens isWhitespace words))

val reversed = (reverseWords "A MAN A PLAN A CANAL PANAMA") = "PANAMA CANAL A PLAN A MAN A"

Some answers to Problem Set 2 - Part 2

(* Part 2 *)
(* a *)
val product = List.foldl Int.* 1

(* b *)
fun even_odd_idx (a: 'a, (b1: 'a list, b2: 'a list)) : ('a list * 'a list) =
    if (length b1 = length b2) then
        (b1 @ [a], b2)
    else
        (b1, b2 @ [a])

(* Any way I can make this point free? *)
fun partition (l: 'a list) :  ('a list * 'a list) = List.foldl even_odd_idx ([], []) l

(* c *)
fun apply_twice_positive (i: int) = fn (f: int -> int, count: int) => if f(f i) > 0 then count + 1 else count
val count_positive_funcs = foldl (apply_twice_positive(~1)) 0

(* This returns 2! *)
val positive_count = count_positive_funcs [fn x => x + 1, fn x => x - 1, fn x => x * ~1, fn x => x*x]

One final thought on Standard ML learning - there are a lot less resources than a popular modern language! I know this is to be expected, but I was very surprised to see that only around 1800 questions had been asked on stack overflow, as opposed to the 1.9 million you see for JavaScript. I am very happy with my progress given that fact, and I am getting close to the goal of being able to fully understand a large "real world" program implemented in SML.

Useful References And Links

One resource I really want to give a shout out to is Crafting Interpreters - this is a really well written and free resource on writing an interpreter for a programming language called Lox. The first part is implemented in Java, the second part is implemented in C.

The only reason I am not using this as my primary learning text is that I wanted a more rigorous text to get started - something that dives a little more into the theory. I wouldn't be surprised to see myself reference this resource throughout my journey though!

Another place that I have been looking at is the programming languages subreddit, /r/programminglanguages I am using it as a gauge to see how much I am learning - at the moment a lot of the topics being discussed are way over my head, but I hope to start understanding the common problems discusses in programming language creation as I learn.

The subreddit also has an associated discord server for a little bit more live discussions.

Until Next Time

If you made it this far, thanks for reading! I think that moving forward I am going to move my "publish" date to a Monday so that way I can spend a bit more time organizing my thoughts. This post felt like it was a bit more of a brain dump than I wanted it to be this time around.

By next time I hope to have finished my initial learning of Standard ML so that way I can talk a bit more about implementing a specific stage of a compiler next time. I am super excited to start!

I will post the next update on January 20th, and all future updates will be on the Monday 2 week after.

If you have any corrections or clarifications to statements I have made, please drop a comment. The last thing I want is to be misleading anyone, even though this is about my journey to learn and not a tutorial.

Modern JavaScript Tooling: Compiling

Brett Martin — Tue, 24 Dec 2019 06:29:19 +0000

Photo by Matt Artz on Unsplash

Also published @ CatStache.io

One thing that I commonly see new JavaScript developers struggle with, and that I found particularly annoying when jumping back in, was this concept of having to compile when developing with modern JavaScript, as well as all of the other tooling in the ecosystem.

In this series, I want to jump into the "why" around having a build for your front end code, and walk through setting up a brand new project from scratch.

What Happened to the Good Ole Days?

If I were to tell you to open the developer tools in your browser of choice, you could easily start writing JavaScript and executing it. In FireFox I can just hop into the console and type const f = a => a * a; and after executing that code I could type f(5); and expect to see an output of 25.

In fact, I can continue this path and open up my favorite text editor, create an HTML 5 document with a script tag, and put most JavaScript that I would want in there:

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>My Test Page</title>
</head>

<body>
    <section>
        Hello, <span id="name"></span>!
    </section>
    <script type="text/javascript">
        const name = prompt("Please enter your name");
        if (name !== null) {
            document.getElementById("name").innerHTML = name;
        }
    </script>
</body>

</html>

This is all fine, and you will see that I made sure to still use "newer" syntax such as const with no issues. But lets say that I want to get a little fancier and use the rest parameters syntax:

<script type="text/javascript">
    function sum(...nums) {
        return nums.reduce((prev, curr) => prev + curr);
    }

    console.log(sum(1, 2, 3, 4, 5));
</script>

When I open this up in FireFox it appears to work as intended, but as a good front end developer I start testing in all of my browsers. I notice that I have issues with IE, Edge, and Safari! A quick consultation of caniuse.com shows me that this syntax is not supported in those browsers 😞

So, what are my options? I can choose to move away from this new syntax and find something that works in all browsers, I can drop support for these three browsers (good luck with that!), or I can look into setting up process that will either polyfill or build backwards compatible code for me!

This article is focusing on the last option, the build option.

I Can Live Without Rest Syntax

That may be true, but what was illustrated above was just one instance of backwards compatibility issues - many more exist today, and this will be an issue moving forward as new versions of ECMAScript are defined.

ECMAScript is the specification that JavaScript conforms to. The latest version as of this writing is ECMAScript 2019 which was published in June of 2019. The rest syntax was added in the ECMAScript 2018 version, so that can show you how browsers lag behind!

JavaScript will most likely always have this issue - new standards will be created and released well before all of browser vendors will be able to update their JavaScript engine. In some cases, like Internet Explorer, new updates will be dropped all together as the browser is sunset. Unfortunately, depending on who your target audience is, you may have to support these browsers long after they have been "retired".

Since we want to take advantage of the new specification of ECMAScript while also targeting as many platforms as reasonably possible we need to look at solutions to help us achieve that!

Babel to The Rescue

Babel is what I am used to using when wanting to have my cake and eat it to. The tagline on the banner of the webpage is even "Use next generation JavaScript, today." which is exactly what we are looking to do.

One thing that I suggest is to navigate to the "Try it Out" link, which will take you to a REPL (Read-Evaluate-Print Loop) with babel enabled. If you write any next generation JavaScript, it will compile it to backwards compatible JavaScript!

An example that scrolls by on the main page takes the following snippet

[1,2,3].map(n => n **2);

and the result is as follows:

"use strict";

[1, 2, 3].map(function (n) {
  return Math.pow(n, 2);
});

Awesome! Take some time to play around with different functionality to see what compiled JavaScript is returned.

And just for completion, and because one can never have enough code snippets, here is the rest operator code we wrote earlier, and its compiled output.

Before:

function sum(...nums) {
    return nums.reduce((prev, curr) => prev + curr);
}

console.log(sum(1, 2, 3, 4, 5));

After:

"use strict";

function sum() {
  for (var _len = arguments.length, nums = Array(_len), _key = 0; _key < _len; _key++) {
    nums[_key] = arguments[_key];
  }

  return nums.reduce(function (prev, curr) {
    return prev + curr;
  });
}

console.log(sum(1, 2, 3, 4, 5));

I don't know about you, but I much prefer writing the first snippet 😉

Basics of Using Babel

Back over on the Babel webpage, there is a section for setup so lets take a look at that! The first thing you will notice is there are a TON of different options, and if you are new to the ecosystem it will be overwhelming. So, lets take a different approach for this first step - we will get back to this phase in a later post specifically about asset bundling , specifically looking at Webpack.

For now, we will focus on the usage guide in the documentation. You will also need to install Node.js and npm on your preferred platform to follow along.

The first step will to be to create a new directory that we will call, build-tooling and we will change into that directory and instantiate a new node project.

mkdir build-tooling

cd build-tooling

npm init -y

That will create a package.json for your project with the default settings! Next we will follow step 1 from the babel usage guide:

npm i --save-dev @babel/core @babel/cli @babel/preset-env

npm i @babel/polyfill

@babel/core is what it sounds like, the meat and potatoes of the compiler! This is what will take the code we write, and output the backwards compatible code we saw earlier.

@babel/cli is the command line interface (CLI) that allows us to run a command that will specify our input directory and output directory.

@babel/preset-env is a preset that makes our lives a lot easier by not needing to micromanage which transforms to apply, and which polyfills to provide.

Next we will follow along and create a new file, babel.config.js to setup some basic properties for our project:

const presets = [
  [
    "@babel/env",
    {
      targets: {
        edge: "17",
        firefox: "60",
        chrome: "67",
        safari: "11.1",
      },
      useBuiltIns: "usage",
    },
  ],
];

module.exports = { presets };

Now we just need to write some code and compile it! Lets write the most interesting code in the world, and place it in src/interesting.js!

const sum = (...nums) => {
    return nums.reduce((prev, curr) => prev + curr);
};

const mergeState = (curr, next) => {
    const updated = { ...curr, ...next };
    return [next, updated];
};

export { sum, mergeState };

This is obviously a very useful section of code, so lets get it compiled so we can ship it ASAP! 🚀

We then go to our command line and type the following command to invoke the babel CLI:

./node_modules/.bin/babel src --out-dir dist

When you execute the command, you should see a successful compilation method with a file called interesting.js in a new directory called dist. If we open up that file, we are hit with this!

"use strict";

Object.defineProperty(exports, "__esModule", {
  value: true
});
exports.mergeState = exports.sum = void 0;

require("core-js/modules/es6.symbol");

require("core-js/modules/web.dom.iterable");

function ownKeys(object, enumerableOnly) { var keys = Object.keys(object); if (Object.getOwnPropertySymbols) { var symbols = Object.getOwnPropertySymbols(object); if (enumerableOnly) symbols = symbols.filter(function (sym) { return Object.getOwnPropertyDescriptor(object, sym).enumerable; }); keys.push.apply(keys, symbols); } return keys; }

function _objectSpread(target) { for (var i = 1; i < arguments.length; i++) { var source = arguments[i] != null ? arguments[i] : {}; if (i % 2) { ownKeys(Object(source), true).forEach(function (key) { _defineProperty(target, key, source[key]); }); } else if (Object.getOwnPropertyDescriptors) { Object.defineProperties(target, Object.getOwnPropertyDescriptors(source)); } else { ownKeys(Object(source)).forEach(function (key) { Object.defineProperty(target, key, Object.getOwnPropertyDescriptor(source, key)); }); } } return target; }

function _defineProperty(obj, key, value) { if (key in obj) { Object.defineProperty(obj, key, { value: value, enumerable: true, configurable: true, writable: true }); } else { obj[key] = value; } return obj; }

const sum = function sum() {
  for (var _len = arguments.length, nums = new Array(_len), _key = 0; _key < _len; _key++) {
    nums[_key] = arguments[_key];
  }

  return nums.reduce((prev, curr) => prev + curr);
};

exports.sum = sum;

const mergeState = (curr, next) => {
  const updated = _objectSpread({}, curr, {}, next);

  return [next, updated];
};

exports.mergeState = mergeState;

Wow! That is pretty neat, and we hardly had to do anything. Lets add a script to our package.json so we don't have to remember the CLI syntax every time we want to compile. Your "scripts" key can be modified to look like this:

"scripts": {
    "build": "./node_modules/.bin/babel src --out-dir dist"
  },

now executing npm run build should take everything in the src directory and compile it and place it in dist.

Now What?

Well, technically you have all you need to write next generation JavaScript and compile it in a backwards compatible way. If you keep creating new files and compiling, you will start noticing that this will just create a new compiled of the same name in the dist directory - that doesn't seem very manageable or scalable. That is where something like Webpack comes into play! Next time we will take a look into what its purpose is, how it can be configured, and we will also take a look at some of the other tools in its space such as Rollup and Parcel.

If you have any questions, need clarifications, or want me to go into more detail about this first tool and process let me know! I want to try and make this a holistic introduction to JavaScript tooling!

DEV Community: Brett Martin

Advent of Code (AoC) Day One

Intro

Day One Solution

Explanation

Scenario

Part One Objective

Solving Part One

countLarger First Definition

countLarger Second Definition

countLarger examples

Wrapping up Part One

Part Two Objective

Solving Part Two

Wrapping up Part Two

Conclusion

JavaScript Bites: Closure

Formal Definition

Scoping

Closure

Whats the Big Deal

Access Data Through Interface

Partial Application

Other Uses

Conclusion

How do you like to learn a new language?

JavaScript Bites: Ternary Operator

Example

Suggestions

Conclusion

Tilld Devlog #1 - Building a Better Knowledge Source

The Problem

The Goals

First Milestone - Clarifying Documentation Through Key Word Detection

Some Wireframes

Initial System Design

Current Progress

General Notes

Data Processors

API

Next Steps

What Are You Building?

Why You Should Write Pure Functions

Defining Pure Function

Side Effects Cause Complexity

Side Effects Are Unavoidable

Extra Bonuses

Conclusion

Higher Order Functions & Functional Composition

What are Higher Order Functions (HOF)?

Extending our isEven function

Cleaner Code With Function Composition

Conclusion

How Do You Name Projects?

JavaScript Array .map() vs .forEach()

Mapping

Using forEach

Conclusion

ES2020: Nullish Coalescing with Babel Plugins

Getting and Adding The Plugin

Simple ?? Example

The End

The First Two Weeks: A Compiler Writing Journey

What Are Compilers And Interpreters?

What Are The Stages of Compilation?

Lets Check Out The "Super Tiny Compiler"!

What I Have Learned About Standard ML

Useful References And Links

Until Next Time

Modern JavaScript Tooling: Compiling

What Happened to the Good Ole Days?

I Can Live Without Rest Syntax

Babel to The Rescue

Basics of Using Babel

Now What?

`countLarger` First Definition

`countLarger` Second Definition

`countLarger` examples

Simple `??` Example