DEV Community: Neil Syiemlieh

ltag: A little CLI tool for tagged text searching

Neil Syiemlieh — Fri, 27 Oct 2023 15:14:40 +0000

The Problem

I have a command reference. It's a text file with lots of CLI commands I've written down because I don't want to memorise them all. For example here's a bunch of network-related commands:

nslookup -type=ns example.com
nslookup -type=txt example.com
nslookup -type=cname example.com
nslookup example.com 1.1.1.1
dig example.com txt
dig example.com cname
netstat --tcp
netstat --program
netstat --listening

When I want to look up a command, I don't always remember the actual command I want. Let's suppose I need to do a DNS lookup. I might not remember that I need dig or nslookup. Or I might just need something to do with networking. How can I search for all DNS commands or networking commands?

Halfway There: A Better Command Reference

I can add headings and sub-headings to the list above. Each heading starts with a hashtag to differentiate it from the actual commands.

# Networking
# DNS
nslookup -type=ns example.com
nslookup -type=txt example.com
nslookup -type=cname example.com
nslookup example.com 1.1.1.1
dig example.com txt
dig example.com cname
# See processes and ports used
netstat --tcp
netstat --program
netstat --listening

This is better, but I'd still need to search for DNS and scan over the following lines to find the command I want. A better way is to associate a bunch of tags with each line. Each tag would place the line in some category. A DNS category, a networking category. But manually writing "#dns" after every DNS command in my list would be tedious. It would also pollute the command reference with tags everywhere.

The Solution: ltag + fzf

So instead I created a tool read the above text, append the heading to every line as a tag, and print out the result. That would convert the above text to the following:

nslookup -type=ns example.com #networking #dns
nslookup -type=txt example.com #networking #dns
nslookup -type=cname example.com #networking #dns
nslookup example.com 1.1.1.1 #networking #dns
dig example.com txt #networking #dns
dig example.com cname #networking #dns
netstat --tcp #networking #see-processes-and-ports-used
netstat --program #networking #see-processes-and-ports-used
netstat --listening #networking #see-processes-and-ports-used

I call this tool "ltag", short for "Line tag". It's written in Go. Here's the git repository: mebble/ltag.

With tagging out of the way, I now have the ability to search.

The CLI search tool I use is fzf. fzf takes in any text stream and spins up a TUI for you to fuzzy search through the text. I can pipe my tool's output to fzf and violà, I can now search by command and by tag!

I still have one problem though. The purpose of looking up commands is so that I can run them. When I select a command through fzf, the tags associated with that command also get selected. I'd like to get rid of these tags once I've found the command I want. So my tool is also able to remove tags from the selected line.

I can now do a quick command lookup using a shell alias:

alias cl="cat ~/.cmd-reference.txt | ltag | fzf | ltag --trim | pbcopy"

Beyond a Command Reference

This tool could help you search through any list of things. And the list doesn't have to be in a text file. ltag reads from standard input, so you can pipe the output of any command to it.

This is my first Go project, and I'm glad I found a way to create something actually useful. So here it is, hope you find it useful as well.

Sirup: A CLI tool to manage your local Git repositories

Neil Syiemlieh — Fri, 18 Feb 2022 09:36:35 +0000

Hey guys I made a CLI tool that might help you if you have a lot of git repositories in your computer.

The Problem

I recently got a new laptop which meant I had to move the 100+ repos I had in my old laptop. I couldn't simply move or copy all the repos because my old laptop was really slow. I thought I might as well just push all the repos to their remotes and then clone them from my new laptop, but then I realised it would be really tedious to cd into each one of the repos, figure out which repo is clean or dirty, which repo is synced to its remote, and so on. So I decided to create a script to help me.

Enter `sirup`

sirup will show you a JSON summary of all your git repositories. You could analyse the JSON using any standard tool to figure out the status of each repo. This helps you know which repos will need to be cleaned up. I personally like to use the interactive Node.js shell for this. Then you could move this JSON to another computer and generate the git repositories over there. In the process, you could also tweak this JSON so that in your destination computer, sirup would pick up these tweaks and behave accordingly.

Here's the project on GitHub:

mebble / sirup

A CLI tool for migrating your git repositories

sirup

Summarise a directory of git repos. Regenerate them from the summary.

Requirements
Installation
- Install and update
- Remove
Usage
Examples
- sirup sum
  - Recipes
- sirup gen
Who Needs This?
Development
- Testing

Requirements

Python 3 at /usr/bin/python3
Git 2.22 or above
Ensure that ~/.local/bin/ is in your $PATH environment variable
jq (optional) so you can use some handy recipes shown below

Installation

Install and update

curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/mebble/sirup/main/scripts/install.sh | sh

Remove

curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/mebble/sirup/main/scripts/remove.sh | sh

Usage

Print the usage instructions by running:

sirup help

Output:

Usage: sirup <command> <args&gt
Commands
    help                        Print these usage instructions
    sum                         Summarise git repos and print the summary in JSON to stdout
        --repos ./repos/dir     The directory containing the git repos
        --log   [optional]      Will output logs to stdout
    gen                         Generate git repos from a summary file
        --from  ./sum/file      Path to the summary file
        --to    ./dest/dir      The destination directory where you want

…

View on GitHub

Hope this helps anyone who runs into a similar situation.

Game of Life with p5 and Svelte

Neil Syiemlieh — Wed, 26 May 2021 05:44:40 +0000

Made a little web app over the last couple of days of Conway's Game Of Life. It uses TypeScript for the algorithm, Svelte for the web app stuff, and p5.js for rendering cells in a canvas element. Hope you find it interesting!

Check it out over at: cgof.netlify.app

bURL: A tiny web app for breaking down URLs

Neil Syiemlieh — Mon, 19 Apr 2021 16:01:20 +0000

I've been building a little web app that would help you edit URLs more seamlessly. It gives you a breakdown of a URL so you could easily manipulate it. Here's the MVP version.

Check it out at burl.bar

It's still an MVP at this point. I'm putting it out there now so I get feedback as I'm building it. Gonna spend the next couple of weeks prettying it up and putting in more features.

Why Did I Make This?

Reason One: Managing Permalinks

Some time ago I had to work with Grafana dashboards. I needed to work with permalinks to the dashboards that would encode a specific dashboard state. I needed to edit parts of the dashboard state like the time range using the "from" and "to" query parameters, and the dashboard refresh rate using the "refresh" parameter. A typical permalink would hence look something like this:

http://mydashboard.com/dashboard/service-metrics?orgId=101&from=now-6h&to=now&refresh=30s

It's possible to manually edit this if you need to, but if there are many more URLs and more query parameters involved, it gets tedious.

Reason Two: Cleaning Up URLs

When you share a post from social media by obtaining a link to that post, the link will typically include utm_* query parameters so that the site could keep track of traffic to that post. I wanted an easy way to get rid of these parameters. Here's the URL used in the MVP screenshot above:

https://www.reddit.com/r/aww/comments/mtv5a3/this_german_shepherd_is_well_known_in_her/?utm_source=share&utm_medium=web2x&context=3

I thought it would be really nice if I could just click a couple of buttons to get rid of them.

So I decided to build a tool that would fulfil both the use cases above and any new use cases one could come up with!

Updates

I'll be posting updates once or twice a week on my Twitter so you can follow me there for updates. I've posted two so far and I'm hoping for some good progress ahead.

// Detect dark theme var iframe = document.getElementById('tweet-1383705756407533569-160'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1383705756407533569&theme=dark" }

The repository is public. I'm open to any PRs, although I haven't been able to afford time to flesh out a contribution guide. But any kind of PRs/issues are welcome!

mebble / burl

Breaking URLs

bURL

Break up URLs at burl.netlify.app

Usage

Open the app with a blank URL:

https://burl.netlify.app

Open the app with some given URL:

https://burl.netlify.app/?u=https://www.reddit.com/r/aww/comments/mtv5a3/this_german_shepherd_is_well_known_in_her/?utm_source=share&utm_medium=web2x&context=3

Dev Requirements

Node.js
yarn

Dev Setup

Clone the project, cd into the project's root directory and install the project's dependencies

cd burl
yarn install

Testing

Run the application in development mode

yarn dev

In another terminal window, open the Cypress runner

yarn cypress:open

In the Cypress runner, select the test file you want to run. This test will be run in a Cypress-controlled browser. Then as you update your tests and the corresponding application code, you can re-run the tests from the Cypress browser.

You can also run the Cypress tests in headless mode:

yarn cypress:run

References

URL structure: https://en.wikipedia.org/wiki/URL#Syntax
URL encoding: https://en.wikipedia.org/wiki/Percent-encoding
URI schemes: https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml

View on GitHub

Plans

Pretty UI

I'm thinking of quickly patching up the UI using a React component library like Ant Design or React Bootstrap. Then later we could go completely custom by using our own CSS. For CSS in a React app, I think a CSS-in-JS solution provides a good dev experience. So I'm thinking styled-components.

Shareability

I want the app state to be shareable. If two people have to manipulate URLs for some reason, and they wanna share a broken-down view of a URL to one another, it should be as easy as sharing a link. So if you wanna share the broken-down Reddit post link to someone, you just prepend burl.bar?u= to it and you're good to go:

This isn't implemented, so the following URL won't work yet.
Update: It works now!

burl.bar?u=https://www.reddit.com/r/aww/comments/mtv5a3/this_german_shepherd_is_well_known_in_her/?utm_source=share&utm_medium=web2x&context=3

Many URL Formats

For now, the app supports only http and https URLs. I'm imagining it would later support other URL protocols as well, such as ssh, ftp, etc. The URL breakdown input fields would be based on the protocol of the URL input.

This tool has already come in handy to me since building the basic version. It might help you out someday too! Thanks for reading.

Event Streams for Reactive Views

Neil Syiemlieh — Tue, 23 Feb 2021 08:17:29 +0000

I remember hearing about Bacon.js one day and checking out their Getting Started page. In it, they demonstrate a counter using the Bacon event streaming library. This is the demo code:

var up = Bacon.fromEvent($('#up'), 'click');
var down = Bacon.fromEvent($('#down'), 'click');

var counter =
  // map up to 1, down to -1
  up.map(1).merge(down.map(-1))
  // accumulate sum
    .scan(0, (x,y) => x + y);

// assign observable value to jQuery property text
counter.onValue(text => $('#counter').text(text));

Well this was something very new to me. In my early days of programming, when using jQuery or vanilla JS with the DOM API, I would manually update the view whenever my state changed. Something like this:

let count = 0;
updateView(count);

function updateView(count) {
    $('#counter').text = count;
}

$('#up-button').on('click', () => {
    count++;
    updateView(count);
});
$('#down-button').on('click', () => {
    count--;
    updateView(count);
});

Then when I heard that frameworks such as React would update the view for me, I thought "Great! One less thing to think about!". My code became something like this:

const Counter = () => {
    const [count, setCount] = useState(0);
    return (
        <div>
            <p>{count}</p>
            <button onClick={e => setCount(count => count + 1)}>Up</button>
            <button onClick={e => setCount(count => count - 1)}>Down</button>
        </div>
    );
};

Ooooh it's declarative. No more fragile imperative logic, right? One less thing to think about! Now I've got less code, which means less clutter and possibly fewer bugs. And now there's no way I might accidentally forget to update my view! I just need to write to the state, and the state would write to the view for me!

Events, State, View

Writing to the state is triggered by the user clicking a button. A user clicking a button is an event. In UI code, we get a lot of events. From the user, from the network, from some background task. With these events we decide if and how we write to the state. Then the state shows us what's changed by updating the view.

This is great. Now we can focus on state management. In many apps, state management is simple enough. If it gets complex, you can try out an event sourcing-ish tool like Redux or a state machine tool like XState.

Events, Transform, Accumulate, View

But it never occurred to me that state management isn't a must. State management seemed like such a smarty pants thing to do, I never asked if you could maybe wire up your view to your events directly.

Get an event, write to the view.

Of course this alone is very limiting. We should be able to transform the event. Remember old events. Process two or three different events to get a single result. Merge and accumulate events. Ignore events that don't matter. If we could do all this, we get all the power we had with state management, without actually doing state management.

Two Sides, Same Coin

My mind was blown when I realised that state management is basically the same darn thing. When you write to the state, you are doing all event-related processing in one go. When your #up-button and #down-button are clicked, they're equivalent to two emitted events, emitted on their own event streams. When they write to the state (i.e. count++ and count--), that's equivalent to:

merging their streams
defining how that merge would affect the events that came before them i.e. it's accumulating them

So the state is like a bucket where you dump all your event-processed results. In the event processing world, the ability to accumulate all events that have happened is equivalent to storing state. Because that's what state is: the state of something after everything that's happened to it.

And that's how I understood what Bacon.js meant when it said:

var up = Bacon.fromEvent($('#up'), 'click');
var down = Bacon.fromEvent($('#down'), 'click');

var counter =
  // map up to 1, down to -1
  up.map(1).merge(down.map(-1))
  // accumulate sum
    .scan(0, (x,y) => x + y);

// assign observable value to jQuery property text
counter.onValue(text => $('#counter').text(text));

This is how we manage a count state in the event-streaming world. State management and event stream processing are two sides of the same coin. You can use either, depending on your requirements. These two paradigms are object-oriented programming's approach and functional programming's approach to the same problem, respectively.

A very popular event stream processing library is RxJS. Their site includes a pretty good guide into this style of programming. There's also rxmarbles, which has interactive diagrams of Rx event streams. I think these are good starting points if you're new to this and want to dive deeper.

Have a nice evening.

If you liked this post, consider viewing it on my site! You'll see other posts I've made and you'll find notes where I document my learnings too!

A QR Code Primer

Neil Syiemlieh — Sat, 20 Feb 2021 09:33:40 +0000

You're probably familiar with the QR (Quick Response) Code. Those little boxes of even littler black and white boxes. You've mostly used them to make payments. You've maybe even used one to add someone's contacts to your phone or to follow someone on social media. But what exactly are they?

Not Just an ID

For the longest time, I had thought that a QR Code was some unique combination of black and white squares like some sort of ID. That a QR Code is meant to be just one of a bazillion possible codes. That no other code like the one you're looking at exists. I had assumed that if you wanted a QR Code for yourself, you'd have to get one registered, like how you would get your own unique vehicle number plate registered. I've only recently learnt that it's not the point of a QR Code. A QR Code is unique in the same way that a random piece of text you'd note down on a piece of paper would be unique. So a QR Code is just written text, written in a form that's readable by a computer. You also don't have to get one registered; you can easily generate QR Codes that contain any content you want. Ever since I learnt this, I've played around with QR Codes a bit and I'd like to write about it.

Playing With Codes

If you scan a code using any of your payment apps, they attempt to make a payment. The QR scanner on your social media apps might attempt to follow a person. They don't show you what content is in the code. Therefore, you need a generic QR scanner app that makes no assumptions about the QR Code's intent. It should just scan the code and show you the contents encoded within. And there are two such apps that you likely already have: Google Lens on Android and the iOS camera app on the iPhone (iOS 11 and higher). These apps are a great way to demonstrate just the interpretation of the QR Code.

There are many websites that could generate a QR Code out of some content you provide. www.the-qrcode-generator.com is what we'll be using here. In the input box, you'll see a bunch of "data types" to pick from. These are the various formats that your QR Code's content could be in. Select the "Free Text" type to get the most freedom on what content you can put in. Let's put in the text "This is a QR code!".

The generated code looks like this:

When we scan the code using Google Lens, the app decodes it and shows us the contents!

Let's put in some different free text, but this time start it with a tel:.

And we'll see that our scanner prompts us to make a call with this number.

What's Happening Here?

Your QR Code scanner has to interpret the code in two steps:

Decode the QR Code to get the contents
Decide what it should do based on the format of the contents (or whether it should do anything at all)

1. Decoding the QR Code

This is the part where the QR Code technology comes in. "QR Code" is a registered trademark of DENSO WAVE INCORPORATED. They invented the QR Code. There's an official site for it where they list out the features of a QR Code, the standards that have been refined over the years, etc. There's also an ISO standard for it. The standard defines the algorithm for encoding text into a QR Code and decoding the code back into text. It also defines the information required in the code like the QR version and error correction. All this information is encoded in dedicated parts of the QR Code. This article gives us very succinct visuals that highlight these parts.

2. Parsing the contents

Once the contents are decoded, the scanner now figures out what it's looking at based on its format. The format, or "data type" of the contents is not bound to the QR Code standards. We're now out of QR Code territory. This step is equivalent to typing out a piece of text and asking the computer what it's looking at. Is it a URL? A date? An email address? Or something else? There are quite a few standards that define these data types. A very popular list of data types is defined in the open-source ZXing library. Stated in its README is:

ZXing ("zebra crossing") is an open-source, multi-format 1D/2D barcode image processing library implemented in Java, with ports to other languages.

Its wiki has a list of all the data types it recognises with some explanation for each one. I had found a link to it in this Stack Overflow thread, so I'm linking that thread here as well in case it helps you out.

In our little experiment above, we encoded the text This is a QR code! and tel:123456789. The first one doesn't follow any certain format, so Google Lens simply showed it back to us. The second one follows the telephone numbers format and since Google Lens recognises this format, it prompted us to perform some action with this phone number. Notice that if we were to omit the tel: at the beginning of the phone number, Google Lens would see it as just plain text.

The format most commonly used in QR Codes is the URL. A URL works great because it's short, it can link to literally any web content (or even app content), and the only user action required to use it is a single click. This works great for restaurant menus or ad campaigns. Your phone's operating system could also decide if it should open the URL directly in the appropriate app. This lets you do things like encode a Google Maps link to a specific location and let the user open it directly in the Google Maps app.

A pretty interesting format though is the MECARD format for describing a person's contact information. Let's put in the sample text that they provided in that page

MECARD:N:Owen,Sean;ADR:76 9th Avenue, 4th Floor, New York, NY 10011;TEL:12125551212;EMAIL:srowen@example.com;;

like so:

and Google Lens reads this as:

I wanted to demonstrate these different formats using just the "Free Text" input of the QR Code generator on purpose. It's pretty clear now that the other input modes the generator gives you such as "Contact" are only meant to give you the convenience of not having to type out all that messy syntax yourself. Under the hood though, it's all just text.

Can They Have Emojis?

In most cases, yes. But before we get into how or why, I need to be a little more honest about what is in a QR Code. This section is concerned with step one of a QR Code: "Decoding the QR Code". It's about how we turn the 1s and 0s in a QR to the characters we see in the QR scanner's output.

I mentioned in the previous section that the contents in a QR Code are "all just text". That's only part of the story. The reality is we can specify one of few "modes", aka the input character set of the contents we put in. Wikipedia has a list of the possible modes:

Numeric only
Alphanumeric
Binary/byte
Kanji/kana

The QR specification says that when you create a QR in binary/byte mode, the character encoding is ISO 8859-1. What is a character encoding? It describes how we're supposed to turn the 1s and 0s into text. ISO 8859-1 is basically ASCII, with some more characters put in. The QR generator we used seems to create the code in binary/byte mode by default. Hence, the scanner detects that the QR is in binary/byte mode and decodes the characters using ISO 8859-1. But ISO 8859-1 is just a small extension of ASCII. It doesn't describe how to turn 1s and 0s into emojis. So then how do QR scanners read emojis?

It turns out that in reality, many scanners will decode the QR Code using the UTF-8 character encoding by making use of an extension that the QR specificaton provides. Through this extension, a QR generator could provide a hint about the character encoding used. The answers on this StackOverflow post explain it well. So I guess our QR generator uses UTF-8 encoding through this extension, and this lets us use emojis in the code.

Nonstandard Standards

Since the format of the contents of a QR Code is independent of the QR Code itself, anybody can come up with their own format and still have a working QR Code. As long as they provide their users an application that could correctly interpret their custom format, there'll be no complaints. And that's exactly what happened. Over the years, as various companies, governments and organisations adopted the use of QR Codes in their applications, they came up with their own formats to meet their use cases. Some have decided to define a schema in a well established format like JSON or XML, while others have gone completely custom. Here's a short list of custom formats used in QR Codes:

Short Payment Descriptor (SPAYD)
European Payments Council QR Code
Mobile deep linking formats on various mobile phone operating systems
QRIS by Bank Indonesia. A valid QRIS code can be found in the header image of this news article.
NPCI's Unified Payments Interface (more on this below)

Go ahead and scan the QR Codes shown in those pages, or copy and paste the sample contents into the QR Code generator. Chances are, your generic QR scanning app would not be able to parse the contents to prompt a user action, and so it would simply show the contents to you.

Unified Payments Interface

The Unified Payments Interface (UPI) is the official standard used to make payments across various payment platforms in India. If you're shopping in India and the shop you're in happens to accept payments through a QR Code, that QR Code is most likely UPI compliant. So naturally, I wanted to see what the UPI-formatted contents in a QR Code look like.

If you scan a UPI QR Code near you using a generic QR scanner, you'll find that the decoded content is a deep link that contains the information necessary to make a UPI payment. Unfortunately I couldn't find any official documentation on the structure of this deep link. The closest I could get to some structured documentation was this wiki on GitHub which seems like the results of someone else's research on this same topic. In its most basic form, the UPI deep link looks something like this:

upi://pay?pn=<payee-name>&pa=<payee-vpa>

This is a URL with a upi scheme, a pay user action and some parameters:

pn the payee's name
pa the payee's virtual payment address (VPA)

If your QR Code scanner recognises URLs that don't start with http or https, the scanner might redirect you straight to the payment page of your default UPI compatible app. Google Lens doesn't seem to do this though.

upiqr.in is a QR Code generator that will generator a UPI-compliant code given the payee's name and the payee's VPA. Again, all this does is it creates a UPI deep link from those two values and encodes the deep link into a QR Code using a QR Code standard. From this, we could tell that the payee's name and VPA satisfy the minimum required information needed to create a valid UPI deep link, and hence, a valid UPI QR Code.

You can also create a UPI-compliant code by manually writing down the deep link in the www.the-qrcode-generator.com we used above.

Enter a valid VPA and you'll be able to make a payment through any UPI compatible app using your very own self-generated QR Code! Include the cu and am fields in the deep link and you've just created a QR Code that's bound to a specific amount of money!

Unanswered Questions

While trying out different QR Code generators, there was something peculiar I noticed. Different QR generators generate different-looking codes for the same exact content. Here's the QR Code from above, the one with the content "This is a QR code!":

Now check out this one with the same content, but from a different generator, www.qr-code-generator.com.

They clearly look different even though they store the same content! What gives? I'm guessing these generators use different values for parameters like the QR Code version, error correction level, etc. That could explain why they look different. Is there a QR scanner that could show us this metadata? That's something I haven't looked into yet.

Inspirations

There two things I had found online a few months ago that inspired me to experiment with QR Codes:

A YouTube video on embedding an entire video game into a QR Code
A Twitter thread on using QR Code for restaurant menus:

// Detect dark theme var iframe = document.getElementById('tweet-1318890523743182851-906'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1318890523743182851&theme=dark" }

Because any data (of a reasonable size) can be stored in a QR Code, the possible use cases seem endless. The official site lists the many ways they can be used.

If you're interested, check out this post at my website! It's new and gonna change quite a bit over time. I plan to post notes as well, where I would document my learnings in software and programming.

What Can Array Folding Do?

Neil Syiemlieh — Sat, 20 Apr 2019 10:07:03 +0000

This is Part 2 of the "Folds" series, where we look at how we could use the simple Fold pattern to perform a variety of array processing tasks.

What Was It Again?

In the previous article, we looked at how the fold works under the hood. Let's see it again as a recap:

const fold = (reducer, init, xs) => {
    let acc = init;
    for (const x of xs) {
        acc = reducer(acc, x);
    }
    return acc;
};

It uses a for..of loop to traverse the list xs, reducing the list each time till we end up with just a single value. This programming pattern is very powerful. When I first learned about the fold, I was sceptical of how such a simple operation could do so much. But it turns out that a lot of problems in programming are reduction problems — we have a list of things and we want to extract a piece of information from that list.

Many of you might be familiar with Python's built-in functions sum, len and max. All these functions are essentially folds. I wanted to see how many more folds I could implement in JavaScript using just the function definition above. That would really demonstrate the various things this seemingly simple little function could accomplish. So below are different functions that we could create using the fold.

Keeping An Eye Out

I want to mention that in each fold shown below, there are two parts worth looking out for:

The reducer: I've defined the reducer for each fold separately instead of inline, like the add reducer for the sum fold. The reducer is passed two arguments, acc and x. The data type of acc would be that of its inital value.
The initial value: Notice how the initial value for every fold's accumulation is an identity with respect to the reducer. For example, 0 is the initial value used in the sum fold, because it is the identity under the add reducer. Remember that from the point of view of the reducer, the accumulation's initial value should essentially hold zero information. It should be void and useless, like how add sees 0 as having no information.

Behold, The Folds

`sum`

sum(xs: number[]): number

const add = (acc, x) => acc + x;
const sum = xs => fold(add, 0, xs);

The sum is probably the very first thing you think about when asked about collecting a list of values into one.

`len`

len(xs: any[]): number

const inc = (acc, x) => acc + 1;
const len = xs => fold(inc, 0, xs);

This is an emulation of the universally loved len, from Python. In the reducer, we ignore every element x, just adding a 1 instead.

`product`

product(xs: number[]): number

const mult = (acc, x) => acc * x;
const product = xs => fold(mult, 1, xs);

The product of a list of numbers. Having even a single 0 in xs would render this fold useless.

`join`

join(xs: any[]): string

const concat = (acc, x) => `${acc}${x}`;
const join = xs => fold(concat, '', xs);

This will concatenate a list of strings, or a list of anything, really! Injecting x into the template string invokes its .toString() method. So me saying that the declaration is join(xs: any[]): string, isn't specific enough. What I actually want is xs to be of type xs: A[] where A is a data type that returns a nicely formatted string when we call its .toString(). Without static typing, we can't do this in JavaScript. We see this feature in other languages though, like through Typeclasses in Haskell and Interfaces in TypeScript. Without it, JS would try to stringify x the default way, which might not work so well for more complex objects.

`all`

all(xs: boolean[]): boolean

const and = (acc, x) => acc && x;
const all = xs => fold(and, true, xs);

I really like how clean the all and some folds look. One problem though is that they don't do break out of the loop when the result becomes obvious. all([false, true, true, true]) will go through the entire list even though the result is known by the very first false.

`some`

some(xs: boolean[]): boolean

const or = (acc, x) => acc || x;
const some = xs => fold(or, false, xs);

`maximum`

maximum(xs: number[]): number

const max = (acc, x) => (x > acc) ? x : acc;
const maximum = xs => fold(max, -Infinity, xs);

maximum and minimum could be used on an array of any orderable data type, like JavaScript strings. But then we'd have to use the appropriate initial value. The one we used here, -Infinity, is only appropriate for an array of numbers.

`minimum`

minimum(xs: number[]): number

const min = (acc, x) => (x < acc) ? x : acc;
const minimum = xs => fold(min, Infinity, xs);

`flatten`

flatten(xs: any[][]): any[]

const concatArray = (acc, x) => [...acc, ...x];
const flatten = xs => fold(concatArray, [], xs);

This one has to be one of my favourites. There's a lot of array copying happening here. We could've mutated the acc using acc.push(...x) and returned it to avoid copying acc all the time, but you gotta admit, the spread operator looks much cleaner. This flattens an array one level deep, just like Lodash's _.flatten.

`merge`

merge(xs: object[]): object

const combine = (acc, x) => ({ ...acc, ...x });
const merge = xs => fold(combine, {}, xs);

The merge is very similar to the flatten, except it works on objects. It behaves just like JavaScript's built-in Object.assign.

`reverse`

reverse(xs: any[]): any[]

const prepend = (acc, x) => [x, ...acc];
const reverse = xs => fold(prepend, [], xs);

Another way we could've done this is mutate the acc using acc.unshift(x) (MDN) and return it instead of copying it through the spread operator.

Caveat: This fold's kind of an odd one out. Remember when I said that the accumulation's initial value was supposed to be an identity w.r.t. the reducer? Well, the one here, [], isn't. prepend([], x) will return [x]. According to Wikipedia's article on the fold:

One often wants to choose the identity element of the operation f as the initial value z.

There's no mention of a strict requirement for an identity element. So maybe some elegant mathematical rules would have to be broken in our messy programming world. Or maybe I just did an oopsie somewhere.

`pipe`

pipe(xs: { (x: any): any }[]): (x: any): any

const composeR = (acc, x) => {
    return m => x(acc(m));
};
const pipe = xs => fold(composeR, x => x, xs);

This one is my favourite. I might've butchered the type declaration for the pipe function here, so you'll have to forgive me. The thing I find interesting is initial value for the acc, x => x. It really drives home the idea that the initial value is an identity with respect to the reducer. As for the reducer, it is like the mathematical function composition, except in reverse.

The pipe takes in a list of unary functions and returns a function that runs them all in sequence. The returned value of each function is passed as the argument to the next.

`last`

const second = (acc, x) => x;
const last = xs => fold(second, null, xs);

I just found it fitting to put it at the end.

More Than Just A Fold

All the examples we've seen so far are folds — they take a list of things and return just a single thing. These next ones are not exactly folds in the same sense, but we can still implement them using the fold. That's right, map and filter can be made from the fold!

They don't just require an xs argument; they also need a function f. So the reducer must be defined inline, so we could capture the f through the reducer's closure. These examples also break the identity rule (see the reverse section above).

`map`

const map = (f, xs) => fold((acc, x) => [...acc, f(x)], [], xs);

`filter`

const filter = (f, xs) => fold((acc, x) => {
    return f(x)
        ? [...acc, x]
        : acc;
}, [], xs);

In both map and filter, we pass in the function f before xs, making them "iteratee-first, data-last". This is so that we could leverage the power of currying to make our code more modularized and composable.

Again, we could've mutated the acc using acc.push, but where's the elegance in that? It would go against the principle of immutability that FP preaches. I'm kidding of course, these are all just experiments. In an actual piece of software, we don't really want to get too functional in our own JS implementations, because JS isn't optimized for it (unless we absolutely know what we're doing). For that, we'd be better off using existing libraries like lodash/fp or Ramda.

A Playground

Every piece of code above has been included in this playground below. I also put in some examples of how we can use these folds together. A slight warning though: It looks very messy on a mobile screen.


    
const fold = (reducer, init, xs) => {
    let acc = init;
    for (const x of xs) {
        acc = reducer(acc, x);
    }
    return acc;
};

// reducers
const add = (acc, x) => acc + x;
const inc = (acc, x) => acc + 1;
const mult = (acc, x) => acc * x;
const concat = (acc, x) => `${acc}${x}`;
const and = (acc, x) => acc && x;
const or = (acc, x) => acc || x;
const max = (acc, x) => (x > acc) ? x : acc;
const min = (acc, x) => (x < acc) ? x : acc;
const concatArray = (acc, x) => [...acc, ...x];
const combine = (acc, x) => ({ ...acc, ...x });
const prepend = (acc, x) => [x, ...acc];
const composeR = (acc, x) => {
    return m => x(acc(m));
};
const second = (acc, x) => x;

// folds
const sum = xs => fold(add, 0, xs);
const len = xs => fold(inc, 0, xs);
const product = xs => fold(mult, 1, xs);
const join = xs => fold(concat, '', xs);
const all = xs => fold(and, true, xs);
const some = xs => fold(or, false, xs);
const maximum = xs => fold(max, -Infinity, xs);
const minimum = xs => fold(min, Infinity, xs);
const flatten = xs => fold(concatArray, [], xs);
const merge = xs => fold(combine, {}, xs);
const reverse = xs => fold(prepend, [], xs);
const pipe = xs => fold(composeR, x => x, xs);
const last = xs => fold(second, null, xs);

// other things we could make through folding
const map = (f, xs) => fold((acc, x) => [...acc, f(x)], [], xs);
const filter = (f, xs) => fold((acc, x) => {
    return f(x)
        ? [...acc, x]
        : acc;
}, [], xs);

const A = [
    [0, 1],
    [2, 3, 7, 8],
    [9, 13],
    [16]
];

// find the sum of each row of A
b = map(sum, A);
console.log('b:', b);

// reverse each row of A and then flatten
c = flatten(map(reverse, A));
console.log('c:', c);

// get half of the absolute value of every number
const nums = [3, -8, 6, 23, -100, 8, 1];
d = map(pipe([Math.abs, x => x / 2]), nums);
console.log('d:', d);

// filter out invalid words and make the remaining go UPPER!!
const words = ['cat', 'sl2k3', 'dog', 'sn@k3', 'bird'];
const validUpper = (ws) => {
    const validWords = filter(w => /^[a-z]+$/i.test(w), ws);
    const upper = map(x => x.toUpperCase() + '!!', validWords);
    return upper;
};
e = validUpper(words);
console.log('e:', e);

Like I said in my previous post, our way of implementing the fold is a hack.

const fold = (reducer, init, xs) => {
    let acc = init;
    for (const x of xs) {
        acc = reducer(acc, x);
    }
    return acc;
};

We're using a for-loop and reassigning the acc variable, which isn't very respectful to the lords of immutability. We'll see how we could do that in the next article.

A few of the ideas for this article were inspired by the following:

Learn To Fold Your JS Arrays

Neil Syiemlieh — Mon, 08 Apr 2019 11:45:05 +0000

You might have come across a situation where you need to take an array of values and "collect" them. By this, I mean performing some operation on the array so we could obtain just a single value at the end. Below are a few examples.

You've definitely had to sum up an array of numbers before:

function sum(numbers) {
    let acc = 0;
    for (const num of numbers) {
        acc = add(acc, num);
    }
    return acc;
}

Or get the product of an array of numbers:

function prod(numbers) {
    let acc = 1;
    for (const num of numbers) {
        acc = mult(acc, num);
    }
    return acc;
}

Or find the largest number in an array of numbers:

function maximum(numbers) {
    let acc = -Infinity;
    for (const num of numbers) {
        acc = max(acc, num);
    }
    return acc;
}

In each of these examples, we took an array of things and performed some operation that collected those things into a single thing.

What Is A Fold?

The above examples have a few things in common. They all involve some very similar parts:

A place that holds the final result, commonly referred to as the accumulation or acc
An initial value for the accumulation (0, 1 and -Infinity)
A binary operation that combines the accumulation and the array item we're currently working with (add, mult and max)

This process of collecting items clearly follows a pattern. We're currently repeating a lot of code, so if we could abstract it into a function, we'd have code that's much cleaner and more expressive. There's a name for such a function, the Fold (Wikipedia). This function is one of the fundamentals of functional programming. What we're going to do is implement the fold ourselves in JS, because why not?

A Few Observations

There are three things regarding the fold that are worth noting.

The binary operations add, mult and max are called reducers. A reducer takes two values—the current accumulation and the current array element—and returns the new accumulation.

The initial value needs to be an identity with respect to the reducer. This means when the initial value is passed to the reducer along with another value x, the output is always x. Examples:
add(0, x) = x
mult(1, x) = x
max(-Infinity, x) = x.
Here, 0, 1 and -Infinity are identities with respect to the reducers add, mult and max, respectively. We need it to be an identity because we want the initial accumulation to be "empty". 0 is empty w.r.t. summation and 1 is empty w.r.t. the product.

All array elements must be of the same data type (say type A), but the data type of the accumulation (say B) doesn't have to be the same as the datatype of the array elements. As an example, this code folds an array of numbers into a string.


    
// nothing


    
const concatNum = (x, y) => x + y.toString();  // concatenates a string x and number y
const numbers = [1, 2, 3, 4, 5];  // elements are of type number
let acc = '';  // accumulation is of type string
for (const num of numbers) {
    acc = concatNum(acc, num);
}
console.log(acc);

Notice how the reducer's interface must be reducer(acc: B, x: A): B, which in this case, was

concatNum(acc: string, x: number): string

Creating A Fold

That was a lot of talk. Let's finally make the fold. The fold is a higher order function (I highly recommend Eloquent Javascript for an HOF intro) that takes a reducer (a function), an initial value for the accumulation and an array (more formally a list, which is what JS arrays are).

We first generalise the add/mult/max reducer, calling it reducer (surprise!). We'll call the inital value init. We then generalise the array of things. It could be an array of anything, not just numbers, so we'll call it xs. We've now defined the fold!

const fold = (reducer, init, xs) => {
    let acc = init;
    for (const x of xs) {
        acc = reducer(acc, x);
    }
    return acc;
};

Do you notice the order of the arguments into the fold? There's a reason we first pass in reducer, followed by init and then xs. It has something to do with currying, which we'll get into some other time. The examples from above now look like this, fat arrow style:

const sum = xs => fold(add, 0, xs);
const prod = xs => fold(mult, 1, xs);
const maximum = xs => fold(max, -Infinity, xs);

Much better.

We can write the reducers inline if we want:

const sum = xs => fold((acc, x) => acc + x, 0, xs);
const prod = xs => fold((acc, x) => acc * x, 1, xs);
const maximum = xs => fold((acc, x) => (acc >= x) ? acc : x, -Infinity, xs);

Here's an interactive editor for you to play with:


    
// nothing


    
const fold = (reducer, init, xs) => {
    let acc = init;
    for (const x of xs) {
        acc = reducer(acc, x);
    }
    return acc;
};

const sum = xs => fold((acc, x) => acc + x, 0, xs);
const prod = xs => fold((acc, x) => acc * x, 1, xs);
const maximum = xs => fold((acc, x) => (acc >= x) ? acc : x, -Infinity, xs);

const numbers = [3, 7, 1, 2, 5];
console.log('sum:', sum(numbers));
console.log('product:', prod(numbers));
console.log('maximum:', maximum(numbers));

Pretty easy, right? Well, we kinda cheated. We used a for-loop (more specifically a for...of loop) in our fold definition, which is a big no-no in the functional programming world. Using a for-loop for data transformation means we're going to have to mutate some objects. Here, we mutated acc by reassigning it in the loop. A real functional implementation of the fold would use recursion and would avoid mutation. We'll explore that in another article.

A Few Notes For The Interested

JS already has a fold, which is a method available on arrays. It's called reduce. So I guess you could say re-implementing the fold ourselves was pretty pointless 🤷‍♂️ (although I hope it helps some FP newbie out there).
Because we used a for...of loop instead of an ordinary for-loop, the fold we made works on more that just arrays—it works on any iterable object.
In general, the fold should work on any source of enumerable data, like lists and trees.
The idea of "collecting" doesn't have to be about combining the array elements, like addition or multiplication. It could be about "find and replace", like max/min reducers, or about "applying sequentially", like a function application reducer to pipe functions (if you're interested). The applications are endless!

A function that takes a bunch of things to return just one thing might seem a little trivial, but we'll see how powerful it actually is by implementing many folds in the next article. We'll flatten arrays, pipe functions and [hopefully] do a lot more with the fold.

Creating Image Filters in JS

Neil Syiemlieh — Fri, 29 Mar 2019 17:57:21 +0000

I spent the last two weeks on a project that I hoped would help me obtain a concrete understanding of how image kernels work. You know those image filters you have in Photoshop or your typical photo editor app that let you sharpen or blur an image? Those are implemented using image kernels. This blog post has by far the best explanation of image kernels I could find. For something a little more formal, here's Wikipedia.

The demo is online and I've given the link to it below. What you'll see is a simple page with some instructions at the top. Feel free to play around with the kernel and sample images.

The image is split into four partitions and each partition is passed to a web worker to be processed so we don't block the main thread. I found that four web workers are only a little over twice as fast as one web worker processing the entire image alone. I expected four times the speed, but I guess I got too optimistic, not considering the overhead in copying the pixel arrays, etc.

I can conclude that I now do have a better understanding of how image kernels work (aside from the fact that I ended up spending way more time working with web workers). If only this could get me extra credit for my image processing course.

Here's the demo and the repository:

mebble / imfx

Apply a filter to an image through kernel convolution

ImFx

Perform client-side image processing algorithms like:

Spatial domain filtering
Bit-plane slicing

Check out the demo.

What it does

Takes an image
Takes a kernel
Applies kernel convolution on the image
Displays the output

What it looks like

The kernel for image sharpening

The kernel for edge detection

Sharpening an image of a cat in a hat (Source: Photo by rawpixel.com from Pexels)

Edge Detection on an image of skeletons on the street (Source: Photo by Iván Rivero from Pexels)

TODOs

Choose updatable kernel from templates
Choose from a selection of images
Split and process image across 4 workers
Log the time taken to process image. Can log the time taken separately for each worker
Implement more image processing stuff
- min/max/median filters
- Separable filters for faster convolution
- Bit-plane slicing
Use WebGL for fast processing in each worker
Upload image from device
Take camera photo and use image
…

View on GitHub

Don't Make This Async/Await Oopsie!

Neil Syiemlieh — Mon, 25 Mar 2019 15:00:00 +0000

Suppose we need to perform some I/O on items of an array, like fetching the owners of cats from the cats' IDs using some API.

const catIDs = [132, 345, 243, 121, 423];

Let's say we decide to use our newly acquired async/await skills to do the job. Async/await gets rid of the need for callbacks (in most cases), making asynchronous code look similar to synchronous code. But if we forget that we're still just dealing with asynchronous code, we might make a mistake that defeats the entire purpose of having concurrency.

We might be tempted to do something like this:

async function fetchOwners(catIDs) {
    const owners = [];
    for (const id of catIDs) {
        const cat = await fetchCat(id);
        const owner = await fetchOwner(cat.ownerID);
        owners.push(owner);
    }
    return owners;
}

What Was Our Oopsie? 🤷‍♂️

Disclaimer: If you know what the oopsie was, then you probably already know what you're doing. You might know a use case for this behaviour, so I guess it's a little unfair to call it an "oopsie". This article is just to familiarise people with this async/await behaviour.

We run the code and all seems to be working alright. But there's a glaring problem in how we've used async/await. The problem is that we used await within a for-loop. This problem is actually indicative of a common code smell, which is the ".push to an output array within a for-loop" method of performing an array transformation, instead of using .map (we'll get to this later).

Because of the await within the for-loop, the synchronous-looking fetchOwners function is performing a fetch for the cats sequentially (kind of), instead of in parallel. The code awaits for the owner of one cat before moving on to the next for-loop iteration to fetch the owner of the next cat. Fetching the owner of one cat isn't dependent on any other cat, but we're acting like it is. So we're completely missing out on the ability to fetch the owners in parallel (oopsie! 🤷‍♂️).

Note: I mentioned "kind of" sequentially, because those sequential for-loop iterations are interleaved with other procedures (through the Event Loop), since the for-loop iterations await within an async function.

What We Should Be Doing 😎

We shouldn't await within a for-loop. In fact, this problem would be better off solved without a for-loop even if the code were synchronous. A .map is the appropriate solution, because the problem we're dealing with is an array transformation, from an array of cat IDs to an array of owners.

This is how we'd do it using .map if the code were synchronous.

// catIDs -> owners
const owners = catIDs.map(id => {
    const cat = fetchCatSync(id);
    const owner = fetchOwnerSync(cat.ownerID);
    return owner;
});

Since the code is actually asynchronous, we first need to transform an array of cat IDs to an array of promises (promises to the cats' owners) and then unpack that array of promises using await to get the owners. This code doesn't handle rejected promises for the sake of simplicity.

// catIDs -> ownerPromises -> owners
async function fetchOwners(catIDs) {
    const ownerPromises = catIDs.map(id => {
        return fetchCat(id)
            .then(cat => fetchOwner(cat.ownerID));
    });
    const owners = await Promise.all(ownerPromises);
    return owners;
}

To further flex our async/await skills, we could pass an async callback to the map method and await all intermediate responses (here, fetching a cat to get its owner's ID) within that callback. Remember, an async function returns a promise, so we're still left with an array of promises as the output of .map. This code is equivalent to the previous one, but without the ugly .then.

async function fetchOwners(catIDs) {
    const ownerPromises = catIDs.map(async id => {
        const cat = await fetchCat(id);
        const owner = await fetchOwner(cat.ownerID);
        return owner;
    });
    const owners = await Promise.all(ownerPromises);
    return owners;
}

What is `.map` actually doing?

.map invokes the callback (in which we make an I/O request) on each cat ID sequentially. But since the callback returns a promise (or is an async function), .map doesn't wait for the response for one cat to arrive before shooting off the request for the next cat.

The requests are shot sequentially, but travel in parallel, and their responses arrive out of order (imagine throwing a bunch of boomerangs with one hand behind your back).

So we're now fetching the cat owners in parallel like we intended to 🙌! Oopsie undone!

This was my very first post. Not just my first on DEV, but my first blog post ever. Hope you liked it.

DEV Community: Neil Syiemlieh

ltag: A little CLI tool for tagged text searching

The Problem

Halfway There: A Better Command Reference

The Solution: ltag + fzf

Beyond a Command Reference

Sirup: A CLI tool to manage your local Git repositories

The Problem

Enter sirup

mebble / sirup

A CLI tool for migrating your git repositories

Game of Life with p5 and Svelte

bURL: A tiny web app for breaking down URLs

Why Did I Make This?

Reason One: Managing Permalinks

Reason Two: Cleaning Up URLs

Updates

mebble / burl

Breaking URLs

Plans

Pretty UI

Shareability

Many URL Formats

Event Streams for Reactive Views

Events, State, View

Events, Transform, Accumulate, View

Two Sides, Same Coin

A QR Code Primer

Not Just an ID

Playing With Codes

What's Happening Here?

1. Decoding the QR Code

2. Parsing the contents

Can They Have Emojis?

Nonstandard Standards

Unified Payments Interface

Unanswered Questions

Inspirations

What Can Array Folding Do?

What Was It Again?

Keeping An Eye Out

Behold, The Folds

sum

len

product

join

all

some

maximum

minimum

flatten

merge

reverse

pipe

last

More Than Just A Fold

map

filter

A Playground

Learn To Fold Your JS Arrays

What Is A Fold?

A Few Observations

Creating A Fold

A Few Notes For The Interested

Creating Image Filters in JS

mebble / imfx

Apply a filter to an image through kernel convolution

ImFx

What it does

What it looks like

TODOs

Don't Make This Async/Await Oopsie!

What Was Our Oopsie? 🤷‍♂️

What We Should Be Doing 😎

What is .map actually doing?

Enter `sirup`

`sum`

`len`

`product`

`join`

`all`

`some`

`maximum`

`minimum`

`flatten`

`merge`

`reverse`

`pipe`

`last`

`map`

`filter`

What is `.map` actually doing?