DEV Community: Florian H.

UNIX pipes and output redirects

Florian H. — Fri, 29 May 2020 11:15:53 +0000

(Initially published on my blog.)

Pipes and redirects are a core concept of using the command line in any UNIX like system¹. With the new WSL² package UNIX like concepts are becoming more and more important for developers using Windows as well. Redirecting inputs and outputs is core to how the UNIX system was designed. The rise in popularity of functional programming³ is a great reason to explore this concept again.

This article will help you dig into the concept, and understand why pipes and redirects are so beautiful and versatile. You will be able to take this simple and beautiful design, and apply it to how you design your application's code.

Commands

When you run an application from your terminal, you are mostly interested in the output of your application. Say you run the ls (or dir on Windows), you expect to see a list of files in the current working directory.

$ ls -l

a.png
b.txt
c.txt
d.txt
e.jpg
f.jpg

There is not much of a concept behind this, it is just the way you would expect things to work.

You could even say, that in this scenario, there are two elements involved: The application ls, which outputs the list of files from a directory, and the terminal⁴, which receives this output as its input and makes the letters appear on the screen.

Pipes

The concept behind pipes is very simple to grasp: You take two applications, and take the output of the first application, and connect it to the input of the second application. Just like in plumbing you would connect a water source to a faucet with a pipe. The second application, will do its work using the data you put into it's input, and generate some kind of output. This outpuf of the second application, will be sent to the input of the terminal, which again makes letters appear on the screen.

$ ls -l | grep "jpg"

e.jpg
f.jpg

In our little example, the ls -l command will output the list of files in the current directory (see above), and the command grep "jpg" will act as a filter, that only allows lines that have "jpg" in them, to pass through to the terminal. Syntactically this is achieved by putting a little vertical bar | (called pipe character) between the two commands.

You could also understand this setup as a list of commands, chained together. You can chain as many applications in a row as you like: As long as you always have some kind of output, to send to the next command as input, your chain will work perfectly fine.

And this – I think – is the beauty of the UNIX pipes concept: No UNIX system comes with an app, that lets you output the list of jpgs in a current directory. But you can easily make this happen by chaining a command that can output all files to a command that can filter filenames including "jpg".

There are endless possibilities of what you could achieve with this kind of setup. Every new command you learn does not only give you the power of the specific command, but also allows you to use this command in context of other commands.

Example: Reading logs

Just to give you one example: Let's say you have a folder full of logs for all your services for the last 7 days. You want to see all the entries from all the files, that were collected at a certain hour of a certain day, and that include a special user id.

First you probably want to output all the logs to see what they look like, using the cat⁵ command:

cat /var/logs/*.log

2020-04-11 11:43:45 - INFO - Opened connection to service X. - UID 42
2020-04-11 11:43:45 - INFO - Opened connection to service Y. - UID 42
2020-04-11 11:43:47 - INFO - Opened connection to service X. - UID 32
2020-04-11 11:43:55 - ERR  - Connection timed out for X. - UID 42
2020-04-12 11:56:32 - INFO - Sending confirmation email - UID 32
2020-04-13 22:56:32 - INFO - Sending confirmation email - UID 32
2020-04-14 09:12:22 - INFO - Sending confirmation email - UID 42

Next, you only want logs from the 11th of April, so let's filter the output:

cat /var/logs/*.log | grep "2020-04-11"

2020-04-11 11:43:45 - INFO - Opened connection to service X. - UID 42
2020-04-11 11:43:45 - INFO - Opened connection to service Y. - UID 42
2020-04-11 11:43:47 - INFO - Opened connection to service X. - UID 32
2020-04-11 11:43:55 - ERR  - Connection timed out for X. - UID 42

Last, let's make sure to only output the rows for the user id 42:

cat /var/logs/*.log | grep "2020-04-11" | grep "UID 42"

2020-04-11 11:43:45 - INFO - Opened connection to service X. - UID 42
2020-04-11 11:43:45 - INFO - Opened connection to service Y. - UID 42
2020-04-11 11:43:55 - ERR  - Connection timed out for X. - UID 42

Even though the command cat /var/logs/*.log | grep "2020-04-11" | grep "UID 42" might look intimidating at first, it is actually quite simple. If you build up your command chain step by step, you have full control over what is happening and can build up the complexity of this command chain one step at a time.

Pipes and filters

Understanding this concept is super helpful in doing almost anything with a UNIX like command line: You don't need to build specialized applications, that can do very specific things. You actually want the opposite: You want to make applications, that are very simple and versatile, so that they can be connected to other applications that are the same. This is the fundamental idea behind UNIX pipes.

And this simple and beautiful concept can (and probably should) be applied to all kinds of challenges, that we try to solve with code: If you need to do a lot of data processing using a Jupyter notebook⁶, you are basically doing the same thing: You build up your data cleaning and filtering pipeline, one step at a time.

If you are building sophisticated Javascript applications, in a "functional" way, you also follow this principle (or at least you should ;)

db.getUsers()
    .filter(user => user.age < 18)
    .map(user => { user.lda = false; return user;  })
    .map(user => db.updateUser(user));

This design principal allows you to design beautiful and well structured applications, and at least in my opinion, it is a great way to think about the problem you are trying to solve.

Redirects

Files are so special, that people usually want to have a way to deal with them in a very convenient way. This is why UNIX has two special "signs", which allow to either write to a file, or append to a file.

Let's say you want to write the output of some command you write to put into a file. I will use our logging example from above:

cat /var/logs/*.log | grep "2020-04-11" | grep "UID 42" > output.txt

The > output.txt at the end of the line will tell the system to write the generated output from the last command in the chain to a file called "output.txt". If this file does not exist, it will just create a file and put the content in their. Notice, that running this line, will no longer give you any output in your terminal, because you redirected the output into a file.

This "sign" also has a sibling, which will keep the current content of the mentioned file, and just append the new output at the end. >> will append, instead of replace the content of that file.

cat /var/logs/*.log | grep "2020-04-11" | grep "UID 42" >> output.txt

Just like the previous command, our new example ending in >> output.txt will show no output on the terminal, as the output is – again – redirected away from the terminal, into your file.

Recap

So to wrap this up: One of the core UNIX principles is to connect the output of one simple and versatile application to the input of another simple and versatile application. As a result, it is very simple, to build complex functionality right from your terminal, by just "chaining" applications together.

You can find this principle in many parts of software development, from number crunching to developing sophisticated functional software.

I hope this little article helped you understand this core UNIX principle. As always, I am happy to hear about your thoughts and opinions.

This includes all UNIXes like openBSD or MacOSX and Linux distributions. ↩
WSL is short for Windows Subsystem for Linux. It is a recent (as of 2020) addition to the Windows operating system and is still under development (Version 2 was just released). It basically is a stripped down version of Linux, running alongside Windows, allowing you to run most Linux applications, especially command line applications. This is a great help for developers, who are using Windows to develop in an environment that makes heavy use of the Linux or UNIX ecosystem. ↩
Functional programming is a concept that is very old, but became popular again in recent years. It is very hard to boil it down into a footnote, but the basic concept is that you try to write your application as a series of functions, that follow a strict ruleset, which ensures that these functions do not create any side effects: They are supposed to receive input and return output based on that. If I call a function with the input "X" today, it should yield the exact same result as it will tomorrow or in two years. There are many benefits for this, and I recommend digging deeper into this starting at the Wikipedia article about functional programming. ↩
Terminal is a term commonly used for an application that allows you to control a computer via text input. There is a lot of history behind it, which is why this term is not 100% correct in many use cases, but it is the term most commonly used. The well-known desktop and server operating systems (Linux, Windows, MacOSX) all come with some kind of Terminal. Mobile operating systems often offer some kind of app to allow you to at least connect to another computer's system via text input. ↩
cat is a very simple unix command, which allows to output the content of one or more files. If you don't pipe its output into another command, it will just output a file's content onto the terminal. ↩
Jupyter notebooks are a system that is heavily used by data scientists and analysts all over the world, to do data analysis. It will allow you to write Python code that loads and processes data and prints graphs. The beauty of it is its nice and simple graphical user interface and portability. ↩

JAM stack – what is it about, and should you consider using it for your next project?

Florian H. — Mon, 25 May 2020 11:01:55 +0000

Originally published on my blog.

JAM stack is a concept, that emerged some time in late 2019 or early 2020. It takes the already established concepts of cloud native web development and SPAs¹ one step further. There is nothing technically new in JAM (which stands for Javascript, APIs and Markup), but by applying a new view on the combination of those technologies, it defines a new (and kind of liberating) approach to building web based applications.

I am trying to give a gentle introduction into what the JAM stack actually is, and try to share my view on how and why it is important to developers. Reading this article should allow you to have an informed opinion about it.

What is a JAM stack?

From a high level perspective, the JAM stack is limiting the technologies you use to three main building blocks:

Javascript
APIs
Markup

So the idea is to use Markup (probably HTML) as a basic foundation and scaffold around your application, fill in the interactive functionality (potentially using SPA frameworks¹) using Javascript and connect to server functionality (like persistence, heavy calculation, payment handling, …) via an API.

What you would not do (among other things) is to render dynamic content on your web server (using some kind of backend framework² or more modern approach like next.js). The only role of the web servers in a JAM stack application is to provide an API for their functionality.

Classic website approach

To get a better perspective on the matter, let's look at a more classic approach to building websites or web apps.

You probably have a bunch of different building blocks, like static files (css, js or images), a database and probably some kind of caching system. You also might use some internal or cloud APIs to get additional functionality (like providing your users with credit card payment etc).

The classic way to build a website like this is to build a big node.js application, which does all the heavy lifting by connecting the different sources of data and functionality and renders nice looking pages for the browser to display.

Whenever a user interacts with the page (by clicking a link or submitting a form), a request will be sent to the server, it will generate a new page (probably using a database) and sent it back to the user's browser.

This is – of course – an extremely classical approach, but I wanted to have something to contrast the JAM stack approach to.

JAMstack approach to building a website

Building a website using the JAM stack approach limits the technologies we can actually use to a set of simple and manageable tools: Javascript, APIs and Markup.

Markup

Most websites and web applications have a good portion of basically "static" markup. Think of your skeleton HTML and maybe your markdown³, that form your "basic" content pages, like the contact or about pages. There is nothing dynamic about them, no need to update them constantly, and no need to run a CMS to generate those.

You can either create these HTML files manually, or use some kind of static site generator⁴ to generate those for you. This markup will then be uploaded to some kind of simple web server or CDN⁵, together with any static assets you might have, like your CSS files of your logo.

Once this is generated and uploaded, it will only ever change, when you decide to deploy updates manually.

APIs

If you build anything but the simplest brochure website⁶, you probably want to run code on your server(s), that provides some kind of unique functionality. Be it saving your user's data in your database, or adding funny bunny ears to your user's profile picture.

Following the JAM stack approach, mean to put this functionality behind some kind of API, probably using HTTP endpoints⁷. The API code can then be deployed in the way you (or your company) prefers, be it on your own hardware in a data center somewhere or as using a FaaS⁸ service.

Additionally to your own APIs, you can (and maybe should) use external APIs provided by other companies. These external APIs, could add any functionality to your application, from a headless CMS⁹ to a payment provider. A beautiful side effect of this is, that in case you want to exchange one provider for another (or even an internal API you built) becomes a lot simpler, as there is only one place where all the APIs are connected: your Javascript running in the user's browser.

Javascript

The third (and last) building block is Javascript. All the dynamic functionality of your application, from loading your latest tweets to allowing user's to pay for the T-Shirt they want to buy, will be encapsulated in your Javascript code.

By doing that, your Javascript code will be the most important part of your application, and will probably need the most attention. Luckily, there are many great libraries, that help to build complex Javascript SPAs¹.

Benefits

The most interesting question remains: Why should anyone use this? It certainly is true, that limiting one's choices already is a benefit in itself, as it forces us to be more creative in our approach.

Most of the benefits of using the JAM stack revolve around its simplicity and lack of things. If you don't have a backend server which partially generates your website's content, you will not end up with a big messy ball of mud¹⁰, that becomes unmaintainable after a few years.

If you don't generate any part of your website "on the fly" for your users, and instead put your application (except for your APIs) on a CDN, hosting will be very cheap because you do not need to run expensive servers. Even in peak times, the CDN will just handle the distribution of your application, so that even Black Friday does not have to scare you anymore.

There is also the benefit of security: The only way to have a secure server, is to not have a server at all. Everything that exists can be hacked, so better not have the server exist at all. 😉 The APIs you will write for your JAM stack application need to be accessible by the user's browser, so you are basically forced to secure them properly, instead of hiding them behind some kind of load balancer / firewall setup.

It is also true, that the JAM stack approach makes it less hard and/or scary to deploy a change to production: What you deploy is "just" a set of static files, containing your markup and Javascript. So if anything goes sideways with a new release, it is very simple to "roll back" and redeploy the old files.

There are certainly more benefits to using the JAM stack, and I would be happy to extend this list with your ideas. Do not hesitate to get in contact with me via twitter.com/__florian or email.

Recap

What I intent to express is:

JAM stack stands for: Javascript, APIs and Markup
It very much reduces the technology choices you can make
It takes a lot of emphasis away from classic server backend driven approaches
Some of the benefits are scalability and security

I hope that you did not only enjoy reading this article, but could also learn something valuable from it. Thank you for spending your time on reading my words.

What is CRUD and how does it help me with my application?

Florian H. — Wed, 29 Apr 2020 07:16:37 +0000

(Originally published at florianherlings.de)

Some concepts are not really hard to understand, but so core to the work of a developer, that it is worth thinking and writing about them. CRUD is one of those concepts: You will find many developers talking about CRUD in all kinds of programming projects and tasks, as a way to express what they are thinking about a feature.

You might find somebody saying: „Twitter is just CRUD without the U in front of a big tweets table with joined users.” and even if this is an extreme simplification, it still expresses a lot of information in a small sentence. Many programmers love to speak like this. 😅

Concept

CRUD is actually a pretty simple concept: It describes a set of actions you can do to your data. Whenever you are handling data (in a data base, on your hard drive, in the "cloud"), you always want to think about: What can a user do to this data.

It usually falls into one of these four actions that can be performed to a data point¹:

You can create a new data point
You can read existing data points
You can update an existing data point
You can delete an existing data point

And this is exactly where this acronym is coming from. CRUD is short for:

CREATE
READ
UPDATE
DELETE

So any application you write, that is dealing with data, can do one of those four things: Create, Read, Update and Delete. Let's look at a few use cases like SQL, the file system and HTTP.

SQL (like PostgreSQL or MySQL)

SQL is the super powerful language, that will help you to "talk" to a relational database² like PostgreSQL. All you can do to your database's data is adding new entries, getting entries, updating and deleting entries.

Here are some example SQL queries, and what their role in CRUD is:

Action	SQL query
Create	`INSERT INTO users (name) VALUES ('Alice');`
Read	`SELECT * FROM users;`
Update	`UPDATE users SET name='Bob' WHERE id=1;`
Delete	`DELETE FROM users WHERE id=2;`

File system

The file system follows the same set of rules: You can create, read, update and delete files. Every action you can do to a file falls into one of those four categories.

Action	Shell command
Create	`touch hello.txt`
Read	`cat hello.txt`
Update	`echo "Hi" > hello.txt`
Delete	`rm hello.txt`

Even moving a file from one directory to another directory is just an update, as you only update it's path from one directory to another.

If you run: mv hello.txt ~/Desktop/hello.txt it will move the file hello.txt from its current location to the desktop, which only means that the file's path will be updated to the new directory (~/Desktop),

HTTP

Even HTTP³ follows this concept. The main four HTTP actions, that a request can have are GET, PUT, POST and DELETE. Those can be easily categorized into the four CRUD actions as well:

Action	HTTP Method
Create	POST
Read	GET
Update	PUT
Delete	DELETE

There are numerous other examples of CRUD being used in all corners of software development. Those three (SQL, the file system and HTTP) are just simple examples to underline the point I am trying to make in this article:

CRUD is everywhere. 😊

Using CRUD as a thinking model in your project

The CRUD concept is – as we have seen – very simple and straight forward. Because of its simplicity we can use it as a foundation for thinking about how we design our software.

Data layer design

Let's say we want to design a data layer for our new web application. The first thing we have to think about are the "Nouns", which means: What kind of "things" exist on our website.

If you were to re-create something like twitter, you would definitely come up with two things: Tweets and users.

The next step is to think about: What can you do with those Tweets and users. This is where CRUD comes in. For every "thing" that exists in your data layer (tweets and users), think about if you want to build and what kind of functionalities these would represent:

Action	Tweet	User
Create	Write a tweet.	Register.
Read	See tweets.	Login and show user profile.
Update	Edit a tweet.	Change user profile.
Delete	Remove a tweet.	Delete account.

With this big list, you can think about which kind of functionality you want to provide to your users. In our example of Twitter, you can do almost all those things, except editing a tweet⁴. So the product team at Twitter sat down with this list and decided to not allow updating tweets.

API design

The same model can be applied when you create an API for your application. Let's say you are working for Spotify and want to create an API for their app.

The "things" in their app that come to mind are: Users, Songs, Albums and Playlists.

For a user, the app API should pretty much allow everything:

	User	Allowed
Create	Register.	✅
Read	Login and show user profile.	✅
Update	Change user profile.	✅
Delete	Delete account.	✅

But what about a Song? It would not be helpful, if any user could create new songs, or delete existing ones.

	Song	Allowed
Create	Upload new song.	❌
Read	Listen to a song.	✅
Update	Change a song's title.	❌
Delete	Delete a song.	❌

With this simple "matrix", you can go through all the "things" in your application and decide which of the four CRUD actions you want to allow.

Thinking this through will give you a great foundation to talk to customers and users about your application as well as a great todo list. Even if you decide that a user should not be able to do some of the CRUD actions, this is still a valuable thing to know and a great thing to decide before you start implementing.

Code structure

As we now know, almost all applications follow the CRUD concept. Even if you decide to not include one or more of the CRUD actions, the general concept is still the same.

This is why, it makes a lot of sense to structure your application in the same way: You probably have routes to create, read, update and delete the things in your database, and you might want to name those routes in the same way.

This means, that potentially, a route to add a new tweet might look like this:

app.post('/tweets/create', async (request, response) => {

    const newTweet = await db.createTweet(request.body.text);
    response.json(newTweet);

});

Because this is such a well known pattern, many web frameworks even come with libraries to support creating CRUD actions for your data.

Ruby on Rails⁵ is the most famous example. It has special commands you can run to generate everything you need to do CRUD for a certain "thing" on your website. The generated code will including routes, controllers, models and database migrations⁶.

rails generate scaffold Tweet text:string

Recap

What I hope you take from this article is this:

CRUD stands for create, read, update and delete
Those four actions can be found everywhere in software development.
When you think about the application you want to build, you can use CRUD as a guideline to think through the different "things" and what a user should or should not be able to do with them.

I hope you enjoyed reading the article and were able to learn something from it.

In this article, I am using the word "data point" as what you could think of as a single thing in your database. Like one user, one song or one Album. ↩
Databases have existed for a very long time in the software development word. This is why many different kinds of databases exist to be provide a perfect solutoin for all the different use cases. The kind of database, that is used most often is called a relational database. If you allow me to simplify the matter a lot, you could think of a relational database as something like a fancy excel file. You have columns to define what kind of fields you want and rows to hold the data entries. ↩
HTTP is short for "Hyper text transfer protocol" and is one of the basic foundations of the internet that we know today. This protocol defines how computers (often a browser and a server) communicate with each other. ↩
As a side note: On Twitter, you cannot actually edit a tweet, you can only delete it and write a new one. The developers and product people at Twitter certainly had/have their reasons to not allow editing a tweet, but many users still complain about the fact from time to time. 🤷‍♀️ ↩
Rails (actually Ruby on Rails) is a very popular web framework, written in the programming language Ruby. It is know for its ease of use. There are frameworks for other languages that try to achieve similar goals. ↩
A database migration is a piece of code, that allows you to update the schema of the tables in your database. This means, that when you have an existing database with data in tables, you don't log into the server and change the table structure manually, but you have a well structured file with the code to do exactly that. Two of the biggest benefits are that you can make those changes on many different computers (like your own computer, your colleagues computer and the server). The other big benefit is that these migrations can often be undone, which helps a lot, if you find out later that you made a little mistake. ↩

Working with history, pushState and replaceState in Javascript

Florian H. — Fri, 24 Apr 2020 11:43:10 +0000

(Originally published on florianherlings.de).

The browser's history feature is something that we use almost every day, without thinking about it too much. When we navigate from one website to another website, we are creating basically a list of websites that we went to. Whenever we find ourselves in a situation where we want to go back to the previous page (or even a few pages back) we can easily do this with the back button in our browser.

The Javascript API¹ to deal with the browser's history is surprisingly easy to use. On the one hand we can move back and forth through the browser's history and on the other hand we can even manipulate the current and future state.

Moving back and forward

Imagine our browser's history as a series of elements: There is always one element, that is our "current" element and it represents the page that we are currently seeing.

Fortunately for us, the functionality behind the back and forward buttons is provided to us by the browser's history object. To go back to the previous page, we can simple call history.back() in our Javascript code and the browser will go "one element back" in our history. This way, the previous element will now be marked as the "current" element.

The browser does not forget about the other elements, so that we can also go forward and make the next element the "current" one by simple calling the history.forward() method.

We can even go multiple steps at a time (in either direction) using the history.go() function. If you provide the value 1 it will go forward one element, while calling history.go(-2) will go two elements back.²

Conveniently, calling history.go(0) will not go anywhere, and will instead just reload the page.

Changing history

Going back and forth between existing entries in our browser's history is exciting and useful. The browser's history API¹ goes even further and allows us to add new entries or manipulate (to an extend) the entries that already exists. This is a great feature that enables authors of SPA³ frameworks to write wonderful things like the React router library.

history.pushState

The browser provides a way for us to add a new entry into the browser's history. Right now all major browser support this feature, but (as of 2020) it is still not 100% where it can be.

Using the browser's history.pushState function, we can add a new entry as the "current" entry of the history list. This way we will have added a new entry and at the same time updated what is our current entry at the same time.

The function itself takes three arguments: a state, a title and an URL:

const state = { user: 12 };
const title = 'My new page';
const url   = '/new-page';

history.pushState(state, title, url);

This example will add a new entry into our history, with the state of an object carrying the user's id, the new title My new page and the URL /new-page. The state parameter is really meant for those people who write libraries that make good use of it, because the browser itself will not do anything with this data. Unfortunately, the title parameter is ignored by modern browsers (as of 2020), but in theory the title of the tab should be updated.

This is why you often see code, uses the null value for the first two parameters like so:

history.pushState(null, null, '/other-page');

The browser will make good use of the last parameter url, though: It will update the address bar and show our new URL.

One thing is interesting, though: It does not reload the content of the page. The browser will not actually go to the provided URL /new-page. And this is the wonderful thing about history.pushState: It will leave the currently displayed page as is, while updating the browser's address bar with the new URL. It adds a new history entry, without changing what is currently on the page.

As a side note: If I actually wanted to go to the page (meaning: Also load the content of the page), I could easily just call window.location='/new-page'; to have the browser load that new page.

Because we are manipulating the browser's history, we can still use the back button in our browser to move back to the previous URL without any problem.

history.replaceState

Another way to change your browser's history state is to use the replaceState function. It works almost exactly as the pushState method mentioned above. The big difference is, that while pushState will create a new entry in the browser's history, replaceState will onle replace the current state.

As a side effect of this, using the replaceState method will change the URL in the address bar, without creating a new history entry. Even though this was not the use case the developers had in mind, you could use this to change the URL in the address bar without any further side effects.

Recap

The browser's history is not only a useful feature for actual users, but also for us developers. You can navigate back and forward, and even go multiple steps at a time. The browser's API will also allow you to manipulate the history state by adding a new entry pushState or overwrite the current one using replaceState. Manipulating the history will not cause the page to reload.

There is a lot more you can do with the history API, but I hope this gave you a good intro.

Difference between encoding, hashing and encryption

Florian H. — Thu, 09 Apr 2020 13:03:04 +0000

When I was starting out as a junior developer, it was very hard for me to tell some concepts apart, because I never took the time to sit down and learn about those. Helping with this journey is the intention of this article. I will simplify and shorten a few things here and there to make the topics more digestible and less hard to understand.

This article will only give you an overview over which is which, so that you can dive deeper into the topics that interest you the most. 🙂

Encoding

Encoding is all about representing some kind of information (let's say a word or a text) in a way that can be conveniently saved or transfered. A popular thing to do on computers is to write a little text (like this one) and save it onto a file storage (like my computer's harddisk). A file storage will only be able to save binary data (think: 1s and 0s), so I need to encode my text in 1s and 0s.

Let's say, I want to encode the German word for the color green in binary form, the result would look like this:

Each letter will be transformer into a series of eight 1s or 0s. Why did I choose to use a German word? Because the German character "ü" does not fit into one set of eight 0s and 1s. This encoding is called UTF-8. You probably have heard this before.

Because using text on a computer is so important for people, people have come up with many ways to encode text, and UTF-8 is the one that is used the most because it can encode almost all language's characters (German, Swedish, Japanese, Chinese,…).

Obviously, when you know that the binary data (1s and 0s) are in UTF-8, you can easily turn them back into your actual characters and display them on the screen again.

The binary data could also represent other kinds of information. A series of 0s and 1s could – for example – be numbers in a spreadsheet or colors in a picture of a puppy. Math (especially in school) mostly uses the decimal system. If our binary data represents data in the decimal system, our encoding would look like this:

There are also other ways to encode data, some of which are base64, HEX or even ascii. Encoding always means the same thing:

I have data and I want to represent it in another system. To achieve this, I encode my data in another system's rules. You don't loose or hide data, it is just in a different format.

Hashing

For a long time, it has been considered best practices to only store the "hashed" version of a user's password in the database instead of the password itself. But what does hashing mean, and what does this have to do with passwords?

Almost all programming languages come with a series of hashing functions (or you can add those functionality by installing libraries). You can provide some kind of input (like a word or short text) to them and they will return a "hashed" version of that input.

One example of those is SHA512, which will return a 512 bit long hash.

How is this different from Encoding? Hashing functions only work one-way: You can turn your input into a hash, but you cannot turn your hash into the input again. It is a one-way-street. You actually loose some of the information by hashing.

The fact that this is a one-way-street is really great for storing passwords: You actually don't want to store the original password of the user, in case somethings goes sideways and an attacker steals your database.

Hashing functions have differents "strengths": With growing computational power of computers, these functions usually become cracked after some years, which is why it is important to use an up-to-date one. Old hashing functions like md5 are considered insecure for years now, and the current (early 2020) favorite is bcrypt (which actually uses a lot of clever ideas to defend itself from being cracked soon).

So hashing is different from encoding, because it intentionally looses some of it's data to become a "one way street" function. Hashes cannot be turned back into the input that produced them, which is why they are great for storing passwords.

Encryption

Keeping a secret a secret is one of the hardest and most valued things in human interaction and by that also in computing. What is definitely true for secret government files, is just as true for private conversations between coworkers or family members. You really don't want your parents to know that they will be getting a new computer as a holiday gift, which is why you want to encrypt your buying list.

Encryption is a way of turning any input (like a word or a file) into some kind of data, which cannot be turned back into the input without knowing the secret of how to do this. A typical secret is a password: You use a password to encrypt your holiday shopping list and save it on your computer.

Later, when you want to see the list again, you have to use the same password to see that list again.

So in the simplest case encryption is a "two way street", which allows you to turn input and a password into an encrypted representation. This representation can later be turned back into the input, if you have the password. There are much more sophisticated encryption algorithms, but they all try to do the same thing: Come up with a representation of your data, that is not useful to anybody, if they don't have the password. Encryption does not loose any data.

Conclusion

So what is the difference between all three? Encoding is just a different representation of your input. Hashing is a one-way-street to get a unrecoverable representation. And encryption is a safe way to store information that you want to look at again later.

	Looses data?	Readable by anybody	Output can be turned back into input	Use case
Encoding	No	Yes	Yes	Saving text into a file, sending data to a server, …
Hashing	Yes	No	No	Storing passwords
Encryption	No	No	Yes	Saving secret files, sending secret messages or emails.

I hope this little article helped you to get a general overview of those topics. It is certainly true, that I made a few generalizations to break the topic down into a more digestible format. From here, you can make a deep dive into any of those topics.

DEV Community: Florian H.

UNIX pipes and output redirects

Commands

Pipes

Example: Reading logs

Pipes and filters

Redirects

Recap

JAM stack – what is it about, and should you consider using it for your next project?

What is a JAM stack?

Classic website approach

JAMstack approach to building a website

Markup

APIs

Javascript

Benefits

Recap

Further reading

What is CRUD and how does it help me with my application?

Concept

SQL (like PostgreSQL or MySQL)

File system

HTTP

Using CRUD as a thinking model in your project

Data layer design

API design

Code structure

Recap

Working with history, pushState and replaceState in Javascript

Moving back and forward

Changing history

history.pushState

history.replaceState

Recap

Further reading

Difference between encoding, hashing and encryption

Encoding

Hashing

Encryption

Conclusion

Further reading