DEV Community: Darren Burns

A short example of how types can make your life easier

Darren Burns — Sat, 29 Jun 2019 18:47:51 +0000

What follows is an example covering of some of the issues I encounter frequently when writing code in dynamic programming languages, and how rewriting the same code using type annotations (and more specifically, enums) makes my life easier. The code snippets below are written in Python, but apply to any language that supports enums, such as TypeScript, Java, etc.

Imagine you work for an online book store, and you've been tasked with writing some code to predict and store a user's book format preferences. In the codebase you're working on, you find a method that sounds useful:

def save_preferred_book_format(format):
    ...

You want to make use of this method somewhere in your code, but at a glance, you can't tell how to correctly call it. It clearly takes a single argument, format, but there's no indication as to what format is. Is it a string, a number, or maybe an object of some kind?

You dig into the method body and see code like this buried in the middle of it, which checks that the argument is valid.

if format not in VALID_BOOK_FORMATS:
    raise ValueError(f"format must be one of {VALID_BOOK_FORMATS}")

You jump through to the VALID_BOOK_FORMATS variable, and see that it is defined as follows:

VALID_BOOK_FORMATS = ["AUDIOBOOK", "EBOOK", "PAPERBACK", "HARDBACK"]

Given that this is a list of strings, and our method checks that the format argument is contained within this list, we can deduce that when we call save_preferred_book_format, we must pass it a string, and if it's not one of the strings defined in this list, our program will crash!

If we hadn't read the full implementation of this method, we wouldn't have known this. The signature of the method does not contain enough information in order to safely use it. In this case, the original developer was kind enough to include a reference to the list of VALID_BOOK_FORMATS, so we were able to use our detective skills to work out how to call the method.

This is a common ritual in code bases written in dynamically typed languages such as JavaScript and Python, and it quickly becomes tiresome if you have to perform it every time you want to call a function. If the function has multiple parameters, the complexity can vastly increase the amount of time required to understand it.

This is problematic. If the one of the key ideas behind clean code is reusability, why should you have to jump through so many hoops just to understand how to reuse a function? Well designed code lets you spend more calling functions and less time trying to work out how to use them.

You shouldn't have to read the body of a function to learn how to call it.

How could the method above have been written to avoid this problem?

We know that a copy of a book be an EBOOK or a PAPERBACK, but it can't be both. Many programming languages come with a handy way of modelling this: enumeration types (enums)!

Here's how we could rewrite VALID_BOOK_FORMATS using an enum instead of a list of strings:

from enum import Enum

class BookFormat(Enum):
    AUDIOBOOK = 1
    EBOOK = 2
    PAPERBACK = 3
    HARDBACK = 4

How you use an enum is similar to how you interact with radio buttons. There's a defined set of options, but you can only select one of them at a time.

Now we can update the original method signature to annotate the previously confusing parameter:

def save_preferred_book_format(format: BookFormat):
    ...

Just by adding this type annotation, we've told the reader of our code that the format parameter has type BookFormat, and so the method expects any of the 4 possible variations: AUDIOBOOK, EBOOK, PAPERBACK, or HARDBACK. We've decreased the size of the set of possible input values from infinity to four, and you no longer need to look beyond the signature of the method to understand how to use it.

save_preferred_book_format(BookFormat.EBOOK)

Not only that, but if we pass an invalid value to the function, our static analyser, editor, or compiler (e.g. tsc, PyCharm, mypy) will tell us about it before we even run our code. If we were to make a typo when calling the method, we'd be told about that before running our code too!

Enums are just one benefit of using a type system. They make code easier to read, save you from having to jump around your codebase to discover how to call functions, and your code becomes far less error prone.

This post is not even close to being an exhaustive exploration of the benefits of refactoring your code to use types, but hopefully it has given some readers food for thought.

A note on Python 3.8

Python 3.8 offers "literal types", which let us specify that our code must be called with a specific literal. We could potentially rewrite our method to inform readers of our code that it will only accept specific strings using literal types as follows:

_FormatType = Literal["AUDIOBOOK", "EBOOK", "PAPERBACK", "HARDBACK"]
...
def save_preferred_book_format(format: _FormatType):
    ...

Now, if we call our code with any literal string that isn't contained within the list, we'll be informed of it before we run it!

Simulating millions of particles to create the Sun

Darren Burns — Wed, 26 Jun 2019 21:18:38 +0000

I recently wrote a blog post about how I was learning to make video games in the Unity game engine. In that post, I built a simple space scene, and added some basic game mechanics. After writing that post, I wanted to try something a little flashier by building a dramatic version of the Sun to sit at the centre of the scene. This project let me dive into the depths of particle simulation in Unity. At first, the simulation was done entirely on the CPU. Later, I discovered the Unity Visual Effect Graph and the incredible performance a GPU is capable of.

My first attempt involved adding a sphere to my scene and wrapping it with an equirectangular projection of the surface of the Sun, just like I'd done with Mars in my previous post. The results were underwhelming. The solar surface is alive and radiating energy, but my sphere looked like a giant tangerine floating through space.

To make things feel a bit more lively, I created a densely packed particle system in Unity, and then created a custom texture using Photoshop using an image of smoke. I created this texture by finding an image of a boat on fire, cropping out a section of the smoke, and adding alpha values to each pixel based on how dark it was, essentially converting darkness into transparency. Here's how it looked with ~1000 particles fading in and out on the surface of the sphere.

The surface of the Sun now has motion, which certainly makes things more interesting than they were before. Unfortunately, that's still not very interesting, and things look a little bit off. I soon realised that the texture I created wasn't actually transparent. Areas of thin smoke weren't rendering with transparency, they were rendering completely opaquely!

The fix took hours to find, with the majority of the time trying to work out how to convert darker pixels to transparency in my image. Working with alpha channels in Photoshop is not intuitive for beginners! After fixing the alpha channels in my image, and figuring out how to import a transparent texture into Unity, here's what things looked like:

Much better! The next thing I tried was to add the corona. The corona is the outer layer of the sun's atmosphere, and it's much dimmer than the surface of the sun. So, I added another layer of particles, but made them dimmer and more orange in colour.

I was really happy with this, but I wanted to take things further. The Sun pictured above contains approximately 1200 particles, all of which are simulated on the CPU. Recently, I'd been reading about the Visual Effect Graph, an upcoming feature in Unity which allows you to simulate particles entirely on the GPU. It turns out the GPU is very good at this task. This project was a great excuse to dive into the VFX Graph and see what it was capable of.

To kick things off, I created a sphere containing hundreds of thousands of particles. Then I started increasing the numbers, eventually simulating 2,000,000 particles at once:

Despite having a huge number of particles rendering on my screen, I was able to fly around the scene and maintain a constant 200 FPS. In order to bring my FPS down to a steady 60, here's what it took (note that the FPS looks lower because my GIF recording software is capped at 15 FPS):

That's an explosion of 14 million individual particles being simulated in real-time.

I wanted to make use of the power of the VFX Graph in order to bring my Sun to life.

I created a spherical volume of particles, and configured the graph in order to have new particle instances spawn on the surface of the sphere. To do this, I created a vector of random numbers, then normalised it so that it had a length of 1. This vector represents a random point on the unit sphere. The sphere representing the Sun in my scene has a radius of 42. To get a random point on this surface, I multiplied the vector representing the random point on the unit sphere by 42. The final result is a random point on the surface of the Sun.

With the ability to spawn miniature particle systems on the surface of my sphere, my next task was to introduce some chaos. I wanted them to look like they belonged on the surface of a star.

The "turbulence" node of the Unity VFX Graph gave me an effect I was happy enough with, and one that I think is beautiful in isolation.

At this point I had constructed a giant hollow sphere of particles, which I hoped wouldn't look out of place when I wrapped them around the Sun I made earlier. I placed the Sun inside this particle ball, then added a script to make the whole thing rotate.

At this point I was happy enough with where I was at, but I realised the motion of particles on and around the Sun were really effective at drawing focus to it. To further draw attention to the Sun, and make things more dramatic, I added a stream of particles which spawn on the surface of the Sun, and travel outwards before fading away.

To finish things off I wanted to add some solar prominence, so I fired up Photoshop and painted another texture. This time I just painted it manually rather than basing it off of a real image. I spawned 50 instances of it such that it wrapped around the Sun, and added so slight variation to the scale of the particles as they spawn. The particles spawn at a random rotation, and slowly rotate around the Sun.

The final result isn't the most realistic, but it's certainly quite dramatic, and I'm pretty happy with it overall.

If I were to continue working on it, I would likely replace the surface of the sun with a shader rather than repeatedly spawning smoke particles, and try to make it appear to "glow" more than it currently does using lighting rather than particles.

Thanks for reading!

Learning to Create Video Games

Darren Burns — Thu, 30 May 2019 13:19:41 +0000

Video games have been a huge part of my life since I unwrapped my first console, the SEGA Dreamcast, on Christmas morning back when I was 5 years old. To me, they are the ultimate combination of storytelling, art, and technology.

The process of developing video games became one of my earliest endeavours when I first started learning to code. Unfortunately, it was a frustrating experience akin to learning to draw without knowing how to hold a pencil, and my motivation quickly faded. At that point in time, my programming knowledge extended about as far as knowing what a for loop was. After a few weeks of trying and failing to understand why almost nothing would work, I threw in the towel, feeling deflated and defeated.

A year later I went to university to study Computer Science and it sparked my passion for programming outside of video game development.

Coding and creativity

Despite being an engineer and the "sciencey" type, I've always been driven by creativity. Until I was about 13 years old I couldn't decide whether I wanted to be an animator or a graphic designer, and couldn't imagine ever becoming a software engineer. How could something as technical as programming be considered enjoyable?

What I realised is that coding is a largely creative process, and one that would allow me to scratch both my technical and artistic itches at once. With coding satisfying my need to express my creativity, I never found myself taking the leap back into video game development.

A few weeks ago I discovered Sebastian Lague's fantastic "Coding Adventures" series on YouTube, where he explores coding concepts in Unity, a popular video game engine. The first video I watched mesmerised me. In it, Sebastian uses a "ray-marching" algorithm to generate a stunning Mandelbulb fractal:

Although there is always creativity to be found in coding, the output of the code I typically write doesn't evoke the same feelings that a piece of art like the one above does, nor does it evoke the joy you get when you play your favourite video game. In my case, the end result is often a value being written to a database, or some data being sent over a network.

I will always achieve some amount of satisfaction from these things, just like I did the first time the words "Hello, World!" popped up in my terminal window. However, when the code you wrote today is similar in purpose to the code you wrote yesterday and the day before that, your vision can narrow, and you might forget that computers are capable of so much more. It's easy to forget that code can create truly beautiful things; art, in a very literal sense.

Making planets

On discovering Sebastian's series, I was inspired to pick up game development where I left off, and to prove to myself I was now capable of doing something which once left me feeling defeated. I downloaded the Unity game engine, and after watching another tutorial from the same creator, decided to try my hand at procedurally generating planet surfaces.

I constructed a spherical mesh (a mesh is a collection of points joined together by triangles which together form a 3-dimensional object) using the scripting API provided by Unity. We can do this by providing Unity with a bunch of 3-dimensional vectors (called vertices) which represent points in the world, and some integers indicating how we wish to join these points together using triangles. With these seemingly primitive tools we can create any 3-dimensional shape imaginable.

To generate the mountains of the planet, I pass in each the location of each vertex to a gradient noise function. Gradient noise has been used in computer graphics for years, for a variety of purposes, and it looks like this:

For each pixel in the image above, imagine that the brighter it is, the greater the altitude of the pixel. So, we can think of areas of pure white as being the apex of a mountain, the grey areas represent somewhere around half way up a mountain, and the darkest areas represent the lowest elevations.

Now lets "wrap" this noise around our sphere like you might wrap a basketball with wrapping paper.

If we were to elevate the brighter points of the sphere away from the centre of the circle, we'd find ourselves with a "lumpy" sphere. We can customise the noise to increase it's density (by "zooming out" of the noise), which would result in a rougher surface. Similarly, we can zoom in to the noise to make the surface of the sphere appear smoother.

At this point, the distribution of the noise over the sphere is entirely uniform. The surface of our "planet", much like Edinburgh, is a bit too consistent when it comes to hills. So, to generate more realistic planetary surfaces, we apply the noise in several layers. One layer will model the sea; another will model land; a third will form hills; and a final rough layer of noise will sculpt the highest peaks of our planet's mountains.

There's a couple of smaller additional tricks involved which Sebastian explains in the series, but the end result was a generator that created relatively realistic planetary surfaces. The image below shows what it looks like when I manipulate some of the parameters relating to how the noise is applied to the surface sphere.

In the end, I never finished all of the tutorials in this series. Paradoxically, my motivation was so high after reaching this point that I didn't want to finish the series there and then. Instead, I wanted to explore as many other areas of Unity as possible, as quickly as I could.

The dance of the planets

Next, I would simulate planetary motion. I wanted to make things a little more realistic this time, so I found myself a space skybox on the Unity asset store. A skybox consists of six images that are placed around the player in a cubic shape, so that the player is completely surrounded by them. In this case, it means that wherever the player looks, it looks like space.

Then, I created a sphere in Unity and applied an image of the surface of Mars to it (from Solar System Scope). The image is called an equirectangular projection, which means it was formed by projecting the spherical surface of Mars on to a flat surface.

I wrote a script and attached it to the planet in order to have it apply a pulling force to things around it, based on the mass of the planet. You know, gravity. I placed a few of these "planets" around the scene, then added a glowing effect around them for no reason other than I had just discovered it was possible and I thought it looked pretty cool.

With the scene set, I sat back, and prepared for the elegance that is the dance of the planets.

Here's what happened...

Then I remembered the fundamental law of gravity which states that when the apple drops from the tree it falls to the ground. The apple does not drop from the tree and immediately begin orbiting the planet. With the realisation that this dance was far too dependent on the choreographer's physics knowledge, I decided to beam off to my next adventure.

Interactivity

Both of these experiments were really enjoyable and gave me a feel for working with Unity, but neither were "games". Without interactivity, they're really just simulations.

Despite the fact that it was basically an advert for why NASA should never hire me, I enjoyed the "feel" of the previous scene. I decided to keep things in space. This also meant that I didn't have to fill my scene with assets in order to achieve a sense of realism. After all, space is mostly empty.

I wanted to build in some mechanics from real-time strategy (RTS) games. Although I didn't have a goal in mind for where I wanted my game to go at the time, implementing interesting game mechanics is great experience whether you end up using them or not. I'd learn the concepts behind creating the mechanics, and find out what works, and what doesn't work.

I started by creating a cube which would represent a space ship. Then I added the ability to click the cube in order to select it. This involved learning about raycasting, where you fire off a ray in a direction, and check what it collides with. In order to implement unit selection, I had to send a ray out from the camera, through the cursor position, and detect whether the ray collided with my cube. Then, with the cube selected, I made it so you could right click to have the cube travel to a point in space after rotating to face the correct direction.

After getting movement down, I upgraded my cube to a cool ship asset I found on the Unity Asset Store, and added the ability to pan, zoom and rotate the camera. It took me almost two hours to figure out how to make my ship not fly with its nose pointing down. After some research I found that this was happening because the 3D modelling software Blender uses a different coordinate system from Unity. The axis that the ship model perceived to be "forward", was Unity's idea of "down". To fix the problem, I placed the ship model inside an empty Unity GameObject, adjusted it so that it was facing forward (for real this time), and applied all future rotations to the parent instead of to the ship directly.

I wrote a small script called Rotator to make planets slowly rotate by slightly increasing the rotation on every physics update, which made the scene feel less static.

public class Rotator : MonoBehaviour {
    void FixedUpdate() {
        transform.Rotate(Vector3.up, .01f);
    }
}

Box selection

Most top down strategy games allow you to select multiple units at once by dragging a rectangle around them. I used the skills I learned from the planet generation tutorial to dynamically construct a plane as you click and drag the mouse. I applied a shader to the mesh that is generated to make it slightly colourful, and a little transparent. A shader is a concept in computer graphics which lets you apply special effects to an object using your GPU, making them really efficient.

Shaders are usually written using a special scripting language, which looks a bit daunting to learn. However, In Unity, you can build shaders using an amazing visual tool called Shader Graph, which is so much fun to use and something I definitely want to dive deeper into in the future.

After implementing this, I realised that performing the selection in 3-dimensional space can be a little tricky, especially if you have multiple units on screen and if the camera has been rotated at all. It can be hard for the brain to quickly understand the rotation the camera is currently at, and to know the angles that the edges of the box will appear at when you start dragging.

In the future, I may update my rectangular selection code so that you draw the rectangle in 2-dimensional space, and project it onto the 3-dimensional space of the game world. This is how it works in games like StarCraft 2, and it definitely feels more intuitive.

Right now it's also not clear which units you currently have selected. In the future I'll decide how I want to convey this information to the player. It's a tricky problem, especially with the potential for many units to appear on screen at once.

Combat

A game with space ships obviously requires that they have the ability to unleash a barrage of missiles at each other! I wrote a script called BasicAttack and attached it to my ship prefab. This script defines things like "how much damage do my attacks do". At first, attacks were performed using raycasting. An invisible ray was fired out from the selected ship towards the direction of the mouse click. If the ray collided with another ship, it immediately deducted the attack damage of the attacking ship from the hitpoints of the ship that took damage. If the hitpoints of a ship falls to zero or below, it gets immediately removed from the world.

Without any visual feedback, this made for a rather dull experience. So I added a few more features in order to liven things up a little.

I attached a LineRenderer (used to draw lines, and comes with Unity) component to my ship prefab. A prefab in Unity is similar to a class in programming. It's defines a blue print for what an instance of it will look like, and can easily be reused anywhere. The start point of the line is set to be the centre of the selected ship, and the endpoint is set to the position of the cursor. If a ship is selected, and the user presses A, the ship will enter "Attack Mode", and the LineRenderer will become visible on screen. I created a shader using Shader Graph and applied it to the line to make it slowly fade in and out over time, just to make things a little more visually interesting.

Then, I updated the BasicAttack code to spawn a missile instead of casting an invisible ray. The missile is just a capsule, with a TrailRenderer (built in to Unity) attached to it, which results in a trail of light which follows the missile as it travels towards its target. For now, the missiles always travel at a speed such that it takes 3 seconds to reach their target. After 3 seconds, the target takes damage. This isn't the most realistic approach, but it works.

To let the player know how much damage the target has taken, I implemented (after hours of headaches) floating damage numbers above the target. I absolutely love this effect in video games. When a target takes damage, the amount of damage it's taken briefly appears above the unit. The size of the font also dynamically scales based on the damage taken.

In my implementation, for damage numbers between 1 and 50, the standard font size is used (it results in a fontScaleFactor of 1). As we start doing more than 50 damage, the font size used to display the damage numbers increases in size, up to a maximum damage of 300. If an attack applies 300 or more damage, the font size will be double the standard size.

var fontScaleFactor = 1 + Mathf.InverseLerp(50, 300, damage);
damageText.fontSize *= fontScaleFactor;

Since a ship can take multiple damage instances in quick succession, I added some random variation to the position these damage numbers spawn at. Without this variation, the numbers overlap perfectly, and it isn't possible to determine how many instances of damage the ship has received. Here's the function I wrote to apply the randomness to the spawn position of the floating damage numbers:

private Vector3 ApplyRandomOffsets(Vector3 vec, float minOffset, float maxOffset) {
    var xOff = Random.Range(minOffset, maxOffset);
    var yOff = Random.Range(minOffset, maxOffset);
    var zOff = Random.Range(minOffset, maxOffset);
    return new Vector3(vec.x + xOff, vec.y + yOff, vec.z + zOff);
}

With these changes, things became a little bit more visually exciting:

Explosions

To give the player more visual feedback that that their attacks do damage, I added explosions that trigger when the missiles reach their target. Explosions in Unity are usually achieved using the Unity Particle System, which lets you create amazing physics based particle effects. Rather than attempting to create my own textures and particle systems, I used some of the particle systems available in the Unity Particle Pack.

My initial attempts at making things explode didn't really go to plan.

After some messing around, I upgraded my weird pink box shapes and arrived at something which looked a bit like a firework. Not exactly the high impact effect I was looking for, but certainly a step in the right direction.

A bit of Googling and some more messing around with settings, and I finally realised the cause of my problems was my use of Unity's newish Lightweight Render Pipeline (LWRP). The shaders used by the explosion weren't supported. After updating them to ones that are supported by the LWRP, the particle system actually started looking like a realistic explosion:

When the missiles reach their targets, the target will play an explosion effect. As an extra challenge, I gave each ship "shields", which is an additional layer of hitpoints. If a missile strikes a ship and it still has shields, the explosion is blue. After a ship runs out of shield energy, the explosions become orange as a visual cue to the player that the ship is low on total health.

I still have lots of ideas about how I could improve the effects here. Randomly varying the missile strike locations, and making the explosion effects spawn at those locations will add to the realism. Instead of hard-coding the explosion to appear at the centre of the target ship 3 seconds after the missile is fired, moving to a collision based system will allow the missile to explode immediately upon striking its target. A blue explosion isn't how "shield" effects are typically portrayed in films or video games either, so a shader which simulates a "force-field" effect might work better.

Moving forward

There's still lots to learn, but I'm really happy with my progress over the course of the past couple of weeks. Although there's no real gameplay in the project beyond "fly around and attack ships", it was an incredible learning experience and one that still makes me feel like I accomplished my goal of learning to create video games.

Going forward, I'd like to learn how to add a UI to the project above, and perhaps implement some interesting gameplay systems. I have a bunch of notes on how gameplay might work, and some ideas I want to experiment with. I may also move onto other projects, such as implementing interesting game mechanics I've experienced from other games, like the portal gun from Portal, or the Leviathan Axe from God of War.

Either way, learning the basics of Unity has opened up a whole new world of creativity for me. There's now so many things I now want to explore and learn to create, and I now have the confidence in myself that I'll be able to succeed at learning them!

In the future I might create more development/learning logs like these discussing the things I've built, and what I learned in the process. If you're interested, follow me on DEV, Twitter, and check out my personal blog at darrenburns.net.

Making Mistakes

Darren Burns — Sat, 25 May 2019 13:27:01 +0000

If you ever struggle with learning to code, I want you to know that my code to generate a CSV file of user IDs has produced the output in the image below. I've been coding for 8 years, and it's one of the many mistakes I've made this week.

When I started coding I had a constant feeling of having no idea what I was doing.

I still have no idea what I'm doing, and make mistakes daily.

Long ago, these would make me frustrated and angry with myself. As I gain more experience, my attitude towards mistakes evolves.

Some of my current thoughts on making mistakes are:

Mistakes are human, and are always an opportunity to improve
Every programmer makes mistakes, even the really famous ones, and even if they hide it well
My team is compassionate and will help me rather than shame me for slipping up

I can't stress enough how important the last point is. It's so important to find a team where you feel psychologically safe. In my current team a mistake often feels like an opportunity to collaborate, learn, and improve our process. It's something to reflect on without shame.

Some teams turn the very human act of making a mistake into exercise in assigning blame. Every line of code is another potential slip up, another layer of anxiety, and another step towards burnout.

This is just one aspect of a toxic company culture and it will have a negative impact on your mental well-being.

For this reason it's important not to be afraid to ask difficult questions about company culture and attitude towards mental health at interviews. If a potential employer isn't receptive to questions on these subjects, you probably don't want to work with them.

5 CLI Utilities to Boost Your Productivity

Darren Burns — Tue, 07 May 2019 13:43:51 +0000

Here are 5 command line tools I've found recently which can speed up your workflow.

`fx`, a command-line JSON processing tool

fx (GitHub) is a CLI tool written in JavaScript which allows you to process JSON using your terminal.

You can explore and modify JSON using small snippets of plain JavaScript, and use your cursor to dive into the structure. For all of the functionality offered by fx, check out the documentation.

Installing `fx`

brew install fx

`http-prompt`, for interacting with APIs

http-prompt (website) is useful for exploring and interacting with HTTP APIs. It comes with autocomplete and syntax highlighting.

http-prompt is written in Python, and builds on top of HTTPie, which was covered earlier in this series.

Installing `http-prompt`

pip install --user http-prompt

`fselect`, for querying files with an SQL-like syntax

fselect (GitHub) is an alternative way to search your filesystem. It lets you use a syntax similar to SQL to find what you're looking for. For example, to find the size and path of all .cfg and .tmp files in /home/user/:

fselect size, path from /home/user where name = '*.cfg' or name = '*.tmp'

fselect also supports aggregation functions, similar to those you find in SQL:

fselect "MIN(size), MAX(size), AVG(size), SUM(size), COUNT(*) from /home/user/Downloads"

The query above will find the smallest file size, the largest file size, the average file size, the total file size, and the number of files present in your Downloads folder.

fselect is written in Rust, and it can do much more than the examples above suggest. It has extensive documentation.

Installing `fselect`

brew install fselect

`ranger`, a command-line file manager

ranger (GitHub) is a command-line file manager written in Python that lets you browse and manipulate your file system using Vim like keybindings.

It offers a multi-column display, the ability to preview files, and lets you perform common file operations (such as creation, deletion, chmod, copying, etc.) from within the ranger interface.

You can extend ranger by installing some Python packages which allow it to preview images, HTML documents, and PDF files. The image below is an example posted on the gallery available on the Ranger website, showing the image previewing functionality in action.

Ranger is a deep piece of software with countless features that don't fit within the scope of this post. More information can be found in the official user guide.

Installing `ranger`

brew install ranger

`tokei`, to view statistics on your code

tokei (GitHub) lets you view code statistics for your projects by breaking down the languages you've used. It's written in Rust, which helps make it very fast.

Here's some example output from running tokei in my Advent of Code folder:

-------------------------------------------------------------------------------
 Language            Files        Lines         Code     Comments       Blanks
-------------------------------------------------------------------------------
 Markdown                1          165          165            0            0
 Rust                   13         1071          854           51          166
 Plain Text             17         4032         4032            0            0
 TOML                    1           11           10            0            1
-------------------------------------------------------------------------------
 Total                  32         5279         5061           51          167
-------------------------------------------------------------------------------

Installing `tokei`

brew install tokei

Conclusion

Thanks for reading! Hopefully, you found something on this page that interests you. If you’re interested in more content like this, follow me on Twitter and on DEV!

The Piece Table - the Unsung Hero of Your Text Editor

Darren Burns — Thu, 24 Jan 2019 00:32:59 +0000

The piece table is the unsung hero data-structure that is responsible for much of the functionality and performance characteristics we’ve come to expect from a text editor. Visual Studio Code has one. Microsoft Word 2.0 had one back in 1984.

Despite being an almost ubiquitous feature of modern text editors, it remains relatively poorly documented. In this article, I’ll explain how a piece table is structured, why they’re used, and how your text editor interacts with it as you edit files.

My aim is to make this post as beginner-friendly as possible, so we’ll explore each concept slowly. You should, however, have a basic understanding of arrays, strings, and objects (or dicts/structs) before continuing.

When you open a file in your text editor, the contents are read from disk and into a data structure in memory. If you were to write a text editor, how would you store an open file in memory?

First instincts - the array of strings

Our first instinct might be to use an array of strings, where each string is a single line in the file. In other words, we’d represent this file…



the quick brown fox
jumped over the lazy dog

…like this:



lines = [
    "the quick brown fox",       # line 1 of the file
    "jumped over the lazy dog",  # line 2 of the file
]

This is an intuitive way to represent a text file in memory. It lets us think about the file in a way that is similar to how it appears on-screen in our text editor.

It’s also a perfectly acceptable approach, and you could argue that the intuitiveness of it outweighs any potential flaws it may have. In fact, Visual Studio Code used a similar model until early 2018.

Unfortunately, this method becomes costly when we’re dealing with large files. To see why, think about what would happen if someone were to insert a new line ("went to the park and") into the middle of the file:



the quick brown fox
went to the park and
jumped over the lazy dog

In order to make room for this new line, all of the lines below it in the array need to be shifted along in memory. For large files, this quickly becomes expensive. As the file grows larger, there’s more data to be moved.

This is just one of the downfalls of this method. In this fantastic blog post by the Visual Studio Code team, they highlight other flaws with this approach such as excessive memory usage and performance issues relating to splitting up the file into multiple strings on newline characters.

An append-only representation

If we append something to the end of an array, we don’t have to shift any data along to make space for it, so we don’t incur the same performance penalty as in the case where we insert into the middle of an array (more technically, appending to an array has O(1) time complexity, but insertion is O(n)).

The piece table is the powerful data-structure behind many text editors, old and new. A key characteristic of the piece table is that it records all of the insertions we make to a file in an append-only manner.

Let’s explore how the piece table works.

When we read a file from disk into a piece table, the text inside the file is stored in a single string which we never modify. We call this string the original buffer.



piece_table = {
    "original": "the quick brown fox\njumped over the lazy dog",
    ...
}

When we add text to the file in our editor, that text is appended to the add buffer of the piece table, which is initially just an empty string.

Adding a copyright notice at the top of the file? Append it to the add buffer. A new function in the middle of the file? Add buffer. An extra newline at the end of the file? Add buffer!

The add buffer is the second of the two buffers in which make up a piece table, and it is append-only.



{
    "original": "the quick brown fox\njumped over the lazy dog",
    "add": "",
  ...
}

By appending everything that gets inserted into the file onto the add buffer, we record the text that the user has typed into the editor, whilst avoiding those pesky mid-array insertions we mentioned earlier.

Let’s open our hypothetical text editor again, and add a line into the middle of our file as we did before. That makes our piece table look like this:



{
    "original": "the quick brown fox\njumped over the lazy dog",
    "add": "went to the park and\n",
  ...
}

The text we added to the middle of the file is in the add buffer. The original buffer has stayed the same (and always will).

These two strings, original and add, hold within them the entire contents of the file that is open in your editor, as well as everything that has ever been contained in the file since it was opened.

When the editor displays an open file, it combines different sections of these two strings together to form the what you see on your screen. Some sections of these strings may be ignored if they’re no longer present in the file (for example if the user of the text editor deletes some text).

In the text below, the middle section comes from the add buffer since it was inserted, and the rest of the text comes from the original buffer since it was part of the original file.

Right now, our hypothetical editor knows the user has inserted the string "went to the park and\n" into the file, but not where they inserted it. We haven’t yet got enough information in the piece table for the editor to be able to correctly display the contents of the file. The missing piece of the puzzle (pun intended) is to track where in the document the user inserted the text.

Piece descriptors

In order to know where the user inserted the text in the file, the piece table needs to track which sections of the file come from the original buffer, and which sections come from the add buffer. It does this by iterating through a list of piece descriptors. A piece descriptor contains three bits of information:

source: tells us which buffer to read from.
start: tells us which index in that buffer to start reading from.
length: tells us how many characters to read from that buffer.

When we first open a file in our editor, only the original buffer has content, and there’s a single piece descriptor which tells our editor to read entirely from the original buffer. The add buffer is empty because we haven’t yet added any text to our file.



{
    "original": "the quick brown fox\njumped over the lazy dog",
    "add": "",
  "pieces": [Piece(start=0, length=44, source="original")],
}

Adding text to a file

Now let’s add the same text to the middle of our file just like before. Here’s how the piece table is updated to reflect this:



{
    "original": "the quick brown fox\njumped over the lazy dog",
    "add": "went to the park and\n",
    "pieces": [
        Piece(start=0, length=20, source="original"),
        Piece(start=0, length=21, source="add"),
        Piece(start=20, length=24, source="original"),
    ],
}

The text we inserted in our file using our text editor has been appended to the add buffer, and what was previously a single piece is now three.

The first piece in the list now tells us that only the first 20 characters of the original buffer (the quick brown fox\n) make up the first span of text in our file (note that \n, representing a line break, is a single character).
The second piece tells us that the next 21 characters of the file can be found between indices 0 and 21 in the add buffer (went to the park and\n).
The third and final piece tells us that the final span of text in the file can be found between indices 20 and 44 (start + length) of the original buffer (jumped over the lazy dog).

The act of inserting text into a file typically results in splitting a piece into three separate pieces:

One piece to point to the text that falls to the left of the newly inserted text.
Another piece to refer to the inserted text itself (in the add buffer).
The third piece refers to the text that got pushed to the right of the newly inserted text.

Things are a little different when you insert text at the beginning or end of an existing piece. In this case, adding text doesn’t “split” an existing piece in half, so we only need a single extra piece to represent the newly inserted text.

Saving and displaying the open file

As mentioned at the very beginning of this post, when we open a file in a text editor, it gets read from disk and stored inside a data structure in memory (very likely a piece table or some variant). Likewise, when we do the inverse operation and save the file, our editor needs to be able to read the piece table and write the contents back to the file on disk.

By reading the piece descriptors in consecutive order, our text editor can transform its internal representation of the file in the piece table into what you see on your screen, and what gets written to the file when you save it.



for piece in piece_table["pieces"]:
    source: str = piece.source  # "original" or "add"
    buffer: str = piece_table[source]
    span_of_text: str = buffer[piece.start:piece.start+piece.length]
    print(span_of_text)  # alternatively, write to a file

Note: In Python, the string[start:end] syntax can be used to return a substring from a string. For example, “hello”[1:3] returns "el".

Deleting text

When we delete some text from a file, we split an existing piece into two pieces:

One piece points to the text to the left of the deleted text.
A second piece points to the text to the right of the deleted text.

The deleted text will still be present in one of our buffers, but with no pieces referring to it, it is not considered part of our file.

You can think of piece descriptors like spotlights pointed at a wall with all of the text ever written to a file painted on it. At any point in time, these lights illuminate sections of the wall, revealing the text there. The illuminated text is considered as part of the file. Although we know there’s more written on the wall, it’s hidden in the darkness and considered irrelevant.

Undoing and redoing

One benefit of keeping all of that text around even when it’s not currently part of the file is that it makes the implementation of undo/redo considerably easier.

Text that is currently “in darkness” may be required again in the future. Wouldn’t it great if we could just adjust our lighting to illuminate that text once again, instead of having to repaint the wall?

In the array of strings approach discussed at the start of this post, an undo operation could result in millions of characters of text being inserted or deleted across millions of different strings. With the piece table method, the cost of this action becomes trivial. The text to be added or removed as part of an undo or redo is already there in one of the buffers, and we just need to update our piece descriptors to point to it once again.

Conclusion

There are several ways we could improve the piece table described above. They’re often combined with other data structures such as trees in order to improve aspects of their performance.

That said, my goal for this post was to leave you with intuition and some appreciation for how your text editor works internally, and those tweaks fall out with that goal.

If you have any constructive feedback on this post, or if you’ve noticed any errors in it, let me know!

If you’re interested in more content like this, follow me on Twitter, DEV, and check out my blog!

References/Further Reading

Data Structures For Text Sequences (PDF) - Charles Crowley
What’s been wrought using the Piece Table? - David Lu (I think)
Text Buffer Reimplementation, a Visual Studio Code Story - Peng Lyu

8 Productivity Tips for GitHub

Darren Burns — Wed, 09 Jan 2019 08:39:47 +0000

With the recent announcement of unlimited private repositories, let's take a few minutes before we push up the code we don’t want anyone else to see, and make sure we’re making the most of what GitHub has to offer.

GitHub is built with some extremely helpful shortcuts and productivity-boosting features. From personal experience, however, it’s clear that these often fall under the radar amongst developers. If I’ve ever witnessed a specific GitHub feature surprise or assist someone, then that feature is on this page. That said, what follows is by no means an exhaustive list.

Quick fuzzy file search in repositories

This is, without doubt, the fastest way to browse a repository when you know what you’re looking for. Open up any repository and press t. You can now search for the name of any file in the repository, and use the arrow keys on your keyboard to move through the results. Press Enter to open the file.

Code change suggestions in pull requests

When commenting on a piece of code in a pull request, you can suggest alternative code using the “Suggested Changes” feature. The author of the pull request will be able to apply your suggestion instantly without leaving GitHub.

To make the suggestion, surround a code snippet with a multiline Markdown snippet, with the tag suggestion:

Now that you’ve made the suggestion, the author of the pull request can immediately apply it to their branch, without the hassle of manually changing the file!

Navigate the code tree like in an IDE

This one requires an unofficial Chrome extension, but it’s a slightly more familiar way to navigate your code compared to the default interface. The Octotree extension lets you browse GitHub repositories with a sidebar tree view similar to what you get in applications like VS Code.

Jump to a function when reviewing code

Unless you’re reviewing a single function, a code review often involves a lot of jumping between function calls and their definitions (and therefore a lot of scrolling up and down). GitHub lets you jump to a symbol by pressing t when you’re looking at files in a pull request.

Creating a permalink to a file

When viewing a file or directory, press y, and the URL will be converted to a permalink, which you can share knowing that the contents of the file will never change.

If you send a link to a file or directory on GitHub without making it into a permalink, you’ll need to accept the possibility that the file could disappear tomorrow, breaking the link!

Viewing the blame and change recency heatmap

When viewing a file, you can press b to view the Git blame and a heatmap showing how recently each line was changed. It’ll tell you who most recently changed each line of code, and give you a clickable link taking you to the full commit the change was part of.

On the right-hand side of the gutter (which contains the commit message and author), you’ll notice an orange vertical bar. The more vivid this bar is, the more recent the change, meaning you can easily scan the file to find the freshest code!

Powerful code search

GitHub indexes most code and offers powerful search functionality over it. If you need to find something in a repository, but don’t plan on making any changes to it. There’s usually no need to check the repository out. Press / to search all the code in the repository.

If your search contains multiple words and you want to search for occurrences of your specific search query, put the quotations around the query. You can filter your searches by other things too, such as file size, extension, the path the file is on, and much more.

Saved replies

If you ever find yourself repeating the same comments, you’ll save some time by creating a saved reply. The next time you find you’re going to type that comment again, you can instead just select it from a drop-down menu:

To perform the above action without using my mouse, I can do ctrl + . followed by ctrl + 1.

Conclusion

Thanks for reading. I hope you found at least one thing on this page that will make you a more productive GitHub user. If you enjoyed this post or have any feedback in general, let me know!

If you’re interested in more content like this, follow my Twitter.

Originally published on my blog.

P.S. You can make your own Octocat for sharing like the one in the cover photo at myoctocat.com!

Power Up Your Command Line, Part 3

Darren Burns — Mon, 07 Jan 2019 06:31:47 +0000

This is part three of the Power Up Your Command Line series. In this article, I’ll showcase five utilities that make common problems a little easier to solve using the command line.

This post was originally published on my blog.

`tig`, for interactively browsing your git repo

tig (GitHub) is an excellent way to browse your Git repository using an interactive interface, without leaving the command line.

tig is simple and intuitive to use, and has different views for things like your stash, staging area (which it lets you quickly alter), logs, and so on.

Thanks for Renato Suero (@renatosuero) for introducing me to tig on DEV.

Installing `tig`

On macOS with (Homebrew): brew install tig

PathPicker (`fpp`), for quickly selecting files

PathPicker (GitHub) is a library by Facebook for quickly selecting files on the command line. The animated example below is taken from the PathPicker documentation.

From the PathPicker website:

PathPicker accepts a wide range of input -- output from git commands, grep results, searches -- pretty much anything. After parsing the input, PathPicker presents you with a nice UI to select which files you're interested in. After that, you can open them in your favorite editor or execute arbitrary commands.

Thanks to Nikolay Dubina (@nikolayid) for suggesting this utility.

Installing PathPicker

On macOS with (Homebrew): brew install fpp

`tldr`, for practical examples of how to use CLI tools

tldr (GitHub) lets you quickly access practical examples (simplified, “tl;dr” versions of man pages) of how to use command line tools.

The examples are maintained by the community in the tldr GitHub repository.

Installing `tldr`

With npm (recommended): npm install -g tldr
On macOS with (Homebrew): brew install tldr

`gron`, for exploring JSON

gron (GitHub) transforms JSON text into discrete assignments, to make it easier to find what you need. I particularly like using it for interactively exploring APIs in combination with fzf (mentioned earlier in this series):

You can also use gron to help you transform JSON objects (examples). This isn’t the main use-case of gron though, and you’d probably be better served using a purpose built tool like jq for such things.

Installing `gron`

On macOS with (Homebrew): brew install gron

thefuck, for quickly correcting command line typos

If you mistype a command, type fuck (GitHub) and you’ll be given a list of corrections to choose from.

The name of the command is unfortunately slightly NSFW, so you may wish to alias it to something else.

Installing thefuck

On macOS with (Homebrew): brew install thefuck

Bonus: explainshell, for explaining shell commands

If you’re in the situation where you already have a complex shell command and want to understand what it does without browsing man or tldr pages, you can use explainshell:

Conclusion

Hopefully you found something on this page that interests you! If you’re interested in more content like this, follow me on Twitter and on DEV.

5 More Tools To Power Up Your Command Line (Part 2 Of Series)

Darren Burns — Fri, 04 Jan 2019 17:03:02 +0000

This is part II of the series “Power Up Your Command Line”, which showcases excellent non-standard tools that will make using the command line easier and more enjoyable.

`peco`, interactively filter anything

Pipe the output of any command into peco (GitHub), and you’ll be able to interactively filter it to find the information you need. Think of it like an interactive form of grep, where the real-time updating of results as you type allows for a more exploratory search process.

You can move up and down through the search results using the arrow keys, and select a result with the enter key. When you press enter, the command outputs the result to your terminal.

Installing `peco`

On macOS with Homebrew: brew install peco

`hexyl`, a hex viewer

If you want to examine a binary file, you typically do so by looking at the hexadecimal representation of the file. hexyl (GitHub) is a command line tool for doing just that.

The view is split into three columns:

An offset column, which tells you how many bytes into the file you are.
A hex column (which itself has a divider down the middle - not visible in the example gifs), which contains the hexadecimal representation of the file.
A representation column (also split in two, and not shown in the example gifs), which contains a textual representation of the file.

Each byte in hexyl is colour-coded depending on the type of byte it is (NULL, ASCII, non-ASCII, etc.) which really helps with readability.

Bonus tip: If viewing binary files or large text files, the output will often be too large for your screen, so you can pipe the output from hexyl into bat or less to give you the ability to paginate through it! You’ll need to use the --raw-control-chars/-r flag for the output to be coloured correctly if you use less:

hexyl is written in Rust, and is developed by David Peter who also wrote bat, fd, and hyperfine, all of which I mentioned in part one of this series.

.ltag__user__id__95033 .follow-action-button { background-color: #61122F !important; color: #FFFFFF !important; border-color: #61122F !important; }

David Peter

sharkdp https://david-peter.de

Installing `hexyl`

Installing hexyl requires a few steps on macOS:

Download the binary from GitHub: curl -L -o hexyl.tar.gz https://github.com/sharkdp/hexyl/releases/download/v0.3.0/hexyl-v0.3.0-x86_64-apple-darwin.tar.gz
Extract the tarball: tar -xf hexyl.tar.gz
Add it to your PATH: mv hexyl-v0.3.0-x86_64-apple-darwin/hexyl /usr/local/bin
Make executable: chmod +x /usr/local/bin/hexyl

`pomo`, a pomodoro timer

The pomodoro technique is a great way to boost your productivity. If you haven’t heard of it before, it works like this:

For 25 minutes, you work and allow for no distractions.
For 5 minutes, you relax and do whatever you want (as long as it isn’t work 😁).
Repeat the steps above 4 times (you can adjust this number to suit yourself), then rest for 15 minutes

If you follow the plan above, the theory is that you’ll get considerably more done in considerably less time. It might not work for everyone, but I can personally vouch for its effectiveness!

pomo (GitHub) is a simple, configurable CLI tool to help you manage your time through the pomodoro technique.

Installing `pomo`

There are a few steps to follow to install this one (on macOS):

Download the binary from GitHub: curl -L -o pomo https://github.com/kevinschoon/pomo/releases/download/0.6.0/pomo-0.6.0-darwin-amd64
Grant permissions: chmod +x pomo
Put it on your PATH so you can use it from anywhere: mv pomo /usr/local/bin
Initialise the database: pomo init

`ncdu`, analyse and free disk space

If you’ve got a lot of projects on your computer and haven’t cleaned up your disk recently, you’ll almost certainly find a rogue folder that is hogging disk space (I’ve found some node_modules folders in older projects to be particularly susceptible). ncdu is my favourite tool for fixing this. In fact, in the process of creating the demonstration for ncdu below, I freed up over 10GiB of disk space!

To use it, run ncdu. It’ll scan all subdirectories from the directory you ran it in, so if you run it in your home directory, it might take some time.

You can use the interactive interface to browse through your filesystem using the arrow keys, or the navigation keys from vim if that’s your thing.

Installing `ncdu`

On macOS with Homebrew: brew install ncdu

HTTPie, a modern replacement for `curl`

HTTPie (http) is a simpler (meaning I don’t have to Google it every time, unlike with curl), feature-rich and beautiful alternative to curl, which will let you call APIs over HTTP from the command line. It’s the most popular utility I’ve reviewed so far and has exceptional documentation.

The output from the http command is what really sets it apart from cURL. JSON responses are pretty printed with syntax highlighting, making them far more readable. If you'd rather use a graphical UI for interacting with APIs, you might like Insomnia, Postman or Paw (Paw requires a license and is macOS only).

Installing HTTPie

On macOS with Homebrew: brew install httpie

Conclusion

Thanks for reading! There are a couple more tools I’d like to mention in the next post in the series, but if you have any suggestions let me know in the comments. If you’re interested in more content like this, follow me on Twitter and on DEV.

Originally posted on my blog.

Debug Faster With Colourful Diffs For Your Failing Tests

Darren Burns — Fri, 04 Jan 2019 01:19:53 +0000

Pytest is quickly becoming the de facto testing framework in the Python community, but I've always found the way it reports assertion errors to rather difficult to quickly parse in many circumstances. In this post, I'll explore how we can use a lesser known part of the Python standard library to build a pytest plugin that outputs colourful diffs similar to those seen in testing frameworks for languages such as Elixir and JavaScript. This isn't a tutorial, but it should give you an overview of the following topics:

Getting started with making a pytest plugin
How to use the Python standard library to calculate diffs between text
A brief description of how to use partial application in Python
Formatting terminal output with escape sequences (programming language agnostic)

Here's what a failing test looks like with pytest-clarity (GitHub) installed:

From this output, it's easy to see at a glance why your assertion has failed. Your eyes are immediately drawn to the differences between the dictionaries, and the colours inform you what would need to be added or removed from either side of the assertion in order for them to match.

If you want to try the plugin, but don't want to read about my experience implementing it, you can install it with:

pip install pytest-clarity

After running this command, you can run your tests as normal and you'll get the coloured output! If you have any issues or want to uninstall for whatever reason, you can do so by running pip uninstall pytest-clarity.

For comparison, I've attached the output for the same test using vanilla pytest below (with very verbose logging -vv enabled).

In my opinion there are many things wrong with this output, but the most glaring issue is that the it distracts from the actual content of the objects under comparison. The representations of these objects are interspersed with +, -, ?, and ^ symbols, making it overly difficult to answer the question "what do the objects I'm comparing actually look like, and how do they differ?"

Modifying Pytest Output With A Plugin

Pytest has a powerful plugin system, and an active ecosystem of useful plugins. The easiest way to get started with plugin development is to use @hackebrot's Cookiecutter template for pytest plugins.

After we've created the project using the Cookiecutter template, we need to find the hook that will enable us to customise how assertion errors are reported.

Pytest provides a number of hooks, allowing us to add to or customise various aspects of its functionality, but in this case we need pytest_assertrepr_compare. By creating a function called pytest_assertrepr_compare inside the pytest_{yourpluginname}.py file created by Cookiecutter, we can override the default output pytest prints to the terminal when an assertion fails.

The hook has the following signature (type annotations are my own):

def pytest_assertrepr_compare(
    config: _pytest.config.Config,
    op: str,
    left: Any,
    right: Any,
) -> List[str]

Note: Unfortunately pytest doesn't appear to pass detailed assertion inspection information into this hook, meaning we can't take full advantage of assertion rewriting unless we inspect the AST ourselves.

The first parameter is a pytest Config object, which we don't need to worry about. The op parameter refers to the operation used in the assert statement in the test. For example, if your assertion looks like assert 1 == 2, then op would be "equal". left and right refer to the values that appear on the left and right hand side of the op. In the preceding example, left would be 1 and right would be 2. The function returns a list of strings, and each of these strings correspond to a single line of output that pytest will write to the terminal in the event of a failing assertion.

A Short `assertrepr_compare` Plugin Example

Here's a minimal example of using the assertrepr_compare hook, which just prints the left and right operands of the assert statement, as well as the operator used to the terminal:

# Basic implementation of the assertrepr_compare hook
def pytest_assertrepr_compare(config, op, left, right):
    return [
        '',  # newline because it looks strange without
        'op: {}'.format(op),
        'left: {}'.format(repr(left)),
        'right: {}'.format(repr(right)),
    ]

# We'll run this test, and check the output
def test_one_equals_two():
    assert 1 == 2

The above implementation of the hook results in the following output after running the test:

___________________ test_compare_strings ___________________

    def test_compare_strings():
>       assert 1 == 2
E       assert
E         op: ==
E         left: 1
E         right: 2

tests/test_util.py:64: AssertionError

We now have all the information we need to alter pytest's output into whatever format we wish. Let's look at how pytest currently calculates the diffs it shows the user when an assertion fails.

How Pytest Calculates Diffs

pytest-clarity will rely heavily on difflib, which is included in the Python standard library. difflib provides classes and helper functions for comparing sequences, and computing deltas (diffs) between these sequences. As it turns out, pytest also uses difflib to display the output you see when your assertion is false. It does so using the ndiff helper function from difflib. This function returns a generator which yields strings, each of which corresponds to one line of the delta output.

import difflib
lhs = "hello"
rhs = "world"
delta = difflib.ndiff([lhs], [rhs])
print("\n".join(delta))

And here's the output, which is identical to the output you'd see in pytest if you were to write assert "hello" == "world":

- hello
+ world

The problem with this approach is that it tightly couples the semantics and the presentation of the delta it generates. It expects that we'll directly output the strings it yields. It'd be nice if we could grab a data structure other than a string which represents the diff itself, and which we can use to generate the colourful output.

Luckily, difflib has the answer!

Enter The SequenceMatcher

The SequenceMatcher class is part of difflib, and it provides us with a means of comparing pairs of hashable sequences A and B. We can use it to find the exact index of every element in A that would have to be replaced, deleted, or inserted, in order to transform A into B. This is great, because we can now access an abstract representation of a diff, and we can present it however we desire (colours, everywhere).

Before proceeding, lets look at how to understand and work with the SequenceMatcher. The method we're interested in is called get_opcodes. This method returns a list of 5-tuples which describe how to transform A into B. The Python difflib documentation has a solid explanation of it, but the code snippet below should give a rough idea of how it works.

import difflib
matcher = difflib.SequenceMatcher(None, "hello", "world")
for tag, i1, i2, j1, j2 in matcher.get_opcodes():
    # tag can be one of 'replace', 'delete', 
    # 'insert', or 'equal', and represents an operation
    # to be performed in order to transform the 
    # left string into the right string.
    # i1:i2 represents a slice of the left string, 
    # j1:j2 a slice of the right string.
    # i1:i2 and j1:j2 are the ranges within the strings that 
    # the operation should be performed on

The get_opcodes method gives us the information we need to determine which colour to write the output to the terminal in.

Formatting Terminal Output

Formatting output to the terminal can be tricky. It works by sending escape codes to our terminal, which are essentially "commands", that we represent using a sequence of characters. If the terminal supports the escape code, rather than printing it, it will perform that command. Terminal escape codes let us do things such as:

Change the position of the cursor
Change the foreground and background colour of the output text
Change the formatting of the output text (italic, bold, etc.)
Make the cursor invisble

These escape codes can be difficult to manage, and capabilities can vary depending on the type of terminal you have. The fact that terminal based software such as Vim exists should give an idea of the power that these escape sequences offer. Luckily there are plenty of libraries available which make the process much easier for us. For this plugin, I used the termcolor library. The colored function it provides makes it easy to write colourful and formatted output.

Aside: Partial Function Application In Python

The colored function from termcolor will be used in several places in the plugin. We'll be passing to it a consistent and verbose set of arguments that we don't want to have to repeat everywhere, so it's a great candidate for partial function application! Partial function application lets us "prep" a function by passing in some arguments in advance. Then, when we want to use the function later, we don't have to pass those arguments in again.

from functools import partial
deleted_text = partial(colored, color=Color.red, attrs=[Attr.bold])
diff_intro_text = partial(colored, color=Color.cyan, attrs=[Attr.bold])
inserted_text = partial(colored, color=Color.green, attrs=[Attr.bold])

Now we can call the functions deleted_text, diff_intro_text, and inserted_text in the same way we would call colored, but we can omit the color and attrs named arguments, since they've been applied in advance. Using partial application can make your code more readable, if you give your partially applied functions meaningful names, but use it only where it makes sense.

Terminal Escape Sequences

By default, pytest outputs each line of the assertion report as red text (see the screenshot at the start of this post). We don't want this, so we want to instruct our terminal to revert back to standard character formatting. Unfortunately I don't think termcolor has a function for this, so we have to send the terminal the escape sequence ourselves.

The escape sequence to clear character formatting for VT-100 compliant terminals (the majority of terminal emulators support this) is \033[0m. ascii-table.com has a handy reference listing terminal escape sequences. The \033 part is an octal representation of the decimal value 27. If you look up an ASCII table, you'll find that 27dec maps to the ESC (escape) character. Then, we have a [, which as far as I know is just a separator. The remainder of the sequence is an alphanumeric code that maps to a function. In this case, 0m maps to the "Clear Character Attributes" command. If we print out this escape code at the start of every line of output, we'll override pytest when it attempts to print out everything in bold red characters, and the terminal will output the text in the default format instead.

def plain_text(string):
    return "\033[0m" + string

Putting It All Together

When a test fails, pytest calls the repr function on both sides of the assert statement, and outputs the diff of these object representations. Rather than relying on repr, we'll use pprint.pformat which will provide us with a nicely formatted string representation of the object that may span multiple lines for clarity, and make the output more parseable. pprint.pformat also sorts unordered collections such as dicts and sets when constructing the representation. This is essential, since if the representation of two dicts being compared had different key ordering, we'd get different output every time!

lhs_repr = pprint.pformat(left, width=width)
rhs_repr = pprint.pformat(right, width=width)

Now that we have our "pretty" representations, we can use the SequenceMatcher from earlier to generate a delta between them, and our colouring functions to print out text.

Here's some code for printing out a split diff (a split diff is where the left and right hand sides of the diffs are printed out independently):

matcher = difflib.SequenceMatcher(None, lhs_repr, rhs_repr)
for op, i1, i2, j1, j2 in matcher.get_opcodes():

        # Deltas can span multiple lines, but we need to 
        # operate on a line by line basis so we can override
        # pytests attempts to print every individual line red
        lhs_substring_lines = lhs_repr[i1:i2].splitlines()
        rhs_substring_lines = rhs_repr[j1:j2].splitlines()

        # Highlight chars to remove from the left hand side
        for i, lhs_substring in enumerate(lhs_substring_lines):
            if op == 'replace':
                lhs_out += deleted_text(lhs_substring)
            elif op == 'delete':
                lhs_out += deleted_text(lhs_substring)
            elif op == 'insert':
                lhs_out += plain_text(lhs_substring)
            elif op == 'equal':
                lhs_out += plain_text(lhs_substring)

            if i != len(lhs_substring_lines) - 1:
                lhs_out += '\n'

        # Highlight the stuff to be added on the right hand side
        for j, rhs_substring in enumerate(rhs_substring_lines):
            if op == 'replace':
                rhs_out += inserted_text(rhs_substring)
            elif op == 'insert':
                rhs_out += inserted_text(rhs_substring)
            elif op == 'equal':
                rhs_out += plain_text(rhs_substring)

            if j != len(rhs_substring_lines) - 1:
                rhs_out += '\n'

    # Return the left and right diffs as lists of strings
    return lhs_out.splitlines(), rhs_out.splitlines()

Conclusion

This post was a quick look at some of the code behind pytest-clarity. The plugin is currently available on PyPI, and you can install it using pip:

pip install pytest-clarity

The full project is available on GitHub (see below). Let me know if you have any questions or issues with it!

darrenburns / pytest-clarity

A plugin to improve the output of pytest with colourful unified diffs

pytest-clarity

A plugin to improve the readability of pytest output.

Features

Enables an easier to parse and more understandable diff for your failing tests.
Can display unified diffs or split diffs, and chooses them depending on the context.
Adds helpful hints to the output for failing tests to help you track down common issues.

With pytest-clarity:

The same test, without pytest-clarity:

Requirements

Supports Python 2.7 and 3.4+

Installation

You can install "pytest-clarity" via pip from PyPI:

$ pip install pytest-clarity

Usage

Install the plugin as described above.

** The plugin will only be activated when the -vv option is supplied to pytest.**

You can choose which type of diff you want with --diff-type (auto (default), split or unified):

pytest -vv --diff-type=split

You can disable hints using the --no-hints flag:

pytest -vv --no-hints

You can configure the width of the output with the --diff-width option:

pytest -vv --diff-width=60

Contributing

Contributions…

View on GitHub

10 Tools To Power Up Your Command Line

Darren Burns — Thu, 03 Jan 2019 08:53:31 +0000

This post is the first in a series of showcases of some of the best non-standard command line tools I've discovered in recent years. If you ever make use of the command line, you'll probably find that at least one item on this page will make your life easier.

`z`, jump around

One of the best and most ubiquitous time-saving features in modern web browsers is the fuzzy and smart search capabilities of the address bar. Want to quickly go to Twitter? Typing "tw" into your address bar and hitting enter is probably enough.

By comparison, the standard way of navigating our file systems via the command line (using cd) seems almost prehistoric. Thankfully, z (GitHub) brings this browser-style navigation to the command line.

After a short learning period, z will allow you to jump to a directory from anywhere, using only a substring of the target directory name. The directory that z will take you to is determined by the string argument you gave it, how frequently you visit the directory, and how recently you visited the directory. They call it "frecency".

z not only increases the speed at which you can navigate your filesystem, it reduces the cognitive load of navigation. With cd, you need to recall precisely where the destination directory sits in the tree, and work out the path to get there. With z, knowing the name of the directory is enough.

Ports of z to other shells (such as fish and zsh) are readily available too. autojump (GitHub) is a similar project.

Installing `z`

Install bash version, on macOS (Homebrew): brew install z
Install fish shell version, on macOS (Fisher): fisher add jethrokuan/z

`fzf`, a fast fuzzy finder

After installing fzf (GitHub), you can press Ctrl + T at any point in time to open up an interactive fuzzy search interface, which will recursively search for files inside the current directory. You can enter a search term, and press the up/down keys to move through the results. If you press the enter key, the selected result is inserted into your terminal:

In the example above, I typed bat (but this could be any command, such as less, cd, etc.), then pressed Ctrl + T. I typed five, hit enter, and the path src/five.rs was inserted at my cursor position. This saves the alternative of (roughly): type src, press tab, type fi, press tab, which doesn't scale to long or hard to remember paths.

Installing `fzf`

On macOS (Homebrew): brew install fzf
Bindings for fish: fisher add jethrokuan/fzf

`bat`, to view files with syntax highlighting

If you want to quickly view a source file with full syntax highlighting, bat (GitHub) is your friend. bat can be used as a drop-in replacement for cat.

If the output is large enough (as in the example above), bat will pipe it's output into less, meaning we get pagination for free!

Installing `bat`

On macOS (Homebrew): brew install bat

`bench`, for benchmarking your code

bench (GitHub) is an incredibly useful tool for benchmarking your code. It's written in Haskell, which makes it the coolest thing on this page. You can pass any command you can run from your terminal to it (in quotes), and it will measure the execution time by repeatedly running the command. When it's finished, it'll output useful statistics to your terminal.

This is a considerably more powerful means of measuring the execution time of your code than the built-in time command.

hyperfine (GitHub) is an alternative to bench written in Rust that you might also be interested in.

Installing `bench`

On macOS using Homebrew: brew install bench

`asciinema` & `svg-term`, for recording your terminal as an SVG animation

The terminal clips on the version of this post on my personal blog are SVG animations (unfortunately dev.to doesn't support SVG uploads)! You'll notice they're much crisper than the gifs on this page. Using SVG rather than a video format or gif has several huge benefits:

Perfect quality regardless of zoom 🔍
We can put them in Markdown files like any other image 😱
Smaller file sizes compared to video formats and gifs 🧐
SVG animations are way cooler than videos 🔥

To record the terminal, I use asciinema. Begin recording with asciinema rec. When you're finished, press Ctrl+D, and you'll be given the options of either saving the recording locally or uploading it to asciinema.org.

If you want to generate an SVG animation from your recording using svg-term (GitHub), and you uploaded your recording to asciinema, you'll have to make it public by visiting the resulting link.

To convert the recording to an SVG animation, you can supply the ID of the cast (available on the asciinema page after making it public - the ID is in the URL), an output file, and numerous other optional arguments. For example, to save the terminal recording at https://asciinema.org/a/219486 to the SVG file you see in the example above, I used:

svg-term --cast=219486 --out ~/somewhere/out.svg --padding 18 --height 8 --width 80

Alternatively, if you don't want to upload your recording to asciinema, you can supply a local cast file directly to svg-term (thanks to Mario Nebl, the author of svg-term for pointing this out to me):

asciinema rec cast.json
cat cast.json | svg-term-cli

Installing `asciinema` & `svg-term`

Installing asciinema on macOS: brew install asciinema
Installing svg-term on macOS: npm install -g svg-term-cli

`wrk`, for benchmarking your HTTP APIs

This is a handy little tool for performance testing your API. To demonstrate, I've got a minimal Python HTTP API server with a single endpoint (GET /hello) running on my local machine at port 8001. We can check how well the /hello endpoint performs using wrk (with 12 threads, 200 connections, for 5 seconds):

You can tweak the number of threads, connections, and the duration of your test to check performance under different loads. It's not a replacement for performance testing tools such as Locust and JMeter, but it's lightweight and will suffice in many situations.

Unfortunately, the command line interface for wrk makes it somewhat awkward to perform POST requests. If you want to do that, you'll need to write a small Lua script and supply it to the command as an argument (there's more information in the docs).

Installing `wrk`

On macOS using Homebrew: brew install wrk

`exa`, an alternative to `ls`

exa is a modern replacement for ls with colour-coded output that's a little easier on the eye, and a larger variety of options for controlling how output is presented.

It supports features such as respecting your .gitignore files via the --git-ignore flag, and printing
out a directory as a tree structure with the -T flag (see above). It even shows the git status of files!

Installing `exa`

On macOS using Homebrew: brew install exa

`fd`, for finding files & directories

If you're looking for a file or directory, you'd usually use the find command to perform a search based on a regular expression. fd (GitHub) is an alternative to find written in Rust which offers a more convenient interface by using sensible defaults, and it's faster to boot!

It'll respect your .gitignore files, and it supports parallel command execution, which lets you execute a terminal command (in parallel) for every file or directory returned for a search. For example (from the fd documentation), to find all .jpg files and convert them to .png files in parallel using the Unix convert command, you can run:

fd -e jpg -x convert {} {.}.png

Installing `fd`

On macOS using Homebrew: brew install fd

`rg` (ripgrep), for finding strings in files

rg (GitHub) is a (much) faster alternative to grep.

rg is written in Rust, and it powers the search functionality inside the VS Code text editor. It consistently outperforms similar tools in benchmarks.

Installing ripgrep

On macOS using Homebrew: brew install ripgrep

Conclusion

Hopefully you found something useful in this post! If you’re interested in more content like this, follow me on Twitter and on DEV.

This post was originally published on my blog.

DEV Community: Darren Burns

A short example of how types can make your life easier

A note on Python 3.8

Simulating millions of particles to create the Sun

Learning to Create Video Games

Coding and creativity

Making planets

The dance of the planets

Interactivity

Box selection

Combat

Explosions

Moving forward

Making Mistakes

5 CLI Utilities to Boost Your Productivity

fx, a command-line JSON processing tool

Installing fx

http-prompt, for interacting with APIs

Installing http-prompt

fselect, for querying files with an SQL-like syntax

Installing fselect

ranger, a command-line file manager

Installing ranger

tokei, to view statistics on your code

Installing tokei

Conclusion

The Piece Table - the Unsung Hero of Your Text Editor

First instincts - the array of strings

An append-only representation

Piece descriptors

Adding text to a file

Saving and displaying the open file

Deleting text

Undoing and redoing

Conclusion

References/Further Reading

8 Productivity Tips for GitHub

Quick fuzzy file search in repositories

Code change suggestions in pull requests

Navigate the code tree like in an IDE

Jump to a function when reviewing code

Creating a permalink to a file

Viewing the blame and change recency heatmap

Powerful code search

Saved replies

Conclusion

Power Up Your Command Line, Part 3

tig, for interactively browsing your git repo

Installing tig

PathPicker (fpp), for quickly selecting files

Installing PathPicker

tldr, for practical examples of how to use CLI tools

Installing tldr

gron, for exploring JSON

Installing gron

thefuck, for quickly correcting command line typos

Installing thefuck

Bonus: explainshell, for explaining shell commands

Conclusion

5 More Tools To Power Up Your Command Line (Part 2 Of Series)

peco, interactively filter anything

Installing peco

hexyl, a hex viewer

David Peter

Installing hexyl

pomo, a pomodoro timer

Installing pomo

ncdu, analyse and free disk space

Installing ncdu

HTTPie, a modern replacement for curl

Installing HTTPie

Conclusion

Debug Faster With Colourful Diffs For Your Failing Tests

Modifying Pytest Output With A Plugin

A Short assertrepr_compare Plugin Example

How Pytest Calculates Diffs

Enter The SequenceMatcher

Formatting Terminal Output

Aside: Partial Function Application In Python

Terminal Escape Sequences

`fx`, a command-line JSON processing tool

Installing `fx`

`http-prompt`, for interacting with APIs

Installing `http-prompt`

`fselect`, for querying files with an SQL-like syntax

Installing `fselect`

`ranger`, a command-line file manager

Installing `ranger`

`tokei`, to view statistics on your code

Installing `tokei`

`tig`, for interactively browsing your git repo

Installing `tig`

PathPicker (`fpp`), for quickly selecting files

`tldr`, for practical examples of how to use CLI tools

Installing `tldr`

`gron`, for exploring JSON

Installing `gron`

`peco`, interactively filter anything

Installing `peco`

`hexyl`, a hex viewer

Installing `hexyl`

`pomo`, a pomodoro timer

Installing `pomo`

`ncdu`, analyse and free disk space

Installing `ncdu`

HTTPie, a modern replacement for `curl`

A Short `assertrepr_compare` Plugin Example

`z`, jump around

Installing `z`

`fzf`, a fast fuzzy finder

Installing `fzf`

`bat`, to view files with syntax highlighting

Installing `bat`

`bench`, for benchmarking your code

Installing `bench`

`asciinema` & `svg-term`, for recording your terminal as an SVG animation

Installing `asciinema` & `svg-term`

`wrk`, for benchmarking your HTTP APIs

Installing `wrk`

`exa`, an alternative to `ls`

Installing `exa`

`fd`, for finding files & directories

Installing `fd`

`rg` (ripgrep), for finding strings in files