DEV Community: David Maidment

'I wonder if...': Moving ASCII art

David Maidment — Fri, 14 May 2021 06:52:17 +0000

'ASCII cinema,' was all that I heard. Sat in a weekly sprint meeting, my mind focused more on arranging my upcoming week of work than paying attention to the person offering a retrospective of their just-completed sprint, my ears perked up at the mention of what sounded like something magical.

The reference was, of course, to asciinema, a tool for recording terminal sessions that was wholly-relevant to the retrospective. But all I heard was, 'You could play The Godfather in the terminal.'

Such technology speaks for itself, surely?

We all joked about my absentmindedness and mused for a few minutes on the practicalities of playing back video as moving ASCII art, then continued with the retrospective. But my brain couldn't let go of the challenge; by the end of the meeting I had already mapped out the basic workings of the codec in my mind's IDE.

For those as lazy as me who just want to see the end result, this is what I managed to code in the space of about half an hour, using a GIF as input, JSON as the storage format and Golang as the programming language:

Make it so

Having decided to write the most [un]necessary codec I could think of, my first thought was how to store the data.

In the spirit of misusing technology, I decided that JSON would be the way to go. The format certainly lent itself to storing necessary metadata such as the frame rate, which I planned to preserve from the input file and match on playback with dynamically calculated sleep commands.

I recently gained some experience working with images on the pixel level for another project, so my go-to approach was to read in an image (a GIF version of a movie seemed to make sense) and grab the RGB value of each pixel to save losslessly in some manner that could later be displayed in the terminal.

To begin with, I knew that I needed to store an initial key frame containing the colour values of all of the first frame's pixels. Coding in Golang, a map of maps of integers (an associative multidimensional array, in other languages) seemed to fit the bill, where the map keys would represent x/y pixel coordinates and the leaf values would be -1, 0 or 1 to denote the pixel's colour—I initially opted for a black / grey / white colour palette.

The JSON for such a key frame would look something like:

{
  "1":           // Row 1
    {
      "1": 1,    // Column 1
      "2": 0,    // Column 2
      "3": -1    // Column 3
    },
  "2":           // Row 2
    {
      "1": 0,    // Column 1
      "2": 0,    // Column 2
      "3": 0     // Column 3
    },
  "3":           // Row 3
    {
      "1": -1,   // Column 1
      "2": 1,    // Column 2
      "3": 1     // Column 3
    }
}

When the values for such a data set are arranged in order, a basic picture emerges:

00000000000000
00001111110000
00011111111000
00111111111100
00011111111000
00011111111000
00001111110000
00000000000000

Subsequent frames would also have their pixels traversed, but only those pixels whose values differed from the previous frame would be stored on the next frame map. For example:

{
  "1":           // Row 1
    {
      "2": 1,    // Column 2
      "3": 0     // Column 3
    },
  "3":           // Row 3
    {
      "3": -1    // Column 3
    }
}

If the only change in a frame was that a character blinked, then only data for the pixels around their eyes would need to be stored.

Playing it all back

As mentioned previously, I opted for black, white and grey as the three colours of the palette. My theory was that you could get a fairly good image out of those three colours by taking the average of each pixel's RGB components, ((R+G+B)/3), and assigning the lower third to black, the middle third to grey and the upper third to white.

Characters that covered an appropriate amount of space on-screen could be used to represent those colours (I chose a space, a slash and a capital W, respectively). Displaying them would simply involve building up multiple strings (one per scanline), flushing them all to the terminal once per frame and then issuing sleep and clear commands.

The first test produced this:

Three 'colours' worked, but not well enough.

The result wasn't terrible, but the image also wasn't as clear as I had hoped. I needed to preserve more colour data. After some googling I managed to find a list of around 70 characters that covered a wide range of screen real estate.

I processed my test video again but the result was disappointing—the animation was mostly recognisable, but the resolution was simply too low to make the extra 'colours' worth it.

After much tinkering, a selection of ten characters seemed to yield the best results. This is what you saw in the first GIF.

Improvements I should have made

There were some obvious improvements that I could have made to the codec, had I not given up once I had proved to myself that, yes, I could watch The Godfather in the terminal, if I really wanted to.

Experimenting with a 500MiB JSON file (converted from a 20-minute TV show), I was faced with a lead time of around one minute while the data was loaded into memory. It strikes me that abandoning JSON and switching to a one-frame-per-line format would enable streaming to begin instantly.

Switching to a non-JSON format would probably also yield lower file sizes, perhaps as small as 200MiB in this instance, assuming that low-impact delimiters were used in favour of the quotes and braces that comprise a sizeable percentage of JSON strings.

It would also be interesting to explore run-length encoding for key frames (and/or those frames with over 50% of pixel values changed). Although I have not yet carried out any experiments to confirm it, I suspect that the small 'colour' palette would lend itself nicely to this kind of compression.

Should you build it, or abandon it?

David Maidment — Thu, 29 Apr 2021 21:28:43 +0000

Over time a product will accumulate many features. Some are core features. Some are proofs of concept that have yet to be validated. Some are proofs of concept that failed. Some were once extremely relevant but are no longer fit for purpose. Some are a work in progress.

The point is: features accumulate, and when it comes to determining their relevance and maintainability, no two are the same. But one thing most features do have in common is that, at some point in time, someone thought they were a good idea.

We've all maintained poorly designed features. If we've been lucky (or maybe unlucky, depending on how you look at it), someone else was responsible for them. We've all looked at something and thought, 'Who built this, and what were they smoking?'

Hindsight is a wonderful thing.

And so the question presents itself: when confronted with the option of building or not building, of maintaining or abandoning, how should you proceed? How can you avoid becoming the guilty party at the other end of the 'What were they smoking?' jibe?

This question is particularly pertinent for early-stage companies, where resources are considerably more finite and the potential impact of every feature (positive or negative) is considerably larger.

It can be helpful to take a step back and answer two simple questions as bluntly as possible:

Did users ask for it?
Are users actually using it?

Or, if the feature has yet to be built:

Have users asked for it?
Have we validated that they actually need it?

The answers to those questions can help you categorise the current state of the feature; it may be: Mission Critical, a Stroke of Genius, a Waste of Time or a reminder that, often, Vision ≠ Reality.

The obvious takeaway may be: Spend most of your time building Mission Critical features and have game-changing innovations. However, it is important to understand that other types of features will end up in your product and to be realistic about your ability to consistently innovate.

By understanding the qualities of each quadrant, and how a feature may be masquerading as a different type, this process becomes easier.

Mission Critical features

These are the features your users have asked for, you have duly built and everyone uses. They are the bread and butter of your business and should be no-brainers. If you are a car manufacturer, these features are engines, airbags and steering wheels.

The decision to build these features comes from a combination of thorough market research, common sense and (if you are doing things right) dogfooding your own product.

However, as obvious as these features seem, it can still be worth challenging your assumptions about them. Especially if they are the product of well established received wisdom.

Do they represent an extremely ineffective way of doing things that your users have come to accept as an inevitable part of life? This is a hard question to answer, as it requires really thinking outside the box.

If the answer is 'yes', then also ask yourself whether an opportunity exists to remove the feature (it may actually be a Waste of Time that people have become accustomed to) or, even better, make a radical departure from the norm and disrupt the status quo by evolving it into a Stroke of Genius.

As the quote often attributed to Henry Ford goes:

'If I had asked people what they wanted, they would have said faster horses.'

Stroke of Genius features

When you shake up the industry and create something no one realised they needed, you have a Stroke of Genius on your hands.

Features (or often, entire products) that fall into this category usually take two forms: entirely new inventions, and things that are suddenly possible because of a larger paradigm shift brought on by those new inventions.

Most of our innovations will not fall into the new inventions category. As much as we like to think that our idea will change the world, very few ever do. So it helps to be realistic about the scope of whatever you are building.

Much of what we build instead rides the waves of new inventions. At the moment, the invention that many products and features piggyback off is the Internet. Many industries have been disrupted in a way that no one ever thought possible, largely down to the new abilities the Internet brought with it. Banking, shopping, photo processing, TV and film are just a few industries that now operate in entirely new ways.

So apply that thinking to your own product. Maybe you'd love to build a widget that helps to automate people's lives. The technology to build it has always been science fiction (maybe some kind of advanced AI?), and no one has asked for it because humans have been successfully living their lives for hundreds of thousands of years without it. Then suddenly the technology to build it becomes available; you're ready to change the world.

Simple, right?

Perhaps not. With a product no one has asked for, you will find yourself fighting against years, decades or centuries of established status quo. You'll have to convince everyone that they've been doing it wrong all this time. And no one likes to be proven wrong.

Just because you build it, it doesn't necessarily follow that they will come. You may even find building the feature to be the easiest part of the process, with huge amounts of time, effort and money required to run a successful awareness campaign to obtain even your first few hesitant users (electric cars might be a good example of this). This is where a Stroke of Genius risks becoming a Waste of Time.

To combat this, aim to work in MVPs and spend the least time possible to get a workable version of an unvalidated idea out into the world. Take feedback, iterate, but don't commit huge quantities of time to it. By operating in this way you can run far more high-potential experiments in a given period of time and will feel less guilty about axing one of them when the numbers don't add up.

It is also worth keeping perspective on the uniqueness of your idea. If it's been possible for a while to build your feature and no one in the market is currently doing so, this may not be an indication that you have a unique idea, but instead that it is a really bad idea that others have already tried and failed at. Take this into consideration when deciding how much time you're willing to sink into an MVP.

Waste of Time features

Your users haven't asked for the feature, you built it anyway and they aren't using it. This is a waste of your time and resources, and such features will always benefit from being put on hold.

A Waste of Time may be an unrealised Stroke of Genius, but probably not. Even if you are certain that you are onto something—something visionary and years ahead of its time—it is still worth limiting the time you spend on these features while you carry out market research (which may take the form of user feedback loops) and explore the implications of an awareness campaign.

For anything that falls short of game changing, do not be afraid of culling these kind of features from your product backlog before anyone gets a chance to build them. If an idea truly is worth building, you won't be able to keep it from reappearing time and time again in the backlog.

Features where Vision ≠ Reality

What a user wants vs what they need. Features that fall into this category are those that are prescribed to be essential by users who do not yet have a use for them. This is sometimes due to ignorance, sometimes wishful thinking.

These features may present themselves as being Mission Critical due to received wisdom, especially in industries that have not recently been disrupted.

Suppose your company builds an advanced web analytics suite. You came from a background in Big Marketing where this was an essential feature. But your clients have ended up being mostly small businesses. Like you, the folk at the small businesses are thinking big; they dream of the game-changing insights you might help them uncover. But in reality, they have almost no data for you to work with and all they really need is a way of correlating banner impressions with online sales.

Features that fall into this category may also be the product of a vocal minority. If those taking part in your market research do not accurately represent your target audience, you may end up building 90% of your product for 10% of your users. Beware those who shout the loudest, and try to avoid launching into Build Mode as a knee-jerk reaction to every shiny new feature request.

Start with the obvious, then experiment

When you are trialling new ideas or deciding what to do with old ideas, objectivity is key.

Build the obvious Mission Critical features and spend the time needed to validate your Stroke of Genius ideas. Work in cycles and advance your bold ideas a bit at a time, doubling down on them when the data shows you're onto something, and cutting your losses when it's obvious you made a mistake.

Always avoid the temptation to build shiny and complex things where Vision ≠ Reality just because they excite you, and make a habit of frequently—and brutally—culling unvalidated Waste of Time ideas from your backlog.

As a business and as an individual, you only have so much time and money, so use both wisely. Keep it simple, triage features with a detached objectivity and avoid the obvious traps.

'Just because I can build this, should I?'

David Maidment — Thu, 18 Mar 2021 08:49:20 +0000

When it first became available to anyone with a telephone line and a modem, The World Wide Web was a very special place. Every trip onto the Web was a voyage of discovery. Content was being created at breakneck speeds, and behind every website was someone who took on the mantle of 'webmaster' for the pure love of the subject matter.

Interested in the ecology of Iceland? The works of Chaucer? The Powerpuff Girls? You could guarantee there would be at least one website on the subject painstakingly curated by someone just as fanatical as you. Someone who cared so much about the subject they taught themselves HTML just to share what they knew. It was fandom-driven-development.

The Web was also a beautiful maze.

There were no search engines. No algorithms telling you what to look at next. Nothing to serve up content specifically to support your worldview. Anything you wanted to see, you had to go out and specifically look for it. And if in the process you came across a little pocket of culture that was foreign to you, you would learn something new about the world. There were no comments sections telling you that what you had just read was wrong and that the author should harm themselves—you were trusted to use common sense to form an original opinion.

Perhaps most importantly, The Web was a place where communities sprang up organically.

In 2021, companies pour huge amounts of time and money into curating communities—Chief Community Officer is a real job title you can hold. It's taken a long time, but the importance of communities is increasingly being recognised by businesses.

But back in the 1990s? Any enthusiast with a modem was capable of bringing together hundreds or thousands of strangers. Those online communities came about naturally, and from them, you could make friends. Real friends. Friends you would stay up until the early hours of the morning talking to about everything and nothing. Far-flung friends who would teach you about their interests, their part of the world and their culture. You never stopped learning.

In those days I ran a number of vibrant communities, none of which were intentional. They just happened, because humans are good at organising into groups. It's literally in our DNA. That evolutionary trick helped us grow from cavepeople to a global civilisation. It put people on The Moon. But working in groups requires trust, and trust is built by getting to know someone.

We tend now to treat the problem of 'community' as though it's a new one, but it isn't. We've always known how to build them. Decades ago, online, they were everywhere. So what changed?

In the early 2000s, The Web took a hard right as Social Media took off. The importance of The Individual over The Community soared, and many of the old vibrant communities migrated to social platforms to die a slow, undignified death.

Why did this happen?

Once advertising became a viable online business model, finding ways to make users return to your site every day took centre stage. Almost overnight, dopamine became the currency of The Web, and interactions became more individual-centric. Where we used to have to get to know someone and form a real, human connection, it became the norm to broadcast short messages about ourselves—Status Updates, which we all knew and loved as the taglines of our IM profiles, now without the need to have the accompanying conversation. And when typing became too much work, Likes simplified the process even further. We collected hundreds of 'friends', knowing precisely nothing about most of them. It was how all the platforms operated, so it became the new normal.

To keep the dopamine flowing, the new generation of community tech worked hard to avoid negative experiences. Keeping you in an echo chamber is the easiest way to do this: see more of what you like, and less of what you do not. On paper it sounds like a good idea, but in practice it deepens trivial political divides and removes healthy challenges to your worldview.

Soon enough lines were being drawn everywhere between imagined us and thems. We forgot that life is messy, people are not perfect and that there is nuance is every single situation; that nothing is simply good or evil; that some things require considered conversations rather than emojis. We stopped learning about each other and started making dangerous assumptions about each other.

It. Was. Devastating. We had lost something truly beautiful.

Growing up with technology, I was in constant awe of the possibilities. With The Web, I saw those possibilities flourish as communities self-organised around fansites, encouraging endless learning, conversations and friendships. The future was going to be something extraordinary, and it was all thanks to technology.

But here's the thing: while technology can be amazing, it is not amazing in and of itself; it requires us to make it amazing.

We make technology. Technology that can go viral overnight and gain a large percentage of the world's population as users. So we have a responsibility to ensure that the things we create are carefully considered at every stage. No one set out to create today's individualistic online culture, and that's the most terrifying part—that it happened not because someone said, 'Let's change online behaviour for the worse,' but because they had to figure out how to monetise platforms that people were increasingly expecting to access for free.

We have a responsibility to ensure that the technology that powers the next twenty years is better than the technology that powered the last twenty years.

Regaining the lost potential of The Web (which I honestly believe is in self-organising, tangentially educational, social groups) is not a complex problem, but it will take a concerted effort to solve. If enough technologists work to reframe what is 'normal' online, we can undo some of the damage accidentally wrought by The Web. And it starts by asking a simple question at the beginning of each new project: 'Just because I can build this, should I?'