DEV Community: Farai Gandiya

Improving Netlify's Default HTTP Cache-Control Headers for Immutable Assets

Farai Gandiya — Mon, 14 Feb 2022 13:06:33 +0000

I decided to use the Recursive font on my site once again and when it went live, they kept popping in between subsequent reloads. Looking into it, it turns out that this was due to how Netlify caches things.

What does Netlify do?

In essence, they include this header:

cache-control: public, max-age=0, must-revalidate

As they put it, the header says please cache this content, and then do not trust your cache. This is useful for assets that change frequently as the browser will always check if an asset has changed before fetching it by verifying the ETag header.

Seems ok. What’s the problem?

The issue I had was that certain assets on this site (like CSS, JS, and font files) have cache busted URLs where the URL either has a hash or version number.

<link rel=stylesheet href=/fonts.1baec3efa7adb966c2c5c92e8258e75a350c5151d2679de09650f4536b185be8.css>
<link rel=stylesheet href=/main.ba62e1ad92ea86749a3f67394100f75f0044a087c2101678a42e21ddffdaaa3f.css>

Note the hashes in the hrefs. That corresponds to the file’s hash. If the underlying file changes, the hash would as well.

There’s no need to constantly revalidate such an asset as it will never change. If the asset did change, it would have a new URL that would be cached on its own. As fast as Netlify’s CDN is and however much they want to include an ETag, revalidation is still pointless overhead for fingerprinted URLs.

How To Fix This

To fix this, I used the immutable cache-control header which tells the browser that the asset will not be updated as long as it’s fresh. Note that the maximum caching age is a year which is about 31,536,000 seconds).

Cache-Control: public, immutable, max-age=31536000

To do this, you need to provide custom headers. I used an _headers file which looks like this:

/*.woff2
    Cache-Control: public, max-age=31536000, immutable
/*.css
    Cache-Control: public, max-age=31536000, immutable
/*.js
    Cache-Control: public, max-age=31536000, immutable

This works on my site since CSS and JS files will always be fingerprinted¹, and the .woff2 font files contain the version in the URL so if the font is updated, so will the URL.

Conclusion

After doing this, the site loaded a bit faster and the pop-in didn’t go away. It’s an improvement, though I need to quantify it at some point. That’s a benefit of going beyond the defaults. I know I put Netlify here, but I’m sure you can apply this to any host that let’s you change HTTP headers

This is all possible thanks to Hugo which comes with a good asset pipeline. If Hugo didn’t have this, or I decided not to use cache busted URLs, sticking to must-revalidate would be fine.

Interestingly, a few days after I solved this, I found a guide by Simon Hearne which goes into caching headers best practices which covers other scenarios.

The next performance optimization I want to do after this is preloading assets. The last time I tried it nothing changed but I hope it’ll be different this time.

Even if I include a library through NPM, I’d still cache bust it. ↩︎

Goodbye 2021, Hello 2022

Farai Gandiya — Sat, 15 Jan 2022 16:59:30 +0000

As 2021 ends and 2022 begins I want to reflect on how things went on my Codelab as well as my programming "career" along with my goals for 2022. In general, despite my many commitment issues, I managed to get some things done and I hope to get more done in 2022.

Writing

In 2021, I published just 4 posts on this blog (excluding links and contributions on other sites) which is far less than I have in previous years. The posts are:

Given that I don't have analytics, I don't know how well any of these posts did, but I think "Imagine if Doctors Relied on Google As Much As Programmers Do" got the most traction, probably more than a mega guide I published that same day (more on that in a bit). I've thought about this statement for years, but I put it into writing because some guy decided to be the main character again, arguing that we shouldn't take pride in looking shit up for our jobs.

Besides these posts, I've also captured links of interesting content I found around the web, most interesting of which was a video on Igalia's fundraising model. It's a bit ironic that you need to fundraise to implement features in browsers managed by million, billion, and trillion-dollar companies, but once you hear their explanation it makes sense.

Posts On Other Websites

In 2021, I started to contribute content on other websites, specifically CSS Tricks. I remember reading how these online web development publications allow people to write on their sites, regardless of experience as long as it can help many other people. I took this up for two reasons.

To gain experience in writing for others under the eye of an editor, rather than just whatever I feel like. I got this suggestion from Cal Newport who suggests that you should do things that require a gatekeeper as it will teach you more about how to do something rather than doing your own thing and hoping people will notice.
They pay. I've been unemployed for nearly 4 years and I need to start making money. This is the easiest option I have now.

I tried to contribute a couple of articles in 2021, but they were rejected. Third (and fourth) time's the charm as I managed to contribute two posts on CSS Tricks:

These posts took far longer than they should have to write and while they aren't as good as they could be, I'm glad I was able to get something published on the site I use as a reference. The post on web feeds is something I hope will get more traffic in the coming years even if it didn't spread as much as I'd hope.

Now that I know I can get published elsewhere, I want to do it more in 2022, mostly for money and experience. I'm working on another post right now and I have a few other ideas so now it's a matter of executing them, preferably in under a month apiece.

I also started a newsletter. Honestly, I don't know what I'm supposed to do with it. If you have any suggestions, let me know.

My Many Unpublished Posts

There aren't as many posts on this site as there were in previous years, and that's because I had a lot of unpublished posts in 2021. By unpublished I mean two things,

unpublished as in not published, and
unpublished as in I published it but then I removed it.

Concerning the posts I never got to publish, there are so many of them covering topics from image formats to armchair data analysis on RSS feeds. Some managed to get quite far even. The reason they didn't get published is similar to why my contributed posts took so much time—I'm lazy and I fell down the rabbit hole and ended up bikeshedding things which resulted in me yak shaving.

The topics I wanted to cover are far more complicated than they looked at first and rather than accepting that I can always fix things if they aren't perfect, I gave up and did nothing for weeks on end.

As for the posts I unpublished, I archived them on 2021.codelab.farai.xyz. I do this to clean this site and tame my content folders. They're still reachable through a 301 redirect as Cool URIs don't change but if the content you want, isn't let me know.

That's it for writing, now onto things I programmed.

Programming

Programming-wise, I haven't made much. Actually, I programmed a lot, I just don't have much to show for it. Besides endless redesigns of my personal websites¹, I must have made the same projects over and over again without actually publishing them.

The reason I struggle with actually publishing and iterating off that is that I'm worried about what prospective employers will think. As much as they say they like personal projects, I feel like putting the source code out there will invite a lot of nitpicking about how I did everything. Each redesign I keep hoping that things will be different, but they rarely are. In 2022, I hope to bias towards shipping and fixing things as I go.

While I haven't got a portfolio to my name per se, I've learned a lot given all the experimentation I did. It's Farai's Codelab after all! From web workers to canvas, I've tinkered a lot. Just look at all the shitty CodePens I've made. Yeah, they're shit but they're a learning experience as well. Expect a write-up on that soon.

Open Source

As for open-source, while I haven't contributed any code, I made a few bug reports to a few projects I use, namely Hugo, LCH Color Picker and the Recursive Font. It isn't respected as much as code, but I like to think that I'm helping developers triage issues.

In 2022, I hope to be a better open-source user and make bug reports whenever I find issues. I also want to help improve Hugo's documentation since it has a lot to improve upon. The first thing to document is Hugo's XML unmarshalling.

YouTube

I've got two YouTube channels, Farai's Codelab which is supposed to be my main channel and Farai's Codelab Notes, which is meant to hold archives to live streams and throwaway videos.

On the "main" channel, I've only published content I made years ago and I haven't published anything new on it. Interestingly, the "secondary" channel has more uploads which are mostly me going through Cracking The Coding Interview. Despite the fact my friend bought it for me as a Christmas gift, I've never used it until now. I hope to go through this book and finish it.

As for analytics, just know that I only got 20 minutes of watch time and 14 views on the secondary channel and 4 hours of watch time and 308 views on my main channel. I was surprised that I got 4 hours of watch time on my secondary channel, but I realized that's because of the one post I made which embedded two videos. Glad to see that they were watched.

Career

It's still nonexistent, but there are some embers, mostly thanks to the posts I wrote for CSS Tricks which have resulted in me making the most money since graduation in 2018. The job search process is very discouraging which is why I haven't bothered to try for a long time. In 2022, I want to give it one last shot. If it doesn't work, I'll try something else. It's unlikely, but unlikely things happen.

I also managed to get some contract work through connections, but they mostly failed. For one, I managed to get an MVP running and I fogot about the project only to meet the client in a bank where he paid me for what I did. Next year I hope to be more disciplined when it comes to such projects. Word of mouth is powerful and the best way for people to say nice things about you is to be dependable, which is something I need to work on.

Another thing I'll need to work on is financial management, but that's another post altogether.

Lesson of The Year: Embrace Experimentation and Iteration

The reason I called this site Farai's Codelab was to give myself a license to experiment with code and share what I've learned. Over the years, that experimental fervor has faded as I got worried that what I produce won't look good to others, which they don't. After many random redesigns, spontanious project ideas and ugly CodePens, I've realized just how rewarding experimentation is. There's a rush I get in figuring things out and though it's ugly, I've learned a lot from it.

As I go into 2022, I not only want to continue experimenting, I want to embrace iteration more. Rather than perfection upfront, I want to ensure that my projects and articles improve over time. Interestingly enough, this very article is an example of that. I initially published it without this section but decided to add it just to share the biggest lesson I learned in 2021.

Conclusion

In all, I'm still languishing. While I haven't accomplished all I wanted to do, I managed to get some things done and make a little money while a lot in the process. 2022 is the year I hope to turn things around, become financially independent, and start a career.

As for what I want to achieve next year, there's a few things:

I want to write more content. Between my unpublished drafts, overactive minds, and possible guest articles, I want to put more content out there.
Take YouTube seriously. Besides putting my posts in video form, I've got plenty of ideas that could do well on YouTube. I just need to take it seriously.
Figure out what I'm supposed to do with my newsletter.
Make the projects I've wanted to make for years.
Understand the basics of web development, lest I make more ugly sites like the one you're looking at now.
Contribute more to open source, mostly bug reports and improving Hugo's documentation.
Finally, and biggest of all, I want to get a job (preferably outside Zim) and start my tech career either programming or writing technical content.

That's a lot, but if you break it down and work on it consistently over many years, it's doable, I hope.

Thanks for reading! If you got this far down, you should know that there'll be another post going through the projects and posts I plan to make in 2022. It'll be a premium exclusive, so sign up to my Patreon if you want to read it. I might also make a post going through my old projects which will also be a premium exclusive.

Yes this site looks like ass right now. It's undergoing yet another redesign. ↩

Working With Web Feeds: It’s More Than RSS

Farai Gandiya — Fri, 17 Dec 2021 17:50:43 +0000

Working With Web Feeds: It’s More Than RSS was first published on Farai's Codelab.

Between Google Chrome experimenting with “following” sites, along with a growing frustration of how social media platforms limit a creator’s reach to their fans through algorithmic feeds, there’s been renewed interest in RSS feeds and they’re primed for a comeback as we get into 2022.

You may have heard whispers that “RSS in dead” around the web, but the truth is that they are still widely used as virtually every podcast uses one. Maybe you used to be an RSS fan and need to get re-acquainted with it. Or maybe you’re like Chris here at CSS-Tricks and still love RSS. Whatever the case, RSS is a web technology like any other, and there are best practices for how to create and curate a feed.

That’s exactly what I’m going to walk you through in this article. Let’s talk about the different kinds of feeds, how to implement them, and what strategies you can use to get the most out of your feed content.

This post took far longer than I expected to write, but it’s finally finished. It only took 3 months instead of 6 this time. Thanks again to the CSS Tricks team for helping me with this!

I plan on adding commentary, but that’ll be an exclusive post.

See Working With Web Feeds: It’s More Than RSS on css-tricks.com

Thanks for reading! If you liked this post, consider supporting my work by:

sharing my work,
reffering me for a job (or hire me for freelance work),
sponsoring me on Patreon,
buying Me A Coffee,
buying Me A Ko-Fi,
sending me a tip on Paypal,
buy something on Amazon with my affiliate link (I’ll earn a commision on this as an Amazon affiliate),
or buy a domain on Namecheap with my affiliate link.

You can also subscribe to my newsletter.

Have feedback? Send an email to gandiyafarai+feedback at gmail dot com

Imagine if Doctors Relied on Google As Much As Programmers Do

Farai Gandiya — Thu, 16 Dec 2021 17:19:48 +0000

"Imagine if Doctors Relied on Google As Much As Programmers Do" was first published on Farai's Codelab.

This is a common refrain programmers have from time to time, but the truth is that doctors do look things up for their job. Might not be though Google, but they don’t just rely on head knowledge.

There was a time my face got paralyzed and I had to go to a doctor. I found one and the first thing they checked for was if I was having a stroke. Thankfully I wasn’t having one (if I was, would I even be there). After that, they sat me down and asked a bunch of questions which they furiously typed into a computer.

A bit later they said I had Bell’s Palsy and invited me over to look at their laptop which had a program outlining the condition, it’s symptoms, causes and course of treatment. With that, I got the medicine and eyepatch and went about the next few weeks struggling to move half my face which made me realize why banning plastic straw is stupid.

This episode made me realize that looking things up, particularly with Google, is just a tool. It has little effect on your competency as a professional, whether it is programming, medicine or whatever. Is it worth your time to know every single API call and function signature? What use is me memorizing the visually-only class when I can just copy paste it from the a11y project when I need it?

It can get annoying when acclaimed programmers sound like they’re just bumbling fools who crib onto Google to get their job done but that can be a strength, can’t it? Knowing how to save time through premade solutions that you can evaluate is a powerful skill.

Given that, I think it would be better to focus less on people googling shit and more on making sure the stuff they Google is correct to begin with along with people being able to evaluate what they find.

“The Internet contains the sum of all human knowledge at our fingertips, but I would like to hire people who don’t take advantage of this amazing resource” is a hell of a weird-ass hill to die on but you do you buddy.

—@jacobian

Thanks for reading! If you liked this post, consider supporting my work by:

sharing my work,
refering me for a job (or hire me for freelance work),
sponsoring me on Patreon,
buying Me A Coffee,
buying Me A Ko-Fi,
sending me a tip on Paypal,
buy something on Amazon with my affiliate link (I’ll earn a commision on this as an Amazon affiliate),
or buy a domain on Namecheap with my affiliate link.

You can also subscribe to my newsletter.

Have feedback? Send an email to gandiyafarai+feedback at gmail dot com

Breadcrumbs In Hugo

Farai Gandiya — Tue, 05 Oct 2021 11:10:37 +0000

Breadcrumbs In Hugo was first published on Farai's Codelab.

Breadcrumbs are a type of navigation which shows pages in a particular order, like viewing history or, in my case, a page’s hierarchy.

Implementing this was frustrating, but I eventually figured it out. In short, you need to do something similar to this:

<!-- In whatever template -->
<p class="breadcrumbs">
    {{ partial "breadcrumbs.html" . }}
</p>

Then in breadcrumbs.html, include this:

{{ with .Parent }}
    {{ partial "breadcrumbs.html" . }}
    {{ if .Parent }}→{{ end }}
    <a href="{{ .Permalink }}">{{ .Title }}</a>
{{ end }}

The important thing is that you:

invoke the recursion by calling the breadcrumbs partial to the .Parent.
provide the separator (given you have one) conditional to whether the current .Parent has a parent and
link to the current crumb.

That’s it.

You can play with semantics based on what you need. Want to use nav > ul instead of p and make them list items so you can use Bootstrap’s breadcrumbs? Go ahead.

I spent far more time than I should have trying to solve this and this is what I ended up with. If you want the full story on how I got to this point, it’s a Patreon Exclusive. Thanks to nfriedli for figuring this out!!

Thanks for reading! If you liked this post, consider supporting my work by:

sharing my work,
refering me for a job (or hire me for freelance work),
sponsoring me on Patreon,
buying Me A Coffee,
buying Me A Ko-Fi,
sending me a tip on Paypal,
buy something on Amazon with my affiliate link (I’ll earn a commision on this as an Amazon affiliate),
or buy a domain on Namecheap with my affiliate link.

You can also subscribe to my newsletter.

Have feedback? Send an email to gandiyafarai+feedback at gmail dot com

Give Your Illustrations Dark Mode With This Simple Trick (SPOILER: It's a CSS Filter)

Farai Gandiya — Tue, 05 Oct 2021 10:37:49 +0000

Give Your Illustrations Dark Mode With This Simple Trick (SPOILER: It's a CSS Filter) was first published on Farai's Codelab.

filter: invert(100%);—that’s it!

Ok fine, it won’t work ~~all~~ most of the time (it’s terrible on actual photos) but it’s good for illustrations with simple color palettes, like xkcd comics.

Wisdom of The Ancients by xkcd Transcript

On my site, I put that CSS in a class invert-hack which I place on any illustrations I want to flip to dark mode.

Thanks for reading! If you liked this post, consider supporting my work by:

sharing my work,
refering me for a job (or hire me for freelance work),
sponsoring me on Patreon,
buying Me A Coffee,
buying Me A Ko-Fi,
sending me a tip on Paypal,
~~buy something on Amazon with my affiliate link (I’ll earn a commision on this as an amazon affiliate)~~,
or buy a domain on Namecheap with my affiliate link.

You can also subscribe to my newsletter.

Have feedback? Send an email to gandiyafarai+feedback at gmail dot com

Web Features That May Not Work As You’d Expect

Farai Gandiya — Fri, 23 Jul 2021 15:24:43 +0000

Web Features That May Not Work As You’d Expect was first published on Farai's Codelab.

As the web gets more and more capable, developers are able to make richer online experiences. There are times, however, where some new web capabilities may not work as you would expect in the interest of usability, security and privacy.

I have run into situations like this. Like lazy loading in HTML. It’s easy to drop that attribute onto an image element only to realize… it actually needs more than that to do its thing. We’ll get into that specific one in a moment as we look at a few other features that might not work exactly as you‘d expect.

This is the first post I’ve written for an online web dev publication. I’m embarassed to say that it took 6 months to get it out, but here it is! Thanks to Geoff and Chris for helping me with this! I do hope to write more content for other publications throughout the rest of the year.

Read Web Features That May Not Work As You’d Expect on CSS Tricks

Thanks for reading! If you liked this post, consider supporting my work by:

sharing my work,
refering me for a job (or hire me for freelance work),
sponsoring me on Patreon,
buying Me A Coffee,
buying Me A Ko-Fi,
sending me a tip on Paypal,
~~buy something on Amazon with my affiliate link (I’ll earn a commision on this as an amazon affiliate)~~,
or buy a domain on Namecheap with my affiliate link.

You can also subscribe to my newsletter.

Have feedback? Send an email to gandiyafarai+feedback at gmail dot com

You Can Give A Website Dark Mode With Just One CSS Property

Farai Gandiya — Sun, 27 Jun 2021 10:02:51 +0000

You Can Give A Website Dark Mode With Just One CSS Property was first published on Farai's Codelab.

I’m too lazy to do all the work required to make a proper dark mode for now so I’m trying to find a quick fix. I tried to use filter: invert(100%) on the body but it would literally invert everything, including images and emojis. I was about to give up on it before I came across Jim Nielsen’s post on the color-scheme property which tells the browser what color schemes a site (or element) supports and changes the default color based on the user’s preferences. To use it, you just need to specify the color-scheme property on the :root element.

:root {
    color-scheme: light dark;
}

This supports both light and dark mode. You can also just set light or dark as well as apply the color-scheme property to a specific element. In some environments (like email), you can also apply a <meta name="color-scheme"> tag.

Note that it doesn’t work in Firefox yet.

Issues

When I tried this the link colors didn’t change in Safari resulting in this ugly and inaccessible style. It worked perfectly fine in Chromium browsers however.

Jim had the same issue. Because of that, I ended up implementing some sort of dark mode using Apple’s dynamic system colors. The CSS Colors Module 4 has system colors corresponding to the OS’s UI but they don’t have good support so I just copy-pasted the RGB values. Temper Temper explains how to “fix” dark mode on Safari better than I have.

Also, looking at color scheme’s spec, there was this scary warning noting that they don’t guarantee a color palette.

The light and dark color schemes don’t represent an exact color palette (such as black-and-white), but a range of possible palettes. To guarantee specific colors, authors must specify those colors themselves. Note also that, consequently, pairing default or colors with author-specified colors cannot guarantee any particular contrast level; it might be necessary to set both foreground and background colors together to ensure legibility.

Given those issues my attempt to implement dark mode lazily has become the start of an interesting journey. Still, it’s worth putting in.

Additional Reading

Thanks for reading! If you liked this post, consider supporting my work by:

sharing my work,
refering me for a job (or hire me for freelance work),
sponsoring me on Patreon,
buying Me A Coffee,
buying Me A Ko-Fi,
sending me a tip on Paypal,
~~buy something on Amazon with my affiliate link (I’ll earn a commision on this as an amazon affiliate)~~,
or buy a domain on Namecheap with my affiliate link.

You can also subscribe to my newsletter.

Have feedback? Send an email to gandiyafarai+feedback at gmail dot com

Converting Video into Animated Images Using FFmpeg

Farai Gandiya — Fri, 11 Jun 2021 03:47:28 +0000

Converting Video into Animated Images Using FFmpeg was first published on Farai's Codelab.

This post has been republished with new information.

This post will supplement an upcoming post on why you should stop using GIFs in favor of newer image and video formats. Here’s how to generate animated images in various media formats using FFmpeg.

Preparing The Source Clip For Further Conversion
Images
- GIF
- WebP
- Sequenced AVIF (Chrome For Now)
Videos
- AVC/h.264 MP4 (widest support)
- HEVC/h.265 (WebKit)
- VP8/VP9 (Not WebKit)
- AV1 Video (Chrome and Firefox)
  - aomenc (Reference Encoder)
  - rav1e (Fastest)
  - SVT-AV1

Preparing The Source Clip For Further Conversion

First thing to do is convert it to an uncompressed y4m video, slicing it up and setting the framerate as necessary:

ffmpeg -ss <start_point> -t <duration_from_start> -i <source_media> -an -vf 'scale=<width>:<height>,setpts=<stretch_factor>*PTS,fps=<framerate>' -pix_fmt yuv420p <raw_input>.y4m

What the option flags mean:

Option	Description
`-ss`	Marks the start position of the video stream as a time duration
`-t`	Specifies the duration of the datastream from `-ss` (or 0) as a time duration. Use both `-ss` and `-t` before `-i` to limit the video input.
`-an`	Removes audio
`-vf`	The video filters
`scale`	Sets the w idth and h eight of the video
`setpts`	Sets the presentation timestamps (PTS) for the video. Used to speed up and slow down video
`fps`	Specifies the framerate for the video.
`pix_fmt`	Sets the color format. Necessary if you’re converting from GIFs.

Note, if you are going to set a framerate for a GIF, it has a delay between frames in hundreths of a second (fps=100/delay). So:

Delay	Framerate
1	10pfs
2	50fps
3	33fps
4	25fps
5	20fps
6	15fps

Images

GIF

ffmpeg -i <source_input>.y4m -filter_complex "[0:v] split [a][b];[a] palettegen [p];[b][p] paletteuse" -loop 0 <output>.gif

Simply put the -filter_complex flag in this case generates a color palette to use in the GIF. See GIPHY’s Engineering blog on how to make GIFs to explain the flag.

-loop 0 makes it loop forever.

You can leave it out, resulting in a much smaller GIF (at the expense of quality).

WebP

ffmpeg -i <raw_input>.y4m -loop 0 -q:v 100 -compression_level 6 <output>.webp

q:v is the image quality, from 0 to 100. It’ll be much smaller, but WebP can get blocky if you push the quality too hard.

Like GIF, -loop 0 makes it loop forever.

Sequenced AVIF (Chrome For Now)

ffmpeg doesn’t support writing to AVIF containers when I wrote this, so you nee to use avifenc to do this.

avifenc <raw_input>.y4m <output>.avif

Videos

From here on out, I won’t go over the options too much since there’s a lot to consider, most of which I don’t understand so I’ll link to the respective encoding guide if you want more options.

AVC/h.264 MP4 (widest support)

ffmpeg -i <raw_input>.y4m -c:v libx264 -preset veryslow <output>.mp4

The preset flag tries to find the best tradeoff between quality and compression at a given bitrate and file size.

H.264 Encoding Guide

HEVC/h.265 (WebKit)

ffmpeg -i <raw_input>.y4m -c:v libx265 -preset veryslow -tag:v hvc1 <output>.mp4

You need the -tag:v hvc1 for the video to play in Safari. Thanks Aaron!.

preset is the same as in AVC.

H.265 Encoding Guide

VP8/VP9 (Not WebKit)

Switch the vp8 for vp9 depending on which one you choose. vp9 is newer so better, although it doesn’t have the support of vp8 (although it’s reasonable).

ffmpeg -i <raw_input>.y4m -c:v vp9 <output>.webm

VP8 Encoding Guide/VP9 Encoding Guide

AV1 Video (Chrome and Firefox)

Update 21 May 2021: When I initially wrote this post, my build of ffmpeg didn’t include different av1 encoders so I tried to encode AV1 videos using the encoders directly. As I was updating this post to add instructions on how them, I noticed that the alternatate encoders are now built in the standard build of ffmpeg.

Encoding AV1 will use a lot of CPU and it takes longer than the others.

ffmpeg -i <raw_input>.y4m -c:v libaom-av1 <output>.webm

If you’re on an older version of ffmpeg, you need to add -strict -2. Note that you can also use rav1e and SVT-AV1 in FFmpeg to encode AV1 videos.

AV1 Encoding Guide.

aomenc (Reference Encoder)

You’ll either have to compile aomenc yourself or you can use aomenc’s Windows builds by Marco Sousa.

aomenc.exe -o <output_file>.webm <raw_input>.y4m

rav1e (Fastest)

You can find rav1e’s builds on GitHub. Note that rav1e only generates the raw video stream (saved as an ivf file) which needs to be muxed into a webm container file.

rav1e.exe <raw_input>.y4m -o <intermediate>.ivf
ffmpeg -i <intermediate>.ivf-c:v copy <output>.webm -hide_banner -loglevel error

SVT-AV1

While there are some SVT-AV1 builds on GitHub, the project is marked as archived and SVT-AV1’s Gitlab Repo doesn’t have any prebuilt binaries so you’ll have to compile it yourself. However you get the binary, like rav1e, you need to mux the raw video stream into a webm container file.

SvtAv1EncApp.exe -b <intermediate>.ivf -i <raw_input>.y4m
ffmpeg -i $destIvf -c:v copy <output>.webm

Thanks for reading! If you liked this post, consider supporting my work by:

sharing my work,
refering me for a job (or hire me for freelance work),
sponsoring me on Patreon,
buying Me A Coffee,
buying Me A Ko-Fi,
sending me a tip on Paypal,
~~buy something on Amazon with my affiliate link (I’ll earn a commision on this as an amazon affiliate)~~,
or buy a domain on Namecheap with my affiliate link.

You can also subscribe to my newsletter.

Have feedback? Send an email to gandiyafarai+feedback at gmail dot com

Converting Video into Animated Images (and Other Videos) With FFmpeg (For The Most Part)

Farai Gandiya — Thu, 15 Apr 2021 17:07:31 +0000

Converting Video into Animated Images (and Other Videos) With FFmpeg (For The Most Part) was first published on Farai's Codelab.

This post will supplement an upcoming post on why you should stop using GIFs in favor of newer image and video formats. Here’s how to generate animated images in various media formats using FFMpeg.

Preparing The Source Clip For Further Conversion
Images
- GIF
- WebP
- Sequenced AVIF (Chrome For Now)
Videos
- AVC/h.264 MP4 (widest support)
- HEVC/h.265 (WebKit)
- VP8/VP9 (Android)
- AV1 Video (Chrome and Firefox)

Preparing The Source Clip For Further Conversion

First thing to do is convert it to an uncompressed y4m video, slicing it up and setting the framerate as necessary:

ffmpeg -ss <start_point> -t <duration_from_start> -i <source_media> -an -vf 'scale=<width>:<height>,setpts=<stretch_factor>*PTS,fps=<framerate>' -pix_fmt yuv420p <raw_input>.y4m

What the option flags mean:

Option	Description
`-ss`	Marks the start position of the video stream as a time duration
`-t`	Specifies the duration of the datastream from `-ss` (or 0) as a time duration. Use both `-ss` and `-t` before `-i` to limit the video input.
`-an`	Removes audio
`-vf`	The video filters
`scale`	Sets the w idth and h eight of the video
`setpts`	Sets the presentation timestamps (PTS) for the video. Used to speed up and slow down video
`fps`	Specifies the framerate for the video.
`pix_fmt`	Sets the color format. Not necessary and the only reason I specified it is because I had issues in an earlier draft

Note, if you are going to set a framerate for a GIF, it has a delay between frames in hundreths of a second (fps=100/delay). So:

Delay	Framerate
1	10pfs
2	50fps
3	33fps
4	25fps
5	20fps
6	15fps

Images

GIF

ffmpeg -i <source_input>.y4m -filter_complex "[0:v] split [a][b];[a] palettegen [p];[b][p] paletteuse" -loop 0 <output>.gif

Simply put the -filter_complex flag in this case generates a color palette to use in the GIF. See GIPHY’s Engineering blog on how to make GIFs to explain the flag.

You can leave it out, resulting in a much smaller GIF (at the expense of quality).

WebP

ffmpeg -i <raw_input>.y4m -loop 0 -q:v 100 -compression_level 6 <output>.webp

q:v is the image quality, from 0 to 100. It’ll be much smaller, but WebP can get blocky if you push the quality too hard.

Sequenced AVIF (Chrome For Now)

ffmpeg doesn’t support writing to AVIF containers when I wrote this, so you nee to use avifenc to do this.

avifenc <raw_input>.y4m <output>.avif

Videos

AVC/h.264 MP4 (widest support)

ffmpeg -i <raw_input.y4m -c:v libx264 -preset veryslow .<output>.mp4

The preset flag tries to find the best tradeoff between quality and compression at a given bitrate and file size.

H.264 Encoding Guide

HEVC/h.265 (WebKit)

ffmpeg -i <raw_input>.y4m -c:v libx265 -preset veryslow -tag:v hvc1 <output>.mp4

You need the -tag:v hvc1 for the video to play in Safari. Thanks Aaron!.

preset is the same as in AVC.

H.265 Encoding Guide

VP8/VP9 (Android)

Switch the vp8 for vp9 depending on which one you choose. vp9 is newer so better, although it doesn’t have the support of vp8 (although it’s reasonable).

ffmpeg -i <raw_input>.y4m -c:v vp8 <output>.webm

VP8 Encoding Guide/VP9 Encoding Guide

AV1 Video (Chrome and Firefox)

Encoding AV1 will use a lot of CPU and it takes a very long time; more so on my shitty laptop encoding at 0.1fps.

ffmpeg -i <raw_input>.y4m -strict -2 -c:v libaom-av1 .<raw_input>.webm

AV1 Encoding Guide.

Thanks for reading! If you liked this post, consider supporting my work by:

sharing my work,
refering me for a job (or hire me for freelance work),
sponsoring me on Patreon,
buying Me A Coffee,
buying Me A Ko-Fi,
sending me a tip on Paypal,
~~buy something on Amazon with my affiliate link (I’ll earn a commision on this as an amazon affiliate)~~,
or buy a domain on Namecheap with my affiliate link.

You can also subscribe to my newsletter.

Have feedback? Send an email to gandiyafarai+feedback at gmail dot com

I Now Have A Newsletter; See The First Email

Farai Gandiya — Sun, 11 Apr 2021 19:01:21 +0000

After hesitating a lot on making a newsletter, I’m pleased to introduce the Farai’s Codelab Newsletter.

Read the first edition in which I introduce the myself, newsletter, my current work and future plans.

The newsletter is availible online as well as an RSS feed of the emails.

Only thing left is to add a fancy subscribe box for the site and to keep it running. My measure of success is when I have to pay for it. Hope to get to that point!

plz subscribe, kthanxbai!

How I Optimize PNGs

Farai Gandiya — Tue, 30 Mar 2021 20:43:16 +0000

How I Optimize PNGs was first published on Farai's Codelab.

Here’s how I optimize images in three steps:

Resize the image using ImageMagick to the desired sizes. The size I usually convert to is a 240px width image for small screen devices like feature phones that fit the media query of (max-width: 400px) and (-webkit-device-pixel-ratio: 1).
- If it’s a non-photogenic image inside a JPEG, I convert the image sizes to PNG using it.
- If I’m converting an SVG, I use Inkscape (specifically inkscape-cli) since it’s very good at rendering SVGs.
Use CompressPNG.com to restrict the color palette as much as I can using a slider. This makes a massive difference in file sizes.
Finally, I then use ZopfliPNG to recompress the PNG for a slight size reduction. PNGs use some sort of DEFLATE compression algorithm to compress all the image data—Zopfli is a very good, but slow implementation of DEFLATE. It’s a ~7% reduction which is just a handful of kilobytes.
Depending on the file size reduction and quality, I might also convert it to WebP or AVIF. This isn’t as common since PNGs ae really good at non photographic images.

This process is based on my naive understanding of how PNGs work. I want to make a comprehensive post on how PNGs work and on how to use that understanding to better compress PNGs. There’s a lot about the compression techniques, color indexing, sampling and the like.

Glossary

non-photographic image—not a photo like illustrations, graphics and diagrams.

Thanks for reading! If you liked this post, consider supporting my work by:

sharing my work,
refering me for a job (or hire me for freelance work),
sponsoring me on Patreon,
buying Me A Coffee,
buying Me A Ko-Fi,
sending me a tip on Paypal,
~~buy something on Amazon with my affiliate link (I’ll earn a commision on this as an amazon affiliate)~~,
or buy a domain on Namecheap with my affiliate link.

DEV Community: Farai Gandiya

Improving Netlify's Default HTTP Cache-Control Headers for Immutable Assets

What does Netlify do?

Seems ok. What’s the problem?

How To Fix This

Conclusion

Goodbye 2021, Hello 2022

Writing

Posts On Other Websites

Newsletter

My Many Unpublished Posts

Programming

Open Source

YouTube

Career

Lesson of The Year: Embrace Experimentation and Iteration

Conclusion

Working With Web Feeds: It’s More Than RSS

Imagine if Doctors Relied on Google As Much As Programmers Do

Breadcrumbs In Hugo

Give Your Illustrations Dark Mode With This Simple Trick (SPOILER: It's a CSS Filter)

Web Features That May Not Work As You’d Expect

You Can Give A Website Dark Mode With Just One CSS Property

Issues

Additional Reading

Converting Video into Animated Images Using FFmpeg

Preparing The Source Clip For Further Conversion

Images

GIF

WebP

Sequenced AVIF (Chrome For Now)

Videos

AVC/h.264 MP4 (widest support)

HEVC/h.265 (WebKit)

VP8/VP9 (Not WebKit)

AV1 Video (Chrome and Firefox)

aomenc (Reference Encoder)

rav1e (Fastest)

SVT-AV1

Converting Video into Animated Images (and Other Videos) With FFmpeg (For The Most Part)

Table of Contents

Preparing The Source Clip For Further Conversion

Images

GIF

WebP

Sequenced AVIF (Chrome For Now)

Videos

AVC/h.264 MP4 (widest support)

HEVC/h.265 (WebKit)

VP8/VP9 (Android)

AV1 Video (Chrome and Firefox)

I Now Have A Newsletter; See The First Email

How I Optimize PNGs

Glossary