DEV Community: Jochen Lillich

How I schedule work and communication with Morgen

Jochen Lillich — Wed, 29 Jul 2026 19:22:24 +0000

Time is my most valuable resource. Unlike most others, it can’t be replenished, and at my age, I’m acutely aware of this fact. The app I’ve been using daily for years now to manage my time is Morgen. I can’t believe this service is still such a hidden gem.

Recently, Zapier published a blog post titled “The 5 best time blocking apps in 2022”, and it didn’t even mention Morgen, let alone discuss it in depth. I only found Morgen by chance myself, but it took me only a few minutes of using it until I signed up for a yearly Pro subscription.

I’ve learned to accept that “To-Do List Zero” is an unrealistic ideal. There will always be work queuing up, especially since being a good husband, father and friend requires time, too. This makes it imperative for me to make sure I prioritize every task and responsibility properly.

Many of these tasks are what Cal Newport calls “shallow work”, for example, responding to emails and support requests. The real value comes from “deep work” tasks like creating a strategy or coding a new product feature. As the leader of a fully remote company with international customers, arranging video calls at times that don’t completely disrupt my focus on these “deep work” tasks is another one of my time management challenges. Morgen, a Swiss-made web application, solves all of them wonderfully.

On the surface, Morgen is a typical web-based calendaring app. If you’ve used one of them, you know they all look just about the same, and that also applies to Morgen. In fairness, I don’t know if there actually is more than one way to display a calendar. The first advantage of Morgen is that it integrates with all major productivity suites, from Google and iCloud to NextCloud and Zoho. Having access to all my personal (FastMail) and work-related (Google) calendars from a single app is essential for me to keep track of my obligations.

Morgen has a few nifty features that stand out for me. “Quick Join” is one of them. If the current calendar event comes with a link to Zoom, Google Meet or a similar video service, Morgen automatically displays a button on its main screen that launches me right into the call.

“Calendar sets” is a feature I know from Fantastical, one of the best calendar apps in the Apple ecosystem. Calendar sets allow you to group calendars and switch between them with a single keystroke. For example, when I’m planning my upcoming weeks, I need to see not only my personal calendars but also my team’s holiday plans and my family’s upcoming events. Being able to easily switch between a lean “work” calendar set with only my most important events and the full “planning” set saves me time and embarrassment.

Of course, I wouldn’t pay for Morgen if all it did for me was display my calendars. It is the app’s time management features that sold it for me.

Morgen comes with its own to-do list feature, but it also integrates with Todoist, my task manager of choice. Again, if all Morgen did was display my tasks next to the calendar view, I wouldn’t rave about it. What I love about Morgen is that it enables me to use the “time-boxing method” to make sure I’ll actually have the time to focus on what’s important. I can simply drag tasks from the to-do list to the time on the calendar when I’m going to work on them. Morgen displays them as special calendar entries with a checkbox, but it also creates regular calendar entries that signal to other people that I’ll be occupied at that time. These time blocks will also be excluded from the scheduling feature that is one of Morgen’s major strengths.

Planning my own work is one thing, but I also have to make time for communicating with other people. (Sometimes, I even want to!) Morgen has a powerful scheduling feature with which I can mark specific times as available for other people to arrange calls with me. Each availability rule has its own URL that I can put on a website or paste into an email. For customers, I have a permanent link that allows them to schedule 30-minute calls during office hours. When I want to schedule a call with friends, I create one-off time slots simply by marking the next few free evenings. Since Morgen also integrates with major conferencing services, it sends out call invitations that already contain the link to the video call.

There are many more features in Morgen that I didn’t mention, so go check them out for yourself! The free tier already has everything you need to get a good grasp on its strengths. And once it’s got you hooked, the Personal or Pro plans won’t break the bank either.

In summary, I can confidently say that Morgen has become one of the pillars of my personal productivity. Or, paraphrasing the German proverb: “Was du heute willst besorgen, das planst du besser in Morgen!”

Making my own Nerd Font

Jochen Lillich — Fri, 17 Jul 2026 12:40:45 +0000

Well-structured status lines in vim and shell prompts with version control symbols are a nice quality-of-life improvement. Unfortunately, not all monospace fonts come with the necessary PowerLine glyphs. For example, my favourite font is Operator Mono, and it too doesn’t have PowerLine symbols built in.

Thanks to the NerdFonts font patcher, I was able to generate a font variant that has the necessary symbols. I simply ran the tool as a Docker container in my fonts directory:

$ mkdir HCo_OperatorMonoSSmNF
$ docker run --rm -v $PWD/HCo_OperatorMonoSSm/OpenType:/in \
  -v $PWD/HCo_OperatorMonoSSmNF:/out \
  nerdfonts/patcher --windows --powerline --powerlineextra

Now my spaceship shell prompt sparks even more joy.

The secret to Linux workstation peace of mind

Jochen Lillich — Wed, 15 Jul 2026 12:57:25 +0000

In our inaugural Deep Dive session in The Server Room, I gave a demo of my Linux workstation setup. (Deep Dive sessions are part of the recurring events program in my DevOps membership community.) In my demonstration, the operating system did, of course, take centre stage. After my community members, it now your turn to learn why I’m installing Bluefin as my Linux OS whenever possible.

First, what is Bluefin? Bluefin is a variant of Fedora Silverblue, which in turn is the “atomic” variant of Fedora’s Workstation distribution. Before we get into what makes Bluefin special, let me explain briefly what an atomic Linux distribution is.

The power of the atom(ic)

Traditional Linux distributions use their specific package manager to set up the initial operating system. Later on, you use the same package manager to add, update or remove packages depending on your needs. This is how most distributions work, but it has significant downsides. Package management can go horribly wrong. One of the best-known examples in recent history was Linus Sebastian destroying his whole Linux installation just by trying to install Steam. Of course, this event made huge waves on his YouTube channel “Linus Tech Tips”.

Atomic Linux distributions aim at eliminating the risk of bricking your machine. They take a page out of the version management playbook by maintaining multiple variants of the operating system installation. Adding, updating or removing packages all happens on a clone of the current OS setup. In consequence, these changes will only take effect next time you reboot the machine. And should the new version have any kind of glitch, you can switch back to the previous working version simply by choosing it from the boot menu.

Separating OS and user apps

The best way to keep this mechanism robust, and at the same time retain flexibility in your software selection, is to install user applications independently of the OS. On Bluefin, the preferred method of installing GUI applications is via Flatpak. Flatpak is a package manager that keeps applications “sandboxed” for improved security. The common way to install command line applications is inside Linux containers managed by Podman. This might sound like a tedious endeavour, but there are two tools that make this process easy and seamless. They do so by integrating the containers tightly with the host system. For example, they automatically mount your home directory into the container, which makes your individual configuration and data files available inside the container. Even if you don’t use an atomic Linux distribution, I recommend you have a look at Toolbx and Distrobox.

The two main atomic distributions from Fedora are Silverblue and Kinoite. They differ in the desktop environment they run, GNOME and KDE, respectively. They’re already great but Universal Blue makes them even better.

Bluefin is the Fedora we deserve

The Universal Blue team builds custom OCI/Docker images to deploy the operating system, a feature that isn’t yet available in official Fedora. In consequence, OS updates are downloaded efficiently from a container registry. The main Universal Blue spins are Bazzite, Aurora, Ucore and Bluefin. They all add specific applications, tools, hardware support, and quality-of-life improvements on top of their upstream distributions. Bazzite, for example, aims at an improved Linux gaming experience, and adds support for a range of gamepads and the proprietary NVidia drivers.

With Bluefin, Universal Blue targets the majority of Linux users. For this reason, it adds a bunch of useful applications and extensions to Silverblue’s GNOME desktop environment. It also comes with support for popular hardware such as NVidia GPUs, Framework and even Surface laptops. Its extended version “Bluefin DX” (for “developer experience”) adds tooling that software developers will appreciate, such as the VS Code development environment and the Homebrew package manager. Homebrew is popular on macOS and a viable alternative to the container-based environments I mentioned above. Finally, if you prefer KDE over GNOME, Aurora is there waiting for you.

Additional to what feels like a very solid software collection, Bluefin adds quality-of-life improvements that make its everyday use a pleasure. For example, it installs a daily background job that automatically performs upgrades on all relevant software. It not only upgrades the base OS, but also the installed Flatpaks, and even all the Toolbx and Distrobox containers. Every time you boot your laptop, you can be sure you will work with the latest software versions.

This is how I think a Linux distribution should function in 2026: robust, practical and convenient. It’s why I’m installing Bluefin on all my machines, and even machines I set up for family members. It’s the most low-maintenance Linux experience I’ve ever had. For more information and a peek into the work of the Universal Blue team, watch Noel Miller’s talk from the F40 Release Party!

If you have any questions on my setup, post them in the comments below. I'll be happy to answer them.

The post The secret to Linux workstation peace of mind appeared first on The Monospace Mentor.

A homelab is a skill booster

Jochen Lillich — Sat, 09 Aug 2025 11:50:04 +0000

In her article “What’s the biggest unsolved problem within Site Reliability Engineering?”, Paige Cruz identifies a critical challenge facing our industry: how do we pass the hard-won knowledge of veteran system administrators down to a new generation that has never worked outside cloud infrastructure?

Cruz makes a compelling case that this knowledge transfer gap represents SRE’s biggest unsolved problem. However, I take issue with one of her conclusions. She argues that “It’s also not acceptable or sustainable or accessible in my opinion to continue to expect people set up home-labs and experiment themselves. We don’t ask veterinarians to practice surgery in their free time at home and I don’t think we should expect that from future SRE.”

I respectfully disagree. A home lab represents the best investment a future SRE can make to secure their career growth.

The home lab advantage

The veterinarian comparison doesn’t hold up under scrutiny. Veterinarians need live animals, surgical equipment, and sterile environments – resources that are neither practical nor ethical to maintain at home. Particle physicists need billion-dollar accelerators. But what does a sysadmin or SRE need to experiment with infrastructure? A few decommissioned Lenovo ThinkCentre boxes and a network switch.

The barrier to entry is remarkably low, and the learning potential is enormous.

Scientific research consistently shows that skill development requires more than passive consumption of learning materials. Growing expertise demands practical exploration – running into problems, analyzing failures, and working through solutions. A home lab provides exactly this environment, free from the constraints and pressures of production systems.

You don’t need a data centre in your basement. One or a few small networked PCs will serve you well. These machines consume minimal power and don’t need to run around the clock. Fire them up when you want to experiment with a new service mesh, test disaster recovery procedures, or break something just to see what happens.

For those who can’t or don’t want to maintain physical hardware, alternatives exist. “Pubmox” – the shared Proxmox Virtual Environment I maintain for members of “The Server Room” – offers free access to virtual infrastructure for hands-on learning.

The learning laboratory

A home lab creates space for the kind of curiosity-driven exploration that builds deep technical intuition. When you’re troubleshooting a networking issue at 2 AM in your home lab, you’re not worried about affecting customers or violating change management policies. You can take risks, make mistakes, and learn from the consequences.

This environment encourages the experimental mindset that separates great SREs from those who merely follow runbooks. You learn to ask “what if” questions and then actually test your hypotheses.

Knowledge transfer requires community

While I advocate strongly for home labs as skill boosters, they’re not the complete answer to Cruz’s original question about knowledge preservation. The solution to transferring veteran sysadmin wisdom doesn’t lie in solitary exploration – it requires collaborative knowledge transfer.

Home labs excel at building technical skills, but they can’t replace the mentorship and context that comes from working alongside experienced practitioners. The real magic happens when hands-on experimentation combines with guided learning from those who’ve solved these problems before.

This is why I’m launching “Basic Linux System Administration” – an instructor-led course backed by “The Server Room” community. Participants get free access to Pubmox, allowing them to practice what they learn while benefiting from my decades of system administration experience.

It’s the best of both worlds: the freedom to experiment and break things, combined with the guidance to understand why systems fail and how to make them more reliable.

A home lab won’t solve SRE’s knowledge transfer problem on its own. But for individual practitioners looking to deepen their skills and advance their careers, it remains an invaluable investment. The combination of practical experimentation and collaborative learning creates the foundation for the next generation of systems experts our industry desperately needs.

The post A homelab is a skill booster appeared first on The Monospace Mentor.

80-hour weeks don’t make a great engineer. They’ll break a great engineer.

Jochen Lillich — Tue, 16 Apr 2024 07:00:00 +0000

I’m not exaggerating when I say I was shocked by a question about working hours that I got during a recent office hour live stream. “So I heard from a developer that he would code and learn for like 80 hours a week when he started in order to be come an amazing engineer. I can barely do 8 hours a day and I want to be an amazing engineer. Advice on that?”

Here’s the TL;DR: Working 80 hours a week is neither healthy nor effective. I’d go as far as to saying it’s impossible. Sitting in front of a screen for 80 hours? Yeah, maybe, if that’s your thing. But doing actual work for 80 hours? You might very well be fooling yourself, but you’re not fooling me.

Photo by Muhammad Raufan Yusup on Unsplash

More negative side effects than benefits

Now, before we get into how many hours of work per week you really should invest into your growth, let’s first consider the side effects of spending 80 hours per week working. That is 16 hours every weekday or more than 11 hours every single day, working on your engineering career. Even if we count necessities like eating and pooping as part of work time, it’ll still leave only between 8 and 12 hours per day for the non-work parts of your life. Parts like, you know, hobbies, social life and sleep.

And it’s scientifically proven that for most people, any amount of sleep below 7 hours a night is not enough. The body needs time to recover physically and mentally. In other words, if you sacrifice sleep for other things for more than a very short time, you’re going to ruin your health. Let’s see how not only your career but your whole life flourishes while you’re dealing with the consequences of burnout (or falling asleep driving your car).

Way to sabotage yourself

Total work time also has severely diminishing returns. In other words, putting in more and more time will yield less and less additional results. Effective work requires effective rest. Why do you think Ford and his fellow industrialists themselves introduced the 40-hour work week? And that was just about physical work. Every athlete knows that muscle growth doesn’t benefit from endless training. There needs to be downtime during which the body can recuperate and grow to meet demand.

For our brain, it’s similar, but it’s even more limited in how long it can work at high performance. For knowledge workers, an 8-hour day is like a 16-hour day for the industrial laborer. It’s just not possible to keep your focus for that long, not only because people keep interrupting you, but also for physiological reasons. The brain needs time and rest to build new tissue such as the myelin that grows around neurons.

It’s not about the working hours at all

There’s also the aspect of how much you can even achieve on your own. There’s that story of someone who shipwrecked on an island with just a chess board, and ten years later returned as a grand master. That’s bullshit, because you need other people to beat the Dunning/Kruger effect, i.e. you don’t know what you don’t know. The only way to find the solutions that your brain couldn’t come up with by itself is collaboration. Grand masters learn from other grand masters, amazing engineers learn from other amazing engineers. That’s why I’m so excited about The Server Room, the community of DevOps practitioners I’m going to launch soon.

The good news is that success is not about the total time you spend working at all. Rather, it’s about how much focus time you can get in. In his book “Deep Work”, Cal Newport explains how he manages to thrive as a busy author and academic. “Three to four hours a day, five days a week, of uninterrupted and carefully directed concentration, it turns out, can produce a lot of valuable output.” Just 15 to 20 hours of focused work per week will do the trick. And you’ll still have plenty of time left that you can spend schmoozing and snoozing.

Please don’t blindly believe those hustle culture tech bros who tout how they built their success on relentless work. Even if there’s a kernel of truth to their claims, what they don’t tell you is that they also had the means to mitigate the negative consequences of their self-abuse. Instead, I recommend that you read the book “Time Off: A Practical Guide to Building Your Rest Ethic and Finding Success Without the Stress“. It’s full of good advice like this one:

But as entrepreneurs, managers, and individuals, it’s a dangerous and vicious cycle if we keep assuming that we are productive and effective only because we put in a crazy amount of hard work and long hours, grinding it out. All too often, we get stuck in mediocrity because we are afraid, or maybe just too lazy and comfortable, to question the status quo and the rules set by societal norms.

John Fitch, Max Frenzel, and Mariya Suzuki, “Time Off”

This article was originally published in my newsletter, “News From the Server Room”. To get my column “Mentor Monologue” fresh when I publish it, subscribe here.

The post 80-hour weeks don’t make a great engineer. They’ll break a great engineer. appeared first on The Monospace Mentor.

Startup lessons: 5 mistakes I made when I chose my brand name

Jochen Lillich — Mon, 06 Nov 2023 00:00:00 +0000

Choosing “Opsitive” as the brand name for my teaching and mentoring business was a mistake. I know that now, and in this article, I’ll explain what made me realize that there’s much more to branding than just being clever. I’ll outline the tests that helped my identify the weaknesses of my current brand and verify that my new choice, “Monospace Mentor”, is going to be much stronger.

A few weeks ago, I discovered “Creator Science”, a podcast for content creators I can already highly recommend. One of the latest episodes peaked my interest in particular: “How to invent a GREAT brand name.”. For years, I’ve been dissatisfied with my hosting company’s branding, so I was eager to learn more about this topic. And learn I did!

On the podcast, branding expert Alexandra Watkins outlined two acronyms she invented to gauge the quality of a brand name; one checks the attributes to aim at, the other points out pitfalls to avoid. These acronyms are “SMILE” and “SCRATCH”, and you can find their detailed explanations over on Alexandra’s website, “Eat my words”.

Failing the SCRATCH test

Applying these checks to our hosting product confirmed my gut feeling – its name bombed spectacularly. We’ll address this issue in the marketing strategy development process that we’ve just launched recently. But then I did the same test with “Opsitive”, and it dropped on me that it was weak sauce, too!

First of all, the term “Opsitive”, while being a play on the words “Ops” and “Positive”, does look like a typo and thus fails the first “S” in the SCRATCH test, “Spelling challenged”. The fact that I kept getting emails meant for an online shop for women’s healthcare should have alerted me early on.

Its wordplay character also makes the name suffer from the “Curse of knowledge”, symbolized by the second “C” in SCRATCH. If you’re not familiar with the fact that “Ops” is short for “IT Operations”, you’ll not get the joke.

Taking a less negative perspective, the name is unfortunately also lacking all of the qualities you should search for in a brand name according to the SMILE guidelines:

It’s not suggestive. It doesn’t say much at all, really.
As a made-up play on words, it’s hard to memorize. Was it “Opsitive”, or “Opositive” like the blood type, or maybe just “Opsitiv”?
It doesn’t conjure any image because it’s abstract at best and nonsensical at worst.
It doesn’t have much legs in that it can be extended to other topics. In fact, its explicit emphasis on “Ops” already puts software development out of scope, an area I definitely want to cover!
For the same reasons for which it fails to spur imagination, it’s also not emotional.

There’s another important shortcoming of the old name. It doesn’t put me in the centre, even though that’s the core of my business model: It’s not about the topics I teach, it’s about how I teach them, about my unique way to guide my mentees to their success.

So I needed a new name.

Finding a new name

Coming up with a new name was hard. At first, it seemed that all the good names were already taken. My main problem was finding a name that could accommodate both sides of DevOps, operations and development, but wasn’t at the same time terribly generic. I used several thesaurus websites to research variations on the themes of teaching and mentorship. After many experiments and letting ideas germinate in my brain for days, inspiration finally hit me. There is one thing that unites all system administrators and software developers: they use terminal emulators, text editors and IDEs. And in those, they use monospace fonts. Nerds love their fonts. There are dozens of websites showcasing the huge spectrum of programming fonts, and some people like me even pay good money for a neat commercial one.

And since I just can’t shake my silliness completely, I went for an alliteration as my new brand name. From here on out, I’ll be the “Monospace Mentor”, and my web presence is now at www.monospacementor.com!

Let’s do the SMILE check again.

“Monospace Mentor” is suggestive of personal teaching, and it conjures the image of a programmer’s computer screen.
It is easy to memorize, helped by the alliteration.
It does have legs for expansion in the area of DevOps education.
Is it emotional? I’m not sure, but it’s definitely whimsical!

Did I eliminate the problems of the old name?

It doesn’t look like a typo.
It’s not confusingly similar to the brand of anyone in the teaching space I know of.
It doesn’t restrict later expansion.
It’s not annoying.
It’s not too tame or bland.
No curse of knowledge issue.
And it’s easy and unambiguous to pronounce.

Much better!

Lots of work ahead

I’ll admit that I’m a bit self-conscious about making this kind of a sharp turn. Curiously, we don’t have these inhibitions when we’re coding. Hey, naming and cache invalidation, amirite? 😄 And after all, learning in public is at the core of what I do.

Since I never had any marketing training, I’m grateful that people like Alexandra Watkins share their expertise so others like me can learn from it. The SMILE and SCRATCH tests were tremendously helpful in finding a name that doesn’t make me anxious it’ll lose me potential business simply for the reason that it doesn’t really work as a brand.

Now I’ve got my work cut out for myself. I have to create new brand assets and update the many places where my teaching business is already present.

I’m happy with the new name. Are you, too? Please let me know what you think on my new Mastodon account or on the Opsitive – wait, no – Monospace Mentor Discord!

Un-breaking breaking changes using feature flags

Jochen Lillich — Tue, 24 Oct 2023 23:00:00 +0000

When we got a feature request that we really wanted to implement because it would be a great improvement to our product, we found ourselves in a bind. The problem was that the change would most likely break the workflows of other customers. In this article, I describe how I managed to roll out the change without triggering a flood of angry emails.

Recently, a customer of our managed hosting platform for Drupal and WordPress platform asked us to tighten the security of their website deployment. We realized that the necessary changes would not only help this customer but also improve website safety for all our customers. We love this kind of universally useful feature request because for operational efficiency reasons, we try to avoid making changes for individual customers only.

However, it was easy to see that the restrictions we had to implement for this feature request came with a high risk of breaking existing workflows. People were accustomed to the current level of restrictions, and when you’re above a certain number of customers, you can be sure that some of them will have set up processes that rely on this exact status quo.

A breaking change requires copious advance notice. But the customer who had raised the issue was more or less blocked until we shipped a solution. So how do you make a change in your production environment, but limit its effects to exactly those customers whom it will benefit, not disrupt?

The solution to this Catch 22 came to me late at night, right before falling asleep. (Let me know if you have the same kind of brain.) It’s called “feature flags”.

Feature flags

Feature flags are used to enable new features for a limited number of users only. Common reasons to limit the use of a feature to a smaller circle, at least initially, are that you’d like to prevent bursts in resource usage, or that you want to quickly discover issues that only show up in production without the risk that they’ll impact all your customers at once.

Technically, a feature flag allows you to add code paths additional or alternative to existing ones, and switch those new code paths on or off based on specific conditions. Because the state of a feature flag is usually based on information outside of your code, you can enable or disable a feature at any time without having to deploy updates.

Usually, state of a feature flag is derived from other data, for example a database field or an environment variable. If you have an early access program, the feature condition could be customer.eap_member == true.

In a gradual rollout scenario, you might want to make the new feature available to only a fraction of your customer base for starters. You could pick those customers in a targeted or random way, write their customer ID’s to a database table with a feature identifier, and define the feature state like this:

feature_enabled?(customer: current_user.customer_id, feature: 17)`

It’s easy to build a simple feature flag implementation yourself. In more complex scenarios, I recommend you check out mature solutions like the Ruby gems Rollout and Flipper.

Feature flags for breaking changes

In my security change situation, I realized that I was dealing with just another feature rollout, only with the tiniest of target groups. I basically had to create an “early security improvement program” with the customer who requested the change as its only member. This would enable us to roll out the change immediately for them, and after a generous notice period, for all our customers.

The resulting logic in our infrastructure code turned out very simple:

set_up_website_environment

if feature_active?(:full_isolation)
  set_more_restrictive_permissions
else
  set_the_usual_permissions
end

By the time of writing, the new code has long been deployed to production. Funnily enough, it’s still completely dormant because our customer just could not find a good time for its activation yet. We might end up doing the full rollout process right away. Thanks to the feature flag being already in place, we’ll apply the change first to customers in our Early Access Program, and a bit later to everyone else — including the customer who initiated the whole project.

Summary

Feature flags allow you to have your cake, and eat it, too. You can roll out new code, but limit its activation dynamically in a data-driven manner.

Next time you have to introduce a code change that might not make everyone equally happy, consider using a feature flag to control its blast radius.

Use the Rails console to do a manual Mastodon link verification

Jochen Lillich — Wed, 16 Nov 2022 16:31:00 +0000

When someone on Mastodon asked for help with a Ruby question, my friend and business partner Markus kindly referred them to me, knowing that I’m always happy to help other Ruby developers. Their question was about executing Mastodon code from a custom script. In this post, I describe how I solved this problem using the Rails console and a custom subclass.

The problem the person wanted to solve was testing the logic Mastodon uses for verifying the ownership of a user’s websites.

For details on how Mastodon verifies websites, see the section “Link verification” in the Mastodon user profile documentation. In short, you have to add a link from your website back to your Mastodon account, using either a <a rel="me" ...> link in the page body or a <link rel="me" ...> reference in its header.

Ideally, this test would use the original Mastodon codebase to make sure it worked across application updates.

Not the solution: standalone script

On my live stream, I first attempted to write a standalone script. My thought was to instantiate the VerifyLinkService class fromapp/services/verify_link_service.rb. I was hoping I’d get away with requiring a few dependencies from the Mastodon application, and maybe replacing a few classes with mocks. But I quickly ended up including ActionSupport and the kitchen sink, which made this approach unviable.

The solution: Rails console

In order to use the original verification code, I had to run it in the context of the Mastodon application. The easiest way to do this is using the Rails console. However, spinning up a Rails console requires a working Rails application–database and all. Before I describe how to solve this, let’s first build the code we’re going to run inside the Rails console to test the verification.

Building the test

Maybe I would have been able to use the original VerifyLinkService class with a few tweaks, but after closer inspection, I was actually only interested in its methods perform_request! and link_back_present?. One fetches the website in question, the other checks if it contains a valid link back to Mastodon.

Unfortunately, both methods use instance variables set in the constructor using Mastodon user data. And, to make matters worse, they’re (rightfully) private methods, which meant that nobody is allowed to call them. And just when I told my viewers, “Well, nobody but the class itself, of course”, inspiration hit me: Not only a class is allowed to call its private methods, but also its child classes! All I had to do is create a subclass of VerifyLinkService, provide our test details in its constructor, and call the two methods required to execute the verification. This is what I ended up with:

class ManualVerifyLinkService < VerifyLinkService
  def initialize
    @link_back = "https://mastodon.social/@geewiz"
    @url = "https://www.geewiz.dev"
  end

  def check
    perform_request!
    if link_back_present?
      puts "Verification successful."
    else
      puts "Verification failed."
    end
  end
end

Spinning up Mastodon

This left the problem of getting Mastodon running. The fastest way to run a Rails application with its auxiliary services and not have to do lots of manual installation is generally with Docker containers, and Mastodon is no exception. Conveniently, the Mastodon code repository already comes with adocker-compose.yml definition that contains everything we need. I found a How-To on the web that describes the launch process well. All I had to do was follow it until, and including, the step where the whole setup is started by thedocker-compose up -d command.

In this state, I was able to launch a Rails console inside the already running web application container:

docker-compose exec -it web bundle exec rails console

Running the test

Into this console, I pasted the definition of my ManualVerifyLinkService class from above.

And finally, I was able to execute the verification by instantiating an object and calling its check method:

v=ManualVerifyLinkService.new
v.check

Since my website is already verified by Mastodon, the result was positive.

This was a fun experiment, and as is often the case when I help someone, I’ve learned something myself in the process.

How I do remote administration using ngrok

Jochen Lillich — Tue, 12 Apr 2022 00:00:00 +0000

This month, a new intern from Spain started at my company. While we used to use Macbooks for our interns, he was the first to get a Linux laptop. Nothing fits the DevOps use case better, after all.

I set up the machine with Manjaro Linux in my office, but I had to solve the problem of later installing and upgrading software after the laptop was handed over. It’ll always access the internet from a private network behind a router, and therefore will not be visible for me from the outside. And I certainly don’t want to commute to Dublin, where we book a coworking space for our interns, every time maintenance is required.

In the end, I chose a minimalist approach by using ngrok. This nifty command line tool, paired with the online service at ngrok.com, allows developers to create network tunnels with an endpoint on the public internet. Its main purpose is for web developers to share access to an application running on their personal machine. In other words, it allows people with whom you share the connection details to connect their web browser to your development laptop, even if it’s inside a closed WiFi network. However, ngrok extends this functionality beyond the HTTP protocol and port 80.

When our intern asked me to install Discord and VS Code earlier today, I asked him to run a simple command and tell me the tunnel endpoint address. The command was:

ngrok tcp 22

It sets up the network tunnel as well as a local website on which it lists the details of each connection. Our intern then gave me the listed address tcp://0.tcp.ngrok.io:14463, from which I constructed an SSH command:

ssh -p 14463 geewiz@0.tcp.ngrok.io

Within a second, I was logged into his laptop. A few minutes of installing new packages and upgrading existing ones later, I was done and asked him to cut the connection. It doesn’t get much simpler than that, does it?

YT channel recommendation for neovim and zsh users

Jochen Lillich — Mon, 10 Jan 2022 13:32:00 +0000

I don’t think I ever copied more vim or zsh configuration from anyone than from ChrisAtMachine. Looks like I found a soulmate. 😊 His YouTube channel is well worth watching!

dotenv and direnv for better developer QoL

Jochen Lillich — Wed, 23 Jun 2021 11:46:12 +0000

A RubyTapas tutorial taught me the difference between the dotenv and direnv tools. While they have significant overlap, they complement each other quite nicely. dotenv simplifies both development and production ops by importing environments variables either from a .env file or the application hosting platform. direnv on the other hand augments the dev environment even further; not only can it add dev-only variables, but also modify shell settings like extending $PATH on a per-project basis.

Verify your assumptions

Jochen Lillich — Fri, 18 Jun 2021 14:42:24 +0000

That’s what I keep preaching to my team but still fail at myself every so often.

This week, I wasted a whole day implementing a fix for a bug that wasn’t there. My task was to add one line to a configuration file in a Chef cookbook. When I ran its test suite, it surprisingly failed and it was clear that my simple change couldn’t be the cause. I assumed it was due to recent changes in software packages installed by the cookbook. So I started adapting our old code to these changes, and one change led to another, and another, until I finally realised that over the course of multiple hours, my one-line bugfix had turned into a full code overhaul. How did I get here?

I decided to start from scratch. git reset --hard HEAD. I quickly fixed my config file and ran the tests. When they failed again, closer inspection revealed that it was actually just the same test failing twice. The cause was a bug that had found its way into the main branch without breaking the CI pipeline (finding the reason for this will be interesting). It had been there all along! After another one-line fix, my change was ready for review. Round 2 had taken me less than half an hour.

The lesson: Don’t rush into battle blindly. Look closely. Identify assumptions and verify them. It might just be windmills!