DEV Community: Magnus Skog

Trello to Notion for free

Magnus Skog — Tue, 01 Jun 2021 18:03:40 +0000

Notion recently make their API public. This has been a long awaited thing that opens up a lot of possibilities for Notion users like myself. My current setup is using Trello as an inbox of sorts and then I move the cards manually to Notion as part of my daily routine. This is of course boring and can now be automated using the new Notion API.

My first version of this used Zapier. Zapier is a no-code solution for things like this. It can connect pretty much anything to everything. You can get a notification to Slack when someone subscribes to your Twitter account, automatically share posts in an RSS feed to your LinkedIn profile, and so on.

They have a great interface for this and it makes things very easy to use. The free version comes with 100 tasks a month. A task a single thing that Zapier does. Moving a Trello card to Notion is a single task for example. However, I also want Zapier to delete the card when it has been moved. This means that I now need two tasks to do this and the free version will not be enough.

The cheapest plan costs $19.99 a month billed annually and includes 750 tasks a month. This would be enough but since I don't use Zapier for other things then this is quite a bit of money just for this.

Automate.io is another option and includes 600 monthly actions for $9.99 a month.

I eventually decided to just use Huginn instead. It turned out that the Notion API is pretty easy to use with Huginn but there is a lot of nested data in it. It took quite some time to get it working so I thought that I'd share my Huginn scenario here so that you can get started quicker.

Click here to get it. It is shared though Github gists.

You will need to fill in the following:

NOTION_DATABASE_ID. The ID of the database to create tasks in. Simply browse to the database in Notion to find it.

NOTION_API_SECRET. Sign up for the Notion API to find it. Do note that you have "share" the database with the API to make it work.

TRELLO_LIST_ID. The list to pull cards from.

TRELLO_API_KEY and TRELLO_API_TOKEN

My scenario is pulling cards from two different Trello boards to two different Notion databases: one for personal use and one for work. Simply delete the Agents you don't want.

Enjoy your free API integration!

Automated morning music with Pipedream, Huginn, and Mailbrew

Magnus Skog — Mon, 29 Mar 2021 08:32:00 +0000

Get a great music album delivered to the inbox every morning, with a Spotify link so that you don't have to find the album yourself. Sounds good? Let's make it happen.

We will break this down into these parts:

Get music albums
Finding out if said music album is any good
Get the Spotify link to the album
Deliver this link every morning. By email or chat.

One of my finds after using this for a while

We can solve the first problem in many ways. We can use the /r/music subreddit on Reddit or even Pitchfork Medias "best new music". What I settled on is Metacritic. They have a simple feed of albums that is easy to develop for. There is also little risk ending up with Led Zeppelin IV in your inbox like you would on Reddit (although that would be good too).

To do this we are going to use Pipedream. It is a more developer oriented Zapier alternative for automating workflows. One of the benefits compared to Zapier is that you can run any Node.js code you like as a step. I've written about Huginn in the past as a self-hosted Zapier alternative. You can do everything in Huginn if you wish but I wanted an excuse to give Pipedream a try.

The pitch for Pipedream

So what you need to do is sign up for Pipedream first. They have a very generous free tier which will be more than enough for this project. Once you've that you go to the sources tab and create a new source. The easiest way to get data out of Metacritic is to use their official RSS feeds, so create a new source and select RSS.

All you have to do now is name your feed and put https://www.metacritic.com/rss/music in the Feed URL. Pipedream will now emit events for every new item in the feed.

Step 1 is now complete. We have a way to get music albums. Step 2 will be a bit more complicated. The album review score is not available in the RSS feed and Metacritic does not make this available through any official means. You can solve this problem by scraping the Metacritic site, but there is a better way. Some awesome person has wrapped a GraphQL API around Metacritic. The thing is available for free over at https://mcgqlapi.com/api.

Using GraphQL we can now get album metadata using a query like so:

query {
  album(input: { album: "Ignorance", artist: "The Weather Station" }) {
    url
    productImage
    criticScore
  }
}

Fantastic! Now we can get critic scores. This works for games, movies, and tv shows as well so you can use this for all kinds of projects. There is a problem though. The RSS feed does not make a distinction between Album and Artist, and we need both to be able to use the API. To solve this we need to scrape Metacritic.

To create a scraper we will once again make use of Pipedream. Select the workflow tab and create a new workflow. Select "USE ONE OF YOUR EXISTING SOURCES" and choose the RSS source we created earlier. Then click the "+"-icon to add an action and select "Run Node.js code" to create a new custom action.

There are many ways to fetch the needed data. I chose to use Cheerio. Add the following code:

const axios = require("axios")
const cheerio = require("cheerio")

async function fetchHTML(url) {
  const { data } = await axios.get(url)
  return cheerio.load(data)
}
const $ = await fetchHTML(event.link)

const album = $(".product_title:first").text().trim()
const artist = $(".band_name:first").text().trim()

return {album, artist}

That should do the trick. The return value will be usable in future steps in Pipedream.

Now, you might have noticed that we could grab the score as well using Cheerio. We don't have to use the GraphQL API for this, so why do it? I did it to learn how to do GraphQL with Pipedream. Feel free to change the Cheerio part above to also include score and then skip the next step. There is also more data available in the GraphQL so have a look around if you want to add something else.

We now have an album. Time to figure out if it's any good! Making a GraphQL request with Pipedream is easy. Choose "Run Node.js code" again and use the following snippet:

const fetch = require("isomorphic-fetch");

const response = await fetch('https://mcgqlapi.com/api', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ query: `
query {
  album(input: { album: "${steps.metacritic_data.$return_value.album}", artist: "${steps.metacritic_data.$return_value.artist}" }) {
    url
    criticScore
    album
    artist
    productImage
  }
}` 
  }),
})

const jsonData = await response.json()
if (!jsonData.data.album) {
  $end()
}
const {url, criticScore, album, artist, productImage} = jsonData.data.album 

if (criticScore < 75){
  $end()
}

return {url, criticScore, album, artist, productImage}

You can access the data from previous steps through the ${steps.metacritic_data.$return_value.album} variables. I decided that I only want albums with a Metacritic score of 75 or above. $end()will end the workflow in Pipedream and it will make sure that we do not proceed to the next step.

Next up: I only want albums that are available on Spotify, because I intend to listen to the album that morning. This is another one of those cases where Pipedream is great since it has built-in support for the Spotify API. Simply select Spotify for the next step and search for "Search". You are looking for the GET /search endpoint. You have to authorize with Spotify to use this.

This step takes the parameter Q. Add the following: {{steps.metacritic_score.$return_value.album}} {{steps.metacritic_score.$return_value.artist}}. Also, make sure you add the "Type" optional parameter as well and put "album" in the field.

Right! Now we have a good album with an artist name, album name as well as a Spotify URL and a nice image of the album. What we want to do now is have this delivered to us every morning. There are so many ways to do this. You can send it by email or you can add it to Trello/Slack/Asana/Monday/Twitter. I chose to have it delivered in my morning newsletter. I like to read that while having my morning coffee so that is perfect.

To do this I use an app called Mailbrew. It can create custom newsletters for you from Twitter data, Reddit subreddits, RSS feeds, and much more. I find it handy for setting up summaries of subreddits so that I don't have to refresh Reddit all day. If you want to try Mailbrew and want to support this blog then feel free to use my referral code to sign up. I would really appreciate it! Now, of course you don't have to use Mailbrew for this. This example will assume that we are using Mailbrew or something similar.

This is unfortunately where we run into an issue with Pipedream. Since we need to output the data as an RSS feed to use Mailbrew then we need to turn the album list into RSS. As far as I know you cannot do that with Pipedream since there is no way to create an "API endpoint" of sorts. Luckily, I got just the thing for this: Huginn!

The first thing you'll need is a Webhook agent to accept incoming albums from Pipedream. Create one and then create another step in Pipedream to post the albums to Huginn. It will be another Node.js step an will look something like this:

Once you've set this up you have completed the Pipedream part.

Of course, we don't want all the albums at once. We want one album every day. So use a Delay agent in Huginn. Set "max_emitted_events" to 1 and it will emit a single event every time it runs.

Finally, add an output agent to add an RSS feed. The full Huginn scenario is available here: https://gist.github.com/mskog/4504fbc4e52df04f2221103bb8306c88

Now you can add this RSS feed to Mailbrew. It looks like this for me:

There you! A lot of effort to save 1 minute every morning. But it's cool is it not? This is what automation is for me! You learn something and you get a little treat every morning in your inbox. Of course this is just the beginning.

Now there are ways to improve this of course. Here are a couple of ideas:

Make sure it doesn't throw an error if the Album is not available on Spotify.
Use the GraphQL API to extract genres for the albums and then only select the genres you like.
Count the number of songs in the album (the data is available in the API response from Spotify) and then don't include albums with only one song
Add a deduplicate agent in Huginn to remove duplicate albums. Just in case something goes wrong and Pipedream sends the same album twice

So this was just a single example of what you can do with automation tools. Since then I've also done something similar with Movies and TV Shows. I get a newsletter every week with all this so that I don't have to check Metacritic. I find it very relaxing to receive emails instead and read them on my terms.

My Ruby on Rails stack for side projects in 2021

Magnus Skog — Mon, 01 Mar 2021 14:32:00 +0000

Why would you start a new side project in Ruby on Rails in 2021? Shouldn't you use something like Next.js or even Blitz? Well, I find that Ruby on Rails is still one of the most productive ways to create an application. It is safe and boring and it gets the job done.

With HTML over the wire becoming popular with tools like Phoenix LiveView and Laravel Livewire, you can't go wrong with good old Ruby on Rails. You can even try Hotwire to stream HTML over web sockets.

With all that said I thought I'd share my recommended stack for Ruby on Rails development in 2021.

TLDR

This is long, so if you don't want to read the whole thing then here is a quick summary:

Ruby and Rails versions : Ruby 3 and Ruby on Rails 6

Architecture : Use "the Rails way"

Background jobs : Sidekiq

Database : PostgreSQL

Asset Management : Webpacker

CSS : Tailwind

Javascript : Alpine.js, Stimulus, and Vue

Pagination : Pagy

Testing : RSpec or Minitest with VCR

Periodic jobs : Clockwork

Code formatting : Prettier for Javascript. Rubocop for Ruby

Exception management : Rollbar or Honeybadger

Sending email : Postmark

Slugs : friendly_id

Full Text Search : pg_search or searchkick

Deployment : Heroku or Dokku

Self-hosted tools : Thumbor, Huginn, OpenFaaS and Grafana

Editor tools : Solargraph, Rails fast nav, Prettier, ruby-rubocop

Monitoring : InfluxDB, Grafana and influxdb-rails

Templating engine : ERB

Admin tool : activeadmin

Ruby and Rails versions

Use the latest Ruby and the latest Rails versions. This is currently Ruby 3.0 and Rails 6.1. They both work well together and I haven't found any reason to stay on the 2.x Ruby versions.

Rails architecture

Caveat : This is just my opinion and if you do things differently then that is great too!

If you are building a side project with Ruby on Rails then I think that you should stick to the Rails way of doing things. Do not reach for fancy things like DDD or hexagonal architecture, especially not if your app is small. There is a very real YAGNI factor when it comes to software architecture and you can waste a lot of time designing something that you don't need.

Don't introduce decorators and view models. Use helpers instead.

Don't extract domain models. Put the code in the ActiveRecord models and the controllers.

Don't reach for interactors to model your domain logic.

Don't try to avoid duplication too early.

I have done multiple apps that need all of the above, but those were huge business critical projects with tens of thousands of lines of code. I've started out with an advanced design from the start so many times and then realized that I didn't even need it.

Once you see a pattern emerging in your app you can start thinking about these things. Perhaps your views got all cluttered with presentation logic that don't really belong in your model. Then, and only then, should you think about perhaps introducing decorators or view components.

Your codebase will eventually tell you what it wants to be and once you are experienced enough you will be able to tell when you need to restructure things.

Background jobs

Sidekiq. I've been using that for many years now and it has been rock solid both for side projects and work projects. I also recommend using ActiveJob with Sidekiq just to make things a bit easier and rails-like. It will make things a bit slower but that shouldn't matter for side projects.

One issue with Sidekiq is that you won't have access to rate limiting and scheduled jobs unless you pay for the enterprise version. If you are running a business then this is probably a good idea but we are going for dirt cheap here. The rate limit part can be solved using something like ruby-limiter for global rate limiting and sidekiq_limit_fetch to limit concurrency per queue. It doesn't have official support for modern Sidekiq versions but in my experience it works fine anyway. You should probably not use that for business critical things however.

If you want to get fancy you can also take a look at Faktory, which is a polyglot job processing framework from the creator of Sidekiq. Then you can write your jobs in any language you want as well.

Database

Use a relational database such as PostgreSQL or MySQL. I do not recommend using document databases such as MongoDB because there is really no reason for it in my opinion. PostgreSQL is plenty fast and you probably won't need things like horizontal scaling for a side project. You can of course try something fancy like FaunaDB, CockroachDB, or any of the many database flavors.

This might come off as old school, but when I develop side projects I need something that stores my data safely and is easy to backup with rock solid performance. PostgreSQL checks all those boxes and it has always served me well. Let me put it his way: There is no reason to ponder "is my data safe" when using PostgreSQL.

Even the logo is amazing.

Asset Management

Two ways: Asset Pipeline or Webpacker. I always use Webpacker. It is modern, you have access to the full NPM ecosystem, and it has all the modern features such as tree shaking and hot module replacement.

Webpacker itself is just a Rails style wrapper on top of Webpack. You probably don't have to worry about configuring webpack if you use Webpacker so that is great.

CSS

OK I realize that this is highly opinionated so I'm just going to give you my recommendation: I'm a big fan of Tailwind. I find that its utility-based approach is great for getting things done. I like it so much that I even paid for the premium component package TailwindUI. It is just one of those things that you either like or don't like. Since you already use Webpacker(right?) you can use whatever you want so find something you like, learn everything about it, and then stick to it.

Javascript

I have divided this into three sections to make it easier to recommend things.

Javascript is one of those things where you can really screw yourself over by choosing the wrong thing. I recommend starting very small and only expand when you know that you need it.

Just a little Javascript

This is when you need things like "show this element when I click this button" and "show the value of this input field in this h2-tag as I type it". For this I recommend using Alpine.js. You don't need to juggle script tags and jQuery with Alpine and it gets the job done when you need something fast and simple. It can however gunk up your HTML quite a bit so when you get to that point you might want to move on to the next section.

Some Javascript

Once you need to introduce NPM packages and need something more organized then Alpine might get messy. At this point I would introduce Stimulus. It is made by the creators of Ruby of Rails. It will clean up a lot of gunk introduced by Alpine and leave you with reusable controllers. Here you can import NPM packages and it works very well with Webpacker. You can of course keep using Alpine as well and use Stimulus for the more advanced things if you wish since they work very well together.

A lot of Javascript

You might not reach this stage and you should be very careful about choosing to go down this path prematurely since it is quite a bit of work. If you need advanced state management, high performance and complicated client side logic then Stimulus might no longer cut it. You may have to reach for a declarative, component based library like React, Vue, or Svelte. All of these, and others, work very well with Webpacker and there are wrappers to make things easier for you. For example we have React on Rails for React integration.

Again, you need to be careful about taking this step because once you start using something like React you have to replace your HTML with React components to use it. You cannot simply have React "use your HTML" as a template. You have to rewrite everything from that DOM node and "down" to be able to use React properly. However, if you use Vue then you can reuse your existing HTML as the template in Vue. This might make the transition easier and much like Stimulus did Vue will become "Javascript sprinkles" on top of your existing HTML. I don't think that something similar can be done with React. Hit me up on Twitter if you know of a way!

But Magnus, what about Hotwire or StimulusReflex? If you send HTML over the wire then you won't need to add any advanced Javascript at all right? The simple answer is that I don't have enough experience with these tools to be able to recommend them. They seem very promising however and I will absolutely write an article about this technology once I have worked with them some more.

Bottom line: Do not reach for something advanced unless you know that you need it. Stick with simple things like Alpine.js until you are absolutely sure that you need something else.

Caching

Since you already use Redis to get Sidekiq going then you might as well use that for your cache as well.

For the actual caching part you should use Russian Doll Caching, also known as Fragment Caching. This is the preferred way of caching things in Rails. I also would like you recommend using Action Caching if you can get away with it. It was removed back in Ruby on Rails 4 but there is a gem that you can use. It was removed from Rails because it can be a hassle to get the cache expiration to work properly and such. I try to use it whenever I can however because it is so much faster.

I wrote more about caching in my article about Ruby on Rails performance so you might want to check that out as well.

Pagination

I've been using Kaminari in the past but I've since switched to Pagy. It has all the bells and whistles that you can think of like countless pagination, pagination using AJAX, and built-in helpers for many CSS frameworks.

Testing

This should probably be a full article. I could write about what to test, when to test it and how much to test. I will probably make this into an article of its own so I'll just stick to writing about which frameworks and tools I recommend here.

This will be a bit of a boring one since you really can't go wrong with either RSpec or Minitest and it is a matter of preference. If you want something that is close to bare-bones Ruby then use Minitest. If you want something declarative and like having DSLs for things then use RSpec. I find that RSpec is a bit easier to organize and its helpers are nice. Personally I use them both but I've been leaning towards Minitest lately since that is the Rails default. It also has parallel tests out of the box while you have to use something like parallel_tests to get that going in RSpec. If you test suite is large then having built-in parallelization is fantastic.

For test setup I use Factorybot instead of fixtures since I find that to be much easier to reason about.

Finally, a word about mocking external APIs. You can do this in many ways from setting up manual mocks using Webmock, to using fake servers with Sinatra or WebValve, all the way to replaying previous calls using VCR. They all have their pros and cons, but for fast development I'm a big fan of VCR. You don't have to muck around with setting up mocks with fake data loaded from fixtures. VCR however can be messy too since you literally replaying past requests in your tests. The fixtures created by VCR can also become large and slow. For very fast development however, it is what I recommend. Once you run into enough problems I would take a look at the fake server approach instead.

Periodic jobs

There are many tools for this. For Sidekiq you can use its built-in periodic jobs feature. That is only for the enterprise version however. There are a number of gems with similar functionality like sidekiq-cron and sidekiq-scheduler. There is also the Clockwork gem that is not tied to Sidekiq. Personally I use Clockwork, since that is not tied to Sidekiq and it has been working well for many years. This will however start another Rails process to do the scheduling so there will be extra RAM use.

Code formatting

You should absolutely use code formatting tools that run every single time you save your files. They take a lot of work out of coding so that you can focus on implementing features.

For Javascript code I use Prettier. It comes with reasonable defaults and a non-approach to settings. It is opinionated, gets the job done and there isn't a million things to configure.

For Ruby I use Rubocop. You can set this up so that it runs using Spring by generating a binstub using spring-commands-rubocop. It will make it run a bit faster. Rubocop also has plugins for your testing framework.

For ERB I use ERB Formatter/Beautify. It works very well if you use Visual Studio Code.

Performing HTTP Requests

I used to use Faraday for this but I've switched to the HTTP gem. I find it to be a bit cleaner and easier to use.

Exception management

I use Rollbar to capture errors in production. It has a very generous free tier and it starts at $1 a month once you go past that. Honeybadger is also a great option.

In development mode you should make sure to use better errors to get nicer error pages when things go wrong.

Continuous integration

If you run your tests and deployment on an external service then you have a lot of options. Personally I use Circle CI for all my projects, Ruby or not. It has a very generous free tier as well as extra stuff for open source projects. The paid tier starts at $30 a month which is reasonable.

Sending email

This is a bit of an odd one to be included perhaps but since getting the emails to arrive safely is pretty then this is quite important. I've used Sendgrid and Mailgun in the past but I have found them both to be unreliable. Messages don't arrive correctly sometimes or they get stuck in Microsoft spam filters. I would recommend everyone to use Postmark for all your email needs. They get the job done and you never have to worry about deliverability. I'm not even getting paid to say that; I just like them that much. They are not free however and start at $10 a month for 10000 emails.

Slugs

Sometimes you want a nice URL and perhaps not "myapp.com/movies/1234". Instead you want "myapp.com/movies/parasite". For this you should use the friendly_id gem. It handles every single conceivable feature for slugging such as generating slugs, multiple languages as well as slug history if you need that.

Full Text Search

I think there are two reasonable options here. You can either use a service such as Elasticsearch or use built-in functionality in PostgreSQL with the pg_search. I find that pg_search is better if you can get away with it because it is simply using PostgreSQL. You will need to think carefully about indexes if you have a lot of data though since full text search can be very slow.

If you decide to use Elasticsearch then you should use the searchkick gem to make it a breeze.

Finally, you can also use an externally hosted service such as Algolia. Don't immediately reach for that however since you will need to manage indexing yourself and it can become expensive if you have a lot of queries.

Deployment

If you are ok with paying a bit for hosting then just use Heroku and don't look back. It is a "push code to deploy" with hosted databases and everything. They have all you need to deploy and run your app, but it can get quite expensive. I went into this in greater detail in this article I wrote.

If you want the cheapest way possible but still easy then you should use Dokku. It is a self-hosted "mini Heroku". All you need is a cheap server, an hour to install it and then you are off to the races. I went into Dokku in greater detail in the Heroku post mentioned earlier.

If you need a cheap and solid server then feel free to use my referral code to get €20 in credit at Hetzner. Read my review for more information.

Self-hosted tools

These are not Rails-specific but they are very nice to have for any app.

Thumbor

Self-hosted image transformation and optimization tool.

Huginn

Self-hosted Zapier/IFTTT alternative.

OpenFAAS

Self-hosted serverless functions. See this article for more details.

InfluxDB + Telegraf + Grafana

Self-hosted monitoring and visualization stack.

A Grafana server monitoring dashboard

I wrote more about self-hosted tools in this article

Editor tools

There are more examples in my article about Visual Studio Code plugins for Ruby. These are the essentials:

Ruby

Well...you'll need this.

Solargraph

"Go to definition" in your editor. Code navigation

Rails fast nav

Navigate your Rails apps faster

Prettier

Run and display prettier from within Visual Studio Code

rubo-rubocop

Run your Rubocop cops from within VSCode.

Monitoring

If you're ok with paying some money then use Datadog. It has server monitoring, log management, and application monitoring. If you want something self-hosted then use the InfluxDB+Grafana stack mentioned earlier and then use influxdb-rails to push metrics. It works really well and Grafana can give you some really solid visualizations.

Templating engine

I used to be a huge fan of HAML, but now we have things like Emmet and autoformatting. I also write HTML in React and Vue so the jump to HAML is not great. I highly recommend that you just stick to ERB which is the Rails default. If you have an app that is full of HAML templates then you can use https://haml2erb.org/ to convert them to ERB.

Admin tool

I've tried most of the the admin tools out there and I find that activeadmin is the best choice. It has a very powerful, yet daunting, DSL for generating admin tools. You can have a very decent admin section with little effort using activeadmin, so I highly recommend trying that before writing your own.

Misc gems

In no particular order:

Down: Safe file downloads

Discard: Soft-delete.

Feedjira . RSS management

sitemap_generator Generate sitemaps with ease

meta-tags . Create SEO tags, Twitter cards and opengraph sharing attributes.

html-pipeline. Tools for doing auto-linking of URLs inside text blocks and more.

Conclusion

Wow that was a long one. Thank you so much for making all the way to the end and I hope that your future Ruby on Rails projects will be successful.

What do you use for a Team Knowledgebase?

Magnus Skog — Thu, 25 Feb 2021 15:33:13 +0000

I've been looking for something to use for a team knowledgebase for a while now. This is going to be used for everything from technical documentation to onboarding and code standards.

Currently we are using the Hugo static site generator with a nice theme but it is quite clunky to have to check out the code every time we need to change something.

We are currently looking at Outline to solve this and so far it seems pretty decent, but it lacks comments and permissions.

Feature wish list:

Easy to update, preferably with markdown support
File attachments
Support for discussions/comments
Integration with tools to make it easy to find things like Slack and Alfred
Versioning
Admin tools to be able to lock documents

Can you recommend something?

LQIP in Rails using Thumbor and base64

Magnus Skog — Tue, 02 Feb 2021 18:30:35 +0000

There are many ways to do low quality image placeholders for applications. This is a simple way using Thumbor and Ruby to generate blurry base64 thumbnails. This way the image can be loaded in the first HTML load and there is no need for additional requests to fetch the thumbnails while the original images are loading.

You will need Thumbor to use this method. If you're unfamiliar with Thumbor then I wrote an introduction here.

The idea is to take the original image and then drop the quality as low we can and then make it small with a blur effect.

Let's start by applying the quality filter and drop the quality to say 5

Next, we take the image size down to say 100 pixels wide and keep the aspect ratio.

Now, if we blow this image up to a reasonable size it will look real bad. So what I like to do is play with the quality and size a bit and add a bit of blur. In this example I changed the size to 200 pixels wide, the quality to 25 and added 5 blur:

The size is a bit larger now but I think the trade off is worth it. Here is the URL for the final image if you want to have a look at the settings:

https://thumbs.mskog.com/200x/filters:format:(jpeg):quality(25):blur(5)/https://images.pexels.com/photos/5733127/pexels-photo-5733127.jpeg?auto=compress&cs=tinysrgb&dpr=2&h=750&w=1260

We can now use this excellent Base64 Image Encoder site to convert the image to base64. The final image will thus be about 3.26kB. Keep in mind that this is text and can be compressed. Gzip compression should be able to take this down to about 2kB once we add it to a website.

Ok, so now we have settings that we like. Next up we need to write some Ruby code to convert the image to base64. This is very easy to do:

require "net/http"

url = "https://thumbs.mskog.com/200x/filters:format:(jpeg):quality(25):blur(5)/https://images.pexels.com/photos/5733127/pexels-photo-5733127.jpeg?auto=compress&cs=tinysrgb&dpr=2&h=750&w=1260"

base64 = Base64.encode64(Net::HTTP.get(URI(url))).gsub("\n","")

That is all you need. We now have a base64 string representation of the image. I like to save this result to a database field on the model with the image. You can also use the Rails cache and generate these base64 images on the fly if you want. This will however be very slow and that is probably not what you want. Instead, just use ActiveJob to create these in the background once the original image URL is saved. Then save them to the model. Another idea is to use Active Storage and hook into the metadata analysis to add your base64 strings this way.

Right, time to put the base64 images to use. What you want to do is to use these as placeholders while the full image is loading. I usually use vanilla-lazyload for this. It is a simple and tiny project that gets the job done. Do mind that if you use Turbolinks you have to make sure that vanilla-lazyload does its thing after Turbolinks is done. If you use Webpacker it will look something like this in your application.js file:

import LazyLoad from "vanilla-lazyload";
const lazyLoad = new LazyLoad();
document.addEventListener("turbolinks:load", function () {
  lazyLoad.update();
});

Now you can use this in your HTML

And there we go. Lazy loaded base64 placeholders that will be replaced by the full image once it is done loading. If you want to see how this looks in practice then I made a test site for it

RSS Bridge and Huginn: Feeds for everything

Magnus Skog — Mon, 05 Oct 2020 18:24:35 +0000

RSS Bridge is my latest find in the automation space. While I already had Huginn installed as well as FAAS for creating and running serverless functions, RSS Bridge has been a great addition to this stack. What it is is a simple way to turn anything into a feed. You can get tweets by a specific user, trending projects on Github, popular shots from Dribble, and much much more. RSS Bridge can output the results as an RSS/Atom feed, HTML, or as JSON. The latter is the real killer feature when combining it with more automation using Huginn for exmaple. RSS bridge will cache the feed so no risk of hitting the underlying site too much either. It can be hosted on just about anything as it is a PHP application with no database. You can easily host it on Heroku or use Dokku like I did. On my VPS it uses about 50 mb of RAM so it is a real tiny little thing.

If you need a cheap and solid server then feel free to use my referral code to get €20 in credit at Hetzner. Read my review for more information.

Each resulting bridge is a simple URL that you can use directly or pass it along for further processing.

One of the issues is that the agents sometimes don't work. Many sites don't want this kind of scraping done to them. Instagram is a great example. RSS Bridge has a bridge to grab images from a user but it does not currently work. There is an issue to resolve it on Github but currently no go. It can be frustrating to see a bridge you want to use and then it doesn't work at all once you use it. However, what works works really well.

RSS bridge shines when combined with Huginn as I will demonstrate with a couple of examples.

Example #1: Dilbert strips to Slack

Say that you want to get the latest Dilbert strip posted to your private Slack channel. To do this you'll need Huginn, RSS Bridge and a FAAS function to extract the largest image from the page.

The Huginn scenario for this consists of three agents:

A Website agent to fetch posts from the feed from RSS Bridge. The feed URL will be something like https://yourssbridge.com?action=display&bridge=Dilbert&format=Json
A Website agent to receive events from 1 and use the serverless function extract the largest image from the page.
A Slack agent to post the images to Slack.

I've added a gist with the scenario below. Do note that you have to use your own serverless function and your own RSS Bridge and Slack URLs.

https://gist.github.com/mskog/092f655ea7013fcd59db7d9c3c1fbe9f

With this you get the latest Dilbert strip image posted to your Slack channel of choice every day.

Example #2: Reddit digest

If you want to avoid refreshing Reddit all day every day then perhaps you would prefer a simple email with the content from your favorite subreddit? It is very easy to do with Huginn and RSSBridge. You do need to add an SMTP service to Huginn to use it. Sendgrid is free for up to 100 emails a day which should be plenty unless you need hundreds of subreddit digests.

https://gist.github.com/mskog/b361447167e0eb7a66ee67f1f1633eb4

Bonus Example: Myfitnesspal dashboard

I use Myfitnesspal to track my macro nutrients and caloric intake. I wanted a nice dashboard to show me this over time and some completely necessary graphs to track things. Myfitnesspal has an API that can do this but it is only for registered partners. Exporting data is only available for paying users and I don't want to do that manually. So, enter our good friend the web scraper.

To do this you need Huginn, FAAS with a Puppeteer function, Influxdb and Grafana. I wrote about the latter here. Feel free to use my function to scrape the data from Myfitnesspal. It is a real hack job but it gets the job done every day. Then all you need to do is to use a Website agent to execute the function and a Post agent to add the data to Influxdb. Then you can use Grafana to make a nice graph like so:

Bonus Example #2: Withings dashboard

Let us stick with the fitness stuff shall we. I own a Withings WIFI enabled smart scale for some reason. I weigh myself daily and the scale will upload the data to the cloud. I would like to have this data in Grafana as well. Withings does have an API but it is an oauth2 enabled atrocity that is quite painful to use for such a simple thing as getting my data. I spent a solid weekend trying to get this working before giving up and resorted to scraping again. An hour later I had my data in Grafana just like with Myfitnesspal. Once again you can use my FAAS function if you wish!

The procedure to get this going is identical to the Myfitnesspal one. Simply use a Website agent to get the data from the FAAS function and then use a Post agent to add it to Influx. I will not post an example graph because I don't want to scare anyone.

Now all I need is to get the data out of my weight lifting tracker app and we're in business.

Advanced Example: Weightlifting stats dashboard

Time for something a bit more involved. My current lifting tracker app can export the data to CSV and share it using the default Android share. Huginn has several agents that can deal with files so as long as we can get the file to Huginn we are golden. I would love to have my own data available so that I can graph it next to the other stuff above.

To do this we need to do three things:

Get the CSV data into Huginn
Format the data and remove duplicates
Import the data in InfluxDB

To solve the first problem we will use Dropbox. Huginn has Dropbox integration built in so this step will not be too hard. You do need to register an application in the Dropbox App Console and make sure the token is set to permanent. You can use a scoped app if you wish. Then you need to setup a Dropbox Watch Agent to watch for changes in the directory as well as some trigger agents to avoid running the workflow when a file is deleted. Finally we use a Dropbox File Url Agent to create a link to the raw file that we can read in step 2.

It gets much easier now since we have a URL for the file itself. Simply use a Website agent to read the file, a CSV agent to format the data, a Trigger agent to select the correct rows, and a De duplication agent to get rid of duplicates. Finally we use a Post agent to get the data to Influx. Ok maybe not that simple.

The diagram for the scenario

This was a lot of work but now I have a fully automated workflow to get the data out of my app and into my own data store. All I have to do now is hit Export in the app and save the file to Dropbox. Do note that we did not need to use anything but Huginn and Dropbox to do all of this.

The complete scenario is available here:

https://gist.github.com/mskog/88a246000d1db4c28753ac76d0481627

Conclusion

I am getting more and more excited about using things like Huginn, RSS Bridge, and FAAS to create complicated automated workflows. I have even started building applications that take advantage of this to do things like scraping websites and processing data. Do you have any good ideas for what to do next? Post a comment!

Prevent blank and initial search with Algolia

Magnus Skog — Thu, 03 Sep 2020 19:12:47 +0000

Algolia recent updated their pricing. You no longer have to pay a hefty monthly fee to get search going. It is now entirely based on your search volume and you get 10000 search requests for free every month. This makes it very attractive for blogs like this one.

I follow the official Gatsby guide to get search going which was easy enough, but I ran into a problem. Algolia will by default make a search request on every single page load and another when you focus the search bar. The reasoning for this is that firing off a blank request make the next request faster. However, for people who are trying to limit the number of requests this is a problem. For a popular blog you'd very easily hit the 10k limit even if no one is using your search.

There is no official way to skip the initial or blank search using the Algolia React client so we have to do it ourselves. You probably have something like this in your code if you use the official client:

  const algoliaClient = algoliasearch(
    "ID",
    "KEY"
  );

  return (
    <div ref={rootRef}>
      <InstantSearch
        searchClient={algoliaClient}
        indexName={indices[0].name}
        onSearchStateChange={({ query }) => setQuery(query)}
      >
        <SearchBox onFocus={() => setFocus(true)} hasFocus={hasFocus} />
        <SearchResult
          show={query && query.length > 0 && hasFocus}
          indices={indices}
        />
      </InstantSearch>
    </div>
  );

To fix the issue all you have to is to create your own search client and hook into the search method like so:

  const algoliaClient = algoliasearch(
    "ID",
    "KEY"
  );

  const searchClient = {
    search(requests) {
      if (requests[0].params.query === "") {
        return [];
      }
      return algoliaClient.search(requests);
    },
  };

  return (
    <div ref={rootRef}>
      <InstantSearch
        searchClient={searchClient}
        indexName={indices[0].name}
        onSearchStateChange={({ query }) => setQuery(query)}
      >
        <SearchBox onFocus={() => setFocus(true)} hasFocus={hasFocus} />
        <SearchResult
          show={query && query.length > 0 && hasFocus}
          indices={indices}
        />
      </InstantSearch>
    </div>
  );

There you go! No more searches being fired on initial page load and no more blank searches.

My experience when changing from macOS to Ubuntu 20.04

Magnus Skog — Sun, 30 Aug 2020 08:38:00 +0000

I've been using a Macbook Pro for work for about 7 years now. My first one was a 2013 model and my current one is the 2018 model. In the beginning it was a breath of fresh air compared to the old Ubuntu based desktop computers we were using for work in the past. However, while I think that the Mac experience was great back in 2013 it has become worse over time. From excessive thermal throttling to bad keyboards and dangerous OS updates, it has been going downhill for a while now.

During the Covid 19 epidemic I've had some spare time on my hands so I decided to give Ubuntu 20.04 a try on my home gaming computer. Here is what I learned.

Bad Docker performance

Once of the major problems I have with development on a Mac is the poor Docker performance. I like to use Docker for development, especially for databases. I keep a docker compose file around that launches the containers I need like Postgres, Redis, and so on. I don't run the actual code in the Docker container but I guess I should be doing that too. Doing this on a mac has some of the benefits but the poor performance is just awful. We are talking about something like a 50% performance loss when running in Docker containers. Not to mention that the docker virtual machine on mac is memory hungry and takes a while to start. We used to run the same setup in docker at work for our development but the poor performance was finally too much to bear. We even tried to move the code to Docker containers as well but that made things even slower.

Once you go back to Linux things will improve dramatically. No more memory bloat, no slow startup, and huge performance improvements. We are talking about native performance here so everything works perfectly,. That was a huge breath of fresh air for me and that alone would probably had been enough to make the switch permanent.

Performance improvements

My Macbook Pro is the 2018 "premium" model. It has a an Intel I9-8950HK CPU with 6 cores in it. My desktop computer has a 6700k. While the laptop CPU has better specs in general that is not the case in real life use. I assume this is because it is a..well...a laptop CPU and of course the thermals. The 2018 i9 Macbook Pro is notorious for having very poor thermals. Of course, there is only so much you can do with a high performance CPU in such a small body like the Macbook but still, it is quite awful. Upon any kind of realistic load it will hit 100 degrees Celsius and start throttling. This is to be expected but it will also do this when running turbo boost on a single core. My desktop computer has a "gaming grade" cooler in it so it runs nice and cool. It will handle full load on all cores no problem. In general use when running tests for example the desktop computer is about 25% faster in. It really does make me want to upgrade the CPU though, perhaps to a Ryzen 3900x or even a 3950x. Not sure why but It seems like a good idea.

Hardware issues

I have been using a Macbook as my daily work computer since 2013 and in my experience the hardware on them works well enough(apart from the keyboard) but if you connect non-Apple hardware to it it might have problems. For starters, finding a docking station that does the job well was not fun. I have two 1440p monitors connected to my Macbook at home and the main issue I'm having is that they keep turning themselves off every hour or so. To fix it I have to reconnect them and sometimes even open the lid on the Macbook itself. I've tried with several different cables and nothing has worked. Extremely annoying.

Another issue is that both monitors have a refresh rate of 144hz which the Macbook cannot handle. If I do set anything about 60hz in the display settings the screen goes black and then I have to reboot to fix it for some reason.

I'm happy to say that Ubuntu 20.04 recognized everything in my desktop computer out of the box with zero problems hardware wise. The drivers for my Nvidia GPU are included in the latest release of Ubuntu, no separate install needed, and worked perfectly. Both monitors work in 1440p with 144hz support. No issues at all. We've made some progress for everyday Linux use for sure!

Linux on the desktop?

Everything was going well until the installation of Ubuntu was complete and I rebooted the computer. I got a login screen even though I set the thing to auto-login. When I logged in it kicked me back to the login screen again in an endless loop. This is a known issue and as far as I know it has not been solved. I do wonder how that made it past testing though...

Software use: Gnome vs macOS

As I mentioned earlier I've been using a Macbook for development for about 7 years now. I don't think that I can remember a single significant upgrade to the everyday use for me in a macOS upgrade since then. Except perhaps dark mode...

Every single system upgrade has broken something in a horrible way for me on the Macbook so every upgrade is dreaded.

I find that the everyday use experience is fine on Ubuntu. The file manager has everything you need and I haven't noticed any problems yet. We'll see what happens next time I upgrade though.

Missing apps

So far Ubuntu seems like a great deal! I do miss some of my favorite apps however. Like Alfred. I've build so many little automation scripts and such for it and I really miss Alfred. I know that there is a replacement called Albert but I haven't looked into it yet. It looks like a good option though!

I also miss Dash. Having all the documentation in the world available from anywhere is a great thing. I know that the Zeal project is a replacement and I will give that a try for sure.

Other than that I don't think I miss any apps except perhaps Transmit for FTP. I don't use FTP that much though so that's fine.

Even though my employer pays for the apps I need I still find it refreshing that every single good application is not behind a paywall. Free software and all that you know.

Terminal emulators

On my Mac I like to use Iterm with the sliding-from-the-top feature to access my terminal with a single keyboard shortcut. Turns out that in the Linux world there are plenty of alternatives to solve the same problem. I'm currently using Tilda for this. It has the same slide-down feature as Iterm and seems to be an overall great terminal emulator. The only issue for me so far is that Tilda doesn't seem to have a "normal window" feature. It is all about the slide thing. Iterm has both. This means that I use both Tilda and the Gnome terminal. A bit annoying but it works.

Conclusion

I've been using the Ubuntu setup for about a month now and so far it has been a pleasant time. It has everything I need for my day to day development needs and the performance has been rock solid. I'm not quite ready to get a "Linux laptop" yet though but we'll see where the Macbooks go once they start using their own ARM CPUs. Perhaps I'll switch once I need to upgrade my space heater to something better.

Autocomplete in Ruby on Rails using Stimulus

Magnus Skog — Mon, 10 Aug 2020 15:00:00 +0000

We are going to build a simple application in Ruby on Rails to autocomplete Reddit subreddits. If you want to view the completed application first then it is available on here. The code is available on Github.

The finished application will look like this:

We will be using Stimulus for this example. Stimulus is a simple Javascript framework that uses the HTML in the application. It is not a virtual DOM framework like React or Vue. I think it works very well when you need something more advanced than say Alpine.js but don't want to move to React or Vue. It works very well with Turbolinks as well so its a great fit for many applications.

Generate a new Rails application using webpacker. Make sure you have Rails 6 installed(rails -v). As this application will not use a database then we can skip it as well.

rails new autocomplete_stimulus --webpack --skip-active-record

Install Stimulus:

yarn add stimulus

We also need autocomplete.js and axios:

yarn add autocomplete.js axios

There are many libraries out there for doing simple autocomplete and typeahead stuff. I am a big fan of autocomplete.js by Algolia. You don't need to use Algolia to use autocomplete.js however and it comes with all the bells and whistles like debounce support.

Add the following to app/javascript/packs/application.js to get Stimulus going. Make sure that you don't delete any of the other content.

import { Application } from "stimulus";
import { definitionsFromContext } from "stimulus/webpack-helpers";

const application = Application.start();
const context = require.context("../stimulus/controllers", true, /\\.js$/);
application.load(definitionsFromContext(context));

Go into config/routes.rb and make sure it looks something like this:

Rails.application.routes.draw do
  resource :home, only: [:show]

  root to: 'home#show'
end

We will be using home#show for our main page so create app/controllers/home_controller.rb and add the following code to it:

class HomeController < ApplicationController
  def show; end
end

Finally we'll need a template for the HTML. Create app/views/home/show.html.erb and add the following:

<div style="margin: auto; width: 25%; padding: 10px">
  <h1>Autocomplete</h1>
  <div data-controller="autocomplete">
    <input data-target="autocomplete.field" />
  </div>
</div>

Here you can see Stimulus in action. The data-controller is where Stimulus will start working. The value of the data attribute is important as that corresponds to a controller named autocomplete_controller. Let's create it now! Create the directory tree and file structure like so: app/javascript/stimulus/controllers/autocomplete_controller.js

Mind that the naming of the controller is very important in Stimulus so make sure you get it right.

Lets start with some boilerplate code in it:

import { Controller } from "stimulus";

export default class extends Controller {
  static targets = ["field"];
}

The targets array contains all the targets of this Stimulus controller. As you can see it is what we added to the input tag in the template. Now we need to setup the Autocomplete part in the controller. The controller should now look something like this:

import { Controller } from "stimulus";
import autocomplete from "autocomplete.js";

export default class extends Controller {
  static targets = ["field"];

  connect() {
    this.ac = autocomplete(this.fieldTarget, { hint: false }, [
      {
        source: this.search(),
        debounce: 200,
        templates: {
          suggestion: function (suggestion) {
            return suggestion.name;
          },
        },
      },
    ])
  }
}

The connect method is run whenever the controller is hooked up to the view. To access the field target we use this.fieldTarget. Mind that this.search() doesn't exist yet. Lets create it:

import { Controller } from "stimulus";
import autocomplete from "autocomplete.js";

export default class extends Controller {
  static targets = ["field"];

  search(query, callback) {
    callback([{ name: "Hello" }])
  }

  connect() {
    this.ac = autocomplete(this.fieldTarget, { hint: false }, [
      {
        source: this.search,
        debounce: 200,
        templates: {
          suggestion: function (suggestion) {
            return suggestion.name;
          },
        },
      },
    ])
  }
}

Mind that we also imported Axios to perform HTTP requests. source expects a function so the search-function will return a function.

We are now ready to give this a try. Fire up the Rails server with rails server

If you go to localhost:3000 you should now see something like this:

If you type something in the input field you should see "Hello" as a suggestion. Stimulus is now working. Excellent!

Now lets move on to actually wiring this up with the backend.

Go to config/routes.rb and make it look like this:

Rails.application.routes.draw do
  resource :home, only: [:show]
  resource :autocomplete, only: [:show]

  root to: 'home#show'
end

We now need to create a controller for the route. Create the file app/controllers/autocompletes_controller.rb and add the following to it:

class AutocompletesController < ApplicationController
  def show
    data = [
      { name: 'Foo' },
      { name: 'Bar' }
    ]

    render json: data
  end
end

We now have a placeholder controller. Now all we have to do is add this to our Stimulus controller to make it request the autocomplete from the server. Change your Stimulus controller to look like this:

import { Controller } from "stimulus";
import axios from "axios";
import autocomplete from "autocomplete.js";

export default class extends Controller {
  static targets = ["field"];

  search(query, callback) {
    axios.get("/autocomplete", { params: { query } }).then((response) => {
      callback(response.data);
    });
  }

  connect() {
    this.ac = autocomplete(this.fieldTarget, { hint: false }, [
      {
        source: this.search,
        debounce: 200,
        templates: {
          suggestion: function (suggestion) {
            return suggestion.name;
          },
        },
      },
    ]).on("autocomplete:selected", (event, suggestion, dataset, context) => {
      this.ac.autocomplete.setVal(suggestion.name);
    });
  }
}

We have imported axios and changed the search function to perform requests instead of having placeholder data.

If you try this in the browser it should now always return "Foo" and "Bar". Now lets do it for real and autocomplete subreddit names shall we. I like to use the HTTP gem for http requests in my Rails application so install that by adding the following to your Gemfile:

gem "http"

Don't forget to restart your rails server after adding the gem.

Change your AutocompleteController to perform requests to Reddit:

class AutocompletesController < ApplicationController
  def show
    response = JSON.parse(HTTP.get("https://www.reddit.com/subreddits/search.json?q=#{params[:query]}&limit=10"))
    subreddits = response["data"]["children"].map do |subreddit|
      {
        name: subreddit["data"].fetch("display_name").downcase,
        title: subreddit["data"].fetch("title"),
        description: subreddit["data"].fetch("public_description").truncate(100)
      }
    end
    render json: subreddits
  end
end

This should be it. If you try this in your browser it should look something like this:

It works! Looks real bad though. Lets add some styling to fix it.

Create app/assets/stylesheets/style.scss and add the following to it:

.algolia-autocomplete {
  width: 100%;
}
.algolia-autocomplete .aa-input,
.algolia-autocomplete .aa-hint {
  width: 100%;
}
.algolia-autocomplete .aa-hint {
  color: #999;
}
.algolia-autocomplete .aa-dropdown-menu {
  width: 100%;
  background-color: #fff;
  border: 1px solid #999;
  border-top: none;
}
.algolia-autocomplete .aa-dropdown-menu .aa-suggestion {
  cursor: pointer;
  padding: 5px 4px;
}
.algolia-autocomplete .aa-dropdown-menu .aa-suggestion.aa-cursor {
  background-color: #b2d7ff;
}
.algolia-autocomplete .aa-dropdown-menu .aa-suggestion em {
  font-weight: bold;
  font-style: normal;
}

Now try again!

A bit better! Of course, this still looks bad and will need some more love to be used for real.

As an added bonus you can easily change the Stimulus controller so that it gets its API URL from a data attribute instead of having it hard coded in the controller itself. This way you can reuse the Stimulus controller for autocompletes all over the place!

All it takes is to change the HTML to this:

<div style="margin: auto; width: 25%; padding: 10px">
  <h1>Autocomplete</h1>
  <div data-controller="autocomplete" data-autocomplete-url="/autocomplete">
    <input data-target="autocomplete.field" />
  </div>
</div>

Note that data attribute that holds the URL. It is also a Stimulus thing so make note of it. Now change the Stimulus controller like so:

import { Controller } from "stimulus";
import axios from "axios";
import autocomplete from "autocomplete.js";

export default class extends Controller {
  static targets = ["field"];

  source() {
    const url = this.data.get("url");
    return (query, callback) => {
      axios.get(url, { params: { query } }).then((response) => {
        callback(response.data);
      });
    };
  }

  connect() {
    this.ac = autocomplete(this.fieldTarget, { hint: false }, [
      {
        source: this.source(),
        debounce: 200,
        templates: {
          suggestion: function (suggestion) {
            return suggestion.name;
          },
        },
      },
    ]).on("autocomplete:selected", (event, suggestion, dataset, context) => {
      this.ac.autocomplete.setVal(suggestion.name);
    });
  }
}

But what if you want to have some other rendering of the suggestions? Perhaps you would like to add some images or a nice description or something? There are a number of ways to do this. For example, you can use a simple Javascript template string with some HTML in the Stimulus controller and return that in the suggestion function. This works but its real dirty to deal with it once the string gets complicated. You could also add some kind of Javascript template system like Mustache in separate files and import those in your controller using a metaprogrammed data attribute. I prefer to keep the suggestion next to the rest of the autocomplete form, so I will show you a way to do it using Mustache and script tags. Mind that you don't really need Mustache for this but it makes things a bit cleaner.

Start by installing Mustache:

yarn add mustache

Now, change your HTML to this:

<div style="margin: auto; width: 25%; padding: 10px">
  <h1>Autocomplete</h1>
  <div data-controller="autocomplete" data-autocomplete-url="/autocomplete">
    <script data-target="autocomplete.suggestionTemplate" type="text/template">
      <h2>{{name}}</h2>
      <p>{{description}}
    </script>
    <input data-target="autocomplete.field" />
  </div>
</div>

Note the new script tag. The browser will not render that so it's safe to use. The curly braces are Mustache tags which will correspond to the JSON data from our autocomplete controller on the Ruby side. We also added a new target data attribute to be able to access it from Stimulus. Lets do that now:

import { Controller } from "stimulus";
import axios from "axios";
import autocomplete from "autocomplete.js";
import Mustache from "mustache";

export default class extends Controller {
  static targets = ["field", "suggestionTemplate"];

  source() {
    const url = this.data.get("url");
    return (query, callback) => {
      axios.get(url, { params: { query } }).then((response) => {
        callback(response.data);
      });
    };
  }

  connect() {
    const suggestionTemplate = this.suggestionTemplateTarget.innerHTML;

    this.ac = autocomplete(this.fieldTarget, { hint: false }, [
      {
        source: this.source(),
        debounce: 200,
        templates: {
          suggestion: (suggestion) => {
            return Mustache.render(suggestionTemplate, suggestion);
          },
        },
      },
    ]).on("autocomplete:selected", (event, suggestion, dataset, context) => {
      this.ac.autocomplete.setVal(suggestion.name);
    });
  }
}

We've added Mustache, the new target and a new way to render suggestions using our new template. It will look something like this:

Ok that looks terrible but you get the idea. Do note that you can still reuse the Stimulus controller for other autocomplete fields. Simply add a new template and URL!

There you have it! A simple way to add a reusable Stimulus component to your Rails applications. The complete application application is available on Heroku and the code is available on Github.

Break bad habits by making them annoying

Magnus Skog — Wed, 01 Jul 2020 17:44:08 +0000

Breaking bad habits can be really hard. Willpower alone is often not enough to get the job done or it simply takes too much of it. Willpower is also often regarded as finite. If you spend all of it on forcing yourself to not go on Twitter then you won't have enough of it left to make yourself to go to the gym later. I often fall into this trap myself. I can quite easily make myself avoid going on social media first thing in the morning but then I am rested and my mind is fresh. After a long day of work it is a completely different story. It's so easy to reach for my phone while waiting for my dinner to get ready(those pizza rolls take forever!). I'm tired, hungry and the work day has taken all of my effort.

I've illustrated this in this graph that I made myself:

For me willpower alone only works when there is some kind of reward or tangible thing waiting at the end of the road. "Stop visiting Twitter in excess and you'll get more things done" is too abstract for me. Yesterday-me is not the boss of me; I'm going on Twitter! Since most of the bad habits don't have any immediate rewards for breaking them then I had to think of something else.

What I like to do instead is to make the bad habits harder. Solving the phone problem after work for example is easy. Just turn the phone off and put it on the top of a bookcase or something. Now I have to go get a stool AND wait for the phone to boot. (Do not climb Billy bookcases kids!). Now it's suddenly annoying to get to my phone to check social media or something so now it is much easier to avoid it.

This article is mostly about how to break the habit of visiting "bad" sites while on your computer but some of these also work for your phone. Let us start with a couple of generic tips shall we:

Do not keep your phone next to you while working. Research has shown that simply having your phone within reach is enough to drain your energy. Keep your phone in another room or use the Billy idea from earlier.
Do not keep your phone next to your bed if you can avoid it. Buy an alarm clock or a Google Nest and use that instead to wake you up. You can also have your spouse pour cold water on you if you prefer the human touch. I recommend the Google Nest because it has the added benefit of listening to you while you sleep.
Do not go on social media first thing in the morning. This can ruin the whole day. It can also make you reactive rather than proactive. You don't want to spend your morning angry about something on Twitter while you are trying to plan your day.
If you know in your heart that you will eventually cave and check your phone then make a game of it. Challenge yourself to stay off your phone for as long as possible during the day. If you can get to say 4 pm then you have done good and probably got a lot of productive hours out of it. Now break that record tomorrow! Once you check your phone it becomes a lot more likely that you will do it again later so try your best to stay away.

Now on to the tips I have about making bad habits annoying. The idea is that if you make the bad habits annoying enough then you will train your brain to avoid them. If you know that you have to jump through hoops just to go on Reddit then you might just skip it. I use a couple of tools and tricks to make this easier:

Motion

Motion is a Chrome extension that nag you whenever you visit a "bad" site. If you visit Twitter for example a modal will pop up telling you to cut it out already. You can then choose to have access to the site for a number of minutes and then the modal will pop up again. Motion also has special functionality to hide Twitter recommendations, Youtube recommended videos, and more. You can also set work hours so that Motion is only active when you are trying to get things done.

I like Motion and it is my primary way to nag me to stop visiting bad sites but there are a couple of issues. My brain has gotten used to clicking away the nag screen so the annoyance factor has faded. Motion will also have the popup on your screen but you can still see the site behind it. This means that you can easily spot something interesting behind the nag screen that will make you want to use the site.

Use Blocksite to block sites on Chrome and on mobile

Blocksite is a Chrome extension that blocks any websites you want. It can also be synced with your phone so that the same sites are blocked there as well. You can put in password protection to keep your from disabling it whenever you want to go on Twitter. It works well to keep you off the worst offending sites.

Blocksite blocking a site. I like the picture. Has a real "come on, wtf are you doing?" vibe.

Improvement: Store the blocksite password in a place that is hard to get to

I found that it was much too easy for me to just punch in the Blocksite admin password and unblock Twitter if I wanted to go there. What I did to solve this was to create a new database in my password manager and put the password in there. I then put the password manager master password in a text file and put that on one of my servers. This way I had to be on my work computer to access the Blocksite settings since that is the only one with SSH access to the servers.

Use a Pi-hole to block the sites

Pi-hole is an ad-blocking DNS server that you can run on a tiny computer inside your own network such as a Raspberry Pi. Its main feature is network-wide ad blocking but it can also be used to block sites in general. The latest version of it also added functionality to block sites on only some devices. Now you can use it to block things on your own devices only while it will leave other devices alone. This is my go-to method for blocking distracting sites since it works on both my computers and my phone as long as they are connected to my home network.

Improvement: Store the admin password for the Pi-hole in a place that is hard to get to

This is the same deal as the blocksite one. Store the admin password on a server or something to keep you from getting to it easily.

Improvement: Use a throwaway Google account to store the password

This one is for emergency use only! I created a new Google account and put the password in there. I then changed the password for the account to some random garbage string and didn't write it down. I now have no access to the account at all. The only way to get access back is to use Google account recovery which I connected to my main email account. The magic part is that Google has a 24-48 hour quarantine on account recovery. It will thus take that long to reset your account and get access. This was so annoying to me that I completely stopped using the bad sites.

Miscellaneous tips

Use a habit chain

Made famous by comedian Jerry Seinfeld. It is a technique he used to make sure that he was writing every single day. He tried to make an endless chain of days on which he worked on his material. He found that if there was a long streak of days behind him then it made it much easier to force himself to work on it today as well. This can be adapted for negative habits. You can use a simple calendar or an app to track it. There a multiple such apps, just search for "don't break the chain app" and you'll find plenty. Simply put in "Don't visit Twitter during productive hours" and then keep that chain going. Making yourself accountable to something is often a useful technique.

Challenge yourself

30 day challenge: Don't visit Reddit at all for 30 days. This can be kind of fun but to me it is especially effective if you combine it with journaling. Write down your thoughts about the site you want to avoid every day in a journal. Are you happy that you can't go on Reddit? Do you miss the cats? Once again I find that making this a challenge only in your mind is not very effective. You need something tangible like a journal.

Hetzner Cloud Review: Revisited in 2020

Magnus Skog — Thu, 18 Jun 2020 12:00:00 +0000

It has now been well over two years since my Hetzner Cloud Review and I thought it was time to post an updated one. I am still a happy customer at Hetzner. There has been quite a few updates since then. Hetzner Cloud now has an even nicer interface, block storage, private networks, dedicated CPU offerings and a brand new server line with AMD EPYC processors. They also have a data center in Finland now.

If you want to give Hetzner a try then feel free to use my referral code to get €20 in credit to use for servers.

Back in 2018 Hetzner Cloud was in my humble opinion the best value VPS provider out there. I believe that this might still be true today. There are a couple of providers now which come close price wise but overall I still think Hetzner is #1.

My current main VPS is a CX21 with 2 CPU cores, 8GB RAM and 80GB of NVM disk. It is running quite a few things and has been chugging along now for a long time. I've had no serious downtime and only a couple of reboots here and there. Overall it has been rock solid and performance has been excellent. One of the fears of 2018 was that since Hetzner Cloud was new it would quickly get oversold and slow down considerably. I'm happy to say that this is not the case and there has been no evidence of any noisy neighbours on any of my servers.

I like to compare things to DigitalOcean since they are really the servers to beat. They are simple, have great features and lots of goodies like Block storage, Hosted databases, and managed Kubernetes.

Let's compare the cheapest servers available to get started. I ran a couple of quick Geekbench 5 benchmarks to gauge the performance of each server as well.

	Hetzner	DigitalOcean
Server	CX11	Standard 1GB Droplet
CPU Architecture	Intel Skylake	Intel Skylake
Disk type	NVM	SSD
CPU MHz	2100	2300
CPU Cores	1	1
RAM	2 GB	1 GB
Disk space	20 GB	25 GB
Traffic	20 TB	1 TB
Geographic location	Germany, Finland	USA, Europe, Asia
Price	$2.71	$5
Geekbench (single core)	648	713
Geekbench (multi core)	682	710

So what are we looking at here? With Hetzner you get twice the RAM, 20 times the transfer, slightly lower clock speed at almost half the price of the Digital Ocean droplet. Now, DigitalOcean has a lot more to offer like different kind of server types, managed databases, object storage, load balancers and so on. Hetzner only has their servers, block storage and private networking. However, it is bang-for-buck unquestionable that Hetzners offering remains exceptionally good.

AMD EPYC

Hetzner has a new kind of VPS with AMD EPYC CPUs. These are brand new and became available in early 2020. These are essentially "souped up" versions of the other servers. Each of the previous servers has an EPYC counterpart. For example, here is a comparison between the CX31 and its EPYC sibling the CPX31. I also added a similar DigitalOcean droplet for comparison.

	CX31	CPX31	DigitalOcean
CPU Architecture	Intel Skylake	AMD EPYC	Intel Skylake
CPU Cores	2	4	4
RAM	8 GB	8 GB	8 GB
Disk space	80 GB	160 GB	160 GB
Traffic	20 TB	20 TB	5 TB
Price	$9.59	$13.37	$40
Geekbench(single core)	729	608	605
Geekbench(multi core)	1431	1822	2077

So what you get is twice the CPU cores and twice the disk space for less than 50% additional money. The performance per core seem to be lower on the EPYC architecture however. I did one final benchmark to compare the CPX31 to the next level of Skylake VPS:

	CPX31	CX41
CPU Architecture	AMC EPYC	Intel Skylake
CPU Cores	4	4
RAM	8 GB	16 GB
Disk space	160 GB	160 GB
Price	$13.37	$17.95
Geekbench (single core)	608	649
Geekbench (multi core)	1822	2168

So what we are looking at here is about 30% more money for 15% additional multi core performance and twice the RAM. This looks like a great upgrade path to me when you do run out of RAM on your EPYC server.

What about other alternatives then? DigitalOcean is not the only VPS provider out there. I did a couple of other comparisons to popular VPS platforms:

	Hetzner	UpCloud	Amazon Lightsail
CPU Architecture	AMD EPYC	Intel Skylake	Intel Broadwell
CPU Cores	4	4	2
RAM	8 GB	8 GB	8 GB
Disk space	160 GB	160 GB	160 GB
Traffic	20 TB	5 TB	5 TB
Price	$13.37	$40	$40
Geekbench(single core)	608	639	605
Geekbench(multi core)	1822	2215	2077

Block storage

Hetzner now also has block storage. They do not offer object storage however. The difference between block and object storage is that block storage is mountable as volumes while object storage is more like Amazon S3. Let's compare Hetzner to DigitalOcean once again. I also ran a couple of naive read/write tests to gauge performance. Please don't take these as exact numbers but as a rough estimate.

	Heztner	DigitalOcean
Storage class	SSD	SSD
Replication	Yes	Yes
Minimum volume size	10 GB	1 GB
Maximum volume size	10 TB	16 TB
Maximum volumes per server	16	7
Read performance	309 MB/s	310 MB/s
Write performance	284 MB/s	168 MB/s
Price/GB	$0.04	$0.1

Hetzner is about half the price of DigitalOcean while performance remains excellent.

Unfair comparison

So we have established that the price/performance ratio of the Hetzner VPS offering is superior to popular platforms such as DigitalOcean, but that is not an entirely fair comparison. DigitalOcean has a lot more to offer than just "bare metal" virtual servers. If you bear with me here we are going to do one final table to compare features:

	Hetzner	DigitalOcean
Block storage	Yes	Yes
Private networking	Yes	Yes
Backups	Yes	Yes
Snapshots	Yes	Yes
Floating IPs	Yes	Yes
Server graphs	Yes	Yes
API	Yes	Yes
Object storage	No	Yes
CDN	No	Yes
Load balancers	No	Yes
Managed databases	No	Yes
Managed Kubernetes	No	Yes
DNS	No	Yes
Docker registry	No	Yes
Data centers on multiple continents	No	Yes

DigitalOcean also offers flexible server plans with better CPU or more RAM and such. They also have managed databases as well as Kubernetes. In my opinion the most important part is that DigitalOcean offers deployment on other continents than Europe. If you choose Hetzner you can either deploy your servers in Germany or Finland. On DigitalOcean you have the option of New York, Amsterdam, San Francisco, Singapore, London, Frankfurt, Toronto and Bangalore.

I don't know what Hetzner is planning on doing in the future, but I don't think it is reasonable to expect them to introduce features like managed databases. Hetzner does have some managed servers but the vast majority of their services are "bare metal" servers. I wouldn't hold out for any kind of expansion into other continents either, especially not these days.

There are some others offering competitive pricing like OVH and Genesis. I might take a look at these in a future post.

Conclusion

Hetzner Cloud remains a very competitive service when you need a "bare metal" VPS at the lowest possible price and you still get amazing performance. If you're ok with hosting your applications in Europe and you don't need anything fancy like managed Kubernetes then you simply cannot go wrong with Hetzner. I remain a happy customer myself and I see no reason at all to look elsewhere at this time.

Self-hosted tools for web development

Magnus Skog — Mon, 18 May 2020 14:20:00 +0000

Having some self-hosted services and tools can make your life as a developer, and your life in general, much easier. I will share some of my favorites in this post. I use these for just about every project I make and they really make my life easier.

All of this, except OpenFaaS, is hosted on a single VPS with 2 CPU cores, 8GB of RAM and 80GB SSD with plenty of capacity to spare.

If you need a cheap and solid server to run all this on then feel free to use my referral code to get €20 in credit at Hetzner. Read my review for more information. I use the CX31 model for all of this but you can probably get away with something cheaper.

Huginn

Huginn is an application for building automated agents. You can think of it like a self hosted version of Zapier. To understand Huginn you have to understand two concepts: Agens and Events. An Agent is a thing that will do something. Some Agents will scrape a website while others post a message to Slack. The second concept is an Event. Agents emit Events and Agents can also receive Events.

As an example you can have a Huginn agent check the local weather, then pass that along as an event to another agent which checks if it is going to rain. If it is going to rain the rain checker agent will pass the event along, otherwise it will be discarded. A third agent will receive an event from the second agent and then it will send a text message to your phone telling you that it is going to rain.

This is barely scratching the surface of what Huginn can do though. It has agents for everything: Sending email, posting to slack, IoT support with MQTT, website APIs, scrapers, and much more. You can have agents which receive inputs from custom web hooks and cron-like agents which schedules other agents and so on.

The Huginn interface

Huginn is a Ruby on Rails application and can be hosted in Docker. I host mine on Dokku. I use it for so many things and it is truly the base of all my automation needs. Highly recommended! If you are looking for alternatives then you can take a look at Node-RED and Beehive. I don't have personal experience with either though.

Huginn uses about 350MB of RAM on my server, including the database and the background workers.

Thumbor

Thumbor is a self-hosted image proxy like Imgix. It can do all sorts of things with a single image URL. Some examples:

Simple caching proxy

Take the URL and put your Thumbor URL in front like so:

https://thumbs.mskog.com/https://images.pexels.com/photos/4048182/pexels-photo-4048182.jpeg

Simple enough. Now you have a version of the image hosted on your proxy. This is handy for example when you don't want to hammer the origin servers with requests when linking to the image.

Resizing

That image is much too large. Lets make it smaller! https://thumbs.mskog.com/800x600/https://images.pexels.com/photos/4048182/pexels-photo-4048182.jpeg

Much smaller. Note that all we had to do is add the desired format.

Resizing to specific height or width

What about a specific width while keeping the aspect ratio? No problem!

https://thumbs.mskog.com/300x/https://images.pexels.com/photos/4048182/pexels-photo-4048182.jpeg

Quality

Smaller file size?

https://thumbs.mskog.com/1920x/filters:quality(10)/https://images.pexels.com/photos/4048182/pexels-photo-4048182.jpeg

You get the idea! Thumbor also has a bunch of other filters like making the image black and white, changing the format and so on. It is very versatile and is useful in more scenarios then I can count. I use it for all my images in all my applications. Thumbor also has client libraries for a lot of languages such as Node.

Thumbor is a Python application and is most easily hosted using Docker. There are a number of great projects on Github that have Docker compose setups for Docker. I use this one. It comes with a built-in Nginx proxy for caching. All the images will be served through an Nginx cache, both on disk and in memory by default. This means that only the first request for an image will hit Thumbor itself. Any requests after that will only hit the Nginx cache and will thus be very fast.

To make it even faster you can deploy a CDN in front of your Thumbor server. If your site is on Cloudflare you can use theirs for free. Just keep in mind that Cloudflare will not be happy if you just use their CDN to cache a very large number of big images. You can of course use any other CDN like Cloudfront. My entire Thumbor stack takes up about 200MB of RAM.

In conclusion I think that Thumbor is a vital part of my self hosted stack and I use it every single time I need to show images on any website or app. Once you have this working properly you never have to worry about image formatting ever again since the Thumbor is always there.

Hosted alternatives to Thumbor: Imgix, Cloudinary

Self hosted alternatives: Imaginary, Imageflow

Searx

Searx is a self hosted metasearch engine. It will strip any identifying headers and such from your searches and then it will use one or many search engines to run your query. It can search on for example Google, Bing and DuckDuckGo. What makes Searx great as a self hosted service is that it has a simple JSON api. Simply tell it to use JSON and your query will be returned as JSON. This will enable some pretty neat combinations, but more of that later. It can also search for images, music, news and more.

Searx in action

This is another killer service. The JSON formatting is what really sells it for me since it can be combined with other services in lots of different ways.

Searx is another Python app and is easily hosted through the use of the official Docker image. It uses about 230MB of RAM on my server.

InfluxDB + Telegraf + Grafana

InfluxDB is a time series database. It is built to receive time based event data from sensors, servers and so on. For example it is very good at things like storing CPU load data every 5 seconds. It also has built-in ways to makes sure it doesn't fill the disk with all this data and much more. There are client libraries for most languages as well as a very simple HTTP API for adding data. It goes very well together with Huginn where you can create agents to poll data from somewhere and then use the HTTP API in InfluxDB and a Post Agent to store it . There will be examples of this later on!

Telegraf is a service that will collect data about your server and send it to InfluxDB. It can also send the data to other databases and such but for this we use InfluxDB. It can collect just about any data about a server that you want, include statistics from Docker containers. It has a very simple out of the box configuration that you can tweak if you wish. I install it on all my machines to send data to InfluxDB, including my at-home NAS.

Grafana is a graphing, analytics, and monitoring tool. It can graph the data from many different sources including InfluxDB, AWS Cloudwatch and PostgreSQL. It also has alerting capabilities for Slack for example. It is a delight to use and you will quickly be able to create some very nice looking graphs of your data. You need to be careful though because I find that it is very addicting to graph all your things.

A Grafana dashboard for server monitoring

There are many Docker setups for this stack that you can find on Github, so hosting this is easy.

Grafana is truly delightful to work with and it is probably the slickest graphing tool I've ever used and it can easily be compared to commercial projects like Datadog.

Docker Registry

This is a simple one but oh so useful. It is good to have a place to store your own Docker images and this is what you need. You can use Docker Hub for this but private images cost about $7 a month. It is however very easy to host your own registry. A Docker registry is a requirement to be able to use OpenFaaS.

Ghost CMS

Ghost is an open source publishing and blogging platform. It was originally kind of a replacement for Wordpress but it has since grown to be something more. I use it as a headless CMS for this blog as well as other websites. It has a great GraphQL and REST API that you can use to pull your articles and pages out to use in a static site or show on another website. I have another article about how my blog works with this if you want to know more.

The Ghost editor while typing this

Ghost has a great editor that makes it very easy to include Twitter posts, images, Spotify links, and so on. It is also hosted on your server so you can write from anywhere. You don't have to deal with markdown files if you don't want to and I find it to be a delight to use and write in.

OpenFaaS

OpenFaaS is self-hosted functions-as-a-service aka serverless. I have another article about OpenFaaS so I won't go into too much detail here. You can use OpenFaaS to easily deploy functions in any programming language without having to setup a microservice. Also, I understand the irony of self-hosting a serverless setup, but it is a strange world we live in so just go with it.

It is very useful for a number of tools and combinations. I have a number of these functions and here are some examples:

Readability

Python function that uses the newspaper3k library to pull out metadata and the article content from any URL. I use it to render snippets from articles, prepare for sentiment analysis, and things like that.

Puppeteer renderer

Sometimes websites will not work at all without Javascript or they have systems in place to prevent scraping and interacting with the sites automatically. Rotten Tomatoes is such a site that will fight back against any automation attempts. Enter Puppeteer, the headless Chrome API. This function simply takes a URL, renders the page with Javascript and returns the resulting body. This is then ready for processing in a scraper for example using Huginn. There will be examples of how to use this later with Huginn later so stick around if you're interested in that.

OpenFaaS is hosted on its own server because that made sense to me. It is a tiny little thing though and doesn't use much resources at all.

Combinations

This is where we unlock the real magic of having all these things. You can combine these in clever ways to create something really neat. Here are some examples to get you started:

OpenFaaS+Searx = First image

This is a combination that I really like. Create a function in OpenFaaS in any language of your choice, I used Javascript, that will search for the given query in Searx, making sure to return the result as JSON. Then parse the results in the OpenFaaS function and return the URL for the first image result.

You now have a function that you can call with any query and it will return the first image result. This is useful in a number of different ways. You can for example search for bryan cranston site:wikipedia.org to get a good image of actor Bryan Cranston. Now you can use some cool Thumbor filters and such to process the image if you want!

Suggestions for improvements: Add more functionality to the OpenFaaS function. For example you can add probe-image-size to your function. You can now reject images which are too small for example.

OpenFaaS+Huginn+Trello = Movie recommendations

This is a simple one which adds movie recommendations to my Trello inbox daily.

Steps to create:

Add a Website Agent to Huginn. Use the URL for the Rotten Tomatoes front page and add the OpenFaaS Puppeteer function to render that URL with Javascript enabled. Scrape the section with new movies.
(Optional) Create a Trigger Agent in Huginn to select movies with a minimum score. Perhaps you want only the movies which have a score of 80 or better in your inbox.
Add a Post Agent to Huginn that will post the movie names to your Trello inbox using the Trello API.

Suggestions for improvements: Add another step that will also link to the IMDB page for the movie. You can use Searx for this. Simple search for the movie like so: "fried green tomatoes" site:imdb.com

RescueTime+Huginn+InfluxDB+Grafana = Productivity graph

RescueTime is automatic time tracking software. It keeps track of what you do on your computer and will tell you when you are being productive and when you are slacking off on Reddit. You can use a Web Site Agent in Huginn to access your productivity data on RescueTime. You can then use a Post Agent to add this data to InfluxDB. Finally you can graph it using Grafana. I use something similar to get data about our hot water bill and such. Once you have Huginn and InfluxDB you can graph almost anything.

Huginn+Slack = Notifications center

If you're like me and you have a lot of notifications then you might want to use Huginn to sort these out. Instead of interfacing directly with Slack or whatever notification system you use, you can instead use a Webhook Agent in Huginn to create an API endpoint. Post your notifications to this endpoint. You can then use for example a Slack Agent to post the notifications to Slack.

What is the point of this then? Well, you can very easily change to using something else than Slack for your notifications without changing it on every site that creates them. Perhaps you want to delay some notifications? You can do that with a Delay Agent in Huginn. Perhaps some notifications should go to Trello instead of Slack? No problem using Huginn. You can even use a Digest Agent to group low level notifications and send them all at once by email or something. Don't forget that you can also graph all of this using Grafana.

Conclusion

This is by no means an exhaustive list of things you can self host to make your life easier as a developer. Do you have any favorites that I've missed? Please reply in the comments below or hit me up on Twitter!