How dev.to dynamically generates social images

It's pretty important that dev.to posts with no cover image have a well-designed "card" that grabs a Twitter user's attention as they scroll through the feed. I'm not sure ours are all that well-designed, but they could be, and that part isn't really about the code.

We get a lot of questions about how we generate these default images dynamically. I've hesitated to write this post because I still think we could do this a better way. But since we have not gotten around to doing it differently, here is a post about our current approach and a description of possible alternative solutions.

Here is an example Twitter card for a #discuss post

What we do

We use Cloudinary as an image management/manipulation service and as a CDN. They have features such as adding text overlays to images which we use to generate images. So it's ultimately a matter of fiddling with the layers until you get it right and then writing the code so our app interacts with the service programatically. It's pretty straightforward, but our creativity is a bit limited by the narrow range of things we can do within the system. It's a great service but we generate image templates that are kind of a pain to change. Code that is hard to change is not good code.

Other approaches we could take

Generate the image with HTML and then using a service to take a screengrab

This would give us more flexibility and control over the design. I think this would be our ideal approach. Cloudinary itself provides this service as a plugin but we have not gotten around to moving to it.

Dynamically generating an image on our server.

ImageMagick is a classic library with lots of extensibility. It's very popular but kind of a pain to work with in my opinion. Services like Cloudinary exist because ImageMagick is a pain. I'm sure there are other libraries depending on your backend ecosystem, but I'm not aware of what the good choices are.

Services like ShareMeow

For a very specific type of generated image, ShareMeow is an interesting self-hosted web service we have played around with.

Takeaways

What we are doing works, but I'm not sure I'd recommend it. If I were you I'd skip straight to generating these images via HTML or some other nice service.

The other interesting part of the whole process

There is more to the process than simply generating the image. We also need a cache-busting bot in order to update the card periodically. If we want to show a growing number of comments over time, we cannot wait for Twitter to bust the cache once per week. It still won't update instantly, but if a tweet has been floating around for a while it will show an accurate comment count even if it is more than when the tweet went out originally.

Twitter does not expose their cache-busting feature via API, so we had to ping it via browser automation.

It's a Ruby file which makes use of Capybara and Selenium to automate the process of pinging the Twitter card cache validation. It makes use of basically these three methods:

  def visit_twitter_and_log_in
    visit 'https://cards-dev.twitter.com/validator'
    find('input.js-username-field').set(ENV['TWITTER_USERNAME'])
    find('input.js-password-field').set(ENV['TWITTER_PASSWORD'])
    click_on('Log in')
  end

  def enter_url_and_click_preview(url)
    find('input.FormControl').set(url)
    click_on('Preview card')
    result = has_content?('Page fetched successfully')
    visit 'https://cards-dev.twitter.com/validator'
  end
end

That second method needs to be pinged a few times with some junk cache-busting. This is the method that calls it:

  def enter_urls
    urls.each do |url|
      3.times do
        enter_url_and_click_preview("#{url}?#{rand(10_000)}=#{rand(10_000)}")
      end
      enter_url_and_click_preview(url)
    end
  end

That's it. Building all of this was a nice way to automate some potentially manual work. It all goes towards making the dev.to service more valuable for more users when they share content.

Comments

[Renan Lourençoni Nobile]

Very interesting post, never thought of how those images were generated (to be fair, I don't use Twitter so I didn't know this was a thing), but it seems so simple and effective I might say.

Isn't the cache-busting bot a primitive way of doing the "cleaning", are there any alternatives to that?

I've seen Cloudinary in action in a few projects me and a friend worked at college, didn't know it had that capability, really awesome feature.

You guys plan on moving to the HTML generated image in a short term?

[Ben Halpern]

Isn't the cache-busting bot a primitive way of doing the "cleaning", are there any alternatives to that?

I'm not totally sure I understand the question. Perhaps you could rephrase? To my knowledge this is the best way to go about it. Twitter associates the image with the URL which cannot be changed after it gets tweeted out. For updating future tweets we could tag a random hash on at the end like some do, but that wouldn't help the first part.

But maybe I'm misunderstanding.

No plans on moving away from this style immediately, but it will happen soon enough.

[Renan Lourençoni Nobile]

That's actually what I wanted to know, thanks.

I mean, I have zero knowledge regarding cache-busting, was just asking for curiosity.

[Richard Orelup]

Thanks for sharing this. Not heard of Cloudinary before so going to check it out. Are you still on the free plan or are you paying for it at this point?

[Ben Halpern]

Paying. One downside is their service is more expensive than a CDN, which you can work around, but at the expense of some convenience. Always trade offs.

[Richard Orelup]

Which metric got you that made it so you had to move off of free?

[Ben Halpern]

For this project it has been the bandwidth. We get plenty of traffic and even though we serve plenty of images from other CDN sources, the Cloudinary ones are still enough to be costly. The economics still work out okay because traffic is associated with revenue etc. but needs to be optimized more in the future.

[Andrei Jiroh Halili]

I want to reproduce social previews for dev.to manually, but where I can get the source files for the social preview generation from Forem or is it specific here? For example, this one below:

Thanks in advance!

[Ariel Bogdziewicz]

I'm testing DEV.TO now and social images doesn't work for my posts but it works for others. Why?!

[Ben Halpern]

Probably an issue with _the first time you load it, it sometimes doesn't load fast enough and Twitter, etc. time out within milliseconds.

If you go to cards-dev.twitter.com/validator and try a couple times (helped by adding gobbledeegook params to cache-bust)...

[Ariel Bogdziewicz]

Oh, thanks for the response. It works for Twitter and Facebook somehow. However Slack is totally not working for my posts. Other users' posts are handled easily in Slack. Is something missing?

[Ben Halpern]

Hmmmmm I'm not sure... My best guess is a caching issue, like once Slack has cached the scenario where it doesn't think there's a social image it keeps rolling with that?

[Ariel Bogdziewicz]

Thank you for your time. It started to work because of unknown reason for me. Maybe I need to wait a little bit more after publishing.

[Rémy 🤖]

I would suggest a much simpler alternative: render your image as a webpage and use a headless browser to generate a screenshot.

The only issue is that the integration of browsers is not equal with all languages and often requires specific things to be installed that are a bit flaky, but in the end that's the same issue as PDF generation right?

[Rémy 🤖]

Also, social images are cached by the platforms themselves so there is no need for a CDN (as the CDN is provided by FB/Twitter/etc)

[Jan van Brügge]

Wouldnt it be a lot simpler to have an SVG template and fill it? You can then have a simple Server (or function for our hipsters) that converts the SVG to an image. You can then use your normal CDN.

[Ben Halpern]

Yes, that approach seems solid.