DEV Community: Marc Roberts

Hello (again) Firefox

Marc Roberts — Thu, 21 Nov 2019 08:47:02 +0000

I have recently returned to using Firefox as my main browser.

I had been a Chrome user for many years. I switched shortly after it launched because it seemed much faster (remember when V8 was new) and more stable (thanks to tab isolations with multiple processes). Over time new features were added, performance continued to increase, and I customised it with extensions. Every time a new browser came out, or a major update to existing one, I didn’t think the time involved to switch would benfit me so continued to use Chrome.

A key thing for me during that time was the rapid iteration of the depth and breadth of the developer tools included, especially in Chrome’s daily release channel Canary which became my development browser. I was doing a lot of web development and I lived in the browser throughout most of the day, building and debugging constantly. Having these powerful features right there, and improving all the time, was a huge productivity boost for me.

This changed a few weeks ago. I don’t know exactly what made me try Firefox again, I don’t think it was even one thing but a number of small things that made me try (Chrome’s ever increasing market share, changes to manifest v3 breaking ad blockers). I was pleasantly surprised at how easy the transition was.

Features

I use pinned tabs a lot, keeping the things that I use all the time (Google Drive, Intercom, Jira, BitBucket, Rollbar etc) open and easily accessible. Firefox has these too, great 👍

I like to keep my personal and work related tabs separate. To do this in Chrome I used to keep two separate profiles open. This worked although when clicking a link in another application Chrome would open it in whichever profile was active most recently, not ideal. It also meant keeping two windows open. With Firefox this situation is much better thanks to it’s built-in containers feature (although you currently need an extensions to manage them).

Firefox "work" container

I can now assign certain domains to my ‘work’ container and they always open in this container. All my tabs are in one window with a small colour indicator added to the tabs to show which container that is open in. Cookies and other persistant storage is isolated within the containers so I can be logged in to my work google account in the work container and my personal google account outside of the containers. 👍👍

Another feature I used heavily in Chrome was syncing of browser data (specifically for me, history) and the ability to see what tabs are open on another device and open them on another device. Signing in to a Firefox account on all my devices gives the same functionality, with the added bonus of being able to send a tab from one device to another. 👍👍

Extensions

I didn’t use too many in Chrome but the ones I did use were pretty vital for me.

I use 1Password as my password manager, and of course they have an extension for Firefox which functions identically 👍

In this era of adtech’s invasive tracking an ad/tracking blocker is indespensible. uBlock Origin was my blocker of choice with Chrome, and while Firefox has some pretty good built-in tracking prevention I was pleased to find that uBlock Origin has an equivalent Firefox extension 👍

Chrome has a bit of a reputation for using a lot of RAM, and I have a habit of keeping a lot of tabs open. Not a great combination! To help here I use an extension called The Great Suspender. This suspends tabs that havn’t been active for a while, freeing up their memory. I found a suitable alternative for Firefox called Auto Tab Discard. 👍

In Summary

I’m very happy with Firefox. It took only a couple of hours to get switched over and setup. I’ve only come across one site, so far, that did’t work in Firefox – Apple Business Manager!

I am still using Chrome’s Canary as my development browser, mainly because I am so used to the layout and behaviour of the devtools there. But I am trying out Firefox’s developer tools and maybe one day (soon?) will switch over to that too.

My current podcast subscriptions

Marc Roberts — Fri, 25 Oct 2019 08:11:05 +0000

I listen to podcasts, a lot of podcasts. Over the last few years I’ve subscribed to a number of them that I listen to in the hours I’m travelling (after school dropoff, or commuting to/from London).

Although I only listen for around 8-10 hours a week, I manage to listen to around 15 hours a week of podcasts thanks to Overcast’s Smart Speed, which skips silent parts of episodes, and listening to most podcasts at either 1.25x or 1.5x speed.

Overcast about screen

I’ve tried various ways of organising them and have settled in to this relatively simple method. I have two playlists, one named Tech and one named Non-Tech. Generally during my morning commutes I listen to the Tech playlist and during my evening commutes I listen to the Non-Tech.

Here’s a selection of my favourites, in alphabetical order:

16 Minutes (tech)

a16z partners, guests, and industry experts disect the week’s tech news in about 16 minutes.

Find out more at a16z.com.

99% Invisible (Non Tech)

Roman Mars talks about the hidden and unnoticed design and architecture in the world around us.

Find all episodes at 99percentinvisible.org, and while you’re there you should totally sign up for their fascinating newsletter 👍.

Accidental Tech Podcast (Tech)

Weekly, Apple-heavy, tech (and often car!) news & commentary from Marco Arment, Casey Liss & John Siracusa. Typically clocking in at around two hours per episode, this is definitely one that benefits from those speedups.

All episodes and notes at atp.fm.

Freakonomics Radio (Non Tech)

Freakonomics looks at the hidden economics behind everyday things.

Read more about freakonomics and listen at freakonomics.com.

Hackable (Tech)

Everything around us is becoming more and more connected. And with more connectivity comes more chances of vulnerability, exploit and malware. Geoff Siskind talks to cybersecurity expert Bruce Snell about the vulnerabilities we can face in this super-connected world, and what we can do to avoid or mitigate them.

Listen at hackablepodcast.com.

Masters of Scale (Tech)

Reid Hoffman, co-founder of LinkedIn, talks to founders and leaders of companies who have experienced rapid growth to uncover the secrets and stories behind that growth.

Read more and listen at mastersofscale.com.

The Pitch (Tech)

This is a Dragon’s Den (Shark Tank for my friends on the otherside of the pond) for tech startups where you get to hear a lot more candid talk from both the pitcher and the investors.

From gimletmedia.com.

Planet Money (non Tech)

Planet Money take complex economic stories and breaks them down in to short, fun and easy to understand episodes. See also their even more regular/shorter spin-off, The Indicator.

Listen to their substantial (almost 1,000 episode) catalog at npr.org.

Reply All (Tech)

Describing itself as “A podcast about the internet” is really underselling! Alex and PJ scourer the internet and report on interesting, funny and/or inspiring stories. One fun repeating segment is their “Yes/Yes/No” where they try to explain weird tweets/memes to their boss.

Another one from gimletmedia.com.

Revisionist History (Non Tech)

Malcolm Gladwell looks at events or people, in great detail, that have been overlooked or misunderstood in the past. In his own words “Because sometimes the past deserves a second chance.”

Listen at revisionisthistory.com.

Science Vs (Non Tech)

Each episode digs deep (with full citations in the show notes) in to trends or recent opinions and distills them down to just the facts.

Yet another one from gimletmedia.com.

Twenty Thousand Hz (Non Tech)

From ring tones to film scores and everything in between, sound designer Dallas Taylor details the stories behind the sounds we hear everday.

Listen at 20k.org

Other honorable mentions

Every Little Thing, The Good Place, The Indicator, The Infinite Monkey Cage,Radiolab, Serial

What do you listen to? What else should I listen to?

Git commit messages

Marc Roberts — Fri, 18 Oct 2019 12:11:05 +0000

A recent post from David Thompson, My favourite Git Commit, hit the top of Hacker News today and I thought I should jot down a few thoughts.

When working on a small project alone, the details of your commit messages may not seem important but if/when that project grows, and more people join, it is useful to have consistent and well formed commit messages. If some developer is trying to understand why something is the way it is, git blame is a good place to start. If that takes them to a commit whose subject is more stuff, well that’s not going to very helpful!

What is well-formed, well lots has been written about this. The key points that I keep coming back to and try to ~~enforce~~ encourage my team to use are the following:

What & Why, not How

The code itself should be clear and self-explanatory (what is and isn’t clear and self-explanatory is something for another time) so we shouldn’t need to explain in the commit how it’s doing it. What may not be clear, and definitely should be in the commit message, is what it does and why it doing that.

This doesn’t have to be an essay, but it could be if that’s needed. Also this shouldn’t duplicate work. If there has been a long discussion in a Jira ticket or a Trello card, then a link to the ticket/card would be sufficient here. The GDS commit mentioned in David’s post is a brilliant example of this, complete with all the steps taken to diagnose it.

Style

Looking through a a git log should give a fairly consistent experience. The subjects shouldn’t be too long, nor too short. Somewhere between 30 and 50 characters, excluding any issue/build tags. It should also be written in the imperative mood, it should complete this sentence “This commit will …”.

Here are some bad examples:

c4fe7753 fix typo
6293a870 removing extra div
a0bd75ea report fixes
99428f3e more report fixes

And some better ones:

ca8355cf Fix the toggle class on chevrons
cf940fc2 Use firebase to attribute links correctly
334654af Fix diffInSeconds call on null
1ab3c47a Update search ranking weights

Gitmoji

I’ve been using an emoji at the beginning of calendar event titles for a long time. This allows me to scan across a weeks/months worth of events and get a quick feel for what’s going on:

🐔 standup (A chicken? yes, ask me why some time)
🖥 video conference
🤓 time for code review
📞 phone call
✏️ time for writing
👨‍👧‍👦 Pickup kids
📚 time for reading

I while ago I found Gitmoji. This is the same principle for git commits. Prefix your message with an emoji to give an at-a-glance summary of the change, for example:

⚡️ Performance
🔥 Removing code
🐛 Bug fix
⬆️ Updating dependancies

Squash

Sometimes when you’re ‘in the zone’ it’s easy to forget the ~~rules~~ guidelines and your commits end up looking like this

xkcd on git commits

There is a solution to this, before pushing these commits somewhere beyond your local repo you can squash these commits in to one. Most git client interfaces have a way to do this via an interactive rebase but it’s also fairly easy to do this on the command line, David Walsh explains here in Squash Commits with Git.

Track traffic exiting your site, without jQuery

Marc Roberts — Fri, 07 Aug 2015 00:00:00 +0000

A few years ago (well, over five years ago!) I published an article that showed you how to use Google Analytics to track traffic exiting your site with a little bit of jQuery. Now while jQuery is great, for smaller sites it’s usually overkill, especially if only included for this one use. Recently on one of these small sites I came up with a different way of doing it, without jQuery.

So how can we do this without jQuery? Well many of jQuery’s features are available in some form in most modern browsers. For example animation can be easily implemented with CSS3’s transition and/or animation features. Most importantly there is widespread support for querySelectorAll in browsers.

Did you know a large part of jQuery is a library called Sizzle which is just used for selecting elements with CSS selectors. This is exactly what querySelectorAll does.

With this knowledge let’s start this again without jQuery. Firstly, we’ll put something in the markup to indicate which links should be tracked. Last time, I used a custom jQuery filter to find all external links but by adding something to the markup we can be more explicit about what is tracked, as well as also tracking non-link elements (buttons and forms for example). I’m going to use a HTML5 custom data attribute for this.

<a href="http://google.com" data-ga-track>Let me google that for you</a>

That little data-ga-track is all we need. Now here’s the javascript, I’ll go through it all in a moment.

;(function(){
  if (!document.querySelectorAll) return;
  if (!Array.prototype.forEach) return;

  function track() {
    var href = this.getAttribute('data-ga-track') ||
               this.getAttribute('href');

    ga('send','pageview', '/out/'+href);
  };

  var trackables = document.querySelectorAll('[data-ga-track]');

  Array.prototype.forEach.call(trackables, function(el) {
    el.addEventListener('mousedown', track);
  });

})();

I’m using an immediately invoked function expression (IIFE) to keep the global scope clean and within the function, bailing quickly if the browser does not support a couple of key features. Obviously we need querySelectorAll and over 97% of browsers support this which is great, but IE8’s support is a little buggy. I’m also checking for the ES5 Array function forEach which is supported by over 95% of browsers. I’m using this later on as a simple way to iterate through the matching elements, but more importantly this is not supported by IE8 so that won’t run the rest of the code even though it does (partly) support querySelectorAll.

Next up there’s the function that actually logs to Google Analytics, which will look for either a value for the data-ga-track attribute or if there isn’t one, look for the href attribute. This value is then sent as a page view to Google Analytics prefixed with /out/. Last time we sent events to Google Analytics rather than a pageview but that’s just down to my current preference, you could easily change this to send an event instead.

I then use querySelectorAll to find all elements with the attribute data-ga-track and stash them away in a local variable (the IIFE keeps this variable local to the function and out of the global scope). Lastly we use the forEach function to iterate through each element and add an event handler for the mousedown event to call the track function defined earlier. I’m using mousedown here to ensure that the tracking happens before the browser starts navigating away.

The mousedown event also fires when a right click occurs on the element, some of these clicks could be someone trying to ‘open in a new tab’ but some won’t be. This will result in a few false positives.

And that’s it. There’s not much too it really. Do you think it can be made shorter, simpler, better? Or do you have any other neat little standalone javascript snippets? Let us know!

Building WordStream

Marc Roberts — Sun, 25 Jan 2015 00:00:00 +0000

Last week I spent a couple of hours playing with some new technology and built wordstream, a real-time word cloud generated from the twitter sample stream. Here’s how.

The twitter streaming APIs are a very efficient way of having the tweets you’re interested in pushed to you. For example you can use the filter endpoint to have tweets matching your filter (author, hashtag, keywords etc) but for this I was more interested in the sample endpoint which sends out about 1% of all public tweets. This endpoint does however have some limitations:

A set of credentials (app/user combination) can only have a single connection open (any further connection attempts will terminate the previous ones). So in order to use it I would either need to have each visitor authenticate with the app in order to create their own streaming connection, or build some sort of servers-side proxy.
The API response is actually quite large and when combined with the hundreds of tweets per second received results in a large amount of data being retrieved (during testing on a Friday morning I was getting a fairly consistent 2 Mbps of data from the API).

Here’s a quick example (capturing the stream for about 5 seconds resulting in 1.3 MB of data, I’ve shown just the firstfew tweets here, a sample of the sample you could say) of the streaming API data:

{ 
    created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954651971600,
    id_str: '559747954651971584',
    text: 'Мосгорсуд оставил под арестом до 16 апреля Александра Кольченко, фигуранта дела ...',
    source: '<a href="http://ifttt.com" rel="nofollow">IFTTT</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2687442584,
       id_str: '2687442584',
       name: 'Галина Никандровa',
       screen_name: 'Byce6A',
       location: '',
       url: null,
       description: null,
       protected: false,
       verified: false,
       followers_count: 210,
       friends_count: 121,
       listed_count: 1,
       favourites_count: 0,
       statuses_count: 73725,
       created_at: 'Mon Jul 28 12:45:30 +0000 2014',
       utc_offset: null,
       time_zone: null,
       geo_enabled: false,
       lang: 'ru',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'C0DEED',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '0084B4',
       profile_sidebar_border_color: 'C0DEED',
       profile_sidebar_fill_color: 'DDEEF6',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://abs.twimg.com/sticky/default_profile_images/default_profile_1_normal.png',
       profile_image_url_https: 'https://abs.twimg.com/sticky/default_profile_images/default_profile_1_normal.png',
       default_profile: true,
       default_profile_image: true,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'ru',
    timestamp_ms: '1422289286660'
},
{
    created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954639384600,
    id_str: '559747954639384577',
    text: 'Beautiful life is so much better than Carry you tbh',
    source: '<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2974152997,
       id_str: '2974152997',
       name: 'Sandra Young',
       screen_name: 'edwardalazobuy1',
       location: 'West Virginia',
       url: 'http://optimizedirectory.com/',
       description: '1D / Glee / T-Swizzle / Narnia / Criminal Minds / KSS 8 / Lucky #18/ #23 / #24 / Directioner / MATTHEW GRAY GUBLER FOR DA WIN! / Louis\' pants',
       protected: false,
       verified: false,
       followers_count: 0,
       friends_count: 1,
       listed_count: 0,
       favourites_count: 0,
       statuses_count: 37,
       created_at: 'Sun Jan 11 06:10:53 +0000 2015',
       utc_offset: null,
       time_zone: null,
       geo_enabled: false,
       lang: 'en',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: 'C0DEED',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '0084B4',
       profile_sidebar_border_color: 'C0DEED',
       profile_sidebar_fill_color: 'DDEEF6',
       profile_text_color: '333333',
       profile_use_background_image: true,
       profile_image_url: 'http://pbs.twimg.com/profile_images/559450280236830720/fGI9TXLt_normal.png',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/559450280236830720/fGI9TXLt_normal.png',
       profile_banner_url: 'https://pbs.twimg.com/profile_banners/2974152997/1422261339',
       default_profile: true,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'en',
    timestamp_ms: '1422289286657'
},
{ 
    created_at: 'Mon Jan 26 16:21:26 +0000 2015',
    id: 559747954672943100,
    id_str: '559747954672943104',
    text: 'Saints win 2-0! Enppi are 0-0 so double chance looking good on this one too.',
    source: '<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>',
    truncated: false,
    in_reply_to_status_id: null,
    in_reply_to_status_id_str: null,
    in_reply_to_user_id: null,
    in_reply_to_user_id_str: null,
    in_reply_to_screen_name: null,
    user:
     { id: 2960224947,
       id_str: '2960224947',
       name: 'The Secret Tipster',
       screen_name: 'Secret_Tipster_',
       location: '',
       url: null,
       description: 'FREE betting tips and £10-£1,000 challenges! \n\n5pts - Strong tip (high stakes)\n3pts - Good tip (medium stakes)\n1pt - Fair tip (low stakes)',
       protected: false,
       verified: false,
       followers_count: 343,
       friends_count: 1588,
       listed_count: 2,
       favourites_count: 104,
       statuses_count: 290,
       created_at: 'Sun Jan 04 14:09:31 +0000 2015',
       utc_offset: 0,
       time_zone: 'London',
       geo_enabled: false,
       lang: 'en-gb',
       contributors_enabled: false,
       is_translator: false,
       profile_background_color: '000000',
       profile_background_image_url: 'http://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_image_url_https: 'https://abs.twimg.com/images/themes/theme1/bg.png',
       profile_background_tile: false,
       profile_link_color: '89C9FA',
       profile_sidebar_border_color: '000000',
       profile_sidebar_fill_color: '000000',
       profile_text_color: '000000',
       profile_use_background_image: false,
       profile_image_url: 'http://pbs.twimg.com/profile_images/551742687452229634/Q2rfimMq_normal.png',
       profile_image_url_https: 'https://pbs.twimg.com/profile_images/551742687452229634/Q2rfimMq_normal.png',
       default_profile: false,
       default_profile_image: false,
       following: null,
       follow_request_sent: null,
       notifications: null },
    geo: null,
    coordinates: null,
    place: null,
    contributors: null,
    retweet_count: 0,
    favorite_count: 0,
    entities:
     { hashtags: [],
       trends: [],
       urls: [],
       user_mentions: [],
       symbols: [] },
    favorited: false,
    retweeted: false,
    possibly_sensitive: false,
    filter_level: 'low',
    lang: 'en',
    timestamp_ms: '1422289286665' 
}

Here’s a few things to note:

There is a lot of metadata about tweets included which I don’t need.
There are quite a few native retweets which include the retweeted text prefixed with RT in the new tweet. Should they be excluded or should the retweet count towards the word count?
There are many different languages, in order to have something meaningful for myself (I only speak english fluently, plus a couple of other european languages poorly at best) I decided to only process english tweets.

All of this meant that it made sense to build a simple back-end service/proxy that created a single streaming connection, processed this data and fed a far more condensed amount of data out to the browser(s). I chose to build something with node.js.

First we need to get the data out of the streaming API. I found a npm module called node-tweet-stream that worked with the filter endpoint, and with a little butchery was able to hook it up to the sample API instead.

var twitter = require('./twitter-stream'), //
    stream;

stream = new twitter({
  consumer_key: 'xxx',
  consumer_secret: 'xxx',
  token: 'xxx',
  token_secret: 'xxx'
});

stream.on('tweet', function(tweet) {
  console.log(tweet);
});

stream.connect();

I often use Heroku for hosting small things like this and Heroku encourages you to store as much of the application configuration as possible in the environment rather your application code respository. To manage this in my Ruby projects I use dotenv to allow me to keep such configuration in a .env file locally (excluding this from the source control). I was very pleased to find such functionality also exists for developing in node. A quick install of the dotenv npm module and a simple require and it was working here.

Logging things out to the console is great for debugging things but no real use. To get the data out to a browser I started to build a simple express app as I’d had some experience with this before but something reminded me of web sockets and socket.io so I thought I’d try playing with them. Again, all that was required was another install/require and a couple of extra lines and now we have tweets being proxied through to the browser(s). The code was now looking like this:

var app = require('express')(),
    dotenv = require('dotenv'),
    server = require('http').Server(app),
    io = require('socket.io')(server),
    twitter = require('./twitter-stream'),
    stream;

dotenv.load();

stream = new twitter({
  consumer_key: process.env.TWITTER_CONSUMER_KEY,
  consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
  token: process.env.TWITTER_TOKEN,
  token_secret: process.env.TWITTER_TOKEN_SECRET
});

server.listen(process.env.PORT || 5000);

stream.on('tweet', function(tweet) {
  io.emit('tweet', tweet);
});

stream.connect();

The main reason for proxying the data was to reduce the amount sent out to the browsers, so now was time to take those massive responses and reduce them to some word lists. Again I found a couple of great npm modules to help with this; keyword-extractor for extracting the important words (or more accurately, excluding the non-important words), and franc for determining the language of the tweet (keyword-extractor only works with english, much like my brain).

While writing this I noticed that the twitter response actually contains a lang field, negating the need to use franc. I hadn’t noticed this at the time, oh well!

Plugging these in, along with some exclusions myself (links, retweets, replies) gives us the final code (find it on GitHub) that was deployed to Heroku:

var app = require('express')(),
    dotenv = require('dotenv'),
    server = require('http').Server(app),
    io = require('socket.io')(server),
    xt = require('keyword-extractor'),
    franc = require('franc'),
    twitter = require('./twitter-stream'),
    stream;

dotenv.load();

stream = new twitter({
  consumer_key: process.env.TWITTER_CONSUMER_KEY,
  consumer_secret: process.env.TWITTER_CONSUMER_SECRET,
  token: process.env.TWITTER_TOKEN,
  token_secret: process.env.TWITTER_TOKEN_SECRET
});

io.set('origins', '*:*');

server.listen(process.env.PORT || 5000);

function exceptions(word){
  if (word.match(/https?:/)) return false; // links
  if (word.match(/^@/)) return false; // replies
  if (word.match(/&|\/|"/)) return false; // random punctuation

  return true;
}

stream.on('tweet', function(tweet) {

  // ignore retwets
  if (tweet.retweeted_status || tweet.text.match(/^RT/)) return;

  // only english for now
  if (franc(tweet.text) != 'eng') return;

  // parse that tweet, extract words
  words = xt.extract(tweet.text,{
    language:"english",
    remove_digits: true,
    return_changed_case:true
  }).filter(exceptions);

  if (words.length > 0) io.emit('tweet', words);
});

stream.connect();

So with less than 50 lines of code we have live tweets being parsed for words and those word lists being sent out to the browser. Now let’s get the browser to render them.

This is going to be almost entirely javascript powered so I’m going to concentrate on that, if you’re interested in the HTML and CSS then take a look at the source and ask me any questions you might have.

Firstly we’ll use socket.io to connect to the web socket and start grabbing the words as they come in.

I’m using the underscore.js library here to get access to some simple helper functions

var socket = io.connect('wss://twitter-word-stream.herokuapp.com/');

socket.on('tweet', function (data) {
  _.each(data, function(word) {
    console.log(word);
  });
});

And there we go, the words are being spat out to the browser’s console, but of course this is of no practical use. Lets count the occurences and displaying that visually. We’ll do this by throwing the words and their counts in to an object and then displaying the most popular ones periodically.

var socket = io.connect('wss://twitter-word-stream.herokuapp.com/'),
    word_counts = {},
    text_nodes = {},
    frame = 0;

function render() {
  var max = 0,
      displayed_words = [];

  // increment frame counter
  frame++;

  _.each(word_counts, function(count) {
    if (count > max) max = count;
  });

  // filter them to just the most popular ones
  displayed_words = _.sortBy(_.keys(word_counts), function(word) {
    return max - word_counts[word];
  }).slice(0,30);

  _.each(displayed_words, function(word) {
    var size = words[word] / max,
        text, node;

    // create the text node if need be
    if (!text_nodes[word]) {
      text = document.createTextNode(word);
      node = document.createElement('span');

      // position kind of in the middle somewhere
      var top = 80*Math.random();
      var left = 70*Math.random();

      // give it a random pastelly colour
      node.setAttribute('style', "top: " + top + "%; left: " + left + '%; color: hsla('+360*Math.random()+',50%,50%,0.75)');

      node.appendChild(text);
      document.body.appendChild(node);
      text_nodes[word] = {
        updated: frame,
        node: node
      };
    } else {
      text_nodes[word].updated = frame;
    }

    // clear expired words
    _.each(text_nodes, function(obj, word) {
      if (obj.updated < frame) {
        obj.node.remove();
        delete text_nodes[word];
      }
    });

    // size it relative to it's occurence
    text_nodes[word].node.style.transform = 'scale(' + (0.2 + size*0.8) + ')';
    text_nodes[word].node.style.webkitTransform = 'scale(' + (0.2 + size*0.8) + ')';

  });

}

setInterval(render, 500);

socket.on('tweet', function (data) {
  _.each(data, function(word) {
    word_counts[word] = (word_counts[word] || 0) + 1;
  });
});

There’s a few things to explain here:

A scale transform is being used instead of font-size to change the size of the words as this results in a GPU accelerated transform, which we can then enhance with transitions with very little impact on performance.
The created DOM nodes are being cached in the text_nodes object so we don’t have to recreate them each time or try to find them.
A frame number is used to note when the elements were last updated so that it’s easy to remove any words that are no longer popular.
The colour of the words are randomised using hsla() as this only requires a single number to be generated (the hue) instead of the multiple numbers required to use rgba().

This works great, but it will count occurences since you first loaded the page, I wanted it to only consider the most recent words (lets say only the last 5 minutes) so I need to store the word lists in such a way that I can easily and quickly remove the older ones. I could have stored the time of each occurence of each word but that would get complicated. I decided instead to store the word occurences in several different objects (I called them buckets), with the one that was incremented being rotated every few seconds. The render method would then only use the buckets covering the last 5 minutes worth of occurences.

var socket = io.connect('wss://twitter-word-stream.herokuapp.com/'),

    text_nodes = {},
    frame = 0,

    current_bucket = {},
    buckets = [current_bucket],

    bucket_count = 30, // how many buckets to remember
    bucket_width = 10; // how many seconds worth of words to keep in the buckets

function render() {
  var max = 0,
      words = {},
      displayed_words = [];

  // increment frame counter
  frame++;

  // get counts of words across all buckets
  _.each(buckets, function(bucket){
    _.each(bucket, function(count, word) {
      words[word] = (words[word] || 0) + count;
      if (count > max) max = count;
    });
  });

  // filter them to just the most popular ones

  displayed_words = _.sortBy(_.keys(words), function(word) {
    return max - words[word];
  }).slice(0,30);

  _.each(displayed_words, function(word) {
    var size = words[word] / max,
        text, node;

    // create the text node if need be
    if (!text_nodes[word]) {
      text = document.createTextNode(word);
      node = document.createElement('span');

      // position kind of in the middle somewhere
      var top = 80*Math.random();
      var left = 70*Math.random();

      // give it a random pastelly colour
      node.setAttribute('style', "top: " + top + "%; left: " + left + '%; color: hsla('+360*Math.random()+',50%,50%,0.75)');

      node.appendChild(text);
      document.body.appendChild(node);
      text_nodes[word] = {
        updated: frame,
        node: node
      };
    } else {
      text_nodes[word].updated = frame;
    }

    // clear expired words
    _.each(text_nodes, function(obj, word) {
      if (obj.updated < frame) {
        obj.node.remove();
        delete text_nodes[word];
      }
    });

    // size it relative to it's occurence
    text_nodes[word].node.style.transform = 'scale(' + (0.2 + size*0.8) + ')';
    text_nodes[word].node.style.webkitTransform = 'scale(' + (0.2 + size*0.8) + ')';

  });

}

function rotate_buckets() {

  current_bucket = {};
  buckets.push(current_bucket);

  while (buckets.length >= bucket_count) buckets.shift();

}

setInterval(rotate_buckets, bucket_width*1000);
setInterval(render, 500);

socket.on('tweet', function (data) {
  _.each(data, function(word) {
    current_bucket[word] = (current_bucket[word] || 0) + 1;
  });
});

And there we have the (more or less) finished code, and here it is running on Heroku

There’s still a few things I’d like to improve when I can:

The positioning of the words is random, which often results in excessive overlapping, the translucency helps with that but it sometimes is quite bad.
It would be nice to have it be a little more customisable, maybe the source being a hashtag, a user or your timeline instead of the sample stream.

It was fun to spend a couple of hours playing around with some new things, everyone needs to be able to do that occasionally.

What new technologies are you most excited about playing with?

UPDATE: The source for all this can be found on GitHub

marcroberts / wordstream

A twitter streaming client to extract words

node-js-getting-started

A barebones Node.js app using Express 4.

This application support the Getting Started with Node on Heroku article - check it out.

Running Locally

Make sure you have Node.js and the Heroku Toolbelt installed.

$ git clone git@github.com:heroku/node-js-getting-started.git # or clone your own fork
$ cd node-js-getting-started
$ npm install
$ npm start

Your app should now be running on localhost:5000.

Deploying to Heroku

$ heroku create
$ git push heroku master
$ heroku open

Documentation

For more information about using Node.js on Heroku, see these Dev Center articles:

View on GitHub

My Favorite Ruby gems

Marc Roberts — Mon, 01 Sep 2014 00:00:00 +0000

Many tasks that would otherwise take hours or days to build can be easily implemented by simply including the relevant Ruby Gem in your project. Here’s my pick of interesting, useful and most-importantly time-saving gems:

What’s a Ruby Gem? Ruby and particularly Rails have an excellent developer community providing numerous open-source (usually) libraries called Gems. Ruby, Gem, geddit?

Firstly, this very blog would not exist without the wonderful Jekyll, a blog-aware static site generator. I also often use Sinatra for smaller sites that don’t require a large framework like Rails.

When it comes to authentication my first port of call is always Devise. This, along with it’s various extension gems, covers any sort of authentication you could want: Username/Email & Password, Oauth (Facebook, Twitter etc), and Mozilla Persona to name but a few.

FlagShihTzu allows you to store numerous boolean fields in a single numeric database field within ActiveRecord and access them all as though they were individual fields. Kaminari has replaced will_paginate as my pagination gem of choice.

cache_digests enabled a russian-doll style nested caching mechanism within view that can produce huge speed improvements. Read more about this on the signal vs noise blog post how basecamp next got to be so damn fast, under point #2.

ActiveAdmin build beuatiful admin interfaces quickly and simply, which is a great place to start when building the back-end services required for any web application. lograge reduces the Rails log output of most request down to a single easily-parsable line, great for searching and troublshooting.

I’m a big fan of Heroku so obviously I use a few gems that are particularly useful here: rails_12factor enforces a couple of aspects of the Twelve Factor App that Rails/Heroku don’t by default, font_assets ensures your font assets are served with the correct headers to enable browsers (mainly Firefox) to use them correctly, and heroku-deflater gzips just the content that would benefit from such compression without wasting CPU cycles compressing content that does not (it will also serve any existing .gz file).

Many of my favourite gems are solely for use during development. quiet_assets will keep those noisy asset related request lines out of your development log files. meta_request and its RailsPanel Chrome extension gives you access to information about each request in Chrome’s developer tools, here it is in action:

better_errors makes the built-in rails error pages about a thousand percent better by giving you a fully interactive console in your browser at the point of the error, or any parent frame. I recorded a short screencast demonstrating this last year:

These are just the tip of the iceberg, RubyGems.org has nearly 90,000 gems listed. What are your favourite gems? Let us know.