DEV Community: Jane Wilson

What is A/B testing

Jane Wilson — Fri, 13 Sep 2019 09:40:10 +0000

To put it simply, A/B testing is comparing one thing to another. Like getting your eyes tested, does version A (with the lens) work better than version B (no lens).

Like changing the copy text on your sign up form in hopes it increases conversion or changing the colour of your CTA button to make it more obvious. Many marketers also use A/B testing with their ads so they can use their budgets more effectively.

No product is perfect on launch (which is also why you should always start with an MVP) and it can sometimes be hard to pinpoint exactly what is preventing users from getting the most out of a website or app. By using A/B testing alongside traditional web and app analytics, you can hone in on problem areas (like a certain form) and make small, measurable changes to increase conversion.

Planning your A/B test

Before you dive into running an A/B test, there are a few questions you need to ask yourself.

Are you running the test on-site or off-site?

Off-site tests tend to be things like ads which bring traffic to your site/app or email newsletters.
On-site tests are, therefore, carried out on elements of your site/app and those tests need to take into consideration how your site works as a whole.

What variables will you test?

Do you want to change when a user is asked to create an account?
Make a CTA button larger to make it more obvious?
Change the text used on a newsletter subscription button?

What results are you looking for?

This is particularly important for digital products. Are you looking for an increase in accounts created? More subscribers to your newsletter, or higher average basket total. You should already know your current baseline for your site/app to compare the results of your experiment to.

Running your test

You should run all of your different versions at the same time. If you run them on different days, or even at different times of the same day, you can’t really do a comparison. You should split your traffic evenly between your different versions too. Whatever platform you’re using to run your test should help with this.

Length of test

This is a bit of ‘how long is a piece of string’. In essence, you should run your test for as long as it takes to gather a statistically significant data set. For some that could be as little as one hour. For others, you could be looking at leaving the test running for a week or more. You don’t want to leave them running too long, however, as that can mean the test is influenced by other external factors.

A/B testing isn’t the sort of thing which can be done last minute. It takes planning, preparation, and patience. For the most accurate results, you’ll only be running one test at a time, and you’ll run it quite frequently (especially if you’ve changed more than one thing at once). If you’re ever feeling a little doubtful in your results, retest!

Testing multiple variables

Although for the most accurate results you should only test one variable at a time, that doesn’t mean you can’t test more than one.
If, for example, you want to test the colour of your CTA button, you can easily run an A/B/C test using three (or more) versions. This would be much more time-efficient than testing A/B, A/C, and B/C separately.

Multivariate testing

But let’s say you want to test the colour of your CTA and its button text (called multivariate testing). That would be much more complicated to run than A/B/C testing. You also need much more traffic coming to your site/app than is needed for traditional A/B testing. To work out how many variations you’ll be testing, use the following formula:

[number of variations on element A] * [number of variations on element B] = [total variations being tested]

For example, element A would be our CTA button colour and element B its copy text.

Benefits of A/B testing

By testing small changes against each other, you can slowly create the perfect version of your site, ad, or sign-up process. You also get a much better view of what your users like which may have been missed during your initial user research.
A/B testing can help you lower bounce rates, increase conversion rates (and value), and increase user engagement.

When helper text doesn't help

Jane Wilson — Fri, 13 Sep 2019 09:27:08 +0000

Like most people in IT, I use Slack at work. It’s excellent. There’s GIFS, and integrations like Google Drive, Jira, BitBucket, and Outlook (if you’re that way inclined). But like most bits of software – its got its faults.

Never mind that you can’t reply to a thread with a GIF even though that would be amazing – today I stumbled across a bit of a clanger in terms of its accessibility.

Accessibility isn’t just for the blind

When we talk about accessibility – we tend to focus on things like colour contrasts, font sizes and weights, and alt tags for screen readers. But that isn’t because accessibility is just for people with visual impairments, accessibility is for everybody.

If you think about someone with cognitive issues, there are a number of issues they could face when using a website or application. Time-limited content might not give them enough time to process what is being asked of them, a lack of navigational features might mean they can’t work out where they are in your website/app, and confusing instructions on a form might leave them unable to fill it in.

Your Slack Profile

Users on Slack have profiles. Pretty standard nowadays, you can upload a picture and add links to your social media etc. And it is in those social media links where our problem lies.

Now the problem might not be obvious when you first look. But look at what the different parts of the form are asking you. The placeholder text is asking for a URL starting with https:// but the helper text under the input is just asking for your username. This might seem like a minor issue but for someone with cognitive issues, this could mean they really struggle to fill out the form.

So I tried to fill this out this morning in my pre-coffee state. It took a while. Now, I don’t have cognitive issues (that I’m aware of… 👀) but this really threw me. I knew one part of the form was wrong but not which bit. I made the assumption that since all Twitter and GitHub URLs follow a format that the form just needed my username.

Ok so it was the helper text that’s wrong. Ugh. 🙄

So I tried putting the link in quickly – all URLs start with https:// so surely I don’t really need to put that but in…

Ok nope. You need to put in the full URL. Even though the helper text is still asking you for your username. 😔

So I got there eventually. This is a relatively minor issue with the form but it’s so easily avoided! Your placeholder text should not be used to convey vital information to the user. It disappears once a user starts typing (or the minute they click in the box) and then the user is left with your helper text. This is all part of helping users avoid making mistakes and is not only of use to those with cognitive issues but to all your users.

Not everyone is going to finish filling in the form in one sitting, they might go and make a cup of coffee and come back and get frustrated that they keep getting it wrong even though they’re following all visible instructions. This might just be a non-critical Slack profile but if the same mistake is made on a critical form, you’re seriously opening your website/app up to a lot of user frustration and potentially losing a user.

Response from Slack

So me being me, I already tweeted @SlackHQ about this issue and they replied!

I would have preferred an “oh yes we’ll get right on this and fix it” but… baby steps.

Accessibility Myths

Jane Wilson — Wed, 11 Sep 2019 15:07:45 +0000

Like a mythic beast rising from a loch – many designers, developers, and product owners seem to treat accessibility as a giant, unknown entity who will drag your project down into the depths of missed deadlines and over-budget crisis.

Which all sounds lovely and terrifying but it's not exactly what I’d call accurate.

Isn’t accessibility optional?

Do want to get sued, because ignoring accessibility is how you get sued.
Both Domino's and Beyonce have been sued due to their websites not being accessible. It is no longer OK to not conform to WCAG standards.

It increases dev time though...

Nope. You don’t need special tools, skills, or new languages to create accessible websites. You just have to build them in accordance with WCAG.

Creating accessible websites isn’t expensive in terms of dev time and money, especially if you take it into consideration from the start (like you should be anyway).

Doesn’t it restrict your design?

If by ‘restrict your design’ you mean ‘I can no longer use super-thin grey text on a grey background’ then yes. However, if you mean ‘makes my website less attractive’ then no.

Accessibility has no real impact on how visually attractive your website is and everything to do with allowing all users to use it. It is 100% possible to create beautiful websites filled with media-rich content which is interactive and engaging. Web features like forms and navigation can all be designed and built-in accessible ways and still look amazing. Just look at Nomensa as an example.

Only a small number of users need it

Really? You got proof of that do you? Didn’t think so.

Let’s take a look at some numbers for a minute (figures from Disability Sport).

11 million people in the UK have disabilities
Almost 2 million people in the UK are blind
40,000 people under 25 are blind (or partially sighted)
Over 80% of people with disabilities acquired them later in life (so they weren’t born disabled)

Are you telling me your business couldn’t do with a potential 11 million more customers? The Purple Pound is worth around £249bn to the UK economy which isn’t exactly pocket change.

Accessibility is for everyone

Oh yeah, and making accessible websites benefits everyone. Not just those with disabilities.

You might not think it's important that your app can be used one-handed because the number of people with only one hand is very low, but what happens when we add in people who have broken their arm/hand? Or people who may want to use your app while they’re holding a baby or trying to drink their morning coffee? Suddenly the potential number of users who benefit from easy one-handed use has increased quite a bit, hasn’t it?

Captions on video used to be rare until someone worked out that you can’t always have your volume up on your phone while you’re browsing Facebook. Now, just about all Facebook videos from content creators come with captions. Everyone benefits.

There are no other benefits

Apart from increased ranking in search engines, increased site traffic, increased usability, and better company reputation… sure no other benefits.

Many search engines are now testing sites for accessibility features and if your site doesn’t have them but your competitor does, your competitor gets a ranking boost while you slip down.

And as any SEO expert will tell you, with increased ranking comes increased traffic. And if your site is easy to use, they’re more likely to spend more time on your site, which also gives you an SEO boost!

Reputation wise, by building accessible websites you’re showing your users (and potential clients) that you take pride in your work and can deliver beautiful, accessible sites for all their customers.

Accessibility shouldn’t be the afterthought of a project or a ‘nice to have’. It should be part and parcel of your design and development lifecycle and ingrained into your company culture.

Actionable Metrics vs Vanity Metrics

Jane Wilson — Wed, 11 Sep 2019 14:59:14 +0000

When developing your product – it is important to focus on what’s important. Often when collecting metrics for a new product we focus on ‘more is always better’ rather than the more useful metrics which might not look quite so good on paper.

Vanity Metrics

Think about the metrics you see companies bragging about. They’ve had 40,000 signups to their closed beta, they’ve got 100,000 downloads in the first week after launch. They sound really impressive but… don’t really mean anything. How many of those 100,000 downloads stick around after 3 days? How many actively engage with the app? And how many actually spend money in-app?

These big impressive numbers are Vanity Metrics. They sound amazing and a marketing department can shout them from the rooftops to try and entice more investors and more users. But for the product team – they don’t tell you anything actionable.

Actionable Metrics

Say you run an e-commerce website. You’ve got a decent userbase and revenue is steady. Unfortunately, your cart abandonment rate is also higher than you would like. You can see in your analytics that most customers are abandoning their carts at the stage where you ask them to input their shipping and billing addresses. You ask your UX team to see if there’s anything they can do to improve this. They come up with a solution – Give the user the option to use their shipping address as their billing address.

This is a pretty clear cut A/B test in the waiting. Does giving the user the option to use their shipping address as their billing address reduce the number of users who abandon their cart at this stage.

The result of this A/B test is an actionable metric. If abandonment at this stage is reduced in the changed version, you can roll out (action) this change to 100% of your users.

Tips on gathering actionable metrics

A/B Tests

The results of an A/B test are the most actionable of all metrics. They explicitly prove a hypothesis (or disprove it) and give you a defined action to take at the end of them.

A/B test can be used on the smallest of changes right down to changing the text on a signup button to see if it increases conversions.

Metrics are people too

While vanity metrics tend to be far removed from your customers, a good actionable metric will focus on your customers. You should be looking at what is happening on a per-customer basis rather than on abstract groups. So instead of looking at a total number of pageviews, look at average pageviews per new user vs returning users. These should be relatively consistent and safe from external influences or massive fluctuations in new users.

Cohort analysis

Speaking of per-customer metrics, cohort metrics are the cream of the crop.

Think about a product which has a free trial period followed (ideally) by the customer converting to a paid account. Your product also has key lifecycle events you track: downloading the app, creating a free account, using the app and finally converting to a paid account. To compare the cohorts you could create a monthly report and track what percentage of users who downloaded the app that month has completed each lifecycle event. If the completion rates remain stable for each cohort, we know that nothing significant has changed. But if the completion rates start increasing, we can track what else has happened that month to see what is causing the increase (like a discount on the paid account or new marketing campaign).

It’s important to note that there’s nothing wrong with collecting vanity metrics. They’re extremely useful in presentations, marketing materials, and potentially to higher stakeholders. However, it is only from actionable metrics that your product can grow and improve.