A/B Test Results and Case Studies for User Experience Design

Posted by Harshad

A/B Test Results and Case Studies for User Experience Design

A/B Test Results and Case Studies for User Experience Design

Posted: 02 Apr 2014 08:01 AM PDT

The process of split testing is not as complicated as you may think. You contrive a specific goal, such as gaining more visitors onto another sub-page or generating more signups. Then, you create various alterations of said webpage and track analytics for which version performs better. These tests are often done privately with Internet marketing firms, but recently there’s been a lot more available data on blogs.

Today, I’ll share some case studies related to A/B testing for website performance. The results are quite insightful but remember that not every website is the same, because the audience and traffic sources generally differ in key areas. Each website should be examined as its own entity and user experience trends are greatly dependent on other elements in the page (navigation, content width, font size, etc).

So methods which work for another site may not work exactly for yours. However, the beauty of A/B testing is that you can always attempt new goals and determine what’s holding you back. Now, let’s take a look.

Customer Testimonials and Sales

Here’s one of the cases on Visual Website Optimizer (VWO) and their A/B Split Testing blog. This work was done on Wikijob’s Aptitude Tests page. The goal was to improve conversions from visitors to the purchase page. The team worked on various implementations and finally settled with adding customer testimonials onto the page.

visual website wikijob ab testing testimonials

This increased product sales by 34%. There were no major design updates, color changes, or repositioned elements. Merely a new box underneath the heading with actual testimonials from past customers.

Goal: Improve the number of conversion from browsing around the site onto the final purchase page.


The value of peer review and social cognition is crucial. Just a few testimonials added into the page makes a world of difference. People visiting your website are interested in what other people think.

Lean & Slim Newsletters

The VWO blog has an excellent case study on lean newsletters and their benefits from higher clickthrough ratings. Based on Sheena Lyengar’s study on the demotivation of too many choices, users are busy and have too many things to choose from, hence they often end up choosing nothing at all.

email newsletters ab testing ux split

The case study options were to limit each newsletter down to a single call-to-action button. This means readers would open the e-mail to see only a single link to one landing page or blog entry. The results showed a 17% increase of clickthrough rates from the e-mail onto the website.

People are more willing to click onto your webpage when your newsletter is offering only a single, straightforward option as opposed to a dynamically generated RSS newsletter.

Goal: Increase the clickthrough rate of e-mail newsletter subscribers.


Less is more. When your subscribers open an e-mail and there are 5 different sections of related articles, it can be overwhelming. By offering a single link back to the site it sends a clear, unwavering message and allows people to immerse themselves into the content.

Masthead Image Backgrounds

The company OpenMile used VWO to test the masthead design on their landing page. Initially, their control had a large stock photograph of trucking, greatly related to their product. The new design features larger text and a deeper focus on the signup form with subsequent submit button. The goal was to measure conversions and see which variation performs better.

ab testing openmile masthead signup form input

Generally, the updated version without a background image has pulled in more signups. This doesn’t mean a single masthead image will always perform poorly. But for this specific case study the designers believed a background image was too overwhelming and made the signup form appear off-putting.

Goal: To increase the number of lead generations from the site’s landing page(s).


Masthead images work nicely when there is not much content other than text. Visitors are more likely to signup with a bright masthead, or by moving the signup form beneath the background image.

Product Listing Pages

This is a case study following the Vancouver 2010 Olympic store. The goal was to introduce a small bit of content into sidebars which would directly relate to visitors with the aim of improving sales and persuading visitors into making a purchase.

olypics testing split a/b ideas ecommerce thumbnails

The control version had no sidebar content, variation A had a left sidebar with product categories, while Variation B had a right sidebar displaying “related products” from a recommendation engine. The last variation performed the best and produced the highest number of sales (7% more than Variation A). These results demonstrate how users are more interested in products which they can directly see right on the page.

Consider this for your own eCommerce webshop. The addition of a recommendation engine can help visitors buy related products with ease; as can extended thumbnail previews for each garment of clothing in a different color which users found helpful.

Goal: To improve conversion rates from visitors into customers.


Offering related products will bring more attention to items which users may not have found otherwise. Just a small update to your eCommerce website layout can bring about a large increase in the number of product sales.

Videos in Landing Pages

A great blog post on the website Unbounce follows the use of videos in landing pages. This is from a startup named VidYard back when it was only a landing page. The tests had a signup form and a demo video which was embedded into the page, displayed in a lightbox, or not even displayed at all.

This method was a test into what video promotion can do for your product when users have no other alternative to learn about what you do.

vidyard split a/b testing conversion rates video landing webpage

The best results were with the video embedded into the page (up 69%) and even higher with the video in a lightbox (up 100%). The biggest point is to make sure that your video actually explains your product and how it can impact the user.

The best results found simplicity is the easiest way to get visitors interested. Make a simple video or animation that people can follow easily and present it in a clear manner on the page.

Goal: How does web video impact conversion rates on landing pages?


Both location and quality of video affects conversion rates. People are more willing to watch a video, if embedded into a lightbox. But they also want to be informed. This study found that bright animations and cartoons tend to keep people interested for longer periods of time.

Stimulating Webcopy

The testing for conversion lifts in user signups follows a website Bettingexpert, with tips and tricks for betting. The original signup form had very basic web copy and the team wanted to bring in more users onto the website.

control signup form webcopy ab split testing article

The in-test actions and subsequent results are fascinating. There were no changes made to the website design itself, only updates to the form text. The question of “why should I fill out this form?” was not orginally answered. But updating the header text and the submit button value dramatically improved signup rates by 32% with a 99% confidence rating.

Just by looking at these two variations, it’s easy tell which one is better and more interesting, isn’t it? All webmasters should keep this in mind from the research done: “The copy you use in your signup form has direct and measurable effects on your conversion rate.”

Goal: Increasing the number of members/new signups.


With a clearer message in your webcopy, you can make a difference. By getting into your visitor’s head(s) and practicing different ideas for webpage copy, you’ll find a solution which is more direct and influential to the average person.

A Valentine’s Day Newsletter

Our next study investigates catching e-mail subscribers with an interest to purchase a certain product based on holidays. The Marine Mammal Center has an online gift shop, which handles a surge of purchases on various holidays like Mother’s Day and Valentine’s Day

During Valentine’s Day in 2013, the team sent out two variations of the same e-mail with different images for the product.

testing ab case newsletter products conversion sales mammal center

The test was to determine which photo would result in more purchases – the related baby seal, or a photo of the product itself. Naturally, the product photo got 6 more purchases and garnered 2 genuine donations from visitors. The product that they’re purchasing is more captivating and it’s no surprise as the results speak for themselves.

Goal: Increase the number of purchases through various holiday e-mail newsletters.


People are more willing to spend cash on items if they can see exactly what it is. Imagery placed inside the e-mail is key. Holidays are also a period where sales tend to spike therefore it’s perhaps worth redesigning a custom layout for the more important holidays.

Primary and Secondary Form Inputs

Although this was recorded and published in August 2007, the results are still relevant today. The article discussing primary & secondary actions in web forms delves into a case study following user’s interactions with submit and cancel buttons.

The theory is that by keeping the buttons closer together users are less likely to get confused. Also that by having the cancel button appear smaller and less in the way, users would not accidentally click and erase all of their input data.

adding input buttons a/b split testing case study values

The team ran several tests over 20+ different people to get their opinion. There were seven form variations and options A, B, and C received the highest praises from users. Option B had the best results with nobody making any mistakes in the form.

Yet many users also confirmed that option A works well using a single text link for the cancel button. Additionally, options D & E only caused confusion. Unless your form is particularly aligned to the right, it’s best to keep these two buttons aligned on the left.

Goal: What is the best location for submit/cancel buttons in web forms?


Visitors will perform best using form interfaces which they are familiar with. Keep the submit button on the left side, and always the same size as Cancel (if you even need it). Don’t use alternate colors for each button. Overall common sense web design works the best when creating long forms with 10+ input fields.

Wireframe Usability Testing

This case study comes from, online testing company Loop11. It’s a wireframing case study that took place at a University in Sweden.

They made two distinct website wireframes for a tourist information webpage. Participants in the project were given either one of these layouts. Each person was asked to complete a number of 6 different tasks. These tasks included finding a city map, tourist attractions, language courses, and other typical website information.

wireframing and split testing ui ab tests usability

They found that having two columns full of navigation links helped more people find what they needed. But this led to users generally taking longer to find what they needed. Potentially an issue with clutter or link text. But using these results and some other ideas provided a great starting point for the new website launch. Read more about it here.

Goal: How to structure a webpage layout for easy access to a lot of information?


It’s best to provide 2 or 3 separate navigation menus if needed. Try to keep pages no further than 2-sub levels deep as this will lead them to have to dig around more. Other wireframe structures may perform better for other websites.

Further reading

Final Thoughts

Web designers who’ve never gone through the process of A/B split testing may find all of this information to be foreign and strange. But as you research more, it starts making sense. I hope that these case studies open doors for even further marketing research. My examples are only some of the best that I could find. If you know any interesting A/B testing ideas or similar links to case studies, why not share them in the comments below?


10 Free Android Video Editors To Help Perfect Your Videos On-The-Go

Posted: 02 Apr 2014 06:01 AM PDT

Recording your own video is so much easier now thanks to capable Android devices. Although it’s easy to record something, editing is another matter altogether. A good video editor is quite necessary to transform your boring video recording into something catchy and fun to watch.

Video Show Android App

We know how hard it is to find a proper video editor for Android, especially for free. But it doesn’t mean that it’s not possible. Here, we share with you 10 Free Android video editors to make video editing on-the-go a reality.

1. VidStitch

If you ever need an app to create a video specifically for Instagram, look no further. VidStitch combines photos and videos into one beautiful collage and allows you to share it on Instagram with ease. Moreover, you can also use your own soundtrack for the collage that you’re creating.


VidStich allows you to choose from 50 different collage frames for your own video collage. You can also adjust the width of the frame, its color and even the saturation to fit your taste.

2. AndroVid Video Editor

This is a standard Android video editor that’s capable of merging, splitting, trimming and transcoding your videos on-the-go. Furthermore, this app enables you to add text, music, and video effects to further enhance your videos. It has all the necessary features to edit your videos.

AndroVid Video Editor

Whether the video’s recorded from your smartphone or you ripped a high quality video from somewhere, AndroidVid can easily handle them. If all you need is a good video editor that efficiently does several things, then this is what you’re looking for.

3. WeVideo

WeVideo is a video editing app that allows you to edit your video on your mobile using its very simple interface. You can choose from a variety of themes, ranging from travel journal to a wedding theme, according to whatever suits your video. Next, just add an audio file and you can immediately publish it to the WeVideo server, with the choice of making it a public or private video.


WeVideo ensures that your video exists in its cloud server so that you can further edit it with its cloud editor. And since your video is in the cloud, you can easily access it from anywhere.

4. VidCutter

Videos may not always be the ideal length that you want. Often at times, you might just want a small part of the original video. With VidCutter, you can trim any of your videos easily and quickly without any issue.


All you need to do is set the start and end time at the part of the video that you want to cut and it will save it as a new video right away. VidCutter support formats in 3pg, avi, mpeg, mp4, and wmv thus making it one of the most reliable Android video cutter that you can get for free.

5. KlipMix

Ever wondered how to turn your videos and photos into a nicely edited video of your own? Try KlipMix, an app that’s capable of mixing videos and even photos together with your favourite music to create your own videos. Iff your recording happens to be upside down (mistakes like this can happen), you can fix it easily with KlipMix since it can handle various rotations of all your videos and photos.


Besides that, KlipMix also comes with a feature that allows you to set the duration of each video or photo as you wish. Now, you can have an edited video of your own that’s even free from watermarks.

6. KineMaster

KineMaster is one of the best Android video editors that you can get on an Android device. It has all the features that you will ever need on mobile, such as color, saturation, brightness and even 3D transition effects. There’s a lot of themes to choose from that comes with specific music and visual effects.


However, there’s a downside to using KineMaster, where your videos will be watermarked when exported. Some find that it’s not really a big issue, but if you can’t live with that, just get the paid app for $2.99 to remove the watermark. Doing that will also give you more theme options.

7. Media Converter

Videos comes in many different formats and often you end up with one that may not play too well on your Android device. Well, this is no longer an issue with the Media Converter app. This app allows you to convert any of your videos into formats like mp4, 3gp, avi, mpeg and flv right away.

Media Converter

You can also set the parameter for your video/audio bitrate, resolution and the audio sample rate. The app layout is pretty straightforward, and it even comes with the capability to convert video to audio with ease.

8. VidTrim

VidTrim brands itself as a video editor and organizer on the Android platform. It performs well for tasks like video trimming, video conversion, frame grabbing and much more. Uploading a high resolution video to your favorite social media usually takes up a lot of your time. Instead, you can use VidTrim to scale down your video resolution to help reduce its overall size.


This is made possible because it comes with the transcoder that helps to convert, resize and compress your video on-the-go. So, if you often record videos in high quality and want to share your videos with the world, try out VidTrim.

9. VivaVideo

So far we’ve shared a few video editor apps but VivaVideo is one that works as both a video camera and a video editor. VivaVideo comes with features like themes, filters, video effects, titles and transitions that you can get within the app for free.


There really is no limit when it comes to perfecting your videos with this app, as you can even use it to capture video recordings with all kinds of capture modes like slow-motion, fast-motion, music video and FX.

10. VideoShow

VideoShow is a video editor that can create videos using a mix of videos and photos. It comes with additional features like voice to video recording, multiple text to video with timings, video effects, music, trimming and exporting. You can easily edit your own videos and export it quickly with minimal effort.


VideoShow also allows you to express yourself in your own edited video using any effect, text, or music. You can even feature your own voice with ease, if you want. VideoShow is perfect for people who want to create a short film in a snap without any watermark.


Easily Animate Text With Textillate.js

Posted: 02 Apr 2014 03:01 AM PDT

In our previous post, we have introduce you to Animate.css, an awesome CSS3 library, that makes CSS3 animation creation easier. In this post we are going to introduce you to a jQuery plugin, built using Animate.css, which is able to animate text, called Textillate.js.

Although you can animate text with Animate.css, it’s only animated wholly, not letter by letter. Even if you may have an approach of wrapping each character to an element and animating them, it’s going to be troublesome and not effective. With this plugin, you can animate every single letter of your text easily.

Textillate.js divides the animation into 2 components, in and out animation. You may apply them with different animations, separately. All the animation effects use Animate.css library effects.

Besides Animate.css, Textillate.js is also built based on Lettering.js. We can say, this plugin is the combination of these 2 powerful tools. Animate.css works on the animation feature, while Lettering.js focuses on the typography.


Textillate.js needs a few dependencies before it is ready for use. They are Animate.css, Lettering.js and jQuery. After you’ve got the files downloaded, include them in your project, like so.

 <script src="js/jquery.min.js"></script> <script src="js/jquery.lettering.js"></script> <script src="js/jquery.textillate.js"></script> 

And don’t forget to put the Animate.css library in the head section.

 <link href="css/animate.css" rel="stylesheet"> 

Basic Markup

To start using Textillate.js, you must define a class of element you want to animate, then add your chosen effect into it. Please note that this plugin works only on text that contain elements, so any element that is not text will not be animated.

As an example, we have an element which contains text here, and a demo class, like so.

 <div class="demo"> <p>Each of this letter will animate.</p> </div> 

We call the class name with Textillate.js initialization, like so.

 <script> $(function () { $('.demo').textillate(); }) </script> 

The text will now animate.

Textillate.js Options

The previous Javascript code will apply only default animation setting to the text. To change it, there are two methods you can use.

1. First, by inputting HTML data api like the following.

 <div class="demo" data-in-effect="rollIn"> <p>Each of this letter will animate.</p> </div> 

data-in-effect defines the entrance effect of the text. While for the exit effect, you can use data-out-effect api.

2. You can also add the options into the Javascript Textillate.js initialization like the following code.

 <script> $(function () { $('.demo').textillate({ in: { effect: 'rollIn' } }); }) </script> 

In addition to effect, there are other useful options you may want to use. To see the full list of other options, head over to the documentation page.

Final Thought

Textillate.js is a very useful plugin especially when you want to attract users to the text you want to highlight the most. It can be your site heading, services, updates, promotion or anything else. And for more enhancements, you can combine Textillate.js with another plugin like FitText.js designated for gigantic display text. Once again, remember to use the animation in the right amounts.


How to Enable CSS3 Border Radius in Internet Explorer 8 and below

Posted: 01 Apr 2014 10:01 PM PDT

CSS3 gives us the ability to create rounded corners with the border-radius property. But as you might already know, this new feature is not recognized in Internet Explorer 8 (IE8) and its earlier versions. So when you apply CSS3 Border Radius to an element, it still appears as a box with pointed tips.

You may find a lot of different advice on the Internet suggesting you ignore IE8. You wish you could do so. But if you’re working on a website for a government institution or a bank, you may not have a choice. You still have to make the website look as nice as it does in modern browsers, which includes the ability to apply CSS3 Border Radius.

If you really have to make your website IE8-ready, while being able to use the CSS3 feature, we have just the tip for you.


CSS3Pie is a kind of polyfill for CSS3 decorative features. Aside from CSS3 Border Radius that we’re going to discuss in this article, CSS3Pie also supports Box Shadow and Gradients, though it’s limited to the Linear Gradient type.

Download CSS3Pie (here). Put it inside your project directory – I placed it in the css folder. As you can also see from the screenshot below, I’ve created one stylesheet as well as one HTML file.

Open the stylesheet in code editor and set one of the elements in the HTML file with rounded corners, like so:

 .border-radius { height: 100px; width: 100px; background-color: #2ecc71; -webkit-border-radius: 50px; -moz-border-radius: 50px; border-radius: 50px; } 

As the element’s width and height are both 100px, and the border-radius is set to 50px, the element will become a circle:

In IE8, as we expect, it would still be a square (sigh), but don’t worry.

To enable the rounded corner effect, insert this line: behavior: url(css/pie.htc);, like so.

 .border-radius { height: 100px; width: 100px; background-color: #2ecc71; -webkit-border-radius: 50px; -moz-border-radius: 50px; border-radius: 50px; behavior: url(css/pie.htc); } 

Note that the file path in url()has to be either an absolute path or relative to the HTML file. You can also add this line in a separate file.

You may have a stylesheet that is designated for Internet Explorer. You can put behavior: url(css/pie.htc); within that file, but ensure that it’s declared within the same CSS selector, like so.

 /* declared in ie.css */ .border-radius { behavior: url(css/pie.htc); } 

Refresh the page in IE8. The rounded corner should now come into effect (ta-da!). It should work in IE7 as well.


Internet Explorer may behave unexpectedly. If this does not work (maybe the rounded corner does not take effect, or the selected element has disappeared), adding the position and zoom property should solve the problem:

 <style> .border-radius { behavior: url(<?php echo get_template_directory_uri() ;?>/css/pie.htc); position: relative; zoom: 1; } </style> 

Using CSS3 Pie in WordPress

You have finished putting your website in HTML form. You also utilize pie.htc in the website to enable CSS3 in IE8. At this stage, you are ready to transform the website into a functioning WordPress theme. In this case, as WordPress is dynamic, where pages may serve in different level URLs, we have to specify an absolute path.

You can either change the path in CSS like this:

 .border-radius { behavior: url(http://localhost/{website}/wp-content/themes/{the_theme}/css/pie.htc); position: relative; zoom: 1; } 

Or add internal styles in the header.php, this way:

 <style> .border-radius { behavior: url(<?php echo get_template_directory_uri() ;?>/css/pie.htc); position: relative; zoom: 1; } </style> 

Final Thought

Having cool things such as CSS3 Border Radius work in IE8 is fascinating, and CSS3Pie makes that happen, while giving us one less reason to hate IE8.



Post a Comment