by Tim Jahn on March 1, 2011
Paras Chopras founded Visual Website Optimizer, a site that makes it dead simple for you to A/B test elements of your website. His tool is so effective that his customer list includes Threadless, Foursquare, Groupon, Microsoft, Rackspace, and more.
Paras is the guy to know when it comes to A/B testing, so I invited him here to teach us why A/B testing is important to increasing sales, and some basic techniques to get started with A/B testing.
Tim Jahn: For those that don’t know, what is Website Visual? I’m sorry, what is Visual Website Optimizer?
Paras Chopra: So Visual Website Optimizer is a A/B testing tool. And by A/B testing I mean, looking at different versions of your website. For example, if you had an ecommerce site and you want to increase your sales on the checkout page, you’d be surprised to know that changing even a single button that is used to check out, that can increase your sales by 20% just by changing the color of that button or the texture on that button.
Tim Jahn: Yeah, I was looking at your testimonials page on the site and the top testimonial says, “Their elegant design and drop-dead simple approach designing, launching and monitoring tests is just sexy. I don’t know how else to describe it.” And those are my thoughts exactly from using Visual Website Optimizer.
It’s just, it’s so easy to point and click and change a simple portion of your page and then run the test and see if that works. Where did the idea for this come from? Do you have a background in this idea of testing, or research or how did you come about making this?
Paras Chopra: So it actually goes back to last year, last year, so I always wanted to do a startup and it was a particular dream after college that I wanted to do some sort of startup. And I did a couple of web apps after college. And after that during, I worked for two years but I — as I said I always wanted to do something of my own. And I started researching on my first serious startup, which field should I first start with?
And online marketing is a field that I thought interested me because it had the right elements of technology and web apps and the marketing. So in online marketing, I prototyped lots of different technologies. So online marketing is huge. You have analytics, like Google Analytics and then you have segmentation, you have behavioral targeting, A/B testing. So I read through lots of different technologies and then zero’d down to A/B testing as the first start up that we should then look.
So the companies name is actually Wingify and A/B testing is the first product that we have launched. The idea is to launch such complimentary products in online marketing space that make small to medium businesses make more money online. So the reason we chose A/B testing as the first product is because existing solutions for example, Google Website Optimizer, though it’s a free product, it’s quite hard to use, it requires you to know a lot of HTML and then it requires tying from you goals doing particular small groups networks on page.
So the whole — so we did a couple of A/B tests ourselves just to know how people are doing that, and we found it incredibly hard to use. It just takes a lot of time and effort and the whole — so it ends up being much more expensive even though it’s free. With ours it’s a good niche to launch a new product because A/B testing is something people have been doing for years. And since the brand we were doing is so hard to use, people will appreciate if we make it dramatically easier, really AB test, and they would be willing to pay $50.00 a month for that. So that’s how — it was a very conscious decision. It’s just not, we didn’t do this just by chance.
Tim Jahn: Why is — so you made A/B testing very simple in terms of setting it up and implementing it. You don’t have to know a lot of technical stuff, it’s all visual. But why is A/B testing so important in general? If you have a website and you’re selling products, why should you A/B test?
Paras Chopra: So as I said A/B testing is something that large enterprises have been doing for years. Since you take Amazon for an example. Amazon is really, Amazon and Netflix. So it’s kind of what organizations are really testing driven. They will be testing each and everything on the page. They’ll be testing how their product looks like, they’ll be testing their checkout pages, add to cart, the logo height, logo size and layout should on left down by — and that’s because they know even small changes can make a lot of difference. So small to medium businesses and startups online don’t realize that A/B testing is important because traditionally web design is done by intuition. It’s all about intuition of the designer. He designs something and then you have a website.
People come to your site, they may or may not buy the stuff, and there is nothing you can really change about it. You have a conversion rate, you can be happy or sad about it, but what are you going to do about it, right? So A/B testing is the matter that you can use to tactually do something about it. So you are not just sitting idle with your one person conversion rate on the site. But you are actually testing different parts of your website. So as to, so as to squeeze the maximum juice out of the traffic you are getting. So paying for traffic, right. Every visitor who is coming on your site, he has a value associated with him. He is a potential customer.
But if you site is not converting as well as it should, you are losing a lot of value. So just by keeping, just by increasing the conversions, you can create a lot of value. And A/B testing is the best method to increase conversions. Because as I said, even small changes can result in a, can have large impact on sales and conversions. So to give you a couple of examples, we have about 20 or 25 case studies on our website. So when you read any of those case studies, you’ll find that very very pivotal changes was out there than almost doubling the conversions. It’s hard to believe when I am saying that but people actually experience that on a quite regular basis. That’s why —
Tim Jahn: What kind of tiny changes would double conversions? Are we talking like simply making the logo a few pixels bigger or are we talking big changes?
Paras Chopra: So of course if you are testing large changes, it has much more potential to give you large impact on the bottom line, on the profits, revenues, conversions. So if you can afford to completely design of the page, it’s always better to do an A/B testing when you are changing something significant on the page. But sometimes, even insignificant or trivial looking changes such as changing the backing color.
So backing color change is very famous for this because typically when people — that is the first easiest test that people particularly do on their sites. That they’ll try changing the call to action button color. If it is in green, they’ll try to change it into red. So it sounds insignificant but they see that it typically increases or decreases the margins by a significant manner.
So people do this from to such significant changes for visitors. So those are the kind of changes. Or you can take for example, a headline change. You had some kind of headline on the page. You are changing into a different kind of headline and it increased conversions by 30% or 40%. So it’s just a headline change, you’re not doing any kind of design change, it’s a textural change. But still, it changes your conversion rate. And it’s very interesting just how small changes can also have an impact. Because we are the designer, a lot of gut feeling and induction I’m always right when our design is the best. But you actually need to test that. Your visitor clicks and actions can only validate your hypothesis whether it’s the best or whether it can be proved even further.
Tim Jahn: What if someone is just — let’s say they just started their store, they have their first few products going. Where is a good place to start testing? Is it, is the button color the best place to start or is there a good place to start or is it different for everyone?
Paras Chopra: It’s different for everyone. So A/B testing ties to your website goals. Even before doing A/B testing, there are a couple of requirements. For example, you need to have at least some sort of traffic on the site. Because if you are just starting out and your daily visitors are like five or ten a day, I think that’s not sufficient to do proper A/B testing because you do, because in A/B testing, you need to calculate statistical significance.
So it’s a mathematical concept. It says that the difference in conversion rate is not random. Because if you have small traffic, people may on your original page, people may buy two products and on your variation page, people may buy three products. So you can’t claim that my conversion rate is 10%.
We’re always going to land 15% on the, on the variation and there was a 50% inclement because they weren’t marked a lot of conversions in the first place. So if you are just starting out and you have low traffic, then doing A/B testing is not recommended. But if you have traffic in say 100 visitors or 50 visitors a day, or thousands or millions. So anything above 50 or 100 visitors per day, then it’s okay to do A/B testing.
Okay, to do A/B testing, that’s my opinion. So that is when and the second is before doing A/B testing goals of the website should be clear. So you can’t just do, you can’t just start doing it on AB testing, let me try A/B testing but let me try AB test the menu, let me try and AB test the headline.
So the first thing is to layout the goals of the website whether you want to increase the sales, whether you want to increase the signups, whether you want to increase newsletter or newsletter signups or whether you want more people to watch the video. Once your goal is clear, the next step is to see which elements on the page are actually impacting that goal. So if you want to increase signups perhaps the best thing would be to tweak your signup form a bit, so try removing some form fields, try changing the form fields, try changing required option form fields or try moving around the form.
So having a clear rule is very important. But having said that, there are a few quick fixes that people can try to have broad goals on the site. For example, headlines and call to actions are very important on a page. So if you don’t have any other idea on what to test, perhaps the best idea is to first test your website headline and the call to action button.
Tim Jahn: How long is appropriate to run an A/B test before making a decision on the results? You know so if I change the color of my button from red to blue and start a test, how long should I let that test run before deciding whether or not I should keep the color change?
Paras Chopra: Most tools are including our Visual Website Optimizer. They have some sort of metric to tell them that the results are statistically significant or not. So there’s a metric or chance to read original. And only if bad metric is more than 95%, you should conclude that your results are significant. And then almost all tools would have a test duration calculator. So based on your conversion rate and other metrics you enter, it will tell you that you need to run tests for x number of days.
So there is not a rule for this, that’s why I am not providing any direct answer. It is based on some calculations and there’s a online calculator available there you can enter your conversion rate, the expected improvement and the number of versions in the test. And it will count out the number of days you should wait on on the test.
Tim Jahn: Okay so like Visual Website Optimizer doesn’t alert you after a certain period of time in the test and say, “Okay, we’ve come up with some results. Now is a good time to maybe use this specific variation.” It’s kind of up to you?
Paras Chopra: Yes, yes exactly. So Visual Website Optimizer will both tell you if results have come and you are, we document those results on the site and it will also alert you saying that enough time has been spent on this test and we don’t think you have any good results there. So you don’t have to do a lot of thinking. You’re tool can handle that; all the mathematics behind it. So you can safely save the test and check reports that it’s good to stop the test whether to keep it running or what to do. So you don’t have to do any kind of manual calculations.
Tim Jahn: And I assume that you’ve done some A/B testing with the Visual Website Optimizer site itself, right, in terms of converting customers?
Paras Chopra: Yeah. I do a lot of —
Tim Jahn: What’s you’re — I’m sorry, what?
Paras Chopra: I say we do a lot of A/B testing.
Tim Jahn: Oh you still do. You, it’s on an ongoing basis?
Paras Chopra: Yes, almost pretty much ongoing. So we were to have one, all the other test almost every time running. That’s what we need to do to make it.
Tim Jahn: And what kind of results do you see on average in terms of improvement in conversions to sales? These ongoing tests, I mean, are they providing just one or two percent improvements or are they providing a much greater percentage?
Paras Chopra: It depends on the, on the designer variations, on the level of changes you are making. If you are making small changes that says our headline is worth easiest A/B testing tool — if I change it to the best A/B testing tool your company can afford, I cannot expect huge amount of changes. So I expect a small but significant change, but I won’t expect that my conversion read off will say five percent goes to 15% because it’s a small change. If I want larger changes, I need to do a complete redesign of homepage or any other page that I’m testing.
Because larger the changes you are making, the more chances you have are the significantly increasing the conversion rate. Because of course, if you are doing a small change, the visitor may or may not notice that change or may or may not — it may or may or may not influence his decision to convert. But if you are doing multiple different changes and a large amount of changes that is completely removing some parts or resigning the coloring and completely designing the whole page, then there are much more chances that you’ll see huge shifts in conversion rates. So you’re conversion rate can jump from five percent to 20% and it’s quite possible.
Tim Jahn: And I — I’m sorry, you go ahead.
Paras Chopra: Yeah, I was just saying for usually the way we work is that we do lots of small changes and a big change every now and then. And that’s what we recommend to our customers also because you cannot always be doing large changes. You cannot be so unpredictable to your repeat or new visitors. So we do lots of small changes and a couple of big changes.
Tim Jahn: Yeah, that was actually my question is what’s a good amount of changes? I mean, is there a point where you’re doing too many changes at once, is it good to just start with a button color change and then maybe add on a headline change at some point rather than completely shifting all the elements on the page at a time?
Paras Chopra: To start off, I would recommend changing one thing at a moment. Because if you — so it depends on our goal whether you want to see which elements are impactful on conversions or whether you don’t care about it and you just care about increasing the conversions. So if you want to know whether my button color or whether my button is significant to conversions, you would just test that button.
But if you don’t care which element on the page is actually responsible for the conversion rate, you can do any sort of changes you want. But we’ll start off with — I always recommend to start changing one thing at a moment, not do a lot of changes. Because then attributing success or failure to anybody until the change becomes difficult. If you have changed three things at the moment, how do you know what mattered in the test?
And it also becomes a — also such kind of tests don’t have any kind of lessons for your next test because if you’re changing in multiple changes you can end up likely saying lessons and learning to other tests, right? Because you do not understand what matters on the page as far as conversion is concerned. So one thing to counteract this is something called Multivariated testing. So multivariated testing is sort of extension to A/B testing. In A/B testing you do multiple different changes on a page as you said, and then try to see increase or decrease in conversions. But multivariated change or what happens is that every change on the page gets combined by every other change you make and combinations are made.
I’ll give you an example. For example, if you’re changing headline and the image and the button, right, you may make two variations of each three elements. So Visual Website Optimizer will create eight combinations in total where the variation headline is combined by original made, and originally made and so and so forth. So if we change the XCombine by every other change and lots of different combinations are made. And then Visual Website Optimizer splits traffic to those, to these combinations. And in the reports you can clearly see which element mattered to you conversions and which didn’t.
Tim Jahn: Oh, I see. Oh so multivariate you’re seeing which of the elements is the most important as apposed to just testing variations of one element?
Paras Chopra: So if you want to change multiple different elements on the page and still want to know which elements mattered to conversions, you should probably go with Multivariated testing. But if you don’t care about what things are mattering on the page and you just care about increasing conversions, go with changing multiple different things with A/B testing.
Check out all our interviews with entrepreneurs!
- 37signals - Jason Fried
- A Space Apart - Jason & Gretchen Goodrich
- Airbnb - Nathan Blecharczyk
- AirRun - Rob Matthews
- Alexis Grant
- Allie Osmar Siarto (2010)
- Allie Osmar Siarto (2011)
- Annie Sorensen
- Apply in the Sky - Emily Chiu and Chiara Piccinotti
- Arment Dietrich - Gini Dietrich
- ArtistData (2009) - Brenden Mulligan
- ArtistData (2010) - Brenden Mulligan
- Awayfind - Jared Goralnick
- BabbaCo - Jessica Kim
- BatchBlue Software - Michelle Riggen-Ransom
- Behance - Scott Belsky
- Bignoggins Productions - Jerry Shen
- Bingo Card Creator - Patrick McKenzie
- Bite Size PR - Ryan Evans
- Black Cat Strategy - Beatriz Alemar
- BlogDash - David Spinks
- Bonsai Interactive Marketing - Danny Brown
- Braintree - Bryan Johnson
- Brazen Careerist - Ryan Paugh
- Buffer - Joel Gascoigne
- Buzz Referrals - Jordan Linville
- Carol Roth
- Catalyst Ranch - Eva Niewiadomski
- Centered Chef - Ryan Hutmacher
- Centro - Shawn Riegsecker
- Cheezburger Network - Ben Huh
- Chegg.com - Aayush Phumbhra
- Chris Bennett
- Cimaglia Productions - Matt Cimaglia
- Code Academy - Mike McGee
- Cool People Care - Sam Davidson
- CoSupport - Sarah Hatter
- Crosstown Scenic
- crowdSPRING.com - Ross Kimbarovsky & Mike Samson
- Dabble - Jessica Lybeck
- DealsGoRound.com - Kris Petersen
- Disqus - Daniel Ha
- Doejo - Phil Tadros
- DreamChamps - Jill Felska and Jenn Krenn
- Dwolla - Ben Milne
- Endagon Enterprises - Logan Lenz
- EnGreet - Adam Weinstein
- Fathead Design
- FeeFighters.com - Sean Harper
- Firebelly Design - Dawn Hancock
- Firespring - Jay Wilkinson
- Flowtown - Dan Martell
- Foiled Cupcakes - Mari Luangrath
- Foodie Registry - Ben Reid (2010)
- Foodie Registry - Ben Reid (2012)
- FoodTree - Derek Shanahan
- Foursquare - Dennis Crowley
- Freckle - Amy Hoy
- FusionCharts - Pallav Nadhani
- Giftiki - Bryan Jowers
- Github - Chris Wanstrath
- GiveForward - Desiree Vargas Wrigley
- Goshi - Jack Eisenberg
- Grasshopper - David Hauser
- Groupon - Andrew Mason
- GrubHub.com - Mike Evans & Matt Maloney
- Grubwithus - Eddy Lu (2010)
- Grubwithus - Eddy Lu (2011)
- gtrot - Brittany Laughlin
- Hello There - Shane Mac
- Hudl - John Wirtz
- Ideal Project Group - Andrew Wicklander
- Inkling Markets - Adam Siegel
- Instant Technology - Rona Borre
- ItStartsWith.Us - Nate St. Pierre
- IwearYourShirt.com - Jason Sadler
- IWearYourShirt.com - Jason Sadler (2011)
- Jun Loayza
- Junto - Marcy Capron
- Life After College - Jenny Blake
- Life In Perpetual Beta - Melissa Pierce
- Lifesta.com - Eran Davidov
- Little Independent - Lesley Tweedie
- Lockboxer - Jennifer Morehead
- Mac 'n Cheese Productions - Saya Hillman
- Maternal Instinct - Kat Gordon
- MCC Recycling - Michael Mills
- midVentures LAUNCH - Jonathan Pasky
- MightyNest - Chris Conn
- Mightyvites - Christopher Stump & Stephanie Stump
- Milk Products Media - Todd Tue
- Mindlight Films - David Miller
- Mineful - Jaime Brugueras
- Ms Career Girl - Nicole Crimaldi
- MyZeus - Patrick Algrim and Brandon Weiss
- NextStep Test Prep - John Rood
- NowSpots.com - Brad Flora
- One Day One Job - Willy Franzen
- OneSheet - Brenden Mulligan
- Paul Singh
- PerkSpot - Christopher Hill
- Photogram - Bob Armour & Brian Hand
- PitchEngine - Jason Kintzler
- POP! Social Media - Jill Felska & Jenn Krenn
- Power2Switch - Seyi Fabode
- Proxibid.com - Joe Petsick
- Red Frog Events - Joe Reynolds (2011)
- Red Frog Events - Joe Reynolds & Ryan Kunkel (2010)
- Reddit - Steve Huffman
- Restaurant Intelligence Agency - Ellen Malloy
- RIPT Apparel - TJ Mapes
- Roeder Studios - Laura Roeder
- Scriptito.com - Chad Stansbury
- SeatGeek.com - Russell D'Souza
- Sevans Strategy - Sarah Evans
- Short List - Jason Goodrich
- Shotfarm - Mike Lapchick
- Sittercity - Genevieve Thiers
- SocialKaty - Katy Lynch
- Sole Search - Brandon Williamson (End of 2011)
- Sole Search - Brandon Williamson & D.J. Grant
- Spartz Media - Emerson Spartz
- SpotHero - Mark Lawrence
- Sproutbox - Mike Trotzke
- StorageByTheBox.com - Phil Murphy
- Storenvy - Jon Crawford
- Storymix Media - Ariane Fisher
- StudentOfFortune.com - Sean McCleese
- StyleSeat.com - Melody McCloskey
- SYDCON Web Development - Dave Devitt
- TandemSpring - Tomer Yogev
- Tatango - Derek Johnson
- The Big Playoff - Chris Vankula
- The Founding Moms - Jill Salzman
- The Printed Blog - Josh Karp (2010)
- The Printed Blog - Joshua Karp (2009)
- Threadless - Jake Nickell
- Threadless (2009)
- Total Attorneys - Ed Scanlan
- Visual Website Optimizer - Paras Chopras
- WeaveThePeople.com - Paul Caswell
- Wedobo.com - Amanda Sudimack
- Wendy Piersall
- Where I've Been - Craig Ulliott
- WooThemes - Adii Pienaar
- Wufoo - Kevin Hale
- Young Entrepreneur Council - Scott Gerber
- YouTern - Mark Babbitt
- Zaarly - Bo Fishback
- ZealousGood - Brittany Martin Graunke
- Zipments - Garrick Pohl
- ZocDoc.com - Cyrus Massoumi