Kyle Rush MozCon 2013 - Win Through Optimization and Testing

Key Takeaways

thank you cyrus feels great to be in seattle i just came from new york city is anybody else here from new york yeah you guys all know what i mean when i say it feels great to be in seattle you guys know how to do this summer with this 77 degree weather this dry heat it's awesome we've got to figure out how to get that in new york city can we get on that um so as tyra said my name is kyle rush i'm currently at the new yorker but before that i was at the obama campaign and i worked on a lot of the product and tech aspects of our fundraising our online fundraising um and obviously we ran a lot of optimization on that so that's gonna be talking to you guys today about um but before we get started i wanna give you guys some context um on on you know what we jumped into the situation on day one at the obama campaign all the uh media outlets at the time were reporting that we were expected to raise 1 billion they did uh like probably 700 million in 2008 so we were expected to raise one billion and just to put that into perspective for you guys amazon's q4 profit uh for last year was only like 97 million so when you spread that out over a year and a half which was the life of the campaign you still only get like half or a little over half what we were expected to raise on the campaign so this is a pretty daunting challenge um but in the end oh i didn't mean to click to but in the end we did 1.1 billion so we exceeded expectations none of us thought we could do it you know obviously that's a lot of money and we did 690 million of it online as cyrus said um so uh another thing that i want to talk to you guys about uh is just an example of one of our online fundraising programs and that was called quick donate and this was a way for our users to save their payment information so that they could do one-click donations on the web and they could also do one-click donations and email which had never been done before so we had to do a lot of you know funky engineering to get that to work but you could also sms donate which was the first for political campaigns it was actually a big achievement for us because the federal election commission said that political campaigns cannot use short codes to fundraise so we weren't allowed to work with at t and verizon to send out short codes and ask people to text those so we had to engineer a way around that and when we launched sms donations it was the first of its kind quick donate brought in 115 million dollars over its lifespan and it had 1.5 million users so this this was a like a thrill to work on but obviously this type of program we optimized right we ran a lot of tests uh and so those are kind of the things i'm here to share with you guys um when you might ask like how did we get here right so we're in 500 experiments um we always had a test running it you know it was it was really really intense the amount of traffic that we had we had we did weeks of user testing and user testing is really simple it's just putting a user in front of a computer and observing them we used a program called silverback i don't know if any of you guys are familiar with it but it records the isight camera and the computer screen at the same time so you can actually see your user making a donation and we learned a lot from this but we did it like on and on and on to the point where we probably did weeks of it sorry this thing is pretty sensitive and we also just did general data gathering which i really like to do because if you're not gathering data then you're kind of flying blind right we did just a data point to show you guys how much data gathering we did we did over 668 million google analytics custom events i'll be talking about those in a minute but that's a ton i don't think that i've ever worked at a place that pushed google analytics to the point that we did on the campaign it was it was pretty intense so you might ask you know what did that all get us it got us a 49 increase on our donation page conversion rate and it got us it got us a 161 percent increase in our email signup page these are two really high level conversion goals for us the email signed up you might not have known about um and we didn't really talk about it but let you in a little secret uh email was responsible for just about 90 of our online fundraising so gathering emails on our list was super important um so we spent a lot of time optimizing email acquisitions so the three things that i want to talk to you guys about today and this is really what optimization means to me is experimentation i think we're all mostly familiar with this this is a b testing multivariate testing the second is observation and that's what i was talking about when i was talking about user testing right you want to observe your users using your product otherwise you're not going to know how they're using it because you're not a user and also just general data gathering which is super important so first up is experimentation sorry this thing's super sensitive so the first like we identified a process that when we're on the campaign i want to share that with you i'm sure everybody has their own processes but this is what really worked for us and the first step for us in experimentation was to identify our goals and i mean this from both a micro and a macro level on the macro level i just talked about some of our goals which was email acquisition and donations right you need money to win a campaign and in our instance we needed emails to get that money but i also you know encourage you to focus on micro goals and this is like conversion goals when you're running tests you should just measure everything so micro goals can be like um the error rate on a form right like how many uh errors do you get when somebody mistypes their email address is that like you know is the label clear enough there so you just really want to measure everything one thing that really blew me away on the campaign is that we started measuring the conversion rate on the follow-up page so when you made a donation and it was successful you got taken to a follow-up ask that asked you to save your payment information and that was quick donate that was the opt-in to quick donate and that was a very critical conversion goal for us because we found out early on that quick donate users were four times more likely to make a donation in the future right so that's like money right there that we needed to focus on and we measured that goal even though we weren't changing that page at all we were changing the donation page and we found out that some of the variations that we ran actually affected the follow-up page so it's really really important to measure as many conversion goals as you possibly can when you're doing your experiments just to get a good sense of what's going on the second step that we would do is develop hypotheses and this is really important it's just basically you know like the scientific process that you guys all learned in grade school right develop your hypotheses and then test them this is really helpful in making sure that you're staying focused it's really easy to fall in this trap when you realize how much you can test you just start to test everything you don't want to make any decisions you just want to test it's like oh what color should the submit button be i don't know test it like don't do that that's not a good idea create high level hypotheses like one of ours for example in the campaign was that less copy does better than more copy for conversions so we tested that on our splash page we tested that on our donate page we tested that on our email sign up page we tested it everywhere on the site and we figured out different experiments to test it and that's actually number three here is to create experiments create many experiments to test your hypotheses you might want to test the same experiment more than one time because you might get different results in the time of the day you know maybe there's all kinds of weird things that can happen so test it multiple times and create several experiments to test your hypothesis wow the fourth and i can't stress this enough is to prioritize with roi i touched on this a little bit earlier but as you start building out your experiments i'll iterate this with a example from the campaign we ran an experiment where on our donate page we had a picture of the president behind a donate form and that was our control but then we added an inspirational quote above the president's head said something like stand with me work with me let's finish what we started and when we tested that we got something like a 17 increase in conversions because it made the page just a little bit more inspirational and made people really want to finish and stand with the president and that was awesome that was just adding copy right that only took us like a couple of minutes to get on the page and actually into production when it won so roi on that is really high our finance team wanted us to implement uh like paying by check because they had some data that said a lot of people don't have credit cards and you know maybe they have uh checks that they can pay with sounds like a crazy idea to me but the data that we got from them said that we could expect a three percent increase in the conversion rate but on the technical side that was kind of a big lift that would take days if not weeks to implement and we're only going to expect a three percent lift so when it comes to figuring out what experiments are going to give you the highest roi just really dig into the data and make sure that you're focusing on experiments like you know the inspirational quote and not things like you know changing your whole uh donation system for just a three percent increase in donations uh fifth one is very easy you know test your ideas and then lastly you want to record results and i can't stress this one enough either because you know on the campaign what happened is we ran so many tests 500 total that we couldn't like always remember what the result from one test was and if we didn't have this awesome google doc that we built out that recorded the time the hypothesis the result a screenshot of the control and the variation and the results and a link to the results and optimizely if we didn't have all of that we like you know we really couldn't have functioned because you just can't remember the results of 500 tests so you can also disseminate that information when you have it in a google doc so just make sure that you're recording your results and so now i just want to talk about four areas where you can experiment and i've ordered these by roi and copy is in my experience by far the highest roi that you can experiment with and you know it's very simple because you don't have to change any code or anything changing copy only takes like a minute or two and the results that you can get can be really awesome so here is um the quick donate opt-in page that i was talking about before this is the page where if you make a successful donation we ask you to save your payment information for next time we uh did a variation of the header so this one says save your payment information for next time very simple right and then our con our variation changed the copy and it said now comma save your payment information so we only changed like a few words around um it's not a huge change obviously only took us like a minute to get this test into production and by making the copy more direct and directing the user into what we wanted them to do we got a 21 increase on conversions so again this is very little development effort but a huge result in conversions or conversion lift i should say and so here you can see if you missed it before what the two what the control and the variation was so after copy um the next highest roi um area of experimentation that i would say is imagery because it's very easy to switch images out almost the same as coffee it takes a little bit longer though um so here's an example of what we did on the campaign with imagery this is our splash page for the dinner with brock contest which is a super cool contest you could actually win dinner with barack like we'd fly you out to washington dc and you'd sit down with barack and have dinner and sometimes michelle would be there and actual people won this contest so after you submit um you would you know just enter it into that so here we have a picture of the president we figured out early on that like big smiling pictures of the president work because people love him um but uh we had a hypothesis that people would be more likely to submit this if they could picture themselves in that scenario right like you can't really see the people that he's talking to doesn't really seem like a real contest like could i really have dinner with barack obama so we came up with a variation uh that you know gave the uh the user a view of a little bit more of the situation those are two actual people on the right that won this contest and they flew him out and they had dinner with barack and michelle so when we the result of this you know putting a more situational image in there gave us a 19 lift in the conversion rate and again this does not take a lot of time to implement it's just a very easy test and we got a huge lift on it and so here are the two different images so that you can see them again another area that i want to talk about is performance this is going to be a little bit techy for in technical but you guys are all probably very familiar with how page load affects conversion rate we were too early on in the campaign we knew that amazon had published a statistic and it's a crazy statistic that even 100 milliseconds of additional latency on page load could drop the conversion rate by one one percent so that's like huge right so we're obsessed with performance we want to make our pages as fast as possible so here is a look at the architecture diagram for the platform that we started with it's very simple it's very basic it was built by a company called blue state digital which is one of our vendors and actually came from there before i started at the campaign and it worked really well for us in the beginning because it was built out of the box and you know as the first engineer there i didn't have time to build a new platform this was already out there and working so the user makes requests to a load balancer and that splits requests to two clusters if you're asking for the page it would send you to the web cluster if you actually hit submit on the form it would send you the payment cluster very simple but there were a lot of problems with this in terms of performance we on average saw five to second page load time which is you know horrendous when you're processing uh 690 million dollars worth of donations um you want something more like you know below two seconds or about like zero seconds gonna get the pages just load automatically um it didn't have a cdn i don't know how many of you people are here are familiar with cdn but that's a content delivery network so if i'm in la and i request the page and the other uh in that architecture diagram the servers were in boston so the data has to go all the way from boston to la but if you put it on a cdn we used akamai there's an edge server in la so it gets it to you much quicker there wasn't any caching in this environment so you know there were a lot of things that we needed to change so we basically started from scratch and built a new platform we asked blue state to turn their hosted platform into an api that we could hit on the client side and so here's what that looked like i'm going to run through it really quickly we put our static assets which is our javascript files our images and our css and such on an amazon aws s3 bucket which is a super simple data store it's awesome and then we put the akamai cdn in front of that so we have really fast access to those and then we generated our html the actual pages for these with a static site generator called jekyll which is built in ruby and it's super simple to work with it's great for front end engineers that don't have to worry about like server-side templates and all that stuff and then we hosted all those html files on aws s3 just like our static assets and we put akamai in front of that and the cool part is the two donation processors like i said before blue state built a donation api for us to post to and then they had load balancing on there and they had two nodes behind their endpoint uh we put uh ours on uh ec2 and we put them in two different regions we put one payment processor in california and then one payment or maybe an oregon but it was on the west coast and we put another payment processor in virginia on the east coast and so if you had an ip address that was in the west coast you'd be sent to the west uh sorry in the western side of the united states you'd be sent to the west coast payment processor and the same for the east if the west coast went down for some reason like you know there was actually a hurricane in virginia and actually caused um servers ec2 servers to go down during the campaign all that traffic just got sent to the west coast and it was great it was very redundant and once we got this system in place there was never a down time for accepting donations we're accepting donations 100 of the time um the new platform the biggest metric i think is that it had an 80 percent faster time to paint and that means it's you know how fast the user puts something on the screen not page load right the browser can start rendering the page and the page load metric can still be going on because maybe it's loading some javascript or something that's not critical for page load so i like to focus on time to paint and we got 80 faster here and to show you what that is or what that looks like i use web page test which you guys should all use if you're not using it now it's super easy to get data like this but the top film strip shows you that that's the fast platform in one second we have a you know a painted screen that's a screen that the user can start filling out a donation that's super fast the only thing that's not loaded is the graphic assets right and those load by two seconds but you can see our old platform doesn't even have anything on the screen by four seconds that's awful so we we did like you know a lot to increase the performance here we had a 63 reduction in page weight so we just threw out all that legacy code and wrote our own we went from something like 720 kilobytes to like 120 kilobytes and then we had a 52 reduction in http requests which is you know one of the most uh you know common things that contribute to page latency so what did we get with an eighty percent faster uh uh uh sorry time to paint we got an increase in conversions by 14 to measure that we made a page on the fast platform that was identical to the slow platform and then we a b tested them with optimizely 14 is not as big as the numbers i was talking about before but this was in the beginning when we first launched this platform this was you know the a b test to put it into production and when you calculate the 250 million that this platform brought in over its lifetime that's 32 million dollars right i'll take that the money raised on the campaign was tight so just by making that 80 faster we got 32 million obviously this takes a lot more engineering and time and effort which is why it's less roi than the copy and the imagery but this is huge this is 32 million dollars that we got just by making that faster the second area of optimization that i want to talk about is sorry sorry this is experimentation and user experience which also takes a little bit more time the screen that you're looking at right now is a donate page that is already like super optimized this was later on in the campaign we had ran hundreds of tests on this page and it was performing brilliantly and we ran a lot more experiments on it to try and increase the conversion rate and we kept failing we couldn't get the conversion rate up so we got really frustrated and we couldn't figure out what to do so we decided to try something big and what we did is on the variation we chunked the donation experiments experience into four parts because if you look at this slide right here you see all four or all 16 fields it looks very intimidating to fill out looks like it's going to take you forever but if you look at this one all you have to do is select an amount right that's a much lower barrier for entry on engagement here and then you know you just go through that and it just guides you through very nicely and so we tested this one and i like to call this the gradual incline instead of the steep slope and we got a 5 conversion lift so obviously that's not as big as the numbers before but like i said we had already picked all that low hanging fruit and so five percent at that point was major because we went like you know a month or two where we couldn't get the conversion rate up at all so that's that was a pretty big win for us like i said it was on an already optimized page and so you can see the two forms here one is obviously much simpler to fill out or it looks like it is um so here are some best practices i want to share with you guys the first is start simple you don't have to make this complicated my motto in any engineering scenario at all is start simple and test up um you don't have to make a really fancy user experience you don't have to make it all ajaxy when you launch just get something out there and get it into production because done is better than perfect and then since you're in production so much earlier you can start experimenting and each feature that you roll out you'll know what effect that has on the conversion rate because you can test it the second is always have a test running if you have traffic coming to your site which you probably do right now and you're not running a test that's just wasted potential right there because you're not learning from there from the people that are going to your site so always have a test running and the third is don't be afraid to fail and i can't stress this one enough i can't actually remember the numbers but i want to say something like only 20 of our experiments on the campaign actually raised the conversion rate a lot of them were statistical tie where it resulted in nothing and some of them even decreased the conversion rate and those are like you know pretty damaging psychologically but you can't you can't let that like get you down and i want to show you an example of this um ignore the amount buttons this is a bad screenshot i don't know how this came about but everything was the same except for a little check mark check box down there that says save my payment information on the variation so somebody had the idea to instead of ask the follow-up like screen to save your payment information we wanted to put it on the donate page because i thought maybe that would increase the conversion rate on saving people's payment information well um the slide is a little out of order uh that actually reduced our conversion rate by 44 um so right when we saw that we stopped the experiment immediately and just moved on right like that's the whole thing about testing is it's not permanent you can just move on and you might not even have thought that that would result in that right so i'll go back to this side but you know if you aren't failing then you aren't testing enough because you're not going to have 100 percent success in your tests it's just not possible the second area of optimization i want to talk about really quick is data gathering and you really cannot gather enough data that's really my motto we on the campaign just we gathered any kind of data that we could think of error rates on forms when people focused in the forms how long it took people to submit the form how long it took for our ajax response when the user hit submit to get a response from the server so we can tell the back-end engineers how long it's taking because we want it to be faster obviously anything we could think of we measured it again here's this number we did over 668 million google analytics custom events and here's an example of one this is an interactive infographic that we put out to uh show you know kind of showcase our one million donors it was pretty early on in the campaign it has a little a lot of little pieces of interactive content there where you can scroll to see names and you know one of the most popular names that people donated under where people are from and one part of that is this little piece right here which you can just scroll through and see like the most popular names right we put google analytics custom events on the left arrow and the right arrow and we found that 82 percent of the clicks were to the right arrow so that left arrow is like unnecessary and it's just cluttering the ui and you know gives the user more options and you obviously want to be guiding the user through what you're presenting to them so we use that learning to optimize our fur our uis further down the road and we just didn't put left arrows on anything because it doesn't really make sense and this is the google analytics custom event note to track that data it's super simple and it's arbitrary the category is one million infographic and the label is name slides right super simple the last area that i want to talk about is user testing this is actually a really cool example because it solved a problem that i don't think that we were going to be able to solve without user testing but this is the last step in a donation process and this is where we're asking for your employer and occupation this is required of us by the federal election commission so there's no choice we had to gather this information well when we put the error tracking um on our donate form we found out that the two most common errors behind people entering their credit card information was employer and occupation so we're like wow that's really weird like how can that be such a hard question so we went through and looked at the data that people were submitting it was like none of your business fu like you know like people we just like oh people just aren't comfortable right um and so that was that like there's nothing we can do to make people more comfortable really and so we just left it at that until we started doing user testing we took a lot of the volunteers that came into headquarters and there was a ton of them there were students there was retired people all kinds of age ranges and we sat them down on the computer until we're back and we asked them to make a donation and um uh sorry i'm cheating a little bit and we found out that the students and the retired people did not know what to put in there because they're not employed again this is us thinking as us as the users right we work for the campaign i know where i work i work for obama for america that's a very simple question for me but to a retired person it's like what do i put in there so they don't put anything and then they hit submit and that triggers the error so that's why the error rate was going up so high on these forms so once we got that feedback from user testing and observing our users uh use our product we put a little tiny i don't know if you guys can see it but it's just a little tiny line that says if you're retired please enter retired in both fields a little tiny bit of copy did not take us a long time to put that in there and adding that fuel hint in reduced the error rate by 63 right so that's just crazy and like i said we would not have known to test that beforehand if we weren't using doing user testing and watching our users um i blog about all of this stuff a lot on my personal website it's kylerush.net i go into a lot more in depth on the technical side and a lot more experiments if you want to check it out that's all i have for you guys thank you let's let's step over here into the light awesome work man i assume you're using the enterprise version of google analytics uh is there an enterprise version uh yes i know that we had a direct line over there where we're like hey our stuff's not loading can you please like do something and they would like refresh it because there was too much data but yes there was a lot going on uh i i one question i did want to ask did you do for your uh testing platform did you build that yourself or did you use an off the shelf no we used optimizely you used optimizer which is awesome if you guys aren't using that use optimizel it's amazing yes question there we go hi my name is amanda stevens i'm from marketing agency in winnipeg canada um fantastic presentation my question for you is you talked a little bit about the design elements and the ux uh changes that you made to the website to add that lift just wondering if you can expand on some other design elements that you incorporated to increase conversions yeah sure um uh i know i don't want to be too harsh on design but in my experience what we tested on you know design embellishments and stuff is just kind of a waste of time that's fine if the designers want to put that in there that's great but like i said when you're testing like button colors and like you know rounded corner versus square corners do not waste your time with that that's not going to do anything it's just going to sink it's a time sink right really when it comes to design and our brand was all about imagery and photos that's where we got the real big increases in design changes is imagery but other than that i wouldn't say that we found anything as far as design goes that had a real impact on the conversion rate cool thank you yep yes so hi i'm alan i'm with three ventures technology and agency and i actually watched dan speak at an analytics conference in san francisco one of the things that i think i was going to actually i'd like to ask you about is why optimizely and not google analytics content experiments with the multi-arm banded approach and basically minimizing the time uh increasing a certain conversion rate a 95 percent probability so i mean the amount of time basically that it would take for an a b test to finish at those rates yeah sure so i can talk about this forever but i'm gonna make it really brief if you're an engineer there's really no other option for you because optimizing makes your life so so easy all it is is running javascript on top of your page and so when you can do that you just like add css classes to the page and it changes the design it's so easy it's like we we actually were tasked with finding other a b testing platforms that were either cheaper or i don't know what the situation was but we've evaluated a lot you know i don't want to dump on other platforms because you know everyone has its use but for us on the campaign optimizely was by far the best one of the problems with google analytics is the data's not live right optimizely gives you a live reporting on the results so you can see right away if your experiment is just dragging your conversion rate through the dirt and you can stop the experiment but it also gives you a lot of customization like you can do really advanced targeting so you can target people based on a cookie you can target people based on their region like it just has so like a javascript expression there's nothing that we couldn't do um and optimizely any idea that we came up with we could do an optimizely we tried it in other platforms there was a lot of limitations so from an engineering perspective that's why optimizely is great and that's mainly why we chose to go with it cool awesome thank you yep and what i think we have time for one more we'll go over here okay um so i work in fundraising and most of the time the relationships that we're dealing with you know in terms of how long a person is going to donate is five or ten years um longer if we're talking about direct mail and so it seems like a lot of what you were looking at is immediate return i don't know if you had an ltv where you were saying you know we got a 60 increase in conversions but it affected the ltv or the even just the length of the relationship by x did you look at things like that yeah we did um i would say it's very difficult to measure something like that because it's not like an exact like you know the user's on the page clicking something but if you think about it we've been raising money or not me personally but the campaign since 2007 right so there is a long-term donation cycle there and the campaign is actually still raising money now they have an organization called organizing for action that um you know exist to support the president's agenda his legislative agenda so we're still raising money um i would say that in a political campaign where it's so crazy and you know like there's a deadline that is election day which usually people do not have to deal with it's more about like the short term but they are still doing like long term stuff um we just didn't have to worry about that as much because it was like november 7th that's the day you know okay thank you kyle thank you so much for coming to seattle you