SPOT The DIFFERENCE. +100% Higher CTR - A/B Thumbnail Test

Key Takeaways

why do you think one of these thumbnails got 23.5% and the other 4610 how about this one 28.7% versus 40% or these 29.7% 32.8% and 37.6 have you guessed it those thumbnails are all exactly the same and those are actual test results from YouTube's new thumbnail testing tool they're calling it test and compare and it's their new Ab split testing thumbnail tool and if you don't know what that is I'll tell you all about it in this video but that is not what this video is about this video isn't telling you what it is this video is telling you why you shouldn't be using it so people who rely on that are going to get flawed results and that scares me as a a YouTube strategist okay no hey I have wanted to like this tool as much as anyone else we have been waiting for it for so long and I know YouTube has put a lot of work into it but we have been testing it for the last seven months and talked to countless other creators and we have concluded that at this point it is practically useless it was just guesswork and it was just placebos and it sounds like maybe it still is in fact it is kind of a joke currently maybe it'll improve and we're going to talk to Nathan about that later in this video but for now I want to explain to you how we know this when we first got access to this tool we were so excited in fact I had just happened to be working on a video about how to make good thumbnails at the time so I was so stoked to have three uniquely different thumbnails to test on it the tests were helping us figure out what was working with thumbnails on YouTube YouTube or so we thought but in the back of our mind we were wondering is it really trustworthy surely it isn't a perfect tool so how much can we trust it and I decided to do a control test to find out which is basically just a way to measure how well it works the way we decided to do it was with an AA test so running the exact same thumbnail two or three times and what we expected to see was something like this either perfect 33 33 33 split or something close 32% 34% 33% that would have made sense of course the tool isn't perfect there's some amount of randomization but if you're uploading the exact same thumbnail they should at least have similar results what we actually found was this so of course we had to run it again and again and again you guys it just kept happening I would wake up every day so excited to test these thumbnails and it was just getting more and more bizarre I feel like I was in some kind of fever dream like YouTube has been working on this for over 12 years they announced it at viid Summit it's finally out we've been waiting so long and it's apparently completely broken because in 2013 or 2012 when I was at the third annual VidCon I was in a room with the engineers and they literally had printouts that they had brought with them and they said hey guys take a look at this what do you think about this what do you think about this feature and I think one of them was like series playlist where you can take a playlist and turn it into an official series of videos and one of the things they showed us on this print out was AB thumbnail testing and they had a screenshot of like here's how it will look we'll have one thumbnail and two thumbnails and it'll have analytics underneath for each one that shows fli through rate and Impressions and watch time for each sort of each column each choice you pick and everyone was like oh that's so great I we could use that today like how soon do we get it do we need to get on a list like when does it roll out in Alpha and they're like oh yeah yeah no soon we're really excited about this it's it's testing well and then you know crickets I I think I actually emailed or followed up you know six or 12 months later and absolutely nothing came of it and then I kind of forgot about it and every now and then someone would mention it on a Reddit post or TW thread I'm like oh yeah that sounds great but they know about it they're working on it I just don't know what ever happened to it or why it took so long but I think your uh sort of um firsthand knowledge and testing that sort of Beed out it doesn't work it might be why maybe the reason they never fully rolled it out is because like they weren't confident it actually had any meaningful uh you know decisions it was just guesswork and it was just placebos and it sounds like maybe it still is but we kept running tests because we wanted to compare what the control tests were doing versus when we actually ran different thumbnails like the tester was intended to do I really want you to understand what is going on here if you ever intend to use this tool on your own channel okay so I'm going to try to explain this really simply and quickly going to get in the data a little bit but it's only going to take 30 seconds just stick with me here so that at the end of it you understand what's going on and how it applies to your channel so we did a bunch of tests where the thumbnails were exactly the same and then we would also about every other video due tests where we had different thumbnails then I compiled all the numbers into a spreadsheet because I wanted to find the spread which is just the difference between the highest watchtime share and the lowest watch time share on each of the thumbnails that were the same so in the control group which is all of the videos we ran using identical thumbnails we had sometimes a spread as high as 22.6% but on average 11.58% then I took all the videos where we ran non-control tests where the thumbnails were actually different from each other and averaged out the spread on those as well you would expect that the difference would be bigger between the highest and the lowest watch time share because the thumbnails are actually different people should have a preference between one and the other if this tool works at all but that's not what we saw when we average those out those tests average to 6.77% on the lowest watch time share and the highest watch time share of each of the tests I hope that was three seconds are you getting this what does that tell you it doesn't tell you that the tool isn't perfect or that you need to use a little bit of your own judgment it tells you that based on these tests it does not work at all it's like if you had a real medication and a fake medication and you gave those to two different groups and the people that received the real medication showed no improvement and the people that received the fake medication showed some improvement what would that tell you it should be the opposite now we did a lot of tests with this and we even had other channels test it too because we want it as big of a sample size as possible and we wanted to test it on channels of all different sizes is some that we're getting way more views and Impressions than we are we kept seeing the same thing but Nathan did a lot of that collaboration and we've been talking about what we're going to do moving forward how we're going to kind of handle this and how we think you should handle this for your own channel so I'm going to get him in here now okay this stuff is crazy I mean I as we've gone through all of this over the past couple months I have just been like honestly devastated that none of this is working um and so you guys probably have the biggest question which is like what do we what are we going to do about this what's the plan here and don't worry we have a plan for you but first the big question is is it even worth using and we're going to totally address that here in just a couple minutes but I want to cover a couple of the other big questions that we've gotten over the last week or so especially as the tool has come out so number one um is the tool going to get better or more accurate uh well we certainly hope so um YouTube there's so much history here but at one point we submitted all of these concerns to the YouTube engineers and the response that we got was well yeah we really just need to get the tool out we're going to we're going to take care of it later okay a little bit problematic in concerning in my opinion but okay typically like YouTube will just throw stuff out there and we' we've seen this with like stories remember when they did stories that shorts is another great example shorts throw it out there and there's like do shorts everybody and so everybody does shorts and they end up killing their Channel like a lot of people people destroyed their Channel because they went all in on shorts and then they built a Short's audience and they built a Long's audience and neither of them crossed over and and so YouTube puts out this this testing tool and they're like hey throw up three thumbnails and we're going to tell you which one does better and now like according to your guys's data you're seeing that it's like it's broken like this is legitimately broken so people who rely on that are going to get flawed results and that scares me as a a YouTube strategist like why would I ever want anybody to to use a tool that's going to give them bad data uh so I agree with you like use it with caution uh because I don't think YouTube thought this one through just like stories and just like shorts you know there you go history is repeating itself that's my opinion that's my thought and then when the Tool did actually come out uh they have talked about adding new features and improvements so all of that said I think that the tool should get better in a perfect world because they are talking about adding new features it is a little frustrating to me though that they want to add all these new features but the tool doesn't even work I don't know how you feel about that but makes me quite frustrated yeah it's kind of ridiculous I don't know if they think people just won't notice I mean surely surely they have done control tests we have all these tougher engineers and data people like they've thought to do this they know what's going on here and you know no I think they do plan to improve it and maybe they need more users to be able to do that but you can actually help here so you have the option to submit feedback when you use the thumbnail tester so if you run control tests with us and I would really encourage you to do that you can submit your own feedback and say hey this is what's going on okay so what to do in the meantime so there are really two ways that you can use the thumbnail tester at this point number one is exactly what Julia just said it's use the ab tester for control tests on your channel that's really the only way that I would use it at this point running a regular AB test with two separate thumbnails it's probably not worth it but if you run these control tests you're actually going to be providing some value number one you're going to get some sort of Baseline on your channel to know exactly how far off the tool is calculating for your exact channel the second thing is as you run these control tests start talking about it like anytime you see a YouTube educator talk about it anytime you see anybody else on YouTube talk about the ab tester you can take the results from your own channel and start saying hey I've been running these tests and it's not not working I think it's really important that we get the word out about this I hate the idea of creators like yourselves wasting time on something that's not actually going to improve your channel there's a lot of people who don't realize that the thumbnail tester is not working at this point yeah the way it's presented is just so matter of fact especially when it says the winner label and like hidden on YouTube's help page is all this stuff about how if it doesn't have the winner label it's not statistically significant but from what we found even if it does have the winner label it is not statistically significant but by any means and my big fear here is that people are just not going to get this you know even if maybe they have the winner LEL maybe they don't but they're still like you see that big difference in the percentages and you just want to believe it it just feels natural to be like yeah that's someone must be better we were doing it at first and even when we ran these tests we're like how is this happening and I'm afraid it's just not going to sink in with people but obviously obviously we're trying to do something about that but help us out please I hope all the data that Julia shared with you guys actually made a difference in like your understanding of what's going on with the tool right now because I know when you started running these tests at first I was like ah it's probably going to be fine it'll probably work out but then you ran test after test after test and I was just like oh no this is not good people have been waiting for this for so long anyway that's the first way that you can use the thumbnail tester if you're kind of nerdy or you're into the data you like doing stuff like that run these control tests he even comment on our Channel and say hey I ran this test here were the numbers that I got we would love to see that and the second way you can use this tool at this point is to honestly just not use it at all and I know that that's probably frustrating but the reality is it's not working and so it's going to give you false and bad data that's not going to help you succeed at all yes so if you're running your control Tests YouTube actually kind of told us not to do this when it was Rel hilarious to me yeah absolutely ridiculous okay so YouTube has this channel called Creator Insider where they make little videos on all their releases and in that video where they are explaining the launch of this tool they say make sure you run T thumbnails with very big different and they give suggestions of like lots of text versus no text or different background significantly different changes now sure if the tool worked and people were just trying to make this tiny little change but then they're only seeing like one or two% differences then yeah make big changes so that you're not getting like a 33 33 spit y but they shouldn't be telling us not to run control tests when it doesn't work because honestly we're getting this huge split even when they're exactly the same so you need to understand what the margin of error is and the only way to do that is to start with control tests to see how far off the top and bottom ones are before you start to run wildly different thumbnails yeah this is something that is a pretty significant problem like Julia said and like we've said before we're going to continue testing this we're going to be running our own control test our additional AB test to see if the tool improves and we'll be letting you know if and when it does uh we're also working with other creators who are running some of these tests it's funny like Julia mentioned some of these creators they ran these tests with us earlier on they were it was just almost comical how bad it was and like these are these are creators with like over million subscribers on their Channel and so hopefully it will get better I know I just we were on this video call with a Creator and we you know we were just talking about the test that we were running and we were just laughing about how almost comical and terrible it is at the same time because YouTube has been working on this for so long and the results are still so terrible so go run your control test let us know how it goes let other creators know about this so that they aren't wasting their time doing something that's not actually going to help them it's sad but it's also kind of fun to be part of figuring out how reliable this tool is so we would love for you to help us okay so go run those control tests for us and if you want to know more about this tool if you have it how it works all those little nitty-gritty details we have a video about that right here go check it out hey guys future Nathan here I just had to tell you this really quick we've been doing some additional testing and working with additional creators and there are some really promising results so far for improvement with the ab tester but we need to do a ton more testing and we're going to keep you guys in the loop as we do that but any additional testing that you can do those control tests especially send us that data so that we can get as much data as possible and know if the tools actually going to work or not thanks for watching