Skeptical Survey Interpretation

Key Takeaways

welcome to data skeptic all about surveys our season about survey design interpretation and methodology let your voice be heard at survey.dataseptic.com the working title for this episode has been odds and ends there's a number of things where either they didn't seem to constitute a full episode or I just could not find any published research that was relatively contemporary to bring someone on to be the guest for example I want to teach you guys about something called the unmatched count technique I'm very enamored of this idea but apparently modern journals aren't so enamored of it there isn't a lot out there I could find maybe that's because it's so simple the purpose of the unmatched count technique is to try and measure something in a survey that people might not want to tell you about have you ever cheated on your taxes are you addicted to heroin did you steal from your employer very few people want to self-incriminate so the unmatched count technique is a survey question that is run in an A B test you give half the people one question for example please tell me how many of these things you have done in the last 12 months one taken an international flight two saw a play or musical in the theater three went skiing four purchased a new car so you should have between zero and four as your number then the other fifty percent you ask them those same four questions but then you add in the last 12 months have you illegally purchased any Pharmaceuticals now in both versions of the question I am a one I went skiing I did none of those other activities in the last 12 months and if I had illegally purchased Pharmaceuticals saying I was a two really isn't particularly incriminating so you give the opportunity for people to kind of confess something or anonymously give you some information then comes the important step the subtraction for the first group that saw only four of those options take the average maybe it's 1.7 and if you look at the group that saw the five options including the Pharmaceuticals one let's say it's 1.9 now we subtract 1.9 minus 1.7 is 0.2 that's a measurement of the degree to which that other group or your population as a whole presumably engaged in that one special item that only appeared in one list and I wouldn't claim this is some perfect technique that doesn't have flaws or attacks that can be done against it but it is a very neat and clever way to anonymously measure something people might not want to tell you about although make sure maybe my list was bad I was only a one I think I'd want options where most people would have at least a two or three those were the techniques and things I was hoping I might get a practitioner on to talk about but maybe some other time now let's talk about how to apply our scientific skepticism to the results of surveys someone does a survey it's a measurement and from there they want to make a claim our survey shows that the country is headed towards disaster okay Perhaps it is perhaps it isn't the first thing I want to examine is the survey itself if you can't see the original survey don't trust it it is very easy to Prime people sometimes even accidentally to think a certain way or answer a certain way based on the questions that come before I've heard some of your own frustrations for times I overlooked the way I designed the surveys we run had questions that didn't fully apply or presumed missing answers now the other almost certainly more crucial area where we need to apply our skepticism is to the panel itself who are these respondents where'd you find them how do you know they're real people how do you know they're sincere if you heard our interview last week you would have learned about some of the procedures the Gallup poll does for their Invitation Only panel you were carefully selected if you get into their group and there are other organizations similar to that that have similar procedures yet at the polar opposite end there are panel companies that almost exclusively recruit from make money fast get cash now online type ads and that's why I zero right in on the panel if a survey result doesn't come from some trusted organization my skepticism begins with the panel provider or maybe I should even explain that in contrast to Gallup that we heard about last week where they recruit their panelists and then have them take I guess the one main survey and there's some follow-up auxiliary type things as well that's everything in-house another possibly more common approach is to divide this up into different companies with different areas of expertise the group that helps you develop your survey and interpret it might be totally different from the people who are sourcing the respondents that take the survey and even though I believe in division of labor rarely does that process work to your advantage as the person conducting the survey it actually provides a layer of separation between the people you know and whatever process a third party does to get respondents for you of course you can inquire about that and you'll probably be told about a myriad of different recruitment techniques the ones you're told about will sound really good the ones you might not be told about will be the ads that are placed on various Banner slots all over the Internet saying get cash now quick payouts earn big money taking surveys now In fairness what else are they to do but that's the starting point come make quick money most of those surveys are incentivized by completing a survey that person is guaranteed some small amount of money generally they have to get past some threshold before they get a cash out if someone is paid to take the survey and in fact paid a very small amount their incentive isn't necessarily to give you great results and be thoughtful on their answers if they're trying to maximize profit they want to take as many surveys as they can and when software gets involved and you start putting restrictions on people they can go and make additional accounts so any survey of this nature absolutely needs some checks and tests in place to measure the quality of that underlying panel and this should be a two-way street any provider who says their system is perfect and doesn't want to give you information probably as something to hide so what are the things you want to check for broadly speaking it's disingenuous responses yet that can come in many forms a common and easy one to detect is what's called a speeder all you do is look at the time it took that person to take the survey compare it to the distribution of everyone who took it if the average person takes five minutes and someone else is completing it in 30 seconds that's awfully suspicious someone who is sped through the survey often also does another key indicator of fraud they exhibit a behavior the industry calls straight lining so imagine a bunch of questions stacked up vertically and they go straight down in one line answering a to every question something along those lines now knowing those techniques it would be quite easy to develop some sort of script or bot that could slowly take the survey and pick randomly rather than picking always the first choice all right someone is now doing the minimum amount of work to defraud me what can I check for next there are a variety of test questions you can put in here's an example of an attention check a survey question that reads on the next question please select the final answer which is all of the above on this one select false and then give true and false as options if the respondent doesn't pick false followed by none of the above seems like they weren't paying attention then there are red hearing replies a common one I always seem to get is which of these brands are you familiar with and there are at least one sometimes it's exactly one but at least one that's a made-up brand name and the last type of question and I'm not going to claim credit for this because I can't imagine I invented it but I did come up with this idea on my own I'm sure someone thought of it before me it's what I call the Black Swan question or really set of questions so a major problem you have in the market research industry is that people want that incentive they want you to pay them to take the survey but unfortunately most surveys have some requirement we only want to survey people who are considering buying electric vehicles so the survey begins are you in the market for an electric vehicle if you say yes you get to take the survey if you say no you're kicked out and they don't pay you it's kind of crazy so I learned this lesson when I saw the Ghostbusters in theater many years ago you know if someone asks you if you are a god you say yes so I'm certainly in the market for an EV I also own a yacht I plan to retire this year I'm curious to learn about time shares whatever it is I say yes yes yes enthusiastically medical fraud is some of the worst do you or anyone you know have mesothelioma for a lot of rare conditions the answer is going to be no statistically people will say yes to earn the five dollars and that's going to pollute medical research now of course people do fit in Niche demographics and those niches are often the people you want to survey but one person shouldn't be in too many niches at the same time so you ask a lot of broad separate questions and look for a pattern of someone who's just a little bit too affirmative in everything and those are all things about questions in the survey of course there's tremendous amounts of opportunity to look at metadata like IP addresses whether or not the traffic's coming from a VPN there are clever ways of using JavaScript fingerprinting and all these things in concert should be collected through the process and used to eliminate some of the respondents in my opinion this should be looked at as a healthy and normal part of the procedure if your panel provider says we got to the end and all the responses are great and legitimate and perfect you're going to need to do a lot more of your own analysis well for the last segment let's move on to the fun stuff how to interpret your survey results let's assume you've already filtered out as best you can all of the disingenuous respondents or maybe you didn't have to if you have a good source like an alumni group or Professional Organization I'm not paranoid I'm just concerned about Anonymous responses given over the Internet Often by Bots but presuming you conducted a good survey and you got a good panel in place it's time to do survey analysis and interpretation oh actually one more thing before we get into that the end size how many respondents do we need for this survey this is a common area of gross misunderstanding that I observed often in the market research industry you cannot know in advance what end size you need anyone who tells you that doesn't understand statistics the end size you need depends on several factors the most important one being what you are trying to measure if you want a measure if a coin is a 50 50 Fair coin or not you can arrive at a conclusion with a great deal of confidence pretty quickly what about measuring the percentage of people in your population who are left-handed it's estimated that 10 to 15 percent of people are left-handed but for you to measure that and get a confidence interval that's decent is going to take a lot more sampling than the coin toss example the rarer a group you want to measure the bigger your sample needs to be all right everybody today's podcast was supported by the University of San Francisco and their new Masters in applied economics degree so if you're considering grad school and you're interested in data science let me tell you a little bit more about why an applied economics degree could be the way to go in the new digital economy everything is about the platform as you may know I wasn't an economics major myself but I did take a lot of econ classes and I gained a strong appreciation for econometrics and that benefited me greatly in my career for example my understanding of the Vickery auction helped me to work in search engine marketing since at least at the time that was the primary auction mechanism of the platforms and as the digital economy grows and evolves I've been excited to participate in projects related to tracking of reputation online experience and causal inference and those are just some of the tools businesses are using to make decisions today not to be outdone the University of San Francisco's new Masters in applied economics degree is also going to teach you machine learning using R and python this is a stem designated program which I appreciate very much you can get an application fee waiver by visiting this link usfca.edu skeptic once more from the University of San Francisco in California usfca.edu skeptic are you a software engineer looking to make an impact with one of the world's Premier data and technology companies where you should look into Bloomberg Bloomberg is building the world's most trusted Information Network for financial professionals and right now they're looking for engineers to join them you know for me there's two critical features when I would consider a new role impact first job security second well in terms of job security you can look up Bloomberg and see how long they've been around but more importantly in terms of impact you'll be part of a team that builds and delivers tools to help the world's leading business and financial decision makers surface relevant information in an ever-changing ocean of data so that they can act quickly on it I once had a job where after completing a really big project that would do fraud detection and elimination in our system the CFO blocked the project because he was worried it was going to hurt our cost of goods sold so my solution went on the shelf and the company I worked at continued to defraud its customers I didn't say much longer and the company isn't around today it was a sad outcome when a really cool solution went on the Shelf but if I had worked at Bloomberg that wouldn't have been a problem their software engineers build solutions that are relied on by more than 350 000 Financial professionals all around the world using them to make critical business decisions Bloomberg believes in using the right tool for the job their stack includes C plus plus JavaScript typescript and python all languages I love the company is committed to building a diverse Workforce full of fresh perspectives finding your new career can be hard but if you find the right company it can be rewarding and fun so if you're in the market or just thinking about what your next move might be learn more about this opportunity by visiting bloomberg.com careers that's bloomberg.com careers now another key factor you'll need to consider is what sort of testing or analysis do you want to do on the data and now before I go all Ivory Tower I am all for a random fishing Expedition go conduct the survey ask a bunch of questions look through the data crosstab at 10 000 different ways and see what the data tells you but if it took you 900 cross tabs to find something that was novel and interesting you're probably just torturing your data to death technically speaking you should do something like the bonferoni correction or apply false Discovery rates but even stuff like that might be a bit overpowered for just looking at how your survey results break out across a couple of different variables perhaps you're surveying a professional group and you want to cross tab against democrat or republican or maybe you want to crosstab against state or region if the phenomenon you're measuring is strongly pronounced in the data that's not going to be a problem for example what state you live in and your political alignment are highly correlated not predictive of but I would have no shame in looking at both analyzes and doing so as if they were independent trials slice and dice the data whatever way you want but be very careful in how you present your interpretation to your stakeholders and this is one of those cases where the data is not enough it doesn't always speak for itself let's say there's one particular survey question you want to delve into and you want to cross tab it against something like political alignment of course your question shouldn't be Democrat Republican it should maybe be independent involved maybe you want to list other parties or certainly you should have some other options like I don't wish to disclose or I don't feel strongly either way or actually relate to both parties on different issues let's say there's three or four choices there and then you want to analyze how that choice meshes up against some other choice the answer to a specific question it's quite easy to build the cross tab to get that data into a spreadsheet and see where your respondents fell out now my recommendation number one is you should normalize that data if you ask Democrat Republican independent independent's going to be the smallest group almost surely unless it's you know a group of Independents you're surveying if you are surveying the general population of the United States you should get a lot of Democrats a lot of Republicans and then some smaller groups so when you look across the options in whatever question you're cross tabbing the raw counts are quite misleading to the average person's eye you should do a row wise normalization so that you see the Distribution on a scale of zero to a hundred percent across each of those different groups if the question is what do you eat for breakfast and you have common choices cereal eggs pancakes Etc it would not surprise me to learn that political alignment and breakfast choices are independent there's no correlation here and I find it easiest to see that in a normalized distribution across those groups although the one scary part is you could hide a low magnitude result you know fifty percent of people eat eggs but if there's only four people in that category that's two people so you should be doing some sort of test for statistical significance as with almost every statistical test you kind of have to examine each situation you're in and determine if it's the right choice but in my experience 75 of the time you're going to use the chi-square test so if you think of those distributions over the breakfast food choices the null hypothesis is the one I stated that your political affiliation does not affect what you select for breakfast and the alternative hypothesis is that it does now even that procedure is a little bit tricky you really should have someone good at Stats to go look at it you can make operator errors if you don't understand the chi-square test pretty well there's some minimum thresholds and things like that to consider but at the end of the day even though I want to apply the rigorous statistical stuff I don't want to get overly obsessed with that garbage in garbage out the statistics assume that you have a well-sampled data set that is almost never true in a survey so when there's a question like oh should we or should we not have applied some correction I'm always in favor of running the analysis both ways a standard sort of sensitivity check if there's a methodological disagreement and disagreeing parties agree on the final result who cares about the argument yet on the flip side a person with an agenda who wants the survey to say what they wanted to to say well that's the person you defend against using statistics for example if you're doing any sort of stratification by groups maybe you want to ensure that you have a representative population by age and gender and ethnicity the majority of people who conduct surveys in satisfying that request are going to introduce a selection bias you don't want in your data set think about it this way you open up the survey you want everyone to take it and the people who flood in first are the most common demographic groups for example in my experience disproportionately women take online surveys I don't know why so if you stratify your sample and you want it representative across genders you'll probably get female responses faster than you get male responses and when that group fills up typically you pause it we got 100 female responses pause that group wait for the other quota groups that have other genders to fill up but in so doing you've also just manipulated the other groups that are still open maybe you have age groups younger people take surveys more often so if you close out the female group now you're saying on the age distribution I'm only accepting men into these higher age groups just because it's slower I don't know if I'm characterizing that well in the audio format it makes a lot of sense visually but if you find the point particularly nuanced well maybe the lesson is that the survey collection process is nuanced we apply the statistics where we can but most of all we need critical thinking so whether you're conducting participating in or analyzing a survey think hard about the data you have how you came by it and how people will interpret your analysis of it and I feel like I'm running a little long here so I'm going to kick it till next week to do analysis of my recent survey about languages more of a poll really is four or five questions I want to know about listeners your programming languages your spoken languages and two other quick questions head over to survey.dataseptic.com and please participate in that survey I'm going to share the results here next week between now and then this has been another installment of data skeptic all about surveys thank you [Music]