#10 RESEARCH DESIGN | PART 1

Key Takeaways

[Music] hello and a warm welcome I am armed interest professor for organizational behavior at the foot baingan University in Germany and this is my course on social research methods so welcome back everybody in this episode we're going to talk about the research design and that's a completely different thing compared to what we have done in the last episodes in the last episode we were talking about how you collect data using a questionnaire of doing observation and doing content analysis all alike the research design is much more about you can put it like it's about the architecture of the study so that sounds a little bit weird now let me maybe have a very simple example that I will use throughout this entire episode and we can use this example that I have just pulled from the air spontaneously we can use this to to make things clear I I pulled a very simple example and let's assume there is the hypothesis that money effects happiness okay that's simple we have maybe the assumption that the more money you have the happier you are right okay now you can have a long philosophical discussion no no money does not make happy yeah there are thousands of reasons why you believe that money does not make happy and there are millions of very wealthy people who are not happy but you know there are a lot of very poor people who are not happy I ever he's it's not so easy so stop the debate we do a study ok we do a simple study and now the question is how do you do this and so to get back to our original idea of how to define hypothesis now we see two variables right we see money and we can also say wealth and and the other is happiness money I stick to the term money even though it does not sound too scientific money is the independent variable it's considered as the cause and Happiness is the dependent variable that's the effect money effects happiness independent variable dependent variable you got this straight okay remember this please independent variable dependent variable so when we do this study we we have to consider both right I mean that makes sense we cannot do a study about money affecting happiness without considering one or both right so we have to consider both but but how do we consider both how do we do this and my main focus now lies with the independent variable with the money okay because here we now can do a lot of different things I've been doing a study happiness let's make that short we let's assume we simply measure this by using a questionnaire okay we could also do content analysis asking people writing an essay and then we'll look how happiness how happy that sounds maybe we can even have we can make images of people and then we analyze the content of the images maybe we can do this but we don't do this we can have a test that is something that we do a happiness test that would be nice yeah happiness test no we use a very simple technique we ask the people on a scale from 0 to 10 how happy are you and we do this as part of a questionnaire in a real study you would do that differently okay but for the sake of simplicity let's assume we simply do it that way ok so now that's the dependent variable that's that's all fine the interesting thing is what do we do with the independent vary with money and now the first thing is we can do two things now right two things now that's very important we can eyes either just measure the money you have in your pocket all the subjects have in their pocket EDS it is currently right so you could just go out and say okay I want to I want to ask people how much money do you earn in a month how much money do you have readily available so it's about the liquidity and then we ask the question how happy are you so so we measure as things are and then we simply analyze is there a correlation maybe something like this okay this is something that we could do that's simple so in that case we do not we do not manipulate the money we simply measure it we compare different groups so to speak so we ask the people how much money do you have in your pocket right and and now you you you compare different groups let's say you have the poor ones the medium for the medium people and the rich people it based on their response we compare these different groups right so and when we compare these different groups we now can look at the happiness level and then we would assume that the poor ones have a different level of happiness than the rich ones so and this is what we named a cross-sectional design we simply compare different groups that's it right we do not manipulate the money they have we just look at how many money do they have and then we compare the groups we compare different groups right in that case we compare different social classes that's what we actually are doing great okay but maybe you do not compare different groups you just compare one group right so you could do this for instance with students so you you you you look at you look at how how things change over time so you have graduates let's say that you have graduates students that graduate and then you measure and you assume that they get richer over time so that's the idea and and then we measure happiness once they graduate and then we measure again after let's say two years and then we measure again after four years and then we measure again maybe do this over let's say a decade or two right so we compare Meachem we look at one single group over a period of time and let's say we only look at those who are employed right that's the only thing that we do so and and then we look at okay how does happiness change over time okay now you might ask but what is it money okay we can control this but whenever you compare one group over the time one the same group over time then we name this a longitudinal study longitudinal study how things change over time so this idea with money and happiness does not work so well with that one I know here the question might be more how does happiness change over time right and and you look at one in the same group and look how things change over time you measure again and again and again okay so this is this is one side right cross-sectional longitudinal and the common thing here is that you do not really manipulate your independent variable unit rate now what does that mean money pull it manipulate would mean the following and that's practically very difficult with money maybe okay but maybe we can do this that's just imagine it for a moment they say okay we have 100 subjects 100 so people who it would take part in our study and now we do the following we split this 100 people into let's say four different groups each group is 25 people okay so the first group they will receive such kind of money for living the other more and the third one even more and the fourth group get the most okay so you have four groups and four different groups get different levels of money so now you see the independent variable money now it's not just measured how it is how it appears in the field in reality we manipulate this one right actively okay you get so much money you get so much money you get so much money and you get so much money so we actively change that one okay now that sounds a little bit experiment ish okay okay now now the question now is when you do this when you many manipulate the independent variable the first question is is there a control group a control group where you do not do something at all in that regard it's little bit difficult but you can imagine this with let's say a pharmaceutical experiment so one crew you want to understand that's that substance really heal a certain illness right so one group gets the substance and the other group does not get the substance I get a placebo so that's the control group so it's one group you do the so-called experimental treatment and with the other group you don't so do you have something like this yes or no yeah okay is that it's the assignment of the subjects really Rand is it really that through flipping a coin or whatever throwing a cube or whatever do you assign the different subject to the different conditions randomly okay is that the case yes okay fine okay last question is there a double pliant method double-blind means that even the experimenter does not know which subject is in which it's not they're experimental but the real experiment that the real person that operates the experiment does not really know in which condition that different subjects are this is what we named double-blind if all these three things apply then you have an experiment if just one criteria does not apply then it's only a classic experiment a capacity experiment that's a very important term it's and it's a it's a study which is a kind of experiment you manipulate the independent variable but it's not actually a real experiment okay so in practice very often we have we have only quasi experiments so if that was too fast I will walk through this different criteria in a minute but I just want to show you that there are some questions that you need to get straight to answer to find out what kind of research design do you actually apply a new study and you you really should know this so you should not only know that there are different and you should only know that these different designs vary in specific kinds you also should be able to name your own research design in whatever study whatever study you are doing so now let's look at this experiment thing a little bit okay and I would like to go into this a little bit more deeper this was just the kind of starter very summary in the beginning okay so when we talk about experiments or quasi experiments there is a kind of way how we articulate the design and what we always have is observation we observe the dependent variable this is what we very often do be in our example we observe happiness right we observe the happiness of people and we observe the happiness while some changing the independent variable through some experimental treatment this is how we name it okay so in our case we we we assign different levels of money to different people we do it actively that's the experimental treatment so generally spoken let's now go a little bit away from our money happiness example you always try to express your design with simple oh and X and O stands for observation and X stands for experimental treatment and now you will find a lot of different designs here and there you know there are there's a universe of different designs but I just want to pick out the the most important ones okay so one very important design is the so called pretest post-test design one group only okay again pretest post-test design what is that you have an observation of the dependent variable then you do something and then you observe again you dependent variable so for this case we again can really apply we can apply our our example of money in happiness so we have a group of people let's say 20 people we measure their happiness okay then we have the happiness let's say average 64 at 6 6.4 okay and then we say hey here's money that's our experimental treatment you get money yeah Pass yeah here's the money and then be after two weeks we measure again yeah and then my memory might have happiness level of 7.10 did decrease nice money helps it's assuming so we measure one-time happiness money experimental treatment then we measure again okay pretest post-test in between we do something okay this is something that we very often find in practice really and I would like to go a little bit deeper into this so here is an example you I once mentioned what a what flight risk is no turnover intention I think that was one in a very early episode now let's let's pull back this topic again companies very often see a high turnover rate what is a turnover rate a turnover rate is the is the proportion of people in an organization that over the course of the year voluntarily leave the organization this is what we named turnover rate so so if you have a turnover rate of let's say 10% then 10 out of 100 employees leave the organization over the course of the year voluntarily this is your turn over rate and very often companies want rather have a lower turnover rate not zero but rather low because whenever somebody's leaving the company we have to replace this person you have to develop this person have to train a new employee and it's sometimes a whole mess so now let's assume a turnover rate is pretty high in an organization let's say 15 ok so so the companies oh we have a problem high turnover rate 15% - and they see that it's much higher than the industry average and now the HR director has a cool idea say why not giving money to the people right or doing some other intervention I mean you can't do a lot of things to retain people but let's go the easy way and say why not paying more pay the people more and then they will stay ok so now you do this you say guys salary increase salary increase and now one year later you measure the turnover rate again and now you see it's 12 you see pretest 15 post-test 12 now you see oh there is a decrease there's a decrease really okay and as an each other I can say you see it works money works really can you can you know you can't you can't why not because maybe and we will talk about this so called threats to internal validity in a few minutes after we have looked at some experimental designs maybe it was not the money maybe it was because there was a global recession in the meantime maybe there is something else right maybe there's something else maybe this 15 percent was so extremely high that I mean whenever you have a very extreme result things drop just through probability yeah this is what we named statistical regression maybe there is something else there was this great Hawthorne study those who deal with organizational behavior they they they know about this study Hawthorne study classic experiment it's part of scientific management the idea was how is it when you change the light in a factory does that affect the performance of people so they did that so they looked at the performance of people at one time then they changed the light and then they looked again at the performance of the people and they found that when they pull off the light and make it a little bit doggone performance increased and they were completely completely confused by that okay now that's that to the test that's increase in another shift so to speak yet that's let's make it lighter so so they measured the performance and they they lightened up the factory and then performance again increased so can you mention it you make it darker performance increase you make it lighter performance increase what's going on there yeah the simple thing was that the people felt that they are observed it was not the light it was the simple fact that there were researchers running around measuring performance and whenever you measure performance and then people know that they are tracked that they are measured that they they are part in a study open observation they changed their behavior and this is what they did so it was not the light it was something else so this pretest post-test design is a very poor design really and the same goes a little bit when you but it's a little bit better when you have a repeated treatment design this is something that you observe the dependent variable then you do something you observe the dependent variable again then you do something and then you observe the dependent variable again so you you have a constant treatment over over again you might do this with you you can also do this with retention measure so you always try to do something track do something track so that's better that's better but still you do not know whether over the course of this entire study maybe there is something else that that is changing right in that case you have really just one group you will compare one group over a period of time and and every time you do something afterwards you measure their repeated treatment design but you could also handle that differently maybe you have different groups that's also something that you are very often having practice you have three different groups Group one group two group three so and in companies for instance you sometimes simply have different shifts so you have two shifts or three shifts that the morning shift the afternoon shift and the night shift yeah and now let's say okay in one shift we do this in the other shift we do this and the other shift we do that the light give them different money do different kind of methods of work you want to check whether a specific leadership style has some effect on performance so in the morning shift you do this leadership style in the afternoon shift you do this leadership style in the night if you do this leadership style yeah you just do different things that's your experimental treatment yeah the independent variable yeah and then you look okay after a while how things are the dependent variable in morning shift afternoon shift night shift and then you might see a difference okay nice but when you have this post-test only design if you only have a post-test only design you might not know how were things before you done your treatment and you might not even know whether these different groups are were different by nature I mean you might have a night shift and okay who is in the night shift maybe that's a voluntary thing and only those are a night shift who are young maybe who don't have families or whatever maybe there are specific types of personality that prefer a night shift so it could be that in this different group there are already its we call it selection effects different types of people you see this so so that can go on and on and on and on and and and the true experimental design that that that is different you have a observation in the beginning and you have an observation afterwards so in the first end it just looks like a pretest post-test design but you not only have one group you have two groups and and that now that already sounds like a real experiment so what I want to share with you here is very you hear students saying or amateurs saying there was an experiment conducted or some would say I'm doing an experiment I can tell you very often you don't you do don't because what is a real experiment let's let's look at this and now I would like to spend some minutes on this question what is real experiment a real experiment is something that I've just have indicated to explain it indicated something where you have two groups you have a measurement before the experiment treatment treatment and after the experimental treatment and a good example really is something that I already have mentioned when you want to do test the effect of a pharmaceutical substance okay so let's say this is a pharmaceutical substance that that is good for for I don't know a chronicle headache yeah something like this you could could take everything right so this is a substance and now you manipulate I mean that's clear the independent variable you do this by by having one people some people who get the substance and some others don't so that's the independent variable the independent variable is whether whether or not you get this substance right this is what you do so and everything else is held constant so the difference between these two groups is only the fact that the ones get the substance and the others don't that's why we use a blood sable because it would not be okay if you just the one group the experimental group gets a pill and the other group gets nothing why is that not okay because then the there are two things between the groups there are two differences between the group one group gets a pill and the substance and the other group does not get a pill and no substance and this getting a pill can already have an effect okay and we know this this is the this is the placebo effect okay and you always have this placebo effect when people believe oh I got something that really helps they say oh yeah my headache went down a bit okay oh yeah you probably believed in that and those those who are in the in the experimental group who get the pill and the substance they have both they have the placebo effect because they believe in you plus the real effect okay it's it's a very interesting thing okay so you have these two groups experimental group and the control group and the subjects in that regard patients are assigned to these two groups randomly by just flipping a coin that's absolutely important random assignment of subjects I mean that's it's also sometimes very hard to be done in practice as I just have shown you with the different shifts also I mean sometimes we do we do studies here in the university you know a psychologist most studies in psychology were done with either rats or students rats or stiffness and for practical reasons some might say okay let's have an experiment and you in one condition we take the business students and in the other condition we take the technology students bad idea bad idea because once you get differences between the two different conditions you will never know is the difference because of of the experimental treatment the pill that whatever you do or is it because business students by nature are different than technology students students in tech right and then you cannot when you cannot ignore such an possible effect then your experiment was not perfect okay okay so here's the final thing that is quite funny it's about the double-blind so when we go back to our idea of the pharmaceutical study there is this person this experimental yeah who who provides the pill or the the placebo to real people the patient so it's a situation where you say okay miss my er er here is your pill in this moment the experimenter should not know whether the pill he gives is the placebo or the real substance the the subject should not know anyway right that would make no say that would be a pliant designed but the blind design but when the subject and the experiment both do not know whether now this treatment happens as the in the control group or in the experimental group then it's a double point and this is what you want to have because what we know and this is the so-called whoreson tile effect goes back to a social psychologist called horizontal who found that people can in a very subliminal way affect the behavior of other in a very unconscious way sometimes yeah so in the case of the experiment it could be that the experimenter gives the pill to the patient maybe knowing that now this is the substance and the experimental ones that the substance really shows in effect and he/she might give this substance so Miss miles here is the substance and to the head maybe not so extreme but for the for the placebo he keeps us up all that so miss Mayer yes pills might not be so extreme but you know now the experimenter should not know that's double-blind also the observers in an experiment should not know this double blind all those who who are in touch with the subt substance subjects all those who interact might interact with a subject should better not know in which condition they operate right in order to keep everything constant right that's very important we know this from from some experiments of Rosenthal said when experimenters have certain expectations you will get what they expect right I mean we know this for instance from we also named this the Pygmalion effect we know this from from studies done in schools when you tell teachers look teachers this time we have assigned the different students to different classes based on their intelligence so this time we did not randomly assign the students to the different classes we assign the student based on the intelligence here's the in group a are the dumped students and crew B these are the smart students okay now go ahead if the teachers believe this what will you get in the end indeed those who are told to be smart will receive the better grades while those who are perceived as this dump will receive birth grades which makes no sense it's just because the students the teacher have different expectations towards other students they will treat them differently they will look at their performance differently they will judge their performance differently and so the crates that's it's very you have to be very very careful here okay so so I would like to leave it to that in the moment what you take home for the moment is that there are different experimental designs wait and there are different research designs I have to be clear and the first question you ask yourself when classifying your own design is whether you manipulate your independent variable or not and then the rest for if you manipulate the independent variable then the question is only is it a real experiment or is it and if it's not a real experiment but you still manipulate your independent variable then it's what we named a quasi experiment okay and what you take home from this episode is that different experiments lead to different problems and there is only one perfect way which is the real experiment which in practice is very hard to be done so when you don't have a real experiment you always have problems you can never say that the really the independent variable affected the dependent variable you never can say so okay and that refers to an idea or to a concept that we name the internal validity and internal validity describes to what extent you really can say that any change in the dependent variable goes back to a change of the independent variable okay so I would like to have a separate episode in the next one where we talk only about this I really want that you understand this it's it's it's also it's it's it's very important that you think these things through it also takes a little bit away you or your naivety I would say that's it's very helpful okay so that's for the moment and we see us in the next episode [Music]