Lecture 19: Practical Issues in Running Regressions

Key Takeaways

what we are going to do today is uh first finish off a little bit of uh uh testing in the linear model so that's the in particular go over the T Test which you will you will always see in any regression output uh so go over a bunch of practical considerations that you will encounter when running regressions so in particular how do we deal with dummy variables issues of functional forms Etc then putting them all together into one very popular techniques that's based on the linear regression which is the regression regression discontinuity design and then if I have time at the end of today I'll give you one more of this uh project preview from a student which is a regression discontinuity design based on data that's readily available and accessible so that's the plan for today um on Monday we'll uh talk about um onen and instrumental variables Etc uh and then soon we'll have the two guest lecture by send on machine learning then we'll basically be almost done the rest will be uh fun stuff that one can do with data like uh some stuff on data visualization little bit on uh conducting surveys online that kind of stuff so that's the plan for the rest of the semester now uh so let's talk about the T tests it's something that uh um s's barle mentioned when she was talking about the various things we can do in aggression and yet uh you always see T Test in regession so where do they come in uh it must have some use um so the T Test is uh um a test where we don't use the uh as a basis for creating critical value the normal distribution but instead the T distribution uh so what when do we need a t distribution you remember maybe that in small sample if this if the if we knew uh the that the errors have normal distribution then we also know that the betas have a normal distribution but so if we knew their variance then any test based on the beta and the known variant would also have a normal distribution but we don't know it we didn't we need to estimate so uh the error variance is typically uh estimated and so when we substitute when con con constructing a test the true Sigma Square by Sigma hat Square uh the the resulting distribution is not normal anymore it's t uh as the sample become large enough it really doesn't matter the T distribution would become very very similar to the to the normal distribution more generally uh we we not always willing to assume that the error normally distribution we normally distributed we don't need it to justify the OS model we just need that they are uh IID so they don't need to be necessarily normally distribution so uh in that case we can still lose a T Test uh essentially as a way of being a bit more conservative uh and not assuming that the the test is not assuming that or doesn't assume normal distribution of the error so what's happening with the T Test is that the the Tails of the distribtion a little bit fatter so the critical value will be a little bit more conservative than with a normal when than with a normal distribution so how does a t test look like well by now it should have a reasonably familiar uh form if we want to test the hypothesis for example that a particular coefficient is C where C might be zero or any other constant that we know the T Test is simply the estimated beta beta hat minus C divided by the um um s by S the standard of beta hat where the standard aror is given by Sigma Square X Prime x - one uh to the square root of that um so that's going to pick the diagonal element of the variance Coan Matrix that are the relevant X squares this is the I I here uh in practice we don't know Sigma Square so we're going to uh to replace it by by Sigma Square hat uh if it's a a more complicated uh matrice more complicated hypothesis which has potential several restriction as we wrote for the uh a more general form of the of hypothesis testing then it's r r beta equal C so it's R the test then is R beta hat minus C over the um SE of R beta hat and SE of R beta hat is again picking the uh the the diagonal the the right element not necessarily the diagonal element the right element of the Matrix are X Prime x - one R Prime which is going to pick up the relevant element so for example if R is if our hypothesis is that 3 beta 1 + 5 beta 2 equal to C the r is going to be a three uh for the the the beta 1 five beta C and that's going to be we are going to have R here and R Prime here so it's going to pick the relevant linear combination of the X prime xus one inverse so why do we care uh when it's a simple hypothesis simple linear hypothesis uh just testing a particular coefficient uh for example BJ equal C zero being a leading square of that the F test and the T Test are equivalent uh the test statistics are the square root of the S test F test IC so it's the same uh so you can use eer you can construct the F test as we as we show uh as we have done at the end of last lecture uh but it's easier to use the T test for a single estimated coefficient because you have right you have you have it right there in the coefficient for you sometimes it shows up in Little Stars uh and uh so you don't have to worry about that um one thing where you do need a t test because that's not not uh this is not something that you can do in the F test formula is when you is when you're trying to to do a one-sided test so you're not testing Zero versus anything but zero you're start you're trying to test project for example h z is the hypothesis that bet G is greater than zero so it's Greater Than Zero versus smaller than zero in that case you cannot this this doesn't fit nicely in the in the framework of of an F test so you're going to construct a t test for that um so in any regression framework you will find an F test uh that's for the uh that you find in the in in sta it's at the very top in R it's at the very bottom what is that F test what's that F test testing the one you get for free like how does he know which one which one you care about how does AR know like f statistic could be for any kind of hypothesis you might test why why does it decide by default which one does it test Z of what the H is H yeah yeah but any particular one all of them so it's the hypothesis that all of the coefficients are equal to zero uh and then what you're getting uh here is so you're getting that for free the F test uh and another thing you're getting for free is the is the T value for coefficient after coefficient uh so when you have just uh one coefficient then uh if you just have one coefficient in the regression then as I explained before the T Test and F test should really be the same thing so hopefully they give you the same answer uh so you should be able to check that if you take the square of that in fact uh it's going to be 15.3 so this with only one single coefficient the T Test that is plot that is plotted here on the graph is the the square root of the s for the entire regression when you have uh lots of regressors the F test is going to be that all of them are zero and the T Test is going to be coefficient by coefficient testing that this one is zero so this is something that is typically of Interest both of these things are typically of Interest now if you're interested in something else like like two coefficients are equal or two coefficient are equal or opposite side you can you you'll have to you'll have to construct those test either as as test statistics or r t test or are as F statistics in practice it's like you're going to ask your software to do it for you and they are going to do it equivalently with an F test or or T Test so before we move on in this case because the T value is greater than 1.96 we we're comfortable that the regression was not due to chance that the effect was not due to chance well what we know is we can reject the so this coefficient here is 02 or we can reject that it's zero uh with uh so at the 95% level because that coefficient is in fact is greater than 1.96 or whatever the the critical value is for the T statistic which is a little bigger than 1.96 another way another thing we can do from these statistics is we can say well let me construct a confidence interval for our coefficient from here if you want to construct a 95% confidence interval how would you what do you what you're doing from how do you construct the confidence interval from having the coefficient from what we have here how do I construct the confidence interval of GSS data here of the conent interval of of the coefficient it's a function of the effect yes and then also the ccore if you have everything you need here to construct I mean with a I guess I should give you the critical value of let's say with your critical value of 1.96 you can construct a confidence interval you have everything you need now to construct a 95% confidence interval for that coefficient so what would it be yep plus or minus 9 six times exactly so that's that's that's the confidence interval what else do we have that is of Interest so we immediately know that uh we can reject the the we can reject the hypothesis that this is zero uh and then we can in fact uh here we have the the probability that is greater than T we can know at what level we are rejecting the hypothesis that it's equal zero we know that it's quite you know this is a large T stat so this is this is a very small P value and that's reflected by having lots of little stars here if you cannot read this number which of course involves a lot of members you have the little stars here uh what else do we want to see in that discussion that would be interesting so another quick question ke values uh are calculated not but not by fer but by the other test what was the other one name in oh no uh no there's no neon here this is all based on uh I'll go back to the relationship between neon and what we get in OLS regression this is based on the same uh on the same on the same T statistics that we have here which involve uh the square root of the square root of n so it's the it's this is the same there's no neon here it's all same it's based on the same T value which is the coefficient minus um zero in this case divided by Sigma Square hat uh uh and then the the square root of n the number of observation I'll give you the relationship between what we did with randomiz control trial which is Nan how is it different from a from the standard T Test of significance it's very very related obviously okay so some uh practical consideration that you might now you basically have all the Machinery to do all the regression that you want uh so some practical issue that you might run into one is the Demi variables I'm going to talk today about Demi variable on the right hand side I'm not going to talk much or at all about when the dependent variable is a demi variable which has which brings a bunch of other difficulties of interpretation that have to do with the fact that the errors are certainly not normally distributed because it's either one or zero so it's by definition they cannot be normally distributed so I'm not going to talk about when the Y is the Dem variable uh but I'm going to to look at situation when the x is a Dem variable to look at other functional form issues that we might encounter uh on the X or on the Y's and then one example of putting all these things together the variable functional forms Etc which is a very popular design called regression discontinuity design so what's a dummy variable a dumi variable which we also call sometimes an indicator variable is um a variable that is one if uh the observation belongs to group a and zero if it's in group b uh examples we can think about very popular example especially with me is the RCT example where a demi variable could be one if it's if the observation is in the treatment group and zero if in the control group but you could also run regressions that have nothing to do with randomized trials where you have dumy variables for example uh you might be interested in the gender so the dumy could be one if male zero if female uh you could run regressions that involve times so you could be one wondering about uh um financial spread before and after the Great Depression so you would have time series data and one would be after the Great Depression zero uh um uh one before the Great Depression zero after or the opposite uh you could be if you're Sara interested in um manufacturing of drugs and pricing of drugs so one would be uh before the generic substitution Act was passed and maybe the the the price of medicine is higher and zero is after uh you could uh describe uh attributes of uh of goods for example one if the if the house is a deck in the backyard zero otherwise so without any control variable if you run this regression it's very easy to verify once you write down you go back to Sarah's early uh uh notation when we have the biar model it's very easy to verify that beta hat when you run this regression beta hat is numerically like strictly numerically the difference between the mean of the group a and the mean of Group B if uh D is a dami that is one when the observation belongs to group a and zero otherwise so in other words uh writing down uh the your observation you know group a and then you have the average of Y and then Group B I'm just writing a table here and this is YB the difference Y A minus YB is uh beta in this regression okay so that means that uh when we we discussed about condomless control trials we were discussing of taking the means between taking the mean of the treatment and the mean of the control sample average of the treatment and the control group and taking the difference well you can do it by hand like that or you can run a regression uh with one for the treatment group and you're going to have the same answer literally so uh and that's what actually a lot of people do I just uh for reasons that you I actually don't know exactly why a lot of people do that but that's why a lot of what's what a lot of that's why that's what a lot of people do in general one little uh wrinkle here is uh the standard error you're going to get uh for the difference ya a minus YB uh if you're doing the nean regression do you remember the standard the estimate the best estimate of the standard error with the nean but not a regression but do you remember what it was variance of the control the number Control Plus the Vari minus one in BO PES exactly so it's estimated variance divided by n of the size of the control group the same thing with the SI plus the same uh minus the size of the of the divided by the size of the treatment group now if you do an OLS regression what is going to be the estimated variance of beta hat if you do ANS regression if you run this as an OLS regression your estimate of beta a beta hat is going to be numerically the difference in the average between the two group but what's going to be the standard error yes and then by what are we going to divide by the full number treatment plus control exactly so the difference is that the uh the regression when calculating the standard error when when calculating doing any uh t St any standard of the estimate calculation will have uh not uh the sum of the standard error for two groups but the standard error of the difference which is going to have as adjustment of of degrees of freedom the N minus one where n is the sum of the two so the uh if the samples are very large it's not going to make any difference but if the sample are bit small and in particular if they if the samples are very large or if they have exactly the same size in the treatment and the control group it's not going to make any difference uh because you get it both sides anyway if the uh if the if instead the samples are have are different sizes for example you have one control group that has five observation and one control group and one treatment group that has 100 observation the uh s estimates of the standard is not going to penalize that for that you for that at all but the nean would because you would have a very large estimated variance in your control group and a smaller in the treatment group so the the so in small sample they are going to be different the estimated of the standard eror and the Nan are correct this one the OLS would be slightly misleading in this case but it's a small sample problem as long as the sample is large enough and in the special case which is quite frequent where treatment and control group are in any case divided equally between the group it's not going to make any difference so that's the relationship between niman and SS some someone was asking about that uh now do you see a relationship between Fisher and just OLS the Fisher exact test and OLS and and what we would do in OLS I don't think Fisher will punish you well it's very different those are very different framework so there really no way that they really look like resemble each other uh but uh what you could do is you could run uh you could do a Fisher test by running regressions instead of taking the difference in mean so you could use as uh you could uh you could you could run your regression you could first run your regression and estimate the true difference between the treatment and control group then draw a different assignment all the possible assignments if you want or 100 or thousand however many you are patient to do and for each assignment run this regression instead of taking the difference in mean and you will have a distribution of the betas uh uh that you will obtain from from this simulation of running uh every uh every possible assignment that you are looking at in this case it will be identical to doing it with the difference in mean because there is at no point do you calculate an estimated of the standard error you're getting it directly from the sample so in practice if you wanted to do a Fisher test you could do a Fisher test with the by running regressions here um now what happens if you don't have a treatment and control R but for example you have three you have one treatment and two two treatment and one control group or you have 51 states so your variable that you're interested in putting potentially on the on the right hand side here is not a one zero but it has value ranging from 1 to 51 or it could be uh you are are you uh disatisfied with the product moderately satisfied quite satisfied very satisfied in which case you have value ranging from one to four um so usually it doesn't make sense in this case to include that variable directly as a regressor in the in the right hand side if it states it doesn't make any sort of sense because there is no way in which the St the states are ranked from 1 to 51 they might be by alphabetical order or you know that that number that numerical number doesn't make any sense it's just a category um in a randomized trial where you might have several treatments you might order them you might decide that uh if you have one control group one treatment which has a low dose you could call it number one one treatment which had a higher dose you could call it number two there you might be tempted to just put that on the right but that would be odd because it's the coating between one and two is a little bit uh arbitrary for example it's not clear the high dose is even twice the low dose so the coding is arbitrary so the the the beta wouldn't make to too much sense so if you have uh one of these categorical variable that states or different 1 2 3 four Etc what would you uh what you what would you do if you want to deal with these variables what would make sense to do arity so before getting to colinearity what would you do with your variable that goes I've given given you a data set that has uh My outcome uh for eal 1 to e equal 100 uh so my outcome is 12 12 15 9 and then I have my uh treatment category which could be zero two one or three one uh one control three groups what do I do how do I what do what should I do before running regression if that's the data set I've given you very small data set it only has two two guys what what should I yeah I don't I appreciate your politeness don't fight yes so what would be the column I would add exactly tal 0 tal 1 Tal 2 and tal 3 so t equal 0 this is a 1 0 0 0 tal 1 this is a 0 0 1 0 tal 2 this is a 0 1 0 0 and tal 3 this is 0 0 0 and one okay so now I've transformed my my variable which is categor which is discrete but uh where I have different categories into four categorical variable which which are now all dummy variables and now I want to introduce this guy in the regression but what's going to happen if I introduce all of all four of them together then we have our friend who was ahead of us then I introduce multicolinearity because uh they sum up to the constant so if I introduce the constant and the four variable uh that's not going to work so in practice what the software would do is to just drop one but which one I'm going to drop they're just going to decide which one to drop uh different software have different convention but the point is you're not deciding which one to drop so that's a bit that's not what you want so what you're going to do instead is decide which one you want to drop for example you could drop here uh uh T1 equal Z and only include three dummies for t equal 1 tal 2 t equal 3 in that case if I run this regression what what would be the interpretation of beta 1 beta 2 and beta 3 if beta 1 if I'm running a regression of Y um I equal equal Alpha plus beta 1 tal 1 plus beta 2 tal 2 plus beta uh um 3 tal 3 what's the interpretation of each of these numbers yep you exactly so it's a difference between it's again numerically the diff the a the difference in the between the average of the outcome in the in the group that that I have that I'm looking at versus the one that is omitted which is why you want to Omit one yourself and not let your statistical package decide which to Omit otherwise you're going to make strange comp you're going to compare to someone you haven't chosen which might not be the one that is relevant so for example in uh in in the case of one control and several treatment we're typically interested in looking at the treatment relative to the control then of course once we run a regression of this form nothing prevents us to then compare for example T1 versus T2 we can just compare the coefficient and we can test whether T1 and T2 have the same effect by testing h0o that T1 equal T2 or T1 minus T2 equal Z or we can likewise do T1 equal twice T2 or whatever it is that that we think is relevant now if we have other variable in the regression uh we could run something like that where we have um an outcome we're regressing on a constant plus beta times a dummy variable plus XI gamma so I sometimes I I write the I put the co the coefficient on the other end so that the Matrix formulation makes sense uh so XI here is a matrix and Gamma is the coefficient so in that case what's the interpretation of beta in that case the interpretation of beta is that it shifts so suppose X had only one one member uh so uh this is X is just one variable then what beta does is that it shift the slope between uh it shift the intercept so this is group a this is X and this is y and this is Group B uh here beta is actually negative if group a is the dumi so beta is the difference between Group B is has shifted down so that's what if we introduce the the Demi variable and we don't interact with anything it's just an intercept shift uh so uh again going back to RCT often people want to so often the way you see RCT presented is that the first table of the paper shows you some x's that uh we don't think assign assignment we know doesn't affect assignment because assignment is is random randomized but might turn out to have been different for a reason or another you know the samples are not immense so sometimes they might might be some difference left over so you see a differences between uh some variables of Interest which we tend to call covariates and then if we see that there are some stuff that are different typically the way you're going to analyze our CS is show one colum where you estimate this equation without the X so that's the standard difference between treatment and control which is justified by the completely randomized uh design typically people cheat on the standard ER and Report the ol standard ER instead of the standard error but that's fine and then you're going to have a second column where you have some control variable introduced and then the interpretation is that the the the the treatment shifts the the in shifts The Intercept so it increases the increases or decreases the value of the outcome in a common way across every value of x so here uh this is estimated under the assumption that the treatment doesn't change the effect of X the treatment just changes puts you one value up okay so for example if the treatment is aspirine uh and it reduces um you know baby aspirin and baby aspirin might reduce um the chance to have a heart attack one might say well you know uh men and people above a certain age Etc more likely to have an heart attack but the the the effect is going to be lower for everyone so that would justify this uh that would justify by this model so that is again something that you're going to see very frequently and of course it's something that it doesn't you know in non air cities as well you're going to often see for example you're interested in whether uh men or women have different uh wages uh you know conditioning on doing exactly the same on you know controlling for the fact that uh women have different level of Education maybe uh um women major in different fields Maybe uh women have different potential experience maybe because some you know some women may have taken some time out uh of the labor for to take care of kids conditioning on all of these things is it the case that women tend to have lower wages than men you would run exactly this regression which is does the fact that being a woman give me a lower intercept to start with and then I have all these other variables that are going to have their effect that might also turn out to be difference between men and women so that's one way to uh to handle other variabl just not be just uh add them separately in which case D the D variable just shifts yep like a practical question when you have a variable like H yeah in in practice is it more common to sort of group them into certain age brackets and have variables so actually ages so it's a great question I'll go back to that because there are uh it's uh other is more general questions of functional form where in the case of age it's not a discrete variable except if if you are in a very small um if you're looking at a very small age range like 20 to 25 it's too large to be discretized such that uh uh 19 20 21 22 are separate uh but you might or might not think that a linear relationship between Y and AG is appropriate so let me go back to that that's kind of other function form issues and and I'm definitely going to talk about that I me that's a very important practical questions and really in this class I'm really trying to go there is no really nothing really theoretical we are all going kind of trying to understand how we use these tools in practice and how research so just interrupt me and ask me any odd questions and then I'll distribute them toing the class or answering answer them right away so just don't be don't be shy please um now another way that you might think of introducing D variable is uh is not just as an intercept shifter but interacted what we call interacted with the with the uh with the X's so a simple way to look at it to start is assume you have uh you have both variables are dumy variables you have for example a model where you have a treatment that might be randomly assigned and then you have a um um group for example you have male versus female and you might be interested in knowing whether the treatment affects men different than women differently than women in which case what you're looking at is not just an intercept shifter but to know whether the effect treatment has a different effect in one group than another in which case you're going to run a model with what we call interaction which is you're going to multiply one variable by the other so uh here we're running for example uh the outcome let's say so uh Rd is a training program uh for people who are unemployed and we why is probability to be is whether or not the person is reemployed after six months uh and M is a dummy for whether the person is a male so I'm running why the person is reemployed after six months on a constant plus beta they were treated plus gamma they are a male uh plus Delta they are treated it's a one this one this one is a product between the two uh it's really literally the raw product so it's going to be a one if the person is a male and if they are treated and a zero otherwise okay so that's that's what we call an interaction and here it's like by categorical variable so how do we interpret the coefficient um what is uh uh what's what is Alpha Beta gamma and Delta in this model what are the numerically I can equal two in term of differences between groups if there are no other variables in the regression these are categorical variables is the difference in simple means again between and control between treatment and control and not quite so it's a difference between treatment and control among female uh gamma is what treat for male differ control it's a difference between male and female in the control group when treatment is equal to zero uh what is Alpha what is going to what's the sorry uh no Alpha uh not quite so Alpha is the constant here in the agression the enre Tre control group it's the average for the control group for female and uh what is Delta grou no difference between that and yes exactly so let me put it in a let me put it in a in a 2 by two box so this is a very popular design called difference in difference or an interaction variables so you're going to see it again and again and again so it's good to uh remember uh the um you can prove what I'm saying but it's it will also once you look at it looked reasonably intuitive to you uh so suppose this is a treat M and this is zero and this is one and this is uh male no we need female and this is male and I'm going to put the treatment as the first IND this and the gender as the second IND this so this is uh y z z the average for a treatment group uh for a control group among female this is y 01 this is y 1 0 and this is y one one in this regression this is going to be this is equal to Alpha uh this is going to be uh the difference between male and female so the difference between the two between male and female is going to be U so this uh I'm going putting the difference which is this one minus this one so one one i0 minus y0 0 that guy is going to be what did I write it in the regression it's going to be um gamma the difference between male and female uh the difference between treatment and cont Ro in the in the omitted group is the is is y 01 minus y 0 0 is beta and then the difference in difference is uh gamma which is uh y1 let me put it y11 minus y01 so gamma is equal to uh y1 0 - y 0 0 y1 1us y 0 1us y 1 0us y0 0 that's a difference indifference so the reason why this works is that like you can calculate it but the reason why this works is that imagine that the the the we are putting both what we call the main effect which are the dumi for each of the groups and the interaction so what this tells you is that uh the beta is the the the if we didn't have anything else and we run it only in the female group in the female group we would get beta to get the effect for male how would we how do we get the effect for male so beta gives us the effect of the treatment for female how do I get the effect of treatment for me yeah that term so how do I get from this one how do I immediately get the treat the effect for me how do I compute it you can also see it in this table the effect for male is y 1 0 minus y1 that's the effect for male right so you can see it from here this is gamma and this is beta so if I want the effect from I just have to add up exactly gamma plus beta gamma plus beta is the effect for male so what this Delta is is the so beta is the effect for the omitted group m is the difference between uh male and female for the control group and Delta is the additional effect for male compared to female so that the total effect for male is simply beta plus Delta if it's not um it could be a I'll give you another example where it's it's not a treatment and control for any situation where you have two groups that you for example you could have a uh law states that pass a law eventually before and after they pass the law and state that never passed the law before and after they passed the law so then you could say that Del D is going to be dummy for pre versus post m is a state that do it versus don't do it beta is the difference between pre and post for the states that don't never pass the law m is the difference between uh states that pass the law and states that don't before the law is being passed and uh Delta is the extra difference after the law is being passed which is going to be in many cases a reasonably good estimate of the effect of the L Yeah question you've got two gamas on the board one one has to be Delta right uh yeah one should be a Delta this one should be a Delta so the G is yeah the gamma is the different no opposite sorry in my it's my uh my my notation is not the Delta is a different the coefficient of the interaction is a different I'll give you an example soon so we went over all of this and we discussed the uh so yeah we discussed how to get uh the treatment effect for male which is beta plus Delta if I wanted to know the average for male from this coefficient how would I get it if I wanted an estimate just from using this coefficient I wanted the average of the mail the average the average of the outcome for the males I could just calculate it from the sample but uh there is also it can also be found from this regression Del plus beta plus Al correct uh no not the the average for the male so depend the average for the male in the treatment group is going to be Alpha plus beta plus gamma plus Delta the average for the male in the control group is going to be Alpha plus gamma and then the for getting the actual average for the me I would have to to take the average between the two just this portion so suppose that I'm interested in the effect of the mail I I need to put a one for uh for the relevant places so if I'm interested for example in the average effect for the male in the control group uh I I'm setting D equal to zero Alpha is the constant so it applies to the male and then gamma is one so the average effect for the male in the control group is going to be Alpha plus gamma now if I want to know the average effect for the male in the treatment group they have one for being a male and they have one for being treated so the average effect for the male in the treatment group is going to be Alpha plus beta plus gamma plus Delta they have ones everywhere now Alpha is the is the average effect for women in the control group if I want to know the effect for women in the treatment group what is it going to be beta plus beta plus Alpha exactly so I think you're getting the hang of it so that's the basic difference in difference model which is very popular in applied the work because often you have a law that is passed for example in some of the states you're not willing to assume that the state that pass the law are different take the current controversy about transgender bathrooms for example and you want to know the effect on the number of uh uh gay um men and women who decid to take a vacation in those States uh so you don't you're not willing to assume that in the absence of the law the the the same number of gay men and women would visit the states because maybe it's not such a nice place to visit anywhere because there is a reason why those laws got passed uh you might not willing to assume that over time before and after the passage of the law uh um the the trend in visiting to this uh uh would have been the same say because gay men and women are more willing to travel in general because life has become easier in general but you might be willing to make the assumption that in the absence of the law the states might have been following the same Trend so in the absence of the law the difference that we observe in uh which ped last car North Carolina uh the difference in in a state that passes the law and in a state that doesn't pass the law in term of the number of gay men and women who visit for vacation is would have been similar in which case you run a regression that is a a like a difference in difference but D is post uh m is whether you pass or not the low and uh M time D is post times past the law and in that case Delta is going to be interpreted immediately as the effect of the law under the assumption that in the absence of the LW the difference would have remained the same because the only reason why the difference has changed is because the low was best so I want to give you one example of uh of using that method from from some of my work and that's you know going to again give us some practice in different difference so uh I looked at yes I was going to ask a quick question just before we move on couldn't you so in this case you have um two groups uh or you have treatment and no treatment and then you have male and female but you also do diff and diff for just a regular just any old regular regression where you have two groups and a treatment or no treatment yes you could do it in everything we only covered diff and diff today but we've looked at lots of regression before today would a diff and diff be more convincing in in in a in an article or a journal article then just looking at the mean at the means of the treatment the effect aition this is just another regression where the regressors are the two uh main effects uh for the groups and the interaction between the two so uh it will uh it is Justified in some cases for example it is Justified in all sorts of cases it is Justified in some cases where you're interested in the differential effect it it doesn't even have to be a treatment it could be that you're looking at for example uh uh having gone to college or not and male versus female none of that is randomly assigned you can still look at whether uh the wages of someone as a function of whether or not they went to college or not they are male versus female and the interaction of the two in that case the interaction is going to tell you whether are men have a bigger premium when they've gone to college than women or maybe the opposite is true so just the college premium is an interesting question potential difference between men and women uh wages is interesting and then interaction of the two is also interesting it's these are different questions that you might you might ask yourself so that would be one setting where you would look at interactions where you're just looking at differential effect another setting where you set up difference and difference is the one I just described where you set it up because you're interesting specifically you're willing to make the assumption that in the absence of your reform you have two groups one of them that eventually passes the reform and one one that doesn't and you're willing to make the assumption that in the absence of the reform they would have been the same in that case the difference in difference gets very convinced an example that you and I discussed in detail and that's sort of the famous examples of using difference in difference I I I'll go through them uh in words now are the Mariel bliff experiment and the New Jersey Pennsylvania minimum wage experiment so what are what is the m bli experiment at some point in a very short period of time uh um for during very few days Fidel Castro authorized whoever wanted to leave Cuba from the PST Port of Mariel to just go so hundreds of thousands of Cubans suddenly reach Miami in very few days and they settled for the most part in Miami because that's where they had landed so certainly you have many more people who've reached Miami and the question that c is asking is whether that affects the labor market outcomes of people who live in Miami anyway so what he does is that he takes a bunch of cities that are you know reasonably similar to miamis in term of their SP their their size their education makeup their racial makeup Etc and he runs a difference in difference I wanted to give you T but he didn't actually run it this way so I thought he would introduce extra complications but he runs a different uh it runs a difference in different uh specification of the type uh the wages of someone who is a native uh from Miami or one of the other cities that he uses as a control on a dami for whether the city is Miami or not D for whether it's pre Mariel Bo Le or post Mariel boti and then the interaction of the two so if you're willing to assume that in the absence of course there is you know business cycle so things change over time but if you're willing to assume that in the absence of the Mariel Bo Lea Miami and the other comparison City would have moved roughly in sync then the difference in difference coefficient Delta here tells you what is the ex ra wages that people in Miami get post Mar B leaf or the hypothesis I suppose is not is that Delta is zero H zero is Delta is zero against the alternative that Delta is actually negative because all these Cuban you know are fighting for jobs with with the native so the interesting result in this paper is that Delta is zero so they doesn't seem to be although lots and lots and lots of new people arrive potentially competing for wages there is no effect on on the on the wage of of of Native Native Miami people so this this has been enormously influential study because it was one of the first use of this difference in difference idea in a very convincing potentially very convincing setup as a result it has also led to considerable debate on whether it was correct or not and and this debate is interesting because you set up you you see all of the issues with this type of technique for example well is it really true that this these other cities would have been on the same Trend maybe they would have been on another Trend we have no way to know so we have to this is an assumption uh another very famous experiment of difference indifference is also by involv card uh David card who is a professor of lior Economics at Berkley and Alan krugger and they look at minimum wage uh so in they compare uh fast food restaurant workers in New Jersey and Pennsylvania and they're looking at employment after an increase in the minimum wage in one of the states so there again compare New Jersey versus Pennsylvania before and after the minimum wage was passed and see whether uh the U people lose their jobs push I me I was wondering would it be if you're not sure whether two cities would be on the same track would be more convincing to choose one city and then that to average and that's an excellent question and the answer is yes uh and in fact uh uh you can do this there is subsequent work on difference in difference that does this very systematically to find an average of the the idea would be to find an average of other cities which is in fact a weighted average you just you don't take just a simple average you take a weighted average such that your weighted average looks very very very much like Miami so there when you run your difference in difference you will get almost you get the beta is zero because you've constructed a uh a synthetic group of cities like which is quite similar to the original City and uh and then there is an effect of post and then there is the interaction so this is some work that was done later by Alberto abadi who is actually coming here um um to teach here next uh next year to join the faculty it's called synthetic I'll write his name down and I'm sure he's going to teach an excellent applied uh uh applied the econometrics class so you can if you want more of his work Alberto abadi and it's called synthetic uh synthetic control synthetic uh control group method where synthetic comes from the idea that we create a control group out of dispersed another thing you can do and I'm going to show that to you in your example is to uh a testable implication that the the that the cities are on the same trend is that they were on the same trend before so you can at least look at what happened before so here is an example of that uh so this is an an example of the use of difference in difference where I look at um School Construction in Indonesia in the in the late 70s uh there was an oil shock the oil shock turns out to be a Bo for Indonesia because Indonesia o producing country so they had Sly lots of money and they decided to use it in part to con construct schools so in five years they constructed 62,000 schools all over the countries and uh before that they had no money so they were not building any schools so that's one uh we have a pre poost which is pre 73 post 73 and then uh the way they allocated the school is they Tred to allocate the school in places where education levels were low to start with so that creates a difference between regions that got less schools fewer schools and regions that got more schools uh I looked at I used data that was collected in much later so we see people who have completely completed their education and what we are going to be interested in today is the uh the effect of education so there are two factors that affect the intensity of the program in this case the year of birth because if you were seven or younger when the schools were built you can you get to go to them otherwise you you you already so if you are 12 or younger when they were built you get to go to them otherwise you have finished school already uh and then your region of birth which is you get more some places get more schools than others so here is the equivalent of this where so you can focus for for now on the top of this table and what you can see on the top of this table is the regions divided by high program regions and low program regions and then we have uh kids who were young in 1974 so they were young enough to go to the school uh and then a group of kids aged 12 17 in 74 so they were too old to go to the schools so first thing we can do is compute the average years of education that people get in each of these groups and then we can compute the difference and then we can compute the difference in difference and here the the standard error that is below the coefficients are from a regression of uh from the from running the regressions so what you can is that the places people who were born in program that got a lot of schools are generally less educated than people who are born in regions that got few schools does that make sense so yes it does make sense because they were building the schools in places where education levels were low and then we also see that over time the younger people are generally more educated than the older people that also makes sense because the country was growing but what we also see is that this growth is faster in places that got more schools so the difference uh in education level is 47 for uh uh the younger kids versus the older kids in high program region and 36 for younger kids versus older kids in low program regions and the difference in difference is2 uh so we could say well if you assume my so my hypothesis my assumption I'm going to maintain is that in the absence of the program uh the difference would have been the same over time or another way to put it is the difference between the regions would have remained constant over time uh then uh if we assume willing to assume that then this point 12 is the effect of uh of having more schools rather than fewer schools one way to test that is to say well let's look at what was happening before we can compare the we can do a sort of placebo experiment by comparing the uh older kids to kids who are even older none of them benefit from the school so hopefully there is no uh difference in difference in that group and in fact that's what we're finding we're finding now a much much smaller difference in difference 013 so this is reasonably this helps uh uh in in it's not really a test of the Assumption because things could have changed later but at least it's uh it it it is a little bit reassuring so that's one uh one use of difference in difference more generally uh if we if our variables X are not U are not U constant but are things that are discrete are not D variable but things that are discrete uh the the interaction between a demi variable and some variable X tells us the extent to which the Demi variable shift the regression function of that regressor so the what I wrote on the board doesn't make all that makes sense but I light it up for you suppose that I have uh one regressor uh one continuous regressor X and my Demi variable if I run a regression y i equal Alpha plus beta di plus gamma x i plus gamma * D * x i then uh what I'm getting what what this gamma tells us here is by how much the slope of the function is changed by the by the D so we have a should be Delta yes something that's not gamma so we have group uh this is say Group B which is the uh D equal Z and then maybe this is the regression function so this is a relationship between y i and x i this is group a so the intercept the shift difference is given to us by uh by by D and the intercept the shiting the the the the tilting of the slope is give