Lecture 22: Experimental Design

Key Takeaways

uh jumping in uh where we left off so we basically discussed the Val estimate which is a ratio of the diff the difference in mean uh for the outcome for the group that has a that is as a one for the instrument minus a group that has zero divided by the same thing for the intervening variable so the ratio of the first the reduced form divided by the first stage now we could have done that in a regression framework uh if you remember when we run a regression when we run a a variable on a on a demi variable simply the we know that that the pi 1 here is going to be numerically the difference in means anyway and same thing for the gamma one so we could have run these two regressions taken Pi 1 and Gamma One taken the ratio and it would have been the Val estimates uh another way to we could have done that and I'm sort of building up to what if for example the the first stage is not one zero then it's not going to be easy to do the Val the not the first St the z i instrument is not one zero then we can't easily do the Val so we have to build up to that so what we could have done in this case is to say well run the first stage take the predicted value for Alpha for AI and then run regression of Yi on this predicted value and again uh the point estimate of beta here would have been the Val estimate so let me prove that to you um the the point estimate of beta would have been the ratio of the co variance between Y and a hat divided by the variance of a hat that's by definition of what the OLS is I'm replacing a hat by its value a hat is simply Pi 0 plus pi 1 Z we divide by pi 0 plus pi 1 Z then I can take by by the properties of uh expect variance and Co variance I can take some terms out uh the Co variance of Y and Pi 0 is of course zero because it's U uh Pi is Pi Z is a constant uh the covariance of Pi Yi and Pi 1 zi is pi 1 times the coant of Yi zi and then here uh we have just Pi 1 square variance of Z this is uh this term here is simply the the OLS coefficient of z z in the Yi regression so that's gamma one and we left with an extra Pi Pi 1 that we use here so this is the ratio Gamma 1 over Pi 1 which is the ratio of the OLS estimates which is the Val estimates okay so basically instead of running the Val estimate we could have uh do uh what is literally two stage Le squares uh first stage you run AI on the instrument second stage you take the pred Ed value and you stick it in there and the nice thing about it is that you could do that even if you don't have a even if the Z is not is not a one zero anymore you can do that for any any type of Z and that's called um two stly square so more generally imagine that your model is the one we started with at the at the beginning of previous lecture where Yi is beta 0 plus beta 1 X1 which is the variable that you interested in and concerned about that's the one that you want to instrument for and then some control variable X to uh that uh you know gender uh region effect whatever that you uh you just want to put in the regression but you don't need to instrument so they are control variable sometimes those control variables are needed because the instrument is only valid when these control variables are included uh now you look for an instrument uh for x uh if x has more than if x is a matrice as opposed to being just one so if you have two things that you want to instrument with at the same time you need at least as many instruments as you have X's uh if uh or you could also have more but at least the same uh remember our instruments where they need to they need to be good they need to be uh of course correlated with x one then need to be uh randomly assigned or as good as randomly assigned so that we can have a causal effect of the instrument on the Y and they also need to not have a direct effect on the on the on the Y so that all their effect on the Y can be confidently attributed to the X let's say we found those instruments then uh they not Z The Matrix of the instruments plus the X2 that's very important that the Matrix of instrument include both the variables Z that are instrument instrumenting for the X1 and the stuff that's going to instrument for itself that we're just controlling for uh in the old days the STA if you're using sta the new sta Syntax for IV regression in my view is horrible because you forget that because it's telling you you have your X's are instrumented for and some z's that are instrumented for the x1's and then you have the other stuff that's not this is silly like in term of Matrix algebra that's underlying this regression it makes no no sense uh R seems to be like like the old data like doing it properly but so the Matrix of instrument is the stuff that is instrumenting for the for the X1 and all of the stuff that instruments for itself just remember that uh and then intuitively what would we do well following what we have done with in the case of there is just one variable we run X1 on The Matrix of instruments that is all of the instruments plus all the x2s and in the second stage we take the predicted value for X1 uh and then we we we we we put in the x2s as well uh so in practice if you do that the point estimate will be correct but the standard error and all the test will be wrong because uh the you have if you do that like mechanically you forget in the estimation that you've estimated the first stage and there for there is additional error that comes from that so you need to take it into account uh so in practice that's not what you're going to do in practice you're going to tell your software Mr R Mr sta that hey this is my uh model and this is my metrix of instruments which is going to include whatever is instrumenting for X1 and the x2s okay and then what is uh are going to do well if you have um the same number as instrument as X so for example if you have one mogeneous variable that is instrumented with one instrument then the Z the the Z Matrix and the X Matrix are the same size then the uh 2 SLS formula uh which is is actually done in one step but is intuitively this twep formula is z prime xus one that Prime I so what you do is you you replace X Prime X of the OS formula with a z what does it do it takes the AIS IT projects it onto the vectors of these and it takes only the part of the AIS that's explained by the Z to uh to uh uh um instrument with the Y so that's the that's the the formula for 2 SLS and the variance is Sigma Square Z Prim x - one Z Prime z z prime xus one so it's this guy is a little bit bigger than in an OS case uh in the OLS case it's uh we don't have these two because we have X Prime xus One X Prime x x prime xus one so these two go away so that's not a very scientific intuition but that's useful the larger these matrices are in the middle the bigger the standard error so basically the fact that these things refuse to collapse is going to um is basically accounting for the fact that the the the first stage is estimated and your standard your standard are going to be a bit larger than if you pretended that the the the X the X hat that we're putting in a regression is know so this is this is what's this is what's basically reflecting that so this is for the case this is the formul formula for the case where you have the same number of instruments as you have variables in the model so for example One endogeneous X and five control and one instrument and five control if you have more uh instruments than variable for example it it could happen that you have two good instrument for so you have more uh Z than than than x the formula is a little bit more complicated I it's a little bit longer no need for you to learn it or don't need for me to put it there but it's really the same idea you're taking the X you're projecting onto the space of these and then you're using this projection as your regressor to Stage the squares done in one uh step so that's the formula uh let's see how how does it uh so you run a regression this is for the uh the Ghana example that we saw before uh so we are loading the data Etc and then we're running a regression this is the the commment so we said total score uh is um is the dependent variables and we want s um whether you complete secondary cool plus a bunch of region for region fixed effect so region. f is my my fixed effect and then uh uh these are my instrument treatment is going instrument for a uh complete and then the region fixed effect so uh R does it properly the syntax of R is correct you have all the X's here and all the uh all the Z's here the entire Z Matrix here okay so and this is what it's going to report something very similar to the to the uh o SLS um something very similar to the OS table uh it's going to tell you which which what was the estimated in the first place then you get your coefficient for completed Secondary School which is 64 so remember when we did by hand this is about what we ended up as well uh and then we have the the the region D that are here instrumented as well we could add more control if we wanted uh and uh here we would expect the control to make no difference because we have a good instrument we have a random randomly assigned instrument and in fact they don't make they don't seem to make any much difference at all the coefficient is still 63 with a standard error of 23 uh the statistic of 2.7 and the the the P value over here very very low so showing very large impact of going to secondary school I mean very significant and quite large impact of going to secondary school on on scores at a cognitive exam which is reassuring if we didn't have that that would be a bit boring that's for formula that's kind of I'm done with formula for today uh in fact maybe forever because on on Monday we'll do visualization I don't think it involves all that many formulas uh but I'm not done with thinking because uh today we are going to think about uh designing experiment or the the notion of like how what people think about when designing experiments it might seem a bit it is actually not specialized to think about experiment because I think if you're good at if you if you can think about experiments you can also think about non about good non-experimental uh uh methods so what do we think what's experimental design like what do we what are the questions we need to answer when we're thinking about how to design a randomized control trials uh so first we need to think about what is being randomized and that means like what are the intervention or interventions that we are looking at so going we need to think about who is being onized that could be a what to so the W could be a school but I I wanted to have different questions so what's the level of randomization we randomize at the individual level at the school level at the Village level at the cell level within the body or whatever if we could do that and then the sample of which we randomize for example uh in the Ghana example we randomized over people who were eligible for the for the scholarship uh it is it has all of these questions have have all of these decisions have implications for the interpretation of the results as we discussed at length so uh then how is randomization introduced uh are you going to do it with your computer are you going to do it in the field are you going to take Bulls from a n Etc and uh what are the type of uh of modification you imply you you will do to the design for example are you going to stratify by creating groups of relatively identical individual and randomize or schools or whatever and randomize within that or not and then the last question we need to answer is how many units will be randomized and these questions we already discussed when we spoke about testing this is uh you're going to Def you're going to um want to have your experiment to have enough power to answer the questions of interest and at the same times you might not want to spend zillions of dollars so that's going to be the how many question which I'm not going to answer at all today since we already discussed when we talked about power calculation so this was already a design questions uh so these are the the type of issues that you the type of things you want to think about uh in advance of course as you do the experiment there are many other things that you need you need to be thinking about as things go along but these are the type of things you need to think before even going to the field and why uh what are we trying to achieve when designing experiments like what are the things that we want to do well first of all we want to be able to do the project so one of the big uh progress uh one of the big progress that has been made in the last uh 10 15 years I would say is finding ways to introduce randomization by stealth so of course one of the big progress is uh uh AB testing which means all of us are constantly subject to experiments that we have no idea we're subject to uh so randomization is very easy in a context of a website because people you know there is a ton of traffic and you can random the the the pages and what people see and even the prices and stuff like that but uh uh in other settings uh until about you know 15 years ago in a lot of cases people would tell you that's very nice but I'm not going to randomize like that's not possible in my business or in my policy business and a lot of the progress that been made uh on is to try and int introduce some element of randomization and nice thing from the point of view of this class is that often the randomization is not perfect in the sense that it doesn't give you the cleanest experiment but it give you some handle with some variation that has to be combined with statistical techniques in particular IV to uh answer the question you really want to answer so that's one of the design of one of the goal of Designing experiments is making the experiment possible anyways in a cont context where it might not otherwise have been obvious and the second of course is making sure that you your should it's actually the first maybe the most important one surely is to make sure that the experiments you're running answer the questions that you're interested in uh it would be very bad to start with an experiment and then think about what question it might possibly answer so typically you start with a question then you design your experiment to answer your question so what I want to do in the rest of today is give you some examples of uh of both of these things of uh thinking about experimental designs to answer both of these I'll start with the introducing randomization where it may not otherwise be obvious and then I'll go over two examples of designs uh to answer that were done to answer to answer specific questions of of Interest where it is the experimental design that made this possible as opposed to good data collection of the pl so the easiest way to randomize of course when you can is simply to randomize with do is a simple completely randomized experiment take a population of uh uh eligible people that's your sample frame your unit of randomization could be people household Etc and then use a software to do in your office randomly assign one group to treatment one group to control or maybe you have several treatment groups so that's the easiest thing if you have this luxury that you can just simply randomize and it's it's often the case actually that it's possible then you can there are usually two questions that you need to answer at the design stage one of them is do you want to stratify and one of them is do you want to Cluster they are almost the opposite going in opposite direction so what is stratifying stratifying is starting by creating ster uh think of stret as buckets uh buckets of people that or schools or units or whatever that are uh that that are going to be treated separately so why would we need to stratify a main reason reason is to randomize within people who are um who are similar uh so for example in the Ghana experiment that we saw we stratify by gender and we stratify by region which is why all of the IV regression had the street adamis uh sometimes we stratify because we also want for example to give different randomization chances in different St I'll talk to that in a in a moment and that's fine then it means the probability to be assigned to the treatment depends on your potential outcomes but it depends on them in a way that we completely control because we within each ster we have a we have a completely randomized experiment uh so why would we want to stratify usually it's a it's just to improve Power by reducing variance because what you're the way you're going to analyze a stratified experiment is by looking first within a strator where people as far as you could tell are quite as similar as you could make them uh they might share a gender they might share some baseline characteristics they might share um the regions they are in ETC you whatever you think is relevant to determine the potential outcome you're going to make them similar that means it's going to reduce the variance in the Epsilon whatever is left that you're not controlling within each stter and then you're going to aggregate the stter and therefore you're going to get an estimate that is more precise than if you had not stratified yeah how is stratification different from having a Expos of doing it exante versus Expos that's a great question so uh the uh the quick answer is that uh if you do it exposed you have to estimate the effect of gender so that's eating one degree of Freedom if I do it exante whatever is the effect of gender doesn't matter because I've control for it completely so I don't have to estimate that effect so if it's just one variable maybe it doesn't make a big difference but if you when you create St you can create St you know you can create very very fine St at the extreme you could do pairwise randomization which is randomize pairs great treatment and control Pairs and just randomize one member of each pair as a treatment and control of people are really looking similar as similar as you could make them if you wanted to estimate if you wanted to just completely randomize an expost control for all of the things that have made in the street there would be a lot of uh a lot of degrees of freedom to to e up so that's the that's that's the reason it's better to control then you don't have to worry about the estimation one more thing to estimate is going to cost me some some data but if you haven't stratified and for some reason there is still there is imbalance at the end because it could happen you can always control uh so that's one thing that uh so that's why there is pretty much no downside to statification it can only help so uh so whenever possible people tend to do it yeah is there a situation you so you know that's a that's a debated question there are there have been papers about that I am pretty sure that the jury is clearly came out and saying no you cannot overdo it at worse it's going to be irrelevant but it's never going to be uh it's never going to be to be bad because you cannot do at worst you're going to uh create St that are irrelevant you're back to a completely randomized design um you could do works by adding tons of irrelevant ster instead of adding one relevant one but that in the same way that if you completely randomize you would be worse than if you had one relevant one so uh yeah so I think the bottom L you cannot overdo it with control variable Expos you can totally overdo it because you can eat up this going to eat up all your degrees of freedom uh so there is a trade-off between adding more control variable is going to uh is going to take away some of the sigma square but at the same time it's you're paying it by degrees of freedom with the stratification there is no such tradeoff um but it's not it's actually it's not an obvious questions people have raised that maybe it would be an issue not an issue uh so clustering is the opposite instead of Random you've created buckets instead of randomizing within the buckets you're randomizing across the buckets which means that any bucket if it's treated everybody in the bucket is treated so for example the unit of riss could be the school or the village or the the the family um so it will Hur power it's always weekly worse than to do it at the lower level at the extreme think of it where you were randomizing uh two regions effectively your sample size is to when you do it's a bit better than that because you have many people within the class world but not much better so and in fact um some people argue that the best way to to analyze clustered experiment is just to treat each cluster as one observation so you can see that then it could really H power a lot to group people into clusters and then randomiz so you never do it uh if you can avoid it but there are many cases where you cannot avoid it either because the cluster is a natural unit of randomization for example a school level intervention is happens in the school or because there are a lot of externalities so anything you did to one person would immediately affect someone else so the stable unit treatment value assumption would be B if you didn't randomize at that group level or some sometimes it's just not practical because you cannot ask for example a nurse to treat people differently uh even if she could in principle she just doesn't have the bandwidth to do it properly and it wouldn't be done so these are kind of the questions you have to ask yourself in the uh on to how to randomize in the simple case where you can just randomize and then there are cases where it seems like impossible yeah like simple rization and um certification in the sense that sometimes I know I need to straty but I I don't know what my hidden variables are so I do a Oh you mean to randomize and then you don't like the result you randomize no just to look at what the variables could be that I could sty for I don't know if age has an effect on what bases would I age or gender is it pure hypothesis or yeah you you it's pure hypothesis the beauty of it is that it doesn't matter because if you get it wrong it's not it's no worse than fully randomized so it's not it's not these are G guesses which don't have a huge amount of impli it can help you you can lose power yeah sometimesy expensive stying is not expensive per se because I mean unless you need to go and collect the data to uh that you don't otherwise have to stratify but otherwise it's just involved telling your software and when you randomize is make sure that you make sure that you first randomize among women and then among men so per stratifying isn't isn't costly the question I thought you were asking is can I suppose I have many variables and I don't really know how to construct my Streeter I'm going to run one simple randomization and then I look at the variable it looks kind of ugly uh because I it turns out that there are you know 70% of female in the treatment group uh uh and I don't I don't like that can I do it again that's another question on which much ink has been spilled uh and uh so my sense is that the jury is also out on this one which is actually you can uh it's a little bit uh less good than stratifying if you knew what the street were because then it's very clear and you can control for them here you're going to have an implicit stratification without having really uh without knowing exactly by what you've stratified it depends what you've looked at but you're still randomizing every time and so unless you randomize there is actually a theorem that AB Bard and Sil wrote saying that unless you randomize more than the number of observations that you have uh it's it's it's fine you can just randomize so so you can it's not going it introduces a tiny amount of bias but compared to enormous sample size doesn't doesn't really it's a small the the the the intuition for it is that if you have many many observations and you randomize say four five times it's it g still gives you very little choice if you randomize so many times that you pick exactly the randomization you like you might actually you you're not randomizing anymore you're choosing um but if you're doing a few times you you're choosing across such a small set that it doesn't matter too much okay so suppose that you don't have a uh you you work with an NGO and they say it's not going to be possible to randomize for whatever reasons and I'll try and go through the reasons so first thing they could tell you is that we have money to cover uh you know 200 schools with our Dew Waring program we are going to work to to cover 20 200 schools and we cannot just go to see some schools and tell them that sorry we're making contact with you because we want to collect data but we cannot randomize it's just not uh they're not willing to do that um so that happens uh actually less often than you might think but it happens and in that case uh a design been very popular is to is phase in design which is to say okay we are going to take this 200 schools and we are going to randomly divide them into a few groups and we're going to end to to introduce the program progressively so for example uh you could say well uh in year one uh group a is going to get the treatment and group b and c will be the comparison group for group a uh in year two group A and B become treatment Group C is the comparison and in year three everybody is treated of course in year three you don't have an experiment anymore uh but you have an experiment in year one and in year two uh so that's been a kind of a practical way of introducing randomization very popular at the beginning of doing randomization development it has a number of caveat actually number of problems the first one is that if the effect persist over time it's a bit mushed up because suppose the effect are you know kind of take two years to fully develop then uh uh when you're comparing uh uh these guys when you're looking at these guys they're actually less treated than these guys they're still treated but less than these guys and uh um and this one is the is still a comparison and then in the in the third year you don't have everybody anymore but the people got treated in different ways so you can deal with that with with the analysis but it's a little become a little bit messy uh another thing that uh another problem with this kind of design is that um typically it's pretty clear what's going to happen so you inform people of when they're going to be treated and in some cases the anticipation of being treated in the future could change your behavior um so for example uh um if you anticipate paid to receive a loan in the future because microc credit is coming to your village you might decide that uh you know you're going to postpone some investment project until the loan comes or on the contrary you're going to start a company like a little business because you know that you'll have loans in the future so this has to be argued away uh in cases where you randomize with this design but sometimes it's kind of really the only way to go or you can just tell the NGO that's what you're going to do but not inform people that it's EXA sequence and so that's one way to proceed it used to be very popular for the two reason that I'm telling now is now people are trying to stay away from it when possible doesn't mean that you don't treat everybody eventually if you think that the program is super effective but you just don't make it a part of the design that is you know known and public and everybody is informed about yep question about that what what is your NGO that want you to do their evaluation they they identify these three groups but they like say let's say your intervention is like a labor trafficking prevention program and they identify group a as like being more vulnerable so they want to give the intervention to them first yeah so then that's not randomized anymore uh so that's not a maybe you can do something maybe you cannot but it's not going to be an r c but let's see one let's let me pull this example to this one uh which is actually a very useful one in these type of circumstances where say you're working with your ngos easier for me to think about the credit a bank for example but I think we could couch it in your example as well suppose you're working with a bank and you say hey we have this great idea we're going want to look at the marginal impact of uh giving money to people so please randomize whether or not you approve credit or not they might look at you and say you must be joking uh there is a bunch of people I will never lend money to uh so I'm not going to go for that so there you can say well yes sure of course I understand but what are you using to to score people and then typically there's some kind of a score so for example for banks it's a credit score taking your example of trafficking it might me but each Community is identified by the fragility score something like that it might be more or less formal or informal with bank and credit cards usually quite formal and then in the status quo what they do is they say that anybody say in my example above the score of 45 gets uh gets the gets credit and everybody below doesn't get credit but it's not that these scores are perfect uh in fact many banks if you talk to the manager they will kind of fully acknowledge that the sces are a little bit you know uh black magic and they're not very sure that they are so great and maybe they could land a little more or maybe they should lend a little less so if there is such uncertainty you can say look anybody below 30 no way you just reject them all anybody above 60 there are great clients you absolutely want to treat them you're just going to treat them all but then in that gray zone between 30 and 60 in my particular example maybe there is some scope for trying out and look you're going to learn something that is useful for your business because if it turns out that the people uh right above the threshold actually are quite likely to default then you might want to ex make the threshold higher in the future on the contrary if it turns out that these 30 to 45 guys are just as good uh it turns out then you can lower the threshold in the future so in this example they might be willing to randomize they still prefer their good guys so let's say the probability of treatment will be 85% in above threshold and uh the probability of treatment will be 60% below thresold uh but um you it could be even lower but the point is that you still have randomization there and here you have two straighter one straighter of above 45 one straighter below 45 and that's your study sample so uh so this randomization around the K or randomization in the bubble has become quite popular because it it works well with Partners who can continue to do their business and get information precisely where there is a lot of uncertainty for them and it is often the type of people we are interested in anyway uh because they are the type of people that would be uh exposed to an increase in this program if the program was made cheaper or it was expanded or something like that so that's also a a useful design third design that comes very handy uh requires to understand IV uh is the encouragement design uh which is particularly handy in situations where uh you have a programs already there anyway uh for example uh a nationwide program imagine a pension program is already there anyway so it's not going to take it away from anybody but even in those cases uh the take up for those programs might not be Universal because there is a bunch of barriers that stand between people and actually getting the getting the the the program for example information or some loopholes some sorry the opposite of loophole some hops to jump and stuff like that so then you can say well I'm going to work with a bunch of activists take take the pension program I'm going to work with a bunch of activist I'm going to send them I'm to identify by eligible people for this program for example old people above the age of 65 uh who are poor and therefore are eligible for a pension program and I'm going to help them apply if they are not currently getting it so then your sample is eligible people who are not currently getting the program let's say your activists are pretty effective they get 45% of people to to apply for the pension in the control group stuff happen anywhere people you know other people in come Etc 20% of people apply for the pension so the difference in 15% is is randomly assigned is due to your randomization when you compare the treatment to the control group that's not the effect of the pension it's the effect of having an activist come and try to help you get the pension but by dividing by 15% 25% by dividing by 25% or multiplying by four you basically have an IV estimate of the effect of the pension so the Ghana program that we studied last lecture is typically an encouragement design program is there anywhere everybody is uh entitled to go to Secondary School uh but by giving people a scholarship by making it more likely of course in the analysis of any encouragement design you're going to have to argue for the validity of your encouragement as an instrument which means a you have a first stage so here it's 20 minus uh 45 minus 20 b instrument is randomly assigned that's usually in an experiment but see it has no direct effect so in the case of the pension program for example you might think that a young activist coming to your house and helping you to get your pension is going to to have to come three four times anyway maybe there is a direct effect of this visits you know you feel better your old person that nobody has ever visited in the last three month and now this very you know Fresh Face young woman comes for for five five times you feel much better so in that case you would walk worry about that so maybe you have like a home visit program as well in this group but you just don't make them apply to pension so encouragement design require one more level of thinking to make sure that your experiment will be valid you have to do that thinking exan because export is too late to introduce the extra the extra layer of of treatment Etc P um is it more common to come up with your own experiment that you want to test and then find that can do it or to see a progr that's already going and you're go uh there is a combination of everything sometimes people come with something that they want to get evaluated uh sometimes you you have something you want to do anyway some and then usually it's some combination of the two which is people come to see you with this idea and you say uh fine but they have in mind some very simple experiment and then when they've left your office after two hours they have this Ultra complicated design in their hands uh that you've managed to to sell to them because you want to do something interesting so this kind of a combination basically to run an experiment you need a Partners to implement the program you need a research team to look at it and you need money in principle these three things need all to be there together in principle initiative could come from either threee it's actually pretty rare that money starts partly because that creates Partnerships that are not very effective usually but sometimes the partner comes for sometimes the research comes for okay so this was like sort of introducing randomization by steal um and now I want to go over the sort of the the second uh type of things I want to discuss about designs which is you know how do you design experiments to answer questions which are economic questions so not just does my program work but an economic question uh and I'll give you two specific example instead of talking in the abstract one on estimating equilibrium effects of an intervention and one is uh unpacking the effect of an intervention to understand better why it has the effect that it has uh the first question is a the first example is a is a program that I worked on a research that I worked on in in France um so all of Europe has has high unemployment level for long time and uh governments are a little bit uh at a loss of what to do uh with this problem so people are trying um various things but one of the popular uh ways to help people is called um active labor market policy which is when someone is unemployed basically uh doing a lot to try and help them get the job so teach them to tie their shoelaces and put on a tie and show up on time at an appoint at a appointment for a job interview giving some phone calls to your contact Etc and so in fact many European governments give that job now to temp agencies that have sort of a employment placement uh Bureau and there have been several randomization of such programs uh and usually the way they work is within a site for example a town uh um unemployed workers are assigned to one group or another so they assigned to the active labor market programs or they're not and those evaluation typically tend to find that people who are helped do better than people who are not helped that's nice and uh and well but an important criticism against those evaluations is that uh the gains could be offset by displacement effect so suppose Leo gets helped and I don't then he gets the one job that there is packing flowers and the fact that he has it means I cannot get it so if we are going to compare Leo to me we're going to find that Leo is doing better than me so it's an experiment that is effective which in a sense is correct but it is not a net effect because it is just a musical chair he took he took there was only one job into the two of us and the program give them give him a little lead to get the job so on the contrary one could say and when can write models where actually the improving the productivity of the search effort actually improve the net amount of jobs that are available in in the economy for example because firms just hate to search for workers uh and they prefer not even to post a vacancy if posting the vacancy is going to to cost them like the equivalent of half of a year of someone's salary to fill it up so when there are too many unemployed people who are who have no idea what they are good at the the that might actually reduce the number of vacancies actually filled so it's an open question and of course it could be a little bit of both so these are what's called equilibrium effect uh which means uh uh in in equilibrium how many jobs are there uh it's going to determine whether or not this is extra unemployment or not so how what when you do that well the only way to do this really is to is to randomize at uh two in two steps in order to find out whether take the example of Leo versus me the fact that there are many Leos in the labor market that are helped and I'm not is it going to does it make me worse off compared to my situation if I were into an entire other town on this side of the on this side of the room where the program doesn't exist so uh so the the randomization design here is to do to go in two steps uh it's done in two steps are done at one time but it's uh you randomize uh you first randomly assign the proportion of treated to areas and then you randomly assign treatment status to individual within areas let me show you a graph of that so basically what we did is we created so we work these type of things can only do be done at a fairly large scales because you need the skill to get the equilibrium so in France people don't like to move very much so uh a town is roughly a labor market people aren't going to move to get another job so what we did is we took we worked within an unemployment agency in about half of France and we divided the towns into a straight of five so literally like full it's fully stratified in that sense so in each Traer which are quintuplets we have five five town and uh in one town in each now in each of this trat we picked randomly one town where we treated nobody uh one town where we treated uh one um so sorry in all of the in all of the ideas backtrack uh in all of the in all of the Quint plate we one town where we treat nobody one town where we treated 25% of people one time we R treated 50% of people one turn we retreated 70 5% people in one town where we we treated everybody in retrospect that everybody one is is is wasn't that useful because it's not everybody it's everybody who is eligible for this program which are the young unemployed workers so to start with they are just there a fraction of the people but anyway that's how we designed it and now you can look at the effect of being helped within one region so the Leo versus Me comparison which is what people have done traditionally by comparing Mr Blue here to Mr White here Mr Blue of course within each Town once we determine the proportion the people to be Rand to be selected are randomly assigned okay so we can do this Leo versus me and then we can also do me versus uh Frankie over there who uh is actually in a town where nobody got treated and that's going to answer the question of whether the fact that Leo has treated Hots me okay by looking at uh uh what is my what is the situation of someone what is the chance of someone to get a job back within six month if nobody around them got this program versus is some other people got this program so that's the that's the idea for the for the for the program I think everything is is here yes so the program is the target population is uh young unemployed worker so people age less than 30 year old unemployed for more than six month uh but with some some college so what we have is then uh these people in the all white area we call them super control group so they are in individual assign 0% area uh by comparing the assigned to control and assigned to Super control we get the displacement effect by comparing assigned to treatment and the super control we get the full effect of the treated of the program by comparing assigned to treatment to control we get the effect within so that's the poent like job stealing effect I mean the equation is here this regression is run uh within places where so this is this is controlling for the DC so it's controlling for the area level so it is basically a comparison of Leo versus me inside one labor market and what it shows is that very little effect for women uh this is the effect the question that is being asked is not very well labeled the question that's being asked is have you found a job uh the outcome is here have you found a job of uh like a a long-term duration job so more than six month have you found a job for more than six months within six months so six months after the program I'm looking at you again and I ask you whether you have in your hand a contract for unemployment for more than six months and we find that uh so about 17% of control people have found a job anywhere way uh but there is a pretty large effect of men of 5 percentage point so this is the within effect and now I'm going to add the uh comparison of the uh control to the super control so this 51% here is still there it's the comparison of Leo and me but now I'm comparing me and Frankie and you see that I'm hurt actually uh the uh there is a negative effect effect of being in this area compared to not being in this area of 3.9 percentage point and then the net effect would be then to compare him to her and the net effect on him is really not that large it's 1.2% uh but um it looks big in the treatment area because he's taken my job so not only so I'm doing worse he not doing that much better but if we did it within we would think that there is a large effect so that's one uh so this idea of randomizing in two steps is going to be useful whenever you think that there are spillovers and you're interested in the spillovers then you can think of it in your mind as one treatment is being directly treated and one treatment is being exposed to people who are treated so going back to the design of of you have a combination of cluster level randomization because you randomize a cluster into a fraction treated and an individual level randomization because within each cluster you're going to randomize who you are actually treated can I just revisit the results to see if I understand so five 5% for men that's considering displacement effect or not considering displacement exactly it's a difference between within a labor market the treated and untreated so it's a difference of you versus me so if we were to generalize then the success of these programs in in the literature is umus given that that would be the conclusion of that we find that uh they what look successful is just the fact that within an area the people who are helped are somewhat more likely to get a job at the expense of people who are not helped that sounds like a very important result how would the government interpret that or or would you do the study again in a different country or are there any lied variables that are considered I can tell you how they interpreted this result is which is not one of my success in term of policy influence so soon after we we came out with this result the program was scaled up but that's not how they should do yeah you should you should think of that as being pretty bad uh so there actually there is a Twist to it though uh just in term of the substance is that uh the it turns out the evaluation span the the the recession that follow the 2008 crisis and you can look separately at places that were most affected by the recession people who are less affected by the recession before and after and what you find is that the displacement effect is the strongest in recession time uh so in recession time it's presumably where firms are not hiring anywhere so you can you you can help the search effort as much as you want it's not going to be all that helpful in uh non- recession time in good times then uh the the displacement effect is much smaller so in that case in non- recession times firm would be hiring if people you know if they found capable people around and there the fact that you are training some people into presenting themselves better and retraining for uh you know some skills Etc helps um so that's the that's that another thing that you're learning from this experiment and you know it's kind of always is is that so this is the evaluation of this particular program but if you're willing to be a bit willing to use models to go one step further you can say generally it suggest that that means that search effort the productivity of search is largely um competition between workers that means in in in particular during recession times which means that maybe we need not be too worried about unemployment benefits like the problem of unemployment insurance is that it makes people not that enthusiastic of looking for a job but that's only a problem if they would find a job should they search for one so in recession time because of uh social insurance concerns you might want to increase the the length of unemployment insurance because that's when people really needed and what this type of Rel suggest is that in addition there probably will not be too much of an efficiency cost of improve increasing the length of Unemployment Insurance in uh in recession time because yes everybody is going to search a little less but uh uh jobs it's not that there are job for them to find anyways so the the jobs that they are are going to be filled regardless by by with the existing search effort so so one lesson that you can take that's a little broader than just this is that in recession time uh both the benefits of increasing uh UI is is is high uh because of uh Insurance motive and maybe the cost is not that high so that would give you a sort of an argument for doing this you know modulating the length of unemployment insurance with the with the macroeconomic cycle um so that's that's the so that's that's of and Beyond sort of this particular topic uh this idea of randomizing into steps has been used recently to look at various effects of speed any kind of speed over here they are coming from equilibrium effects but could be from adoption conion Etc can be looked at with similar similar designs uh the last example I want to give you today is uh also very popular way to to to do things is this uh kind of answering Jo this question is that who comes up with the program is it the implementing partner and then you're kind of evaluating whatever they are interested in or is it the researcher and often actually some combination of the two and I'll give you one very nice example from a paper by AB Bard benen who are both here and REM Han is at the at the kendi school and they're looking at a program Indonesia called the Raskin program uh the Raskin program provide eligible households which are poor households with 15 kilogram per month of heavily subsidized rice and that's a pretty terrible program uh it's full of corruption many many rice gets disappeared Etc it goes to the wrong people uh there are reasons to there are many reasons for that one of the reason is that but for whatever reason the government likes this program they're not going to get rid of it so that's not on the table but they want to try and make it better and right now one uh problem that the program has is that the information among citizens is low so only uh survey that they did suggest that only 30% of eligible household know that they are uh Raskin eligible and uh the beneficiaries also believe that the cop is 25% than it actually is probably because someone helpfully informed them of that somewhat higher number and so as a result the eligible people for the subsidy only received about a third of the intended subsidy between the fact that some of th