Why teaching will never be a research based p

Key Takeaways

foreign [Applause] 's okay can you hear me at the back is that fine thank you okay so uh there's rather provocative title I'm often told I'm challenging and it's people seem to assume that I actually set out to wind people up but I really don't I I just seem I just find things that find I find interesting I share them with people and some people find that challenging is I guess so what I want to convince you of is that um I want to talk about what it might mean for a profession to be research based I want to show you why I think education falls short of that ideal I want to spend a little bit of time then um what I think educational research can and should do and I want to spend a little time right at the very end thinking about the role of teachers in this process so it seems to me that in a research-based profession professionals would for the majority decisions they'd need to take be able to find and access credible research studies that provided evidence that particular courses of action that would implemented as directed be substantially more likely to lead to better outcomes than others that's my definition of what it would mean to be research-based and we could spend some time going through each of those conditions so for example we've recently been talking about getting teachers access to journal studies that's a straightforward thing to arrange the trouble is that you know when teachers go to the educational research cupboard the cupboard is generally bare and one of the problems is that incentives aren't very well aligned in the system and I'll talk about what we can do that later about that later on but it seems to me that that is what a reasonable definition definition of what it means to be research based and I don't think we come even close to that I don't think teachers could have the majority of the decisions they're taking in the course of a day informed by research and you know we're always held up um against medical research for example so educational research is often conferred compared unfavorably with medical research and you know medical research if you listen to Ben gold taker this afternoon you'll hear there's a lot of problems with that but you know we've done made quite a lot of progress in working out how to treat certain conditions here's what's interesting 40 percent of GPS admit to prescribing antibiotics for viral based conditions things they they prescribe drugs that they know will have no effect because it makes people feel they've got a prescription in their hand as they work out they feel they've been heard they feel they have a solution some doctors are much better than others at getting patients to adhere to drug regimes so what I'm saying is that when we when we compare medical research and educational research and educational research comes up up short that's usually because we're actually comparing the successes of medical education research medical research with the failures of Education research and we look at the reality in doctor surgeries it's very messy how do you get people taking their drugs how do you get people taking more exercise those things are very messy and medical research hasn't made much progress there the other thing I want to stress is that educational research can only tell us what might be uh by the way these slides are all available I've already uploaded them to my website dylanwilliam.net you have to spell my name correctly it only has one L in my last name how many L's do you need so for example the research on homework shows that most homework does no good but then most of the res most of the homework the teacher said is crap so what the research really shows is that crap homework does no good big shock there but what the research also shows is that the homework the teachers set most frequently is the least effective preparation for future learning is actually the most effective form of homework it's just very much more difficult to organize so the fact that people have said that the homework research shows that homework doesn't do any good doesn't mean homework can't be good in education what works which is what politicians would love to know about is not the right question because in education everything works somewhere and nothing works everywhere the interesting question is under what conditions does this work so we need to approve a causal relation you know we know that with with um anything in social science correlation is not causation we need to establish causality and you know David Hume I think probably said it as well as anybody in slightly archaic language in 1748 um David Lewis more recently said that if c and e are two actual events such that e would not have occurred without C then C is a cause of e so we have to prove a causal relationship the difficulty is that ultimately proving a causal relationship is a counter factual claim we want to be able to say is that because we did this this resulted and had we not done this it wouldn't have resulted so what we typically do is correlation research given C the cause e happened the problem there of course is that as a Latin phrase says post-hoc Ergo propter hook the mistake is if it's after the event is therefore because of the event so the counter factual claim if C had not happened e would not happen but problem is C did happen so we need to create a parallel world parallel world where seated at the cause did not happen now you could do the same group at a different time but the problem is then you have to assume stability over time the idea that we actually resort to most of the time is a control group we we actually have two groups the control group stands in for what would have happened had we not treated the experimental group to the experiment and that's the logic of randomized control trials I want to make it clear I think randomized control trials are wonderful I've even done some in my in my own work if you can do randomized control trials you should do so because they actually are provide a very strong but not perfect warrant for causal inferences oops I want to stress some of the problems with randomized control trials first is educational data is clustered if you want to use the mechanisms of statistical inference you have to assume that students in a particular treatment are independent of each other and apart from the treatment they received and the fact is that in education that rarely is the case because you've got 30 kids being taught by the same teacher so those kids experiences aren't randomly independent now I'm quite worried about the study I've just heard about because they've got 30 we heard about a study with 30 schools in the control group 30 schools the experimental group I don't know whether that experiment is big enough to have a significant effect this is the second Point statistical power statistical power is the chance that we actually find a big enough effect if it's there is our experiment big enough to capture this if we have a slightly biased coin and we toss it 10 times we get six heads and four tails it might that might be evidence that is biased but the problem is it might not it could just be a chance variation if we throw it a thousand times and get 600 heads and 400 Tails then I think we're a much safer ground concluding that the coin is biased so the size the experiment matters and the problem is in education the sized experiment is hardly ever the student it's often the number of teachers in the experiment and because teachers talk to each other in the staff room it kind of it's generally the number of schools to get independence in your assumptions you need to have different schools participating you have one school in the treatment group one school in the experimental group your experiments get very big um power so implementation one of the original things is that teachers do not do as they're told it's quite funny that so a number of randomized controlled trials very expensive randomized controlled trials have been conducted in the US and hardly any of them have produced significant results so for example Rick stiggins wrote a book called classroom assessment for student learning CASL and a randomized controlled trial was undertaken of this and they randomized at the school level they did it absolutely right Unfortunately they found a nut and a Zero Effect the point is that the effect wasn't big enough to be statistically significant they then discovered that not a single teacher in the study had required that had received the required amount of time so the program specified 100 hours of professional development the average was about 60. and so because these people hadn't been given time to work through the materials what all you proved is that if you do not reproduce an intervention you are unlikely to get his benefits and it turns out that getting people actually to do this in a complex system like a school is very difficult and then finally I want to say something about context context is really important we can say that yeast causes bread to rise but yeast only causes you know do the causality thing delete the yeast the bread no longer Rises so therefore the yeast is a cause of the bread Rising but the yeast only causes bread to rise in the presence of moisture and heat and a whole lot of other factors which you don't understand that well including the right amount of salt and so for example we might get a randomized control trial which shows an effect but we have to understand the context let me give you an example I did some work in Trenton in New Jersey which is where near where I live now in the U.S and we wanted to help their High School improve the graduation rate from the school and in New Jersey kids have to take a test called a high school proficiency assessment usually called aspa and the success rate in this in Trenton Central High School for the hesper was about 17 percent so it was the second lowest rate in the entire state and we went to talk to these teachers to find out what was going wrong and we found that these teachers didn't actually help kids by teaching them the things that were going to come up on the hesper so these kids were taking a test which determined whether they actually got a high school diploma or not but the teachers weren't teaching the test because they were teaching their own curriculum and we said why why don't you prepare the kids for the test they said that would be teaching to the test so we actually produced an intervention to help them align what they were doing in the classroom with what was going to come up on this test and in two years we improved the graduation rates for young people in Trenton by 40 percent but the point is that intervention only worked because of what the teachers were not doing in another school where teachers were already well aligned to the outcome measures this would have had no benefit at all if you don't understand the context in which you're implementing a solution you can't be conclude that it's going to work elsewhere so I think randomized control trials can be very powerful but they're also hugely problematic even when they're well conducted and implemented and I want to take a little swipe at meta-analysis um people often cite the big publication that Paul black and I wrote about formative assessment in 1998 as a meta-analysis and that shows that that just shows they haven't read it because Section 8 of the paper is called No meta-analysis Such and I just did a little bit of homage to Ben goldhagen who's on this afternoon um I think you'll find it's a bit more complicated than that and as a remark to say to people whenever they talk about educational research you're almost always correct in saying I think you'll find it's a bit more complicated than that so you've seen this the educational Diamond foundation's toolkit and feedback which tops the list I'm going to come back to feedback in a minute um I like this list because it shows that the things I've been talking about like feedback students owning their own learning and students who are helping each other come out as a top three I want to go over to um the last few pages of this is slowing down slightly I think no it's died I also do manually okay teaching assistants have no impact on student achievement apparently but then in most schools teaching assistants are deployed thoughtlessly the teachers they work with don't get time to plan with them what they're going to do in many schools because I do insect with schools a lot the teaching assistants are excluded from staff development days because it costs extra to pay them to be there so this comes back to this idea of educational research will never tell you what might be it only tells you what was ability grouping is presumed here to have a negative effect typically the research ability grouping shows that ability grouping gains results in gains for the highest Achievers at the expense of losses for the lowest achievers and typically the losses for the lowest Achievers are greater than the gains for the highest Achievers so the net effect is to slightly lower student achievement but is the work that I did with Joe bowler and Margaret Brown in London schools showed the fact is that some of the things that happen in ability grouping aren't necessarily to do with ability grouping so for example what we saw were two things we saw curriculum polarization we saw set ones going far too fast and teachers saying things to kids like well you should be able to do this you're set one so they were completely ignoring what the kids could actually do they were saying well you're set one kids set one kids should be able to do this and so what typically happened we saw girls in particular wanting to be moved down to set two because they wanted to understand what they were doing and they couldn't understand what they were doing because a teacher kept on going too fast so step one goes too fast set eight and there were some set eights we found tends to go too slowly we're not clear why this is but one mechanism that seems to be important is the teacher's reference group teachers always queue I mean you have to do this to survive teachers cue their lessons on certain students you kind of these are the students you go to to see whether this is working or not and in the mixed ability group there's often the ones in the middle in the top sets it's often the highest Achievers and in the bottom sets it may well be the ones who are low most likely to kick up and make a fuss so what happens is the top set goes too fast and this is quite a general finding going back to the workers Charles de Forge the bottom set goes too slowly so the sets move apart so we say to kids ah well if you work well in set two you can go back into set one but they can't because within a couple of months set one is so far ahead of set two that the kid wouldn't make that transition there's a wonderful episode of The Simpsons where Bart has put a special class to help him catch up and he says let me get this straight we're gonna catch up with the other kids by going slower than them the other thing that we saw was the best teachers were given to the top sets and now of course this research was six or seven years ago now it's more likely to be the middle sets but it's hardly ever the case that schools systematically group kids viability in order to give the best teachers to the lowest achieving students would just be quite a smart thing to do because it turns out that the best teachers have more of an impact on low achieving students than higher achieving students so by definition good teachers are the ones whose kids learn more but the difference is greater for the lowest achievers so you know if school's adopted ability grouping as a way of getting the best teachers to the students who need the most then it might well be different from this result it might be a substantial Plus it's political suicide of course because the reason that parents like ability grouping so much is middle class parents assume their kids are going to be in the top set try saying to a bunch of parents yes we we do we do setting in maths in the school we make sure of course the best teachers are given to the bottom sets do you still want us to set my ability no I thought not I want to come back to this point about feedback so feedback tops the Hit Parade in the education endowment foundations tool toolkit let's look in more detail at the research klugandanisi reviewed 3 000 research reports on the effects of feedback in schools colleges and workplaces this is an extraordinary piece of work they went back to 1905 and tracked down a copy of every single research study on the effects of feedback published in the English language some of them didn't have control groups they were just people tested before and afterwards why is that a problem well without the control group you don't know whether the Improvement was caused by the intervention or by maturation they could have got a better without the feedback so there'd be one group given feedback and another group not given feedback tested before and afterwards sometimes feedback was combined with an intervention like Target setting so if it's successful you can't be sure that it was the target setting rather than the feedback that caused the intervention so they excluded those some of them only had 10 participants fewer than 10 participants there were a couple of studies from medical education with just one or two participants and therefore errors of sampling are significant in other studies performance was estimated qualitatively rather than quantitatively so thus brings in some subjectivity which obviously is a danger in terms of interpreting the results and finally sometimes not enough details of the size of the impact of feedback was given for the researcher to calculate an effect size they found just 131 out of the original 3000 made the cut there were only 131 reliable studies on the effects of feedback that was surprising what was even more surprising was that although on average feedback increased performance the effects sizes across the studies was very variable the standard deviation was actually one and in 50 out of 131 studies giving people feedback made their performance worse this is one of the most counterintuitive results in all of psychology in every single one of these studies feedback was intended to improve performance but in almost 40 percent of them you would have been better off shutting up than giving the feedback so this sounds I think as a cautionary Tale because it is true that on average feedback improves performance but if you don't understand what kinds of feedback improve performance and we encourage teachers to give more feedback because the toolkit says we should give more feedback you might actually be making things worse and this is my concern I I have to say I think the toolkit is a great piece of engaging teachers with research the danger is we are treating teachers in Lawrence stenhouse's memorable phrase like intellectual navies who are told where to dig but not why and the reason that's important is this the navies who don't know what why they're digging where they are can't make appropriate adjustments if it turns out you can't actually dig exactly where you thought you were going to dig and in the same way if you don't understand why you're reproducing an intervention because it'll never go completely according to plan you have to make smart smart adjustments you have to understand which features of intervention are really important and which features can actually be let go if implementation is proving difficult so I want to take you through some of the real problems of the meta-analysis and this is a bit um math C so I just want to point out that typically meta-analysis uses something called a standardized effect size which is the difference between how well the experimental group did divided by you know it's subtracted off how well the the control group did subtracted off that and then divided by a measure of spread in the population and we usually use standard deviation and I want to draw attention to four problems with metanalysis basically this is why you should be very skeptical When anybody tells you they've analyzed 6 000 studies and the answer is X the file draw problem the fact that some studies don't get published the variation in population variability the selection of studies and how sensitive the measures used in the studies are to the effects of the intervention so um with the statistical experiments we typically have a certain effect we're looking for so we have a certain size of effect if you want a big if you want to you know a cast iron proof you say I want to be absolutely sure that this is a real result I'm going to set my significance level at one percent if you're a bit more exploratory you might set it at 30 but the important point is the more strict you are about your burden of proof the bigger experiment you need besides the effect education experiments have small effects there's a man called Jacob Cohen who wrote a book called statistical power for the social sciences and he wrote a book in which he recommended that effect sizes of 0.3 standard deviations were regarded as small and therefore people think that effect sizes of 0.3 are small in education they are huge an effect size of 0.3 in a middle school or a secondary school would be doubling the speed of student learning I think doubling the speed of student learning is worth having and yet according to the statisticians that's a small effect size and of course then the size the effect it turns out that the typical experiment in Psychology has an effect has a power of 0.4 that means only 40 percent of the experiments will produce a significant effect if the effect is as big as the people believe it is in neurosciences point two an education we're not quite sure but it's about 0.4 here's the important takeaway only lucky experiments get published there's a wonderful cartoon by XKCD which shows somebody's researching whether jelly beans cause acne and he tries it doesn't work he has another idea let's try red jelly beans doesn't work let's try purple jelly beans doesn't work let's try yellow jelly beans doesn't work and he tries 20 different experiments with 20 different and he finds that Green Works so he says green jelly beans cause acne but of course that's exactly what you'd expect if they didn't cause acne and you had a five percent confidence interval statistical significance level and you said okay we do 20 experiments one of them will be significant just by chance if you do enough experiments you're going to get a significant result and this is a very big issue in the medical Sciences where you enter this uh as pointed out that a large number of the experiments are actually in the literature are skewed towards the upper end of the distribution so they over estimate the size of the real impact Okay so so variation of variability this shows what one year's growth looks like at different ages so for a five-year-old one year's growth is about 1.6 standard deviations and by the time the kids are in secondary school it's down below 0.4 why does that matter well if we had an intervention that increased the rate of student learning by 50 percent we get an effect size of 0.76 if our experiment happened to get done on five-year-olds but if you did the same experiment with the same effect on 15 year olds we get an effect size of 0.1 so the point is that if you put studies with different age students into the same meta-analysis you're producing garbage because you're your results are uninterpretable okay so studies with younger children will produce overestimates study studies with restricted populations gifted students special needs students will also produce biased estimates because now the population standard deviation is restricted selection of studies this is in some ways the most disturbing um Maria Ruiz primo and Min Lee reviewed 9 000 papers on feedback in mathematics Science and Technology um they found that only 238 were actually worth reading um but then they actually looked at those 238 in some detail and here's what's rather extraordinary they def they looked at what kinds of feedback was involved they produced a very nice typology which I'm not going to spend a lot of time on you can read it later but here's what they looked at look at the first row of the table here feedback treatment is a single event lasting minutes 85 percent of the studies that researched feedback in math teaching and 72 percent of the ones in science feedback event was a single event lasting minutes now that might be interesting psychology professors most of these studies were done by psychology professors on their own students psychology undergraduates are the most researched species on the planet because professors are lazy and students will do anything for course credit so the problem is that if you do meta-analysis and you lump together all these studies some of which the feedback was lasting a minute and others where teachers were changing the way they were giving feedback to either students over the course of the year your results are meaningless and the final one and maybe maybe even more importantly I'm not sure about this is this thing called sensitivity to instruction the fact is that different outcome measures differ in how well they capture what it is that teaches change and so Maria Ruiz Primo and our colleagues developed a typology of what you're asking students to do as an outcome measure so for example the immediate one science journals a close one if the experiment is kids working out how long it takes for one swing of the pendulum a close measure would be how good they are at working out how many per minute so it's a very very closely related measure and so these basically this is a typology of how far the assessment is away from the thing that was actually taught and then they looked at the effect sizes in different kinds of studies and I've used a box and whisker diagram and here you can see that where it's a close measurement close to the to the teaching the effect sizes are much bigger than their their more distant so again the results you get from metanalyzes depend on what kinds of outcome measures are used that's where I think met analyzes aren't very useful and coming back to um what we need to be doing I think Aristotle had a thing to say about this he pointed out that there are three kinds at least three kinds of intellectual virtue one he called epistema the knowledge of Timeless Universal truths that's like the base angles of an isosceles triangle are equal if you proved that yesterday you don't need to check it today it's still true was the ability to make things there's not one perfect table there's a you know but the ability to make tables is an important intellectual virtue but here we go to the highest intellectual virtue as phronesis practical wisdom and what he pointed out is you often get young people who are precocious at episteme you hardly ever get people who are precocious in terms of phronesis you need to know the science but you also need to temper it with judgment flipberg pointed out that in education and other social sciences we tend we need more research focus on phrenesis rather than science we're not going to look at you know what is the right answer here we say where are we going is it desirable what should be done and I think this actually is very useful for Education research I think the purpose of educational research should in the vast majority of cases be to move teachers to more effective action like I said so I can spend the rest of time talking about this now most people think that research can tell teachers what to do Michael Palani in a very important book written in 1958 pointed out that you can in matters of of expertise and quality in particular you can't generally tell people what to do he talks about maxims and he said maxims are rules the correct application of which is part of the art which they govern the true maxims of golfing or poetry increase or insight into golfing poetry and may even give valuable guidance to Golfers and Poets but these maxims would instantly condemn themselves to absurdity if they tried to replace the golfer skill or The Poets aren't maxims cannot be understood still less applied by anyone not already possessing a good practical knowledge of the art they derive their interests from our appreciation of the art and cannot themselves either replace or establish that appreciation and that's the research trap we have for teachers for example it turns out that teachers the good teachers have smooth episodes between transitions in their classrooms and less effective teachers have chaotic transitions between episodes in their classroom do you think telling weak teachers that they need to smooth out the transitions in their classrooms will help no it doesn't too often the refinings of research are descriptions of what good people can already do and the people who aren't very good can't yet do and don't understand and this has been appreciated in the world of business under the role of tacit knowledge and Michael arrows written Lots about this in education and one model I find particularly helpful is from nanaka and takayuchi to organizational theorists in a book called the knowledge creating company they proposed that the most important kinds of knowledge in organizations is often ignored because it's tacit some some stuff you know that you know some stuff you don't know you know so when somebody has tacit knowledge sorry explicit knowledge and gives it to somebody else that they call that combination so I go to a new school I say what time's morning break and they say 10 55 what time does it finish 11 15. I now know what they know somebody else's explicit knowledge has been given to me as explicit knowledge often however that's not the truth of the matter it may be that the official Break Time end is 11 15 but actually nobody really worries about it teachers get up from the staff from a kind of envelope towards their classroom but let nobody expect anything to start before 11 20. and that's something you pick up not by being told that but being socialized into it so other people's tacit knowledge gets picked up by you as tacit knowledge through a person's socialization two other processes that I think are even more powerful externalization when I first started owing PGC students in my classroom I wasn't very helpful because I used to tell them don't just just do what I do it works for me apparently because I never trained as a teacher and I was of no help at all and so through having PGC students in my classroom I was forced to Think Through more clearly what I was doing I don't know whether to help them but it certainly made me a lot better teacher being forced to think through what I was doing through a process of externalization and the complementary process is one of internalization where you take some advice from somebody else and you think you understand it and then sometimes months later it all clicks into place I remember Jerry Hardy a teacher and Christopher in school where I first started tell me I needed to relax I didn't know what he meant until I suddenly realized that I was being relaxed in my classroom I wasn't putting on a show anymore I wasn't putting off a face it was me but the point was I thought I understood it when he told me it I didn't really understood it to understand it until I could do it and so what they Nona Kentucky says that if we're serious about respecting knowledge and organizations we have to nurture all these kinds of knowledge conversion through a process of dialogue networking learning by doing and sharing experience and the traditional approach of academic research is concerned entirely with the kind of knowledge that goes in the bottom right hand corner that's important but it's not more important than the other kinds of knowledge that teachers have and I think a paradigm for Education research is nicely illustrated by this framework provided by Churchman he points out that different inquiry systems rely on different things as sources of evidence so mathematics is the classic example of a leibnizian inquiry system does it is it correct does it follow logically from the from the axioms that's the kind of thing they do in in maths that's the live knitting inquiry system a lockian one looks at observations and and Compares that with Theory so basically in a lock-in inquiry system you make a prediction you do the experiment and then you find out which of your possible hypotheses is correct a Content inquiry system understands that there's no such thing as reality even Verna Von Heisenberg once said what we learn about in our physics experiments is not the nature of the world itself but nature exposed to our line of questioning so basically what you find out is driven by the questions that you ask this is taken further in the hegelian system whereby you try to tell the strongest story on one side of the argument and then try to tell the strongest story on the other side of the argument I often ask people what would have to be different about the world for you to believe the opposite of what you do believe if the answer is not very much then you don't have very strong grounds for your beliefs and the final one is the seeing area in about inquiry system where the Bedrock is values ethics and practical consequences so the Locking Inquirer displays the fundamental data that all experts agree are accurate and relevant and then builds a consistent story out of these the country Inquirer displays the same story from different points of view emphasizing thereby that what's put into the story by the internal mode of representation is not given from the outside but The hegelian Enquirer using the same data tells two stories one supporting the most prominent policy on the one side the other supporting the most promising story on the other side and I think that's a useful Target for educational research I think as researchers we should be trying to Marshal evidence that supports the idea that we want to conclude is what is is correct but we should also be trying to show that plausible rival interpretations of our data are less warranted and we should constantly be looking at the ethical consequences of these education research is not value free there's a wonderful example of this a history student at a university in France was awarded a PhD from a university in which he questioned the historical basis for the Holocaust so what he was doing in his thesis was saying that the available historical documentation does not support the idea that six million Jews were killed in the Holocaust and because it was a fairly good piece of History the PHD thesis was awarded it was then revoked by the state the state revoked the award of the PHD thesis not because it wasn't good history research but because the whole Venture was driven by what they thought was an immoral or unethical purpose and I thought that was quite interesting example of the ethical basis for something overriding the claim to knowledge and the basis for our warrants the assumptions we make and the ethical basis for defending these consequences are constantly open to question so we can make assumptions in our starting points for research we have to do that because we can't get started but we also have to accept that the the starting points that we choose for our research are themselves open to question we should be prepared to research to Define our choices another example from a colleague of mine Margaret Brown many of you will come across the csms research done in the 1970s one of the things that the csms research never did was produce breakdowns by gender they had the data they chose not to publish it because they chose that they believed that if they had the gender results people would fixate on those rather than what they thought were the more important messages about math teaching in general now that's a moral choice what you choose to publish what you choose not to publish what you choose to research is a moral choice that's why I think people in universities who are given time I get through Government funding to inquire into education have a moral obligation to pursue things that are relevant to teachers and schools not because the truth of knowledge for its own sake isn't a wonderful thing it is but the fact is there's a moral basis I think you have to acknowledge and that's why I think that most educational research should be in pastor's quadrant this is a model developed by Donald Stokes where he looks at is it a quest for fundamental understanding Tico brahe didn't bother with a fundamental understanding he just chartered the Stars Thomas Edison was doing applied research didn't care whether the results were used Edison did but Edison was always a theoretical he just wanted to know what he what worked Neil's boar was completely uninterested by considerations of use he was just trying to find out the truth about the nature of matter and nature of the universe but pastor the argument goes is use inspired basic research and I think that should be our aim which we try to work in Pasteur's quadrant so I think the way we wrote we we undertake educational research should be as follows I think all teachers should be attempting to improve their practice through a process of disciplined inquiry I'm not saying that all teachers should do research I think all teachers should be trying to improve their practice because when teachers do their job better their students will be healthier live longer and contribute more to society it's that moral imperative some may wish to write their work up and share it with others some may wish to publish in academic journals some may wish to pursue research degrees and some may even want to become researchers that's fine those are things that teachers can choose to do if they want to but the non-negotiable I think is that every teacher should be improving their practice in their own classrooms that's not research because if the inquiry is conducted in a way that does not transcend the context of data collection then it's not research for me if it only applies in that one teacher's classroom it's not research it's disciplined inquiry I hope but it's not research the role of researchers then is to help and Paul black and I I think would own up to the fact that over the last 15 years informative assessment we haven't been leading practice we have been following behind the best practitioners try to make sense of what they do trying to theorize it trying to present it to others in a way that's more accessible so making helping teachers make their findings applicable in context other than the context of data collection so for me that's the way that research researchers and teachers can work together teacher's job is to get as good as they can at their teaching some may want to do other things the job of researchers is to help understand what really works in practice in different contexts maybe theorize it so it works in different contexts and then share that with others thank you very much [Applause]