Rachel Glennerster on cost effectiveness, and tackling systems issues at scale

Hello and welcome to the RISE podcast series, where we aim to explore the stories behind education, research and practice as part of the multicountry Research on Improving Systems of Education endeavour, funded by UK Aid, Australian Aid, and the Bill and Melinda Gates Foundation.

Hello and welcome. My name is Laura savage from the UK Foreign, Commonwealth and Development Office, the FCDO. Today I'm speaking with Rachel Glennerster, the Chief Economist at the FCDO, and member of the RISE Delivery Board. We'll be discussing Rachel's reflections on how good interventions can work in poor performing education systems, why we need to go beyond evidence of what works to think about cost effectiveness, and how to build incentives to tackle systems issues at scale. Welcome, Rachel, it's wonderful to have you with us.

It's wonderful to be here, Laura.

Let's start off with a question that everyone working in global education wants an answer to. Why do you think that foundational learning levels are so shockingly low and persistently low, in so many countries?

Thanks, it is maybe a bit of a puzzle, but I think it's really important for us to understand how recent the revolution in access to education is. And I think that really helps to explain why education levels are so low, if I think about some of the contexts that I'm most familiar with, in Sierra Leone, and rural Sierra Leone, and there, the vast majority of adults have zero years of schooling. And yet, most children are now in school. So you've seen this complete revolution within the space of, you know, a generation. And so you are trying to introduce this huge scale of education in a country and in communities, where very, very few people in the community are educated. And so that means it's hard to find teachers who are, you know, who have a lot of levels of education, it means parents aren't able to provide the support to people, you know, children just don't see it around them. And also, you're building an education system in a very, very short time. So maybe in that context, it's not so surprising. I mean, we now have levels of children going to school in the poorest countries in the world, which were only achieved in Europe, in the 1940s and 1950s. So, you know, it's really a very dramatic change that we're seeing. I think the other reason is that education systems, as they are implemented in many poor countries, are kind of direct copies, often, of systems that come from very, very different context. So too many countries sort of cut and pasted an education system, from, you know, the UK or France, who had designed their system for a very, very different context or with, you know, you had parental support at home where you had children coming into school, who had, you know, many of whom knew how to read by the time they started school, they weren't trying to cope with a third language. Often in countries we're talking about their people are being taught in their third, like second or third language. That's not what happens in the UK for most children. And yet, we sort of cut and pasted the system from there. And thirdly, pedagogy is just often very bad. So we just the ways in which people are teaching is just quite bad, really, which I think partly reflects the first two problems, but it's very fundamental.

So RISE has done all sorts of sophisticated analysis of learning levels. But after this question on why foundational learning levels so low, the next question that everyone policymakers/practitioners ask is, what should we do differently? How do we drive learning levels upwards? You are a founding member of the Global Education Advisory Panel, which last year released a report on smart buys in education. This sets out a list of interventions that have been shown to be cost effective at raising learning outcomes. Now, some would say that the smart buys are the antithesis of a systems approach to education. Can you tell us a bit more about the story behind the smart buys, and why you actually think that they're complimentary with the systems approach.

So the smart buys work came out of the desire to help countries, you know, with very practical solutions about what they could do to the learning crisis. And it came out of the observation that there's now quite a lot of evidence about what's effective and improving learning. But the most cost effective thing is often 100 times more, you know, 10, 50, 100 times more cost effective than something that works. So simply saying does something work or not, is not enough, because we have limited education budgets, and we need to get the most impact from those education budgets. So as I was looking at the data and seeing that there was this huge range of cost effectiveness. And we had started working on that question in the FCDO, DFID as it was then. And, you know, we were also talking about it with the World Bank and they had a similar attitude. But they said, well, you know, let's bring together a group of people who really know the evidence and can give sort of really authoritative guidance on piecing together all the different evidence, and there have been some discussion about, you know, different literature reviews sort of pointing in different directions, so let's get the real experts in the room about the data. And we also have implementers in the group, you know, people who know the politics on the ground, so a lot of real different perspectives in the group. Now, every single person in that group believes passionately that the system, education systems, are incredibly important in making change. And we had lots of discussions in the group about how the individual, the evidence from individual studies, shows just how problematic the education system is. So I'll give you an example. A lot of the most cost effective things that you can do to improve learning, try and fix the problem that the curriculums are, as we were talking about before, often kind of cut and paste from much richer countries in very different contexts. And, therefore, a lot of children are far behind the curriculum. So if you look at all these studies, you say, you know, yes, there are individual things that you can do to fix the problem that children are far behind the curriculum. But the real answer is to do something about the curriculum, and make it you know, at the moment, it's often just completely unrealistic. And teachers, just kind of overwhelmed by giving, being asked to do something that's completely unrealistic. So all of us, you know, looked at that evidence and said, these are specific things that you can do to fix the problem. But it's all pointing to this systems level problem, that the curriculum is often to, you know, unrealistic. So that's part of the complementarity, and why in the smart buys paper, we spend quite a bit of time talking about the systems and the importance of the system. The other point is that these are all our recommendations are things to do to improve foundational learning. But nobody is going to do those things unless you have a system that cares about foundational learning. We can recommend what to do if you care about foundational learning. But if the system doesn't care about foundational learning, nobody's going to take those recommendations. So that's, that's an another example of where there's complementarity you need. You need incentives in the system for people to worry about foundational learning. And if you do, then here's some specific things that you can do about it. I do want to just add, though, that sometimes people take away the assumption that if you don't fix the system as a whole, you can't do anything. And that is just not supported by the evidence. There a lot of these individual interventions that were tested in, frankly, quite dysfunctional systems. And yet they still made impressive gains. All the individual evidence is pointing to the fact that there's a huge problem at the system level. Improving incentives at the system level encourages you to do the end of fictional interventions. But don't despair, keep working on the systems. But don't give up if you can't change the system, because there's still specific things that you can do.

So you talk about fixing the system, as the bigger piece that needs to be done. And again, thinking of the next questions to be asked. The follow on is, well, what do we do when foundational learning isn't the primary motivation or when we come up against the smart buy that says, let's reform curriculum so that it addresses the issue of curriculum being over, overly ambitious, you then need to also think about teacher training, the content of teaching learning materials, and it just starts to feel overwhelming very quickly with the size and scale of the issue? I'd like to ask you a question about whether you've come up against these big system, public policy issues and some of your other research. If you think about the breadth of other research that you've led, or you've been involved with, what are the top three things that you think shape whether an intervention is going to work or not to produce intended outcomes? And how then can we avoid these pitfalls of the sort of the intervention, individual intervention, being effective at one scale, but then not working at large scale? Or indeed, not then interacting with or affecting system change to the extent that that these questions around intended outcomes and massive system scary system change, go away?

I think the main thing is, you've got to design interventions that are realistic about the practical constraints on the ground. And that, frankly, is one of the biggest failings of development work, not only by outsiders, it's not just a donor thing. I see it actually just as common when I see, you know, governments or local NGOs, you would think they would be more aware of the constraints, and some of them very much are, but it's a cross the board problem, that people design interventions that just don't really, aren't realistic about the constraints on the ground. Let me give you you know, I work very closely early on with Rukmini Banerji from Pratham, who's, you know, been a huge influence on my thinking about education and, you know, really treasured the months that I spent with her on the ground in schools, literally sleeping on floors, travelling rural India with her. And what she said in some of those conversations was, we have to design something that copes with, not very good teachers, is robust and will work even if the teachers aren't motivated, even if they're not highly qualified, because that is the reality. And so I think that that's an example of designing something that really understands the constraints on the ground. The second thing is, if you're going to have a realistic chance of having impact at scale, it has to be cheap. Because frankly, when I've seen things scaled up, people cut the budgets, they never spend as much per child. So you need to design something that is cheap, or just not going to be scaled. And that's one of the reasons why the deworming is one of the things that scaled up so, so rapidly, it was just cheap. And it was also pretty easy, you know, it's not easy. There's lots of complications, but it's a lot easier than a lot of other things. And thirdly, you need something that doesn't rely on really good people to implement, which is sort of, you know, related to my first point, but I pull it out because it's so important. A lot of the things that are tested at small scale, you know, rely on some incredibly passionate, you know, revolutionary, insightful person. And it's fine if that person designed something that can be implemented by your run of the mill persons. But the implementation cannot rely on having inspirational people in every school because you just won't get that. So those are the three things that I look for. Now, in terms of what, you had a lot of things in your question or in terms of what I've learned about, you know, sort of, are there things that you can do that change the system, I guess, I think in two different directions. One is, it's remarkable how if you can demonstrate change, you can demonstrate that actually, if people do this, there will be a real change, that you can bring people along with you. People get very cynical about politicians. And, you know, I can be cynical too about people who are implementing, you know, people making decisions about budgets. But if you can, if you can prove that you can deliver at scale, a real benefit to their constituents, and it's not going to cost them very much money, they're actually pretty pro. If you can demonstrate you can change foundational learning, right, it's not going to solve all the problems that they mainly care about the elites. But if you can demonstrate that there is something you can do about this problem that they've been facing for a long time and kind of feel is just, there's no hope, you do actually get people to respond. And the second direction, which is where actually most of my research lies is other things that you can do to change the view of the politicians and like politicians will be responsive to, which is kind of a bit outside the educational sphere. But that's the stuff that I do to really try and change systems. And, you know, there's a whole literature on that. And I'm actually quite excited about what you can do to make politicians more responsive. And we ought to be testing that and I am.

So my next question for you was going to ask you to change the lens and shift perspective to apply those reflections to your current role. And perhaps leaving aside the final point, about politician responsiveness, we'll come back to your future research focus. But the points you've raised here around addressing realistic constraints and inspirational people and demonstration effect, in the last three years, as you've been Chief Economist at DFID, and now the FCDO, has any of your work in this time on the various programmes, and indeed, the trends and practices within the FCDO, do any of this challenge your priors challenged your research findings on how public policy systems work?

So I think there have been two ways in which it has changed or reinforced or, you know, I think a bit differently about things. So first, I have to say I was very impressed when I arrived at DFID, just how much people knew about the evidence. So, you know, I came in saying, "Well, I know about the evidence, I'm going to kind of teach people about the evidence." And frankly, there's a lot of people who know the evidence very well. And I think the thing that I learned more about were the barriers between what people knowing the evidence and actually being able to implement programmes that really draw from that evidence. And I knew that that was a barrier already, but I kind of learned a bit more about that. And that and I, and I still puzzle a little bit about it. So why does that not always translate? I think one of the things that doesn't always translate is that people see the evidence somewhat compartmentalise so they see, you know, this, this programme that works. And if I can implement that programme, great if I can't, you know, this, this research is not relevant to me. And what I've really worked on trying to get people to do is think about there's this research base and you need to pull from it more generalizable principles that you can actually apply in a much wider range of contexts than just, you know, lift and shift this programme. And people get that, but then they don't always know how to operationalize that. They still tend to kind of reach for the comfort of, well, there's this programme, and I can try and advocate that it happens in my country. But if that doesn't happen, then I kind of haven't got anything to learn. So it's stretching that okay, what are the implications that I can apply to something which isn't exactly this programme, but I still have something to apply? You know, so like that the curriculum point we talked about, which is taking the fact that all the different studies that are most effective are all about the fact that children are behind the curriculum and saying, well, therefore I need to do something about the curriculum. And that sort of leap, I think, is still quite hard for people. I think, frankly, also, it may be easy for me to say, but I think more junior people tend to know the evidence best. And they find it hard to push back against more senior people who don't know it, and just kind of make assumptions. And as long as the senior people are so ingrained in what they want to do, that's just their assumption, and then they haven't really had time to think about it. And I think one of the roles that I've played is, be the advocate. So those more junior people to say, "No, no, like, actually, you're wrong. And and we shouldn't do it this way." And I, and I hope I've helped some people who were real experts kind of speak up and try and make their voice heard. I think the other thing I've learned about is the institutional incentives that prevent us working at scale. So we are a huge organisation with a lot of money, I mean, not compared to government systems. But I'll compared to anything else, that any budgets that I've been involved in before, and some of our programmes, you know, spend quite a lot of money. And yet, there's a real incentive to do, make schools in a small part of the country really good. Rather than improve by 10%, or a small, you know, a smaller amount of improvement over a much larger range. And even though, you know, if you added that up, that would actually improve things a lot. And the institutional, there are a bunch of institutional incentives that push you in that direction, which, including, you know, if you slightly improved education, and a lot of schools, you can't see it, it's not visibly obvious. So it's not rewarding for you, it's not obvious to your partners that the schools the UK is working in are so much better. Because they're not, it's just a very large number of them a slightly better, you can't take a minister to a beautiful school, because there aren't any beautiful schools, they're all just slightly better. And the partners we're working with on the ground, so the NGOs don't like it, either, because they also can't take pictures and take people around a beautiful school, because they're not beautiful, actually slightly better. So that's the thing, we need to introduce some incentives that will really push us towards scale. And that's the main institutional thing I've taken away is just how ingrained the incentives are, to work in a little patch, and try and make it beautiful, rather than try and improve the system as a whole, like I try and, you know, slightly improve outcomes for every school in the country.

So bluntly, what could some of those things be to shift those incentives? Because taking the taking the long view and the big picture of aid effectiveness and the role of donors and recipients, what is it that you know, in this cyclical world of efforts, tried and new things that that come on the block. RISE represents in a way one piece of that sort of the new word on the block, systems. And systems thinking, systems approaches or the the language that we're talking about the new paradigm of aid, the new paradigm of development. What within this echoes with you, that could be some of the is any of this and applicable to, to addressing some of those incentives that you talked about?

So I pushed very strongly that if we have targets, for education and for girls education, they should not be about what we as FCDO do and how much girls are learning in the schools supported by our FCDO, but how much they are learning in the whole country. And that forces us to think at scale, because otherwise we're not going to bump those. So I very deliberately argued for a target that was at the national and international level, because you're not going to make a dent in that. If you do 20, 50 schools beautifully, it's just not going to change those numbers. The only way you're going to change those numbers, is if you try and work at changing the national system and getting pedagogy changed in all schools. So that was my contribution. And I argued quite hard for it. It's, it's hard. And people will want to default to attribution and being able to say, "Well, what did the UK do? What did France do to deliver this target." And when you start doing that, you start going back to, "Well, what what's happening in the schools that we specifically funded?", which encourages us to put our money into specific schools rather than into systems. But we have been successful in getting the UK and now the G7, to measure their success by looking at the total, not their own. And so, you know, that's a big win. So the G7 have just agreed targets for girls going to school, but girl's learning. So again, that was a big discussion. But that's that we achieve that and I'm very proud of that. And I hoping that will lift everyone's eyes to the scale issue, and really have an incentive to focus at scale. Now, there are intermediate indicators that we're tracking that try and get a little bit about attribution, but they are not so much what's happening in our funded schools, but in are we getting people to do the things that we think will change things at scale?

You're do to move on from the FCDO at the end of the summer? I don't quite know, what we're going to do without you as a champion for education. But can you tell us a bit more about what's next for you? Particularly, I'm keen to hear about what research direction. You've mentioned the, the responsiveness of politicians. Is it that and more?

Yeah, so I am going to start at the University of Chicago in August, very excited about that. I will be teaching and doing research. I will continue to work on education, I think, you know, frankly, I'm so busy trying to wrap things up in my current job that I haven't spent a huge amount of time working on thinking about what's next, other than my teaching, which I don't have to. But I do want to keep going on the cost effectiveness work in education and the drawing out the broader implications from the research and also work on scaling up so I I'm already involved with Teaching at the Right Level, Africa. You know, mentoring the team there, and they, so I will get more involved in that I'm sure. And that's really exciting work bringing the pedagogy that was has been so successful in India and has been tested in Africa, but really bringing it to scale. So really talking about exactly those scale issues that we've been struggling with them and, you know, measuring at scale and all those challenges. I would love to get more involved in trying to work in Pakistan where I think there's huge opportunities for a lot of great people on the ground in Pakistan, but less scaling of the evidence and there has been in India but equal need and, you know, fantastic people to work with. So, but a lot of my other research has been about information and how information can change behaviour. And you know, information as a route to empower women. I have a long run project in Bangladesh, following women who were involved in an empowerment programme and also had an incentive to delay marriage. So we have exogenous variation, so literally randomly induced variation in the age at which girls marry in Bangladesh. And so following those girls, they/re now women and age 25. They're having kids so we can see whether how much, what is different for the lives of a woman in Bangladesh, who married later. And that project, frankly, I hope will continue as long as you know, for many, many years, because there are so many questions to ask about what is different about women who are empowered as adolescents, and also women who, who married later as adolescents and avoided child marriage. I've done quite a lot of work, as you were saying on information on politicians. I don't have a specific project on that. But it's one of the pieces of research that I am most excited about. I still have a project that we did on the last election in Sierra Leone that we haven't written up. So I need to write that up, but it's all related to this, you know how information can improve accountability. And so we held debates in Sierra Leone. And then it's really the politicians speaking in their own words, and actually changed how people voted. In a country where everyone will say that people just vote on ethnic lines. And yet hearing a debate between politicians changed how they voted. And at the last election, we looked at demand for information on politicians. So can you you know, can you kind of set up a market? Will people walk? Will people pay to get that information? And the answers are quite encouraging. So there's just a huge vein, very rich vein of work to be done on improving accountability. People ask about, you know, does democracy work or not? Or is it changing? And that's not the question for me, the question for me is, how do you improve the effectiveness of democracy? What can you do to make democracy work better? And I think we're seeing that there are practical things that you can do. That's the thing that brings everything together for me. I want practical solutions.

It sounds like there's a future conversation with RISE and applying some of your research to the questions of information and accountability at the at the core of the RISE work. So although you've been on the RISE Delivery Board, for the last few years, now I have a feeling we'll be we'll be calling you to come back and continue that conversation.

I had a whole project set up in Sierra Leone about improving accountability in education from local counsellors with information, and Ebola came in and shut it down and we've never restarted it. So yeah.

There is yet time, there is a time post pandemic. We have a final question which we ask all guests on this podcast. And it's a simple and difficult one at once. What is the one thing that you wish people knew about education systems?

The main thing I wish people knew about education systems is this problem of just how big a gap there is, between what the education system is pumping out in terms of what children ought to be learning, and where they actually are. Just that gap between where the system assumes they are and where they actually are. If I could, you know, my second one is, and there's something you can do about it.

What a wonderful point to end on. Thank you so much, Rachel.

Thank you for listening to our podcast today. And if you liked it, be sure to check out our research at riseprogramme.org or follow us on social media at RISE Programme. You can find links to the research mentioned and other workshops and the description for this podcast episode. The RISE podcast is brought to you by the Research on Improving Systems of Education (RISE) Programme with support from the UK Foreign, Commonwealth and Development Office, Australia's Department of Foreign Affairs and Trade, and the Bill and Melinda Gates Foundation.