Education Research - From Systems Thinking to a Science of Implementation

RISE Programme: 00:04

Hello and welcome to the RISE podcast series, where we aim to explore the stories behind education research and practice as part of the multi -country Research on Improving Systems of Education endeavour funded by UK Aid, Australian Aid and the Bill and Melinda Gates Foundation.

Julius Atuhurra: 00:23

Hello and welcome to this episode of the RISE podcast. My name is Julius Atuhurra, a Research Fellow on the RISE Programme. In this episode, we bring you a recording of a panel conversation that happened at the University of Oxford's Blavatnik School of Government during the RISE Annual Conference in September 2023. And for purposes of clarity and length, this podcast is an edited version of the conversation. The panel featured Nompumelelo Mohohlwane, also known as Mpumi, from the Department of Basic Education in South Africa, Rachel Hinton from the UK's Foreign, Commonwealth and Development Office, and former RISE Research Director, Lant Pritchett.

Julius Atuhurra: 01:04

This conversation was moderated by Laura Savage from the International Education Funders Group. In what indeed turned out to be a great fun conversation, as Laura had predicted from the very beginning, the panel started by looking back at the questions that existed at the start of RISE and whether enough was now known about those questions 10 years later. This led into a deeper reflection on the difference between the motivating questions for RISE and the What Works Hub. The panel then debated what a commitment to learning really means and what cultural shifts are needed for this commitment to materialise. And connected to this, what implementation science really means. Finally, they concluded with a reflection on how the thematic shift from systems to implementation will likely play out at the What Works Hub in days ahead. I trust you will have great fun listening to this conversation.

Laura Savage: 02:02

Hi everyone, my name is Laura, and I've got a great fun conversation lined up. We're really going to talk about the shift from systems research to implementation science. And I think it is worth just stopping and reflecting for a moment. And I think there were a number of us who sat around the table 10 years ago. The questions that are being asked by this program are the ones that we're still asking today. I say that and yet, are they? Because some of the questions I heard today are still the questions that we were asking at the start of RISE. So, the start of RISE was going back to Tessa (Bold), Justin (Sandefur) and everybody else's paper thinking well sure like an NGO can use contract teachers and the government can't? Like how is that not working or seemingly obvious things that are done in education that don't seem to be working either at scale or just as Paul (Glewwe) demonstrates again and again, at all. And I think there are still some of those questions today. We're hearing a lot of a lot of organizations, people, researchers, practitioners still searching for this seemingly invisible impossible thing called scale. But there are a lot of questions. I mean I heard today even things like we have suffered through, this is Renata (Lemos) saying, we've suffered through the implementation. Baela (Jamil) you're saying were things designed to fail today? Ricardo (Estrada) how do you create the political environment in which actors are going to support what's been designed? And I think those questions were there at the start of RISE and one of the goals of RISE was to enable everyone, you know, enable us all to take a step back and think about the system.

The word system was very core to RISE at the start. So, the conversation today is going to go through that journey of 10 years of questions--no pressure--and look ahead to what comes next and we've had a few hints of that today.

It's going to be a conversation. I'm going to kick off with some questions and then we'll come out and extend the conversation after that. And really Mpumi, I want to kick off with you because as many of you will know Mpumi’s got like a dream job. It's not the easiest job in the world I can imagine, but one that really is connecting research generation to life. And maybe just as a sort of baseline, where are we then? My first question to you is, what are the questions in your and your colleagues' day jobs that you still don't think you've got answers for?

Nompumelelo Mohohlwane: 04:30

Too many. So, I think firstly we have a sort of formal research agenda that we've solicited across the department to ask questions on different aspects. So, literacy, violence, parental involvement, nutrition, etc. So that's a public good that's aligned with the medium-term framework. So that's like the government plan, five-year questions that's sort of aesthetic and publicly available, but I think aside from that questions that still remain, after all is said and done are some very specific ones and broader ones.

Some of the specific ones are still after participating in years and years of international and regional assessments, we still don't know what children know in the early grades. So, I can tell you that 81% of children, according to polls, can't read for meaning. I can't tell you anything beyond that. So, what do grade ones and twos and threes actually know and then how does that fit into strengthening the curriculum? So, there's still questions around that.

We still have big questions that have largely remained ideological around language. When should we be switching languages of instruction? Is the current policy appropriate or not? If you look at the African context, there's a lot going on, contradicting stories. Lots of countries switching to English too early, in my opinion. UNESCO, which told us to switch in grade three, years ago. And then current questions around, are we switching too early? Those questions have largely remained ideological. They're around politics and who's in power and not really the evidence of when to switch. So, there's a bunch of those. Then I have a second level of questions around when you are designing for a second intervention. So initially we had these RCTs. How do you teach reading for meaning for the first three grades? But how should we be thinking about teachers now? They've had some intervention. Do you have the same dosage, same depth? Do you come back in one year versus three years? What are the impacts you expect to see on the second cohort and the third cohort? So, some of that. And then some long -term studies. Where are the effects of these early interventions that we've had? Do you see them later and maybe a last version, which is a slightly higher level? How do you get things like the Auditor General, who is, I guess, a watchdog of the state, to not just measure inputs, but to measure outcomes? And how do you scale that up? How do you get treasury to be driving, delivering outcomes instead of telling us to cut teacher budgets? So those are still bigger questions than literacy or numeracy.

Laura Savage: 07:19

Sometimes I think the word system and the system's challenges make it feel like everything is a problem. And then how do you drill-- I mean, because the answer to any one of those questions feels like it's a research program in itself, right? But why is it then that you think you're not getting the answers?

Nompumelelo Mohohlwane: 07:35

I mean, some are, like I've said, ideological. Some are around what evidence exists. So, you have a lot of diagnostic work, but not a lot of interventions that are measured. So, you know what would work at scale? And some of them are around political economy. They're not around education, actually. They're about the systems, how decisions at a broad level get made, what are tradeoffs? For example, we have a presidential youth initiative that's massive and focuses on employment but is not focused on education. But we do have real unemployment issues in South Africa. So how do you decide whether you want to keep more young people employed or you want to halve their numbers and use that money for education?

Laura Savage: 08:16

So yeah. And Rachel, that-- I mean, it makes me think a little bit of some of the conversations that we had, and for those who don't know, I was at then DFID at the start of RISE and Rachel was my boss for many years. I won't ask any incredibly difficult questions. But one of the things at the start of RISE was not only about what the questions were, but who should be answering them and trying to bring people from different academic disciplines into looking at education. Economists obviously being one large cohort but also some of the really exciting stuff that I've loved through rise has been some of the anthropology, the public admin, Mark’s (Moore) stuff you know on innovation like there's been some really interesting angles coming at education problems and you at the time had a graph that set out this is the state of education research.

My feeling today is we don't have as a global education community a view on the state of education research today but given that you were the kind of author of that of that graph where do you think we've come from then and I guess a little bit about what's RISE's role in all of that been.

Rachel Hinton: 09:26

Thank you and maybe the clever IT could put the graph up for you because I think you know we actually have made incredible progress and sitting here with Lant I think it's actually testimony to an economist who was prepared to listen to the anthropologists, you know, Yamini (Aiyar) I think is probably one of the people you often talk most about Lant and I think that your convening of an interdisciplinary group has been quite phenomenal and I think it genuinely has shifted the game and I think Laura also we, you know, we had at the time on the axis at the bottom, we talked about quantitative research and that was you know mainly economists but there weren't so many of those. They were, you know, there were fairly few studies at the time, and I think one example of how far we've come is actually if we think about the GEEAP panel, you know, the Global Education Expert Advisory Panel and the Smart Buys report. Just in the last two years between the last report and the one we have just put out, 250 new studies that are in there. Forty-seven thousand downloads of those reports, I think it’s quite phenomenal that we’ve really grown that body. And I think, you know, the other axis from micro to macro we were mainly doing lots of micro studies, particularly the qualitative side, we weren’t really talking about scale, we weren’t really looking at things through government systems and I think all credit to the RISE community and all of you who have been part of that trying to really shift how we do that and how we look at that and I think that’s been really key. And another indicator of that, do you remember, and I think we have Christine Beggs in the audience here and I think we’ve got Luis Crouch, US Aid and the Bank at the time, and the UN, there were just four of us saying we needed to increase the quality of evidence. And Maria, who is here somewhere was coordinating it all, Maria Brindlemeyer, over there. Four members, it’s now 40 members, Laura, and 6 interest groups because people aren’t content with just having the general panels, they want to deep dive into the topics, you know early childhood learning, teacher professional development, climate, education in emergencies, and so on. So, I think that’s been one area. And what I also think has been fascinating has been the evidence uptake and what’s happened there. So, RISE has shifted the narrative and I think that’s partly because of the different voices. The WDR 2018, I think the fact that the RISE conceptual framework has been so influential in some of these and thinking about learning trajectories. And for many, they were talking about access, and they weren’t talking about learning. And we can forget that that about 10 years ago, particularly donor driven programmes weren’t thinking about what happened inside the classroom.

t was really telling that the: 2020

Laura Savage: 17:26

Thank you for that. Lant, I would like to pose a similar question to you. And Lant you’ve said that from the start of RISE and many years before the global community has known that learning has been an issue, that learning has been embedded into the discourse and narratives on education, no one intended for access to take over. But the starting point of mine and the original business case for RISE was we don’t know enough about what works to improve learning at scale. Do we now?

Lant Pritchett: 17:56

So, I’m going to ignore that question. For a second. Because I was thinking about how to illustrate kind of where we are on the system versus implementation. And I want to tell the story of an unrequited love of my youth, which was with a car. I love the Ford Mustang. I was born in 1959, the 1966, '67, '68 Ford Mustang was the dream automobile. You know, Steve McQueen drove a Ford Mustang, it was just. So, and the Ford Mustang was the epitome of cheap oil. And no concern about emissions. So, to make a fast car, you just made a huge engine, right? And then all else was like around that. Then when I was 14, so just about ready to drive, all of this relationship with the Ford Mustang was imaginary, of course, to a teenager. The OPEC shock hit, price of oil, quintuples. In addition, the environmental movement, all of a sudden says, you know, our air is really, really dirty from all these cars. And so, the automobile manufacturers were under enormous pressure, including legislation, to increase miles per gallon, increase fuel efficiency. So, in 1975, my older brother's best friend bought a brand new 1975 Ford Mustang, and it was a piece of ****. It was the betrayal of my years of lust, because--this is a little too personal, I guess, but anyway—but you know, because the way the Ford had solved the problem of the trade-off of having a new objective of miles per gallon was they just put a four-cylinder engine in a Ford Mustang. But without really a fundamental rethink, without any improvement in efficacy in the way the system was designed, and so it was just, it was underpowered. It wasn't fast. It wasn't quick. It wasn't sexy. It wasn't anything. It was a piece of crap. I am still, as you can tell, bitter about Ford Motor Company having betrayed my love.

Now, what's the point, you may ask? The point is a lot of what we're hearing is on the assumption we can just drop a modular component into an existing education system and produce a better system.

And that, I think, is the overall—now I'm getting back to the actual question you asked, the motivating question—and I think the word scale gets people confused, right? The motivating question of RISE, it was, why do some education systems, as a result of the routine operation of the system, endogenously produce excellent learning outcomes and other systems don't? Now, that is intrinsically a question of scale, but it's not the question of how do we take this particular known known thing and scale it in the system.

It's the opposite of that. It's how does the system come to produce, as again, as a routine operating, as the routine endogenous outcome of the way in which the system operates, how does it routinely identify, assess, design, and implement programs in the system. So interestingly, as yet today, no one's used the word Vietnam. Now Vietnam is a really interesting word because Vietnam achieves OECD levels of learning roughly. At roughly the cost per pupil, at least in the past before education expenditures expanded rapidly in Vietnam with its economic growth, it was producing OECD levels of outcomes at roughly equivalent expenditure per pupil that many countries today already have. And so, the way the question is like how the hell does Vietnam do that, whereas other countries with similar expenditures don't, right? So, I think we are much further along in fleshing out what the functional characteristics of systems are, but they're not unique mappings of one -to -one of design characteristics, right? It's, you know, functional systems do these things, but that doesn't elicit the granularity of how do these things get done.

So, I think we're more towards this. But when we talk about implementation science, there's two radically different interpretations of that. Right? One is the science of how do we get governments to implement what we think we already know they should be implementing to achieve desired outcomes. And that inevitably leads to the deliverology type approaches. And the other is, what is the science of systems that, again, work such that they implement well and achieve the outcomes we want? And those are radically different approaches. I was in a conversation where we're often in a situation where we've had massive progress in science, but we don't actually have a science of progress because we don't actually have a science of how does progress happen?

Laura Savage: 25:06

But I think, in a way, the same thing has happened with systems research. And the word system has been interpreted and misinterpreted in so many ways, I think, from the test last year to all these documents. I mean, almost every strategy now from any funder implements the language, the zeitgeist language of the moment, is systems. And it really wasn't 10 years ago. I'm not going to blame RISE for that, but—

Lant Pritchett: 25:30

Oh, I mean, I'm happy to blame RISE. I mean, the question is, yeah, I don't know. I mean, you've got to get people to use the word. I don't know. I-- this is a big question of a science of progress. Do you get people using the word system and then gradually get them to understand what the hell they're talking about?

Laura Savage: 25:48

But see, I think systems change has been interpreted as your first example, that systems change is to do something we already interpret and therefore need to mould the system to do. And in a way, I've had the same question from—as you, RISE. You and the RISE team have been evolving the idea that started quite early on around coherence for learning and recognizing that purpose was going to be key and that commitment, the commitment of a certain number of actors in the system was going to need to be there. My questions early on were, well, how do you achieve that coherence because I can see a way that that would also be interpreted to be from an external actor or indeed various internal actors within the system with more right perhaps to say, well, how do we create it? And I think that is a sort of, there is a, there is a, I still have today a real question. Can we create coherence for learning? Can we create a sense of purpose in the system because who's going to tell any teacher or district education official that they don't have purpose?

Can we create commitment not only to a certain set of policies but also to the implementation? Is it possible to do that? And do you think RISE has helped us understand how to not have a future set of actors running out exactly trying to create that in wrong ways?

Lant Pritchett: 27:13

I mean, what do you mean ‘we’? So, when you ask the question, can ‘we’ do X, Y, or Z, like who exactly is we? And I think if ‘we’ as people in this room, the answer is for sure, absolutely not, right? If we ask, can we, people in this room, contribute to assist at the local, national, state level, can we help people create this? Probably with a program of research and understanding what constraints they face. But, you know, ‘we’, this is where the challenge of the What Works Hub and implementation is just radically orders of magnitude more difficult than RISE. Because RISE was a research program. And we produced research. And we did a good job of producing research. But producing commitment is a really challenging thing. And you have to say, you know, no amount of commitment from the global community is going to create commitment in South Africa or commitment in India that has to be created locally.

Laura Savage: 28:39

Mpumi, what's your take on that? Is there commitment to learning in South Africa? Can you create more of it? And is there, I guess, building off this sort of separation of a commitment to the goal versus commitment to the implementation that the last panel talks so much about,

do you see distinction? Is there commitment to one and not the other?

Nompumelelo Mohohlwane: 28:59

I mean, I think there's a commitment to learning. I think I'm yet to find what people have in their mind of when you think of a dysfunctional system. It means you stand at the gate, and you just see everyone's running around in a school. People are sitting in staff rooms, children at home. That's not what you see in a typical school. You walk in there all day. They're doing something. I mean, I don't think they're doing it to just pass the time. I think there's a different idea about how to do it better, et cetera. And that can be shifted. But I think Lant is right around how you shift that.

And I think we have maybe two examples. One is around a design failure, which Jacobus (Cilliers) and Stephen (Taylor) were part of. We used to call it informed and empowered when we started it soon ended in three months and it was uninformed and disempowered when we renamed it. And it was essentially this idea, global idea, give schools information, give parents information, they'll make different choices. No, that's not, was never going to work, we learned. Did a little qualitative study quickly early, the principal was happy to convene parents the first time and then once we presented the hey your school's doing worse than the school next door, you could be doing better, you have the same inputs etc. The school principal just didn't call them back like and that was the end, finished. Does that mean that school principal wasn't committed to learning or were we using levers that didn't build credibility for the school principal and for what they may perceived our mission to be? I think that's probably what happened there. Where's a different example where we've had a successful intervention that worked, we did it with an NGO, we had training at the beginning of each term,

it happened on time as planned. The current version is we're scaling up with a province and training happens but there are lots of constraints and we've learned to live with that, that is the system and so the budget constraints, the time cycles, academic time cycles that clash with financial time cycles, that clash with the other district programs, do I think that means they're not serious about learning outcomes? No, I think they are serious about learning outcomes but it's around how do we take our programs or ideas that we think work, get by in but also be flexible about how they get actually implemented and so I'm more keen to see the results from that rather than this external everything's perfect but you actually are not in the system at all kind of work.

So yeah, I think there is a commitment, what it looks like, how you map it out, how you measure it, its timing is different depending on what level of the system you're working at.

Laura Savage: 31:49

But you say that is the system, it is full of constraints, how in the design of something like that and then the kind of testing of the early stages of it, how do you work out which of those constraints you can move, which of them are movable and, yeah.

Nompumelelo Mohohlwane: 32:05

I think that's the, in our case, the advantage of being in the state, that we have a better understanding of who the various players are. And there's some trust that they have around sharing what the constraints are. And it's just really engaging them on what they think is going to work. So, I mean, it's even things like, yeah, engaging them effectively. I mean, the curriculum policy says you should be teaching a 12 -week term. If you speak to the actual subject advisors, they tell you you're wasting your time. You should probably aim for eight to 10 weeks, because in reality, they are assessment week, they're funerals, choir competitions, etc. And I mean, you could do the policy thing and have a 12-week curriculum that never gets taught. You could argue with them, let's make it 10 weeks so we can do a bit of extensions. But them feeling heard and accommodated in the curriculum means that they're your partners, rather than in opposition to your curriculum. And they're actually implementing the curriculum, rather than the design curriculum. Someone was presenting design versus implemented curriculum. So, yeah.

Laura Savage: 33:18

So, I mean, I don't hear a huge number of examples around the world of one really strong listening and kind of identification of various different people's views of what the problem is, but then also not policy design that tests it, brings it back and says, actually, after three months, as you say, disempowered, we're going to change the name, we're just going to stop the whole thing.

Rachel, turning to you on the challenge, as Lant put it, the challenge of the What Works Hub and implementation science, is that how implementation science looks like? You know, to enable governments and practitioners to pilot something, to listen first, to pilot something. So, it's the whole old kind of PDIA, Test, Learn, Adapt model.

How, I guess, two sort of sub -questions. how can a research—mostly research programme—come in to support the enabling environment in order to have that happen, and if it was so easy and Lant, has said it's not easy, then how has it not happened more already?

Rachel Hinton: 34:23

Big questions. I think let me start with the first because I'd like to suggest that perhaps it's not just about the top-down trying to create the commitment, but perhaps it's a totally different model we need in terms of the bottom up, and another little story, surprisingly, sometimes the most unexpected things happen in a building like this. The third floor coffee room at BSG, always good for a bit of extra caffeine, a bit of boost of the concentration, but sometimes also surprising things can happen there, and I had actually just been having a conversation with Kat Patillo who had been telling me about the incredible example of Sobral (Brazil), where if we talk about systems, that's where I think of genuine system reform, where you're talking about political commitment and trade unions and communities and your finance ministries, and Kat had shared that how they went from Grade 2, going from 40% learning to 92% learning, just in three years. And this is one of the poorest states in the country of Brazil, incredible progress. So how did that sort of commitment happen?

Anyway, back to the coffee machine, I was up there just minding my own business, and of course suddenly there was Dennis (Mizne) from Lemann Foundation, you're always meeting amazing people here, and I thought okay this is my moment to say it was just the start of the pilot for the What Works Hub, and we were testing different sort of different ideas and I thought what about taking a group from a couple of our pilot countries like Kenya and Pakistan. So, what about taking a group to see this as I found that as quite a powerful mechanism working in Nepal and India that exemplars working from the bottom up are often more inspiring than evidence form the top down. So, I thought let’s see if this works and after a bit of persuasion, Dennis said, well you know we really just do the scaling in Brazil. I said, yes, but wouldn’t sharing that story more widely, wouldn’t that be more interesting? Finally, he said yes, but what was interesting they said, no it’s not for you as the donor to go in there and identify who should go in terms of the individuals. We’ll do that and actually a coordinating organisation in the country will do that. And I was a bit like, hang on, we’re paying for this in my usual kind of way, OK, OK, we’ll leave it to the anchoring organisations. Unsurprisingly, the organisations chosen ended up, with independent criteria and assessment, ended up being PAL Network members. Independent, respected, understanding evidence but not politically aligned to any one party. And for the first time really in my career I saw a group of different people from different elements of the system coming together. What did they do? Lant, when you took everyone out of their comfort zone, and let’s not forget how difficult this programme was at the beginning. It was hard. And Lant used to take everyone out of their comfort zone and do things like go to climbing walls and make us all dance. And do you remember the time you brought your wife and made us all do some art? I mean, at the time the thing I hated most was that dreadful improv session, here in this very building, how to humiliate people. Anyway, Lant’s way of putting people out of their comfort zone was much more pleasant, now in retrospect, than what they did in Brazil. They took everyone for three days into the basement of some vanilla hotel in Sao Paulo. We didn’t have a window in the whole room. And for three days they did all of this psychological: What is your worst characteristic? What are the traumatic things in your childhood? And I was like, after three days, I thought we were going to see system reform. I thought we were going to see some children. What are we doing? However, there was a method to the madness. And those people who when they walked off the aeroplane didn’t know each other and didn’t want to talk to each other because the trade unions did not see eye-to-eye with the trainer lead or the Minister of Finance who was there on that course, they walked back not only with common respect and as comrades, as we might say, but they walked back with a common mission and an outcome that they believed in that wasn’t pre-determined but was based on what they believed to be the problems. And I think it was a really inspiring model that I haven’t see before. So, what does that mean? I think it means we need a shift in culture. We need a shift in culture in how we use this evidence, I think we need a shift in culture around using that test, learn, adapt. And I think we need a shift in culture in terms of the local ownership of it. During the course I was also told I couldn’t be a Fellow. I thought, well I believe in this change just as much as, why can’t I be a Fellow on this programme? And I said, “Well at least my Kenyan counterpart, they’re Kenyan, they can be a Fellow. And my Pakistani counterpart, they can be a Fellow.” See Baela (Jamil) is laughing over there as the anchor organisation in Pakistan. No, you’re still a donor and you are usually dogmatic about determining the agenda. And this is a different level, and this is a different approach. And this is about a locally owned and locally led problem identification and solution. So, I think it’s really interesting and I think the challenge therefor perhaps for the What Works Hub is more about how do we then identify what technical assistance might be demanded and us being responsive to that to help with then with a gold-standard, if you like, of the change that they’re trying to enact. So, ensuring your alignment, Lant, and thinking about Sobral, so the Kenyan team went back saying their mission was going to be to create the Sobral of Kenya. And they wanted at scale, in terms of district, to try and model what they had seen. The challenge is that you go back, and you don’t necessarily have the best assessment system, you don’t necessarily have the teaching and learning materials, they might not being produced or locally created. So how do you then provide the appropriate support and so on to help do that. And at the same time, help support sufficient evidence generation to do your test, adapt, learn. Which again is a challenging thing to do.

Laura Savage: 41:37

So, you’ve got me really nervous about the trip that some of us are going on to Sobral in November. Get ready for your stories. But I want to put your challenge almost back to Mpumi and say if that is what the What Works Hub needs to do, if that's what implementation science looks like, getting researchers, practitioners, government to be working hand in hand almost, you know, kind of changing that culture of practice in a way, is that possible? What would it take? I mean, you've given two examples where you're starting to see it, but what we're saying here really, and what I'm drawing from what you're saying, Rachel, is that a change in the practice of evidence can help contribute in and of itself to improvements of learning. Do you think that's possible, feasible, and what would it take?

Nompumelelo Mohohlwane: 42:30

Yeah, I think so. So, I think the first thing is research is still underfunded in general. I know even the work we do, which has a lot of work, has a lot of external funding. The internal funding is sort of for our salaries, and hopefully, I don't know, desktop work, analysing existing data sets, but innovation, actually trialling things, piloting, doing the case studies, etc, needs money, actual money, which we often don't have in the budgets. And in times like now, we have budget crunches, which are in fact saying we should travel even less, intervene even less, monitor remotely, etc. So, that kind of support financially is needed. I think there is a recognition around technical aspects of what we don't know. So, I think most governments are open to, we don't know how to teach robotics and coding. That's the thing you want to support.

I'm not going to comment on whether you should do that. But there are technical questions that are unknown, and I think people are willing to admit that. There are aspects where people know something but need resources to deepen their knowledge. So, for example, the teaching of reading in African languages in South Africa is not something we'd want to outsource. It's something you can fund that's happening in the country. How you roll out, etc. could then have some technical aspects to it. And then there's some behavioural research that we can bring that can be world -class around how you change people's beliefs, how much exposure to people need to start believing children can learn, what do we learn from coaching, for example, how long does it take for someone to actually change their practice. Those are things you can learn from the global north or other places.

And then there's, I guess, the recognition that's necessary from researchers, donors, et cetera, that they don't know everything, right? So, there's an undermining or mutual respect that happens both ways. And that can be explicit or it can be implicit. You go to the training and you just keep doing whatever you are doing. But I think if we get to those kinds of things that are less like counting and ticking boxes and more relational, there is an opportunity for collaboration, a big need. We have scarce resources, we have large problems that need to be resolved, and best minds applying themselves to that is a useful thing. So, I think definitely doable, but should be thought through carefully.

Laura Savage: 45:05

Lant, I think you have views on that question too, in a way, right? Well, yeah, I mean, take your pick, but will the study of implementation, from your point now as sort of research director of RISE, looking forward to the questions, as I say, we heard many, many questions today on implementation, many more I think on politics, I have to say, but I'm biased on that one, you know, there's a lot of kind of the politics didn't work. But can implementation be studied? Can that culture shift happen? Will it help?

Lant Pritchett: 45:43

Strangely enough, I just read a document that quoted me as saying something I had never said. But which I liked, so now I did, which is that, you know, a discipline is a group of people who agree on what counts as a question and agree on what counts as an answer to a question. And a very serious problem, any commingling of what would be academically published research and implementation, is that there isn't a discipline in which, what these questions that we're answering, count as questions.

And in particular, there isn't a discipline that has agreed upon what counts as an answer to these questions that is sufficiently scientific and my fingers just can't help do this scientific and rigorous.

And that is a first order problem. It's a first order problem because academia is completely, totally controlled by the disciplines. And so, if you can't ask a world class researcher as a junior, tenured person trying to make their way in the world to not respond to his or her chosen discipline, and if you're a discipline like my second love after the Ford Mustang was economics, who I met just before my wife, but chronologically, some disciplines work backwards. They say, here's what counts as an answer. Therefore, the only questions that count are the questions that can be answered with the method we have decided is the method of what counts as an answer.

And today and every day in which economists talk, we're obsessed with causal identification. And if you don't have clean—and I love the word clean because it's like a ritual purity clean, they mean clean like a religion means clean—if you don't have clean identification, it can't be an answer to a question. And yet the questions that we're asking are going to be incredibly difficult to frame into a way in which the associated disciplines that dominate academia will recognize as an answer. And since they don't--and if you've defined what counts as a question by the questions that you can answer, which has very much happened—then you can't get research in the academically publishable, academically rewarded way embedded in this. And, you know, looking at Noam (Angrist), this is a first order challenge of attempting to integrate research and practice.

Laura Savage: 48:59

I'm going to open out to any questions in the room now: Tahrir (Andrabi).

Tahir Andrabi: 49:04

You know, generals fight last, what does it say, the previous generations battle? So, before we start, what works? We've got to fight the RISE battle and work through the system before we can get to implementation.

No, Lant, seriously, I mean, I think this is a good point, right? But there are ways of thinking about systems which are not ad hoc or anything, right? Mathematics has dynamical systems. And we have this idea of a state variable which is slow to move and then the control variable and the elusive search for control variables, which at the World Bank became like growth regressions, anything on the right -hand side could move and all that and, you know, which went nowhere. So, the point is that, you know, some of it is how do you move a state variable? And I'd like to think that, you know, some of it is the hard problem which you are describing is what we call an equilibrium. You know, when things are in equilibrium and particularly start thinking of intuition of a Nash equilibrium where, you know, it's self-enforcing. Nobody has an incentive to deviate and all that. It's a stable equilibrium of some sort. And all throughout the economic history, other things, you know, we don't know even if Vietnam exists and Pakistan exists and both are equilibria, right? Nash and many others have said, just nobody knows really how to move. Theory doesn't tell us, at least, too much on how to move from one equilibrium to another one. And I think that's a real problem. I mean, I think that's a deep question about social change. So, this is not just about technocratic solutions to various things. And this is a problem which in economic history has been studied, different disciplines, sub -disciplines are studied. And I think this is a good way to think. And we did try to do it, right? And our idea about thinking about the village as a closed market, and therefore, we could look at an equilibrium very locally.

So, it was an equilibrium, right? A market -based equilibrium, you shake it. It takes 20 years to study it, right? I mean, we are still waiting to see whether it comes down. So, part of the report guard study is, but it has to be some idea of a general equilibrium, right? We don't know what the relationships are fixed. So, my definition of a system really comes back to economics as an idea of a general equilibrium. So, my sense is you move things, and then things have to work out endogenously afterwards. And then the question is, what does the Schumpeter said, a new circular flow, as they said, right? I mean, that's what you're talking about. I mean, I think it takes time, and you have to really think about it. And then implementation is maybe just one of the shocks that you can try out. But you've got to have some idea of figuring out how to study the system.

And I don't think that in a macro, the beloved field, it's really difficult to figure out what's going on with systems. The debates are endless, right? I mean, how do systems evolve, change? So, my sense is that you've got to think in localised ways of where the conceptual idea of the system still is there.

But at least you can study it, you can shock it, you can trace it, you can see its evolution, you can see whether it comes to a new resting place. The challenge is you put an impossible question, right? I mean, can we replicate China? Can we replicate Vietnam? Can we do this? Either you become like a change the discipline, become comparative politics or something, which we, so I think that those are good questions. I think that we will have to think about it. I think that RISE at least forced us to think about a system. What do we mean really, even conceptually? And I think this will, you know, I don't think we have an answer of what is implementation science, but at least we can start thinking about it. And where does it fit in into some of these kind of disciplinary concepts? And is there a way to think about it? So, okay.

Laura Savage: 52:41

Thank you. I'll turn back to Lant and then Mpumi and Rachel. Any thoughts on those comments?

Lant Pritchett: 52:46

I'm gonna use a word that I barely know what it means, but ontology. Ontology is actually really important in deciding how one goes about researching something, 'cause the ontological character of what you're researching influences what a science of that ontological entity would be, right? So, it is. And there's behavioural sciences, which are about the behaviour of human beings. And human beings, we understand really, unbelievably well, right? Intuitively, right? And then there's the science of objects, which again, we understand, we have amazing progress in the science of objects. But both implementation science and systems are talking about ontological things that are neither agents who are driven by a teleological desire to achieve certain ends. And then we understand agents in terms of why they're doing what they're doing, but we understand agents and the why, we understand objects like this table in terms of the what, but neither organisations nor systems are ontologically agents, nor are they ontologically objects, and so that creates a whole series of really deep questions of what a science of organisations looks like and in the word implementation science it's also not obvious what the unit the most important unit is. Is it the organisation? Which is at least a legally fiction that's identifiable. Or is it a system and a system is by definition almost an amalgam of very different ontological objects So anyway, so I'm just coming to even a bare minimum understanding of what the phrase implementation science might even mean in terms of what's the ontological character of the reference of the thing isn't settled by any stretch of the imagination.

Laura Savage: 55:03

There are yeah, there's a number of challenges. And I wanted to ask Noam (Angrist) if you wanted to finish on anything related to What Works Hub.

Noam Angrist: 55:10

Our goal for the What Works Hub is to make progress and to make sure more children are learning and (that) we understand rigorously how to do that. So one question I have for the panel is now that we're shifting from this theme of systems to implementation, what do you think we should keep common so we don't lose at the amazing work that's happened over the last period and what do you think we actually should do differently as we move to Implementation? So, that was my question, and I also realised I didn't actually say who I was. I'm the academic director of the What Works Hub.

Laura Savage: 55:47

So, thank you, why don't we finish with each of you giving 30 seconds on the response to Noam?

Nompumelelo Mohohlwane: 55:52

Sure, I think the systems thinking aspect is still key. It's understanding what your program was what you're doing and at every level So, you know, you have you having a design failure, a delivery failure, a bureaucratic failure a service provider failure, etc. And so, that whole systems thinking, systems strengthening, systems scaling I think is important. I don't think it should be a competition between systems and implementation.

Lant Pritchett: 56:20

So, I think in order to create a sub discipline of implementation science you need to change the Conception of research from evaluating the impact of an intervention to evaluating the impact of the initiation of a process and then trace the process as part of the research. So it isn't just Implementation of did they do it because otherwise you're intrinsically in top-down mode, right, because the intervention is pre-designed and implementation is a separate stage and so anyway, and figuring out how to make that an acceptable rigorously methodological thing is I think the first-order issue for the research and academic component of the What Works Hub because that's how things happen. They happen as a process. You know, I have jargon about this like crawl of the design space versus, you know, most RCT is we're going to attempt to evaluate a specific design as opposed to we're going to initiate a process in which the system is crawling the design space to find something effective. And how you research that process is I think the key.

Rachel Hinton: 57:55

I would say three things, Noam, to your question. I think let's continue to be producing that incredible quality of robust evidence that we saw in the graph. But let's make sure it's costed. And in that costing, let's also consider equity. The second thing is, I think we need to do what Lant has said, is actually—perhaps it's wrongly named as the What Works Hub—to think about this being the How Works Hub. And I think that's a challenge to you and the team, Noam. And I think the last thing is, on this discipline, let's be ambitious on implementation science.

As Tahir and others said, you know, we're at the beginning of the journey of even thinking about what implementation science really means. But perhaps we need to be ambitious and have even a journal.

I don't think special issues will cut it. We need journals where people can publish on this, where they have the academic incentives that allow them to be professional in the space. And I hope that that Community of Practice that has been just so phenomenal in RISE continues to grow. And it grows with all of those you nurture from the global south as well, so that in the future, they are leaders, and their leaders support the children of the future.

Laura Savage: 59:19

Thank you. Thank you so much, everyone.

RISE Programme: 59:27

Thank you for listening to our podcast today. And if you liked it, be sure to check out our research at riseprogramme.org, or follow us on social media @riseprogramme. You can find links to the research mentioned and other works shared under the description for this podcast episode. The RISE podcast is brought to you by the Research on Improving Systems of Education (RISE Programme), through support from the UK's Foreign, Commonwealth and Development Office, Australia's Department of Foreign Affairs and Trade, and the Bill & Melinda Gates Foundation.