We Can Do Better: Data, Educational Outcomes, and Equity

Challenges in areas ranging from education to the environment, gender to governance, health to housing don’t exist in a vacuum. Each month, Abt experts from two disciplines explore ideas for tackling these challenges in our monthly podcast, The Intersect. Sign up for monthly e-mail notifications here. Catch up with previous episodes here.

With the Education Sciences Reform Act up for reauthorization, there’s an opportunity to revisit how we use data to promote equitable educational policies and approaches. Dr. Karen Gray-Adams and Dr. Rebecca McGill-Wilkinson discuss building equity into the evaluation process—from community involvement to planning randomized control trials—to ensure we can identify the best way forward for every child.

Read the Transcript

Eric Tischler: Hi, and welcome to The Intersect, I'm Eric Tischler. Abt Global tackles complex challenges around the world, ranging from improving health and education to assessing the impact of environmental changes. For any given problem, we bring multiple perspectives to the table. We thought it'd be enlightening—and maybe even fun—to pair up colleagues from different disciplines so they can share their ideas and perhaps spark new thinking about how we solve these challenges. Today I'm joined by two of those colleagues, Dr. Karen-Gray Adams and Dr. Rebecca McGill-Wilkinson. 

Karen is an educational psychologist who has experience in research and technical assistance, with an emphasis on educational equity. Her areas of expertise include culturally responsive pedagogy, equity, and educator effectiveness. Becky is a researcher with expertise in education research, developmental psychology, research design, and statistics, with a special focus on evaluation of education programs. She's particularly skilled in conducting applied educational research within a school system. Thank you both for joining me!

Karen Gray-Adams: Thank you. Hi, Eric. It's good to be here.

Rebecca McGill-Wilkinson: Thanks so much. Happy to be here.

Eric: As states and school districts implement interventions to improve student outcomes, they’re relying on the field of education research to provide evidence to inform their program choices. The Education Sciences Reform Act—which is coming up for reauthorization after 22 years—has essentially been driving the field of education research in the United States. And in the past two decades, we've produced a body of evidence about what works and doesn't work in education. However, literacy and math scores are declining, particularly among those who have historically been left behind. So Karen, let's start with you: What are we doing wrong?

Karen: So, thank you for that question. I think that when we're talking about implementing evidence-based practices, we really need to understand that context really matters. And I think that we need to think about additional studies that are targeted because we want to expand our knowledge base. So part of that is really thinking about interventions and research designs that are representative of all kids. And I think when you think about it, RCTs are one of them. And I know Becky has expertise in this. So, Becky, if you can talk a little bit about that, that would be great.

Becky: Sure. RCTs, or randomized control trials, are sort of what we consider to be the gold standard in conducting evaluations. The idea here is that you randomly assign students or groups of students to either receive an intervention or to not receive an intervention, and then you look at outcomes for both groups and compare them to each other. The idea behind an RCT is that these studies are designed to account for all of the unknown and unmeasurable variations between people. And these studies have been used in many fields for decades, but the field of education research was really growing post-ESRA authorization. Researchers began using RCTs more consistently to look at the impact of evaluation. One of the things that we want to consider when we think about how we can use RCTs to look at the impact of evaluations in different contexts is to think about the populations and the settings that we're using to study.

So, while the field of education research has grown exponentially over the last 22 years, a lot of the research that we have conducted has been conducted with homogenous groups of students, when really these homogenous groups of students don't represent the diversity of students in the United States. And, in order to really understand what works and what doesn't work for all students, we need to pay attention to the populations and the settings and how those things create a context for the research that may or may not match the context that students are actually learning in. One of the things that a lot of researchers will do is replicate a study. So, if a study has already been done with a homogenous group of students, the idea behind replication is to do essentially the same study again, but with a different group of students in a different setting to see if the impact of the evaluation is the same on different groups of students in different settings.

Karen: And so, when you say that, Becky, that just makes me think about this is a really great opportunity to think about the intersections of students and building in equity from the beginning. Because, when you look at the diversity of students and perspectives across the United States, we want to try to do the best we can for all students so that they have an opportunity for success. So, in thinking about this, it's really important to reflect and take some time to really think about what the RCT is going to entail and how we have to figure out how researchers need to think about the intersections of all students and what they're really going to be looking at when they're looking at a particular intervention. So, if it's not working for some students, we have to reflect and figure out why it's not working for students. And that means that we need to just think about the whole thing from the beginning to the end in order to make that change.

Becky: I don't want to say that no education researchers have done a good job of implementing, of studying their interventions in a variety of contexts and with a variety of students, because there are certainly people who have done a fantastic job of doing that work. I think the reauthorization of ESRA really gives us the opportunity as a field to push the envelope further and to think more carefully about how we are incorporating equity into the evaluation work that we do. I think it becomes more and more crucial. We know, based on statistics, that the diversity of this country is increasing over time, which means that over time there are more and more students of different socioeconomic or racial or ethnic backgrounds, language history, immigration status, all of these various characteristics that we just don't know what works for them and what doesn't. And if we don't know, then we need research to help figure out what's working, what context it's working in, so that teachers and principals and superintendents can choose programs that are going to help their population.

Eric: Okay, great. So, let's go from there: If a reauthorization is going to broaden the horizons for what we can do in education research, what should that look like?

Becky: I spoke about replication. And I think that that has been an important first step towards thinking about, “Well, we know that this works for some students, let's see if it works for other students and in other settings.” And I think that's a great first step. I do think, as Karen mentioned already, that it would be beneficial for the field to really think about incorporating equity from the very beginning. When you conduct an RCT, researchers are very intentional from the beginning about thinking through exactly what their study is going to look like, where they're going to recruit their schools or their students, how they're going to randomly assign them, what kind of measures they're going to use to demonstrate an impact on student outcomes. And I think equity needs to be part of that intentionality and built in from the beginning.

Karen: And Becky, thank you for saying that because when you talk about equity, it's a process and an outcome. And so you really do need to understand the process in order to provide context for the outcome. So if you don't do that, you're missing the boat. So it's really important to think about both. And to your point about implementation and impact studies going hand in hand, that's really important.

Becky: I just wanted to expand a little bit more on what you said about process and outcomes, that it's not just that we want to show that programs work for a diverse set of students. We also want to ensure that the practice of conducting research is equitable as well.

Karen:  I think when we think about RCTs, I think we also have to give ourselves grace and be flexible and understand that they may not be applicable in all situations. And so I think that that's an important consideration for researchers in the field. Because if you're thinking about particular groups that you want to do this research study and maybe that they're not receptive, I think we have to think about innovative ways of engaging people. And it may not be ... depending upon what the intervention is or whatever thing that you want to study, maybe the particular group is not wanting to look at ... have a comparison group and then just the intervention, just be in one particular ... So you just have to think about what the intervention is. Is it going to work for the particular group of people or the community? Making sure that the community understands what's happening, whether if it's the school, if it's the district. And you have to take in their perspectives and their understandings of it in order to move forward because it's not going to be of high quality or beneficial for anybody if people are not engaged in it. It goes back to the engagement and understanding the benefits of it.

Becky: And I think, also, Karen, it goes back to what you were saying earlier about building relationships with communities and with individuals.

Eric: How does that intentionality look different moving forward than it has in the past? Because, Becky and Karen, you both said, I think, that RCTs ... people are intentional at the outset but that intentionality has not been paying off. How does it need to be different moving forward?

Karen:  Well, this is where context also comes in: You have to understand the context. So, not everyone is coming from the same experience. So, I think when Becky was talking about, yes, RCTs are well-thought-out and planned, but now we have to embed the context of the individuals that are a part of the study as well. And research questions ... I mean, it starts from the beginning. The research design, the recruitment, the questions, your analysis, your interpretation of the data is important. And how are people going to receive the information and making sure that the audience is taken into consideration as well. So it's all of those pieces together that we have to really think about.

Eric: When we’re thinking about making evaluation intentional in terms of equity, what should researchers be considering?

Karen: Well, I think that what we need to be intentional is “What is the purpose of the particular intervention and what do we want to be able to say about it?” And it's okay if it's a smaller scope. I think sometimes people want to cover every single thing in a study and you can't do that. You have to really decide what is the focus of this particular study, what the purpose is. And people talk about what the purpose is, but then it ends up being larger or people want to generalize more, and you can't do that when you're being intentional. So, it's okay to think smaller and be intentional and get the high-quality data that you're looking for.

Becky: On the point that Karen just made, I think that we also need to think about the evidence we're producing as a body of evidence. And we can't expect one study to provide all of the evidence that we might need about the impact of an intervention. And it's really about multiple studies in different contexts, some with smaller groups. I think there's a place for qualitative and mixed methods research here to really understand, not just the impact of a particular intervention, but also if it works, why is it working? If it doesn't work, why isn't it working? What improvements could be made to the intervention? What revisions could we make to make it work for different populations of students in different communities? I think one of the important things that we need to consider more as a field moving forward is really thinking about pairing impact studies with implementation studies.

Implementation studies are the pieces—they often are paired with impact studies—but these are the pieces that really give us information about what's happening in classrooms, how interventions are being implemented, what the response from teachers and students are as they're implementing the program. And I think that's also really important for understanding the context of why interventions might work for some students and not for others. And I think to just do impact studies without the implementation piece, without understanding the “why,” we're missing a big opportunity to really improve programs overall to serve the students in our country.

Karen: And that's increasing your knowledge, the knowledge base, and that's what we have to get back to. So it's that you need these multiple studies with different groups of students maybe looking at similar research questions, or this is where you have meta analysis, if you have systematic evidence reviews. You can have all of those things but you have to have the studies in order to do that.

Becky: Exactly.

Karen: That's Becky's point, that you need multiple studies. And if you don't, then we're just going to be in the same place, where we are right now.

Becky: I don't want to give the impression that this only matters because we need better data. Because, as a field, we do need better data and schools need better data and principals need better data for when they're choosing programs for their students. But I would also argue that there is a bigger issue here that's a larger equity issue in general and not just about research. It's not just so we can have better data. It's also that we can ensure that we are treating students the way they need to be treated and that we're serving all students equitably.

Karen: Right. So it's circling back to why were the interventions created in the first place, and we want to get back to the student so that we're making positive changes for students so they have opportunities for success.

Becky: Yes. Thank you, Karen.

Karen: And we need to keep that in mind in all of the aspects of all the things that we're doing. So from the beginning, that is why it's important to embed equity in thinking about context. And thinking from the beginning, from the research design to the analysis to any dissemination of these studies and what the implications are and reflecting on it because we want to make positive change for students. That is the ultimate goal: positive change for students and their families and communities. And in order to do so, this is a great opportunity to add to the research base and expand so we can have that knowledge.

Eric: Well, if that's not a great summary, I don't know what is. Thank you both for joining me.

Becky: Thank you so much, Eric.

Karen: Thank you. We appreciate it so much. Thank you.

Eric: And thank you for joining us at The Intersect.