What if I told you there’s a simple way to know with the highest confidence possible that a curriculum, program, or policy in your school district is having the impact you intend on the students you designed it for—and that district personnel could do the work in-house without having to hire an outside consultant? Well, there is such a tool, and it’s called random assignment.
How students are typically assigned to specific programs
Before we talk about random assignment, though, let’s look at how students typically end up receiving a particular curriculum or enrolling in a program (say, after-school tutoring or another special program). Often a school district will roll out a new curriculum or an intervention districtwide so that all eligible students receive it. This practice makes it difficult later to quantify, or put a number on, the impact (effectiveness) of the new curriculum. Why? Because all eligible students are receiving it, so there’s no basis for comparison. Even going back in time to compare previous students under the previous curriculum against current students under the new curriculum is problematic because these are different students in a different context. Comparison with past students is murky at best due to the many other factors—known and unknown—that may be different between the two years’ groups: different teachers, different principals, a different schedule that makes children hungrier by lunchtime—you see how complicated it can get.
Alternatively, if a program or intervention is offered through enrollment, the way students wind up in the program is usually not random, either. It may be informed by teacher recommendation or a student’s “academic profile.” In other cases, it may be that parents who are more involved in their child’s education are doing what they can to ensure that their child enrolls in a particular program.
Any nonrandom process can make it difficult to quantify program impact because the students in the program may be systematically different from other students against whom you would be tempted to compare them. For example, imagine a reading intervention that enrolls students on the basis of teacher recommendations. If students in that reading intervention have more engaged teachers (e.g., if they have an English language arts teacher who stays after class to help them, follows up with their parents about their reading progress, searches for novel solutions to help struggling students achieve, etc.) compared with similar students who are not in the reading intervention, this can obscure the effect of the reading intervention. That is, it may very well be the more involved teachers, not the intervention, that produced any measured impact on student reading outcomes.
How random assignment works
Now that we’ve looked at examples of how students are typically assigned to a new curriculum or program (and how that nonrandom assignment can adversely affect the ability to accurately calculate the impact of the curriculum or program), let’s look at random assignment. What is random assignment? In contrast to the nonrandom way in which students are typically assigned to a curriculum or educational program, random assignment happens when participants are placed in different groups using a random process (such as flipping a coin or using a random number generator).
There are many ways to evaluate a curriculum, program, or policy using random assignment, but the most simple (and common) case involves two groups into which students are randomly assigned. One group, often called the “treatment” group, is the group that will end up receiving the new curriculum, program, or policy that the district is considering adopting (and therefore would like to know if it’s effective). The other group, often called the “control group,” will end up not receiving the new curriculum, program, or policy. Instead, the “control group” will get what students typically get (i.e., nothing different will be done with this group).
Of course, it may not always be practical or make sense to randomly assign individual students to programs. Luckily, students are already in groups, such as classrooms. If there are enough of them, these existing, or intact, groups can be randomly assigned into either the treatment group or the control group, and individual students don’t have to be shuffled around.
For example, if your school district is considering a new curriculum for third grade reading, and you have 30 third grade reading classes across the district, your district can randomly assign 15 of those classes to the new curriculum and 15 to the current curriculum, and then compare the performance of these classes on relevant outcome measures (e.g., through the state’s spring reading assessment or a district-created assessment) after the treatment students have been exposed to the new curriculum.
What makes random assignment so useful
Here are the main reasons we think random assignment is an extremely powerful, if underutilized, tool for school districts:
- It controls for known confounding variables.
- It controls for unknown confounding variables.
- It simplifies analysis.
So what are confounding variables? These are things that are related to both the treatment and the outcome, and their presence can make it difficult to accurately calculate the impact of the treatment. For example, a reading intervention may be targeted to students who scored poorly on a standardized reading assessment administered last year (so that the standardized reading assessment score is related to whether or not the student received the reading intervention), and students who scored poorly last year are more likely to score poorly this year (so that the standardized reading assessment score last year is related to this year’s reading assessment score).
Control for the confounding variables you know about
With random assignment of larger groups, such as whole classrooms, the treatment and control groups tend to be very similar on confounding variables that you know about and measure (e.g., free and reduced lunch, English language learners, baseline standardized test scores). This similarity is beneficial because it takes these factors “out of the equation” (i.e., they “cancel out”), allowing you to be more certain that any differences between the two groups after the curriculum or program are a result of the treatment.
Control for the confounding variables you don’t know about
Although districts have a great deal of information about their students, lots of factors influence academic success (teacher pedagogy, amount of time students spend studying, etc.), and many of these factors are not measured and recorded by school districts. Fortunately, with random assignment, these factors also tend to balance out between the treatment and control groups.
When a school district hires Parsimony to conduct a quantitative impact evaluation of a program, we often find that random assignment was not used; instead, the district typically either provided the program to all targeted students, or it assigned students to the program in a semi-systematic way (e.g., on the basis of teacher recommendations, last year’s spring assessment scores, etc.). As a result, we often have to use complex statistical models to try to accurately calculate a program’s impact. Ernest Rutherford, the “father of nuclear physics,” put it bluntly: “If your experiment needs statistics, you ought to have done a better experiment.”
In many cases, our complex statistical models are used to compensate for the fact that random assignment was not used. With random assignment, because confounding variables tend to be balanced out, you can just compare the groups and rest assured that any difference in the outcome between the groups is due solely to the curriculum, program, or policy in question—because that is the only thing known to be different between the two groups. When we do have a client district that has used random assignment, we get a big smile on our faces, because we know our job has gotten much easier, and both we and the client can have much greater confidence in the calculation of the program’s impact.
Because random assignment controls for known and unknown confounding variables, when you use it and then see a difference between the treatment and the control group on the outcomes you care about, you can be assured that the treatment caused that difference in performance. This is the same reason that the Institute of Education Sciences (the research arm of the U.S. Department of Education) places such a high priority on random assignment when it reviews the quality of the evidence described in a research study.
Have you ever tried to use random assignment to evaluate a curriculum, intervention, or other educational program, or do you have other thoughts about this article? Leave a comment at the bottom of this post.
Find this blog post helpful? Then sign up for our newsletter (if you haven’t already) so you can get notified of new posts. Just click on the Subscribe button below and enter your email address.
Parsimony is a small consulting business that helps school districts quickly, accurately, and affordably calculate the impact of curricula, programs, or policies on important student outcomes through quantitative impact evaluations. Go to www.parsimonyinc.com to learn more.