Despite rapid growth in the adoption of behavioral science over the last decade, some have argued that the field may not be reaching its full potential. When behavioral scientists focus on identifying new applications for existing frameworks, we risk falling short of innovative growth into new technologies and approaches. There is certainly still room to improve and scale the existing suite of behavioral science tools. But if we want applied behavioral science to magnify its role in creating and sustaining long-term social impact in a rapidly changing world, it’s time to expand our toolkit.
Innovative machine learning methods for estimating heterogeneous treatment effects (for example, causal forests) enable behavioral scientists to strategically deliver interventions to those most likely to benefit from them. Crucially, these methods also reveal who is not likely to benefit from an intervention, allowing us to design alternative solutions that will work for these individuals and meet the diverse needs of a heterogeneous population. This is a departure from the status-quo, in which we design and test behavioral interventions to see if they are effective on average. Existing methods do allow us to explore differences in response to treatment between pre-specified subgroups, but they do not capture the complex and intertwined factors that truly drive variation in how people respond to interventions.
In 2019, with support from Schmidt Futures and The Alfred P. Sloan Foundation, ideas42 collaborated with the Golub Capital Social Impact Lab at Stanford University to apply machine learning methods to a set of real-world behavioral interventions. We chose a set of prior ideas42 experiments that had positive average treatment effects (we knew they worked well, on average), and set out to explore heterogeneous treatment effects (for whom do these interventions work?). Ultimately, the goal is to use information about how different people respond to treatment to make strategic decisions about who should receive an intervention in the future.
The culmination of our first year of work is the new report: Computational Applications to Behavioral Science. It is designed to be a guide for researchers, policy-makers, and practitioners who want to use machine learning to enhance their own field experiments – with a particular focus on behavioral science interventions. The report includes technical descriptions and tutorials for how to implement these methods, two illustrative case studies, and reflections on which experimental contexts are best suited for this particular application of machine learning.
In one of our case studies, we revisit data from a five-year partnership with the City University of New York (CUNY). Of the 13 randomized controlled trials (RCTs) that we ran with CUNY, four tested behavioral interventions encouraging community college students to secure their financial aid by renewing the Free Application for Federal Student Aid (FAFSA). On average, students who received behaviorally-informed text messages and emails were between 4 and 9 percentage points more likely to renew FAFSA by the June 30th priority deadline. With our existing set of tools, we could see that the intervention was effective, but lacked insight into whether the campaign was equally effective for all students.
Using machine learning, we discovered that students who unenrolled during the semester had significantly lower treatment effects (by about 4 percentage points) compared to students who remained enrolled. This insight serves two purposes from a policy-maker or practitioner’s perspective: first, we see evidence that those who unenroll during the semester need more intensive interventions (for example, personal outreach from advising staff) to encourage them to renew the FAFSA and help them overcome a multitude of other barriers to persistence. Second, we can make an informed decision about whether it is worthwhile to send messages to a group that is not benefitting from them (but is also not being harmed). This tradeoff becomes especially important in cases where the intervention is costly, or if we think inundating students with ineffective messages might cause them to ignore other important notifications.
Having learned something about who benefits most from an intervention treatment, how do we target the right messaging to the right students? School administrators, of course, do not know ahead of time who is going to unenroll during the semester. Fortunately, we can use machine learning to help predict how each student will respond to the intervention and assign treatment to those most likely to benefit. We can also compare this approach to other ways of assigning treatment (for example, randomization) to determine which is a more “optimal policy” (i.e. leads to better outcomes). In this case, assigning treatment based on predicted treatment effect appears to be the “optimal policy.”
Through this process, we also learned a lot about how to scope behavioral science interventions and designs that incorporate machine learning methods successfully. In the report, we provide tips for designing experiments that allow for detecting heterogeneity if it exists. This application of machine learning will be most impactful in settings where multiple treatment arms are possible, where the data are both long and wide, and where there are behavioral – not just demographic – data (for example, information about somebody’s past engagement with a program or resource). The potential for impact is especially high in cases where some treatment options are expensive or hard to implement, but may be more effective for some people; when we know more about who really benefits from an expensive intervention, we can reduce the overall cost of implementation or maximize outcomes within cost constraints.
Our initial case study with CUNY holds significant promise for this approach. If we scope, design, and test evidence-based, thoughtful behavioral interventions with machine learning methods as part of our toolkit, behavioral science is even more well-positioned to positively impact millions of lives in the future.
Read the full report here.
