Of Mice and Mountains: The Case for a Pragmatic Evaluation

“The best laid plans of mice and men often go astray.” An 18th century poet like Robert Burns could scarcely have imagined something akin to the model that ideas42 and our partners at IntraHealth International designed to integrate family planning services with immunization days at health posts in Senegal, but he couldn’t have offered a better characterization of the challenges we faced implementing this intervention.

Our best laid plans

The months following childbirth are a critical time for women to adopt a family planning method and space out the births of their children, which has been shown to have long-term impacts on the health of both the mother and her children.

We know that over 90% of postpartum women in Senegal come to health facilities to access immunization services for their newborns. We also know that more than three in five Senegalese women have an unmet need for family planning. This gap in health services during the pivotal postpartum window – when health workers could be offering family planning to women when they are visiting the health facilities anyway and about to be fertile again – presented a key opportunity for ideas42 and IntraHealth International to design a program called Innovation Comportementale dans l’Integration (ICI) that integrates immunization services with family planning services.

Based on a thorough diagnosis of the behavioral barriers that prevent providers and clients from discussing family planning options during routine immunization services, we pursued an iterative design process in partnership with stakeholders from the Ministry of Health and Social Action to develop a three-pronged intervention, including:

  1. A series of referral cards that prompt health workers to discuss complementary family health behaviors with clients, including a cue for clients to set a specific time to speak with a provider about birth spacing.
  2. A series of mobile voice messages through which local actors act out a serial drama called théâtre mobile that dispels myths about family planning and gives clients timely information about important health topics during the postpartum period.
  3. A mobile training course for health workers delivered via interactive voice response (IVR) that builds and tests their competency in discussing immunization, child nutrition, and family planning with clients in this critical postpartum window.

Once the process was in place, we needed to measure how well this intervention was or wasn’t working. A rigorous impact evaluation had always been our intended capstone for this work. Although recognized as a promising high-impact practice by the World Health Organization, the integration of family planning and immunization services remains a largely untested proposition. Eager to gauge the impact of our diagnosis-driven model, we organized a cluster randomized controlled trial (RCT) at 78 health centers across two regions in the country. We distributed over 20,000 referral cards, recruited over a dozen trainers and data collectors, and facilitated trainings for nearly 200 health workers on incorporating our intervention materials into routine service delivery on immunization days at their health posts.

The plow levels a mountain

But just as we were about to launch the field test in April 2018, a labor dispute arose between public health worker syndicates and the Senegalese government, prompting a strike in health worker services. As the strike escalated in scope, we had to halt our implementation of the field test — after all, you can’t integrate family planning counseling with immunizations, when there are no immunization services taking place — and rethink how we were going to evaluate our behavioral model.

As a health worker later told us, “La montagne a accouché d’une souris,” or the mountain has given birth to a mouse – a French turn of phrase highlighting the contrast between our efforts in preparation for the field test and its underwhelming outcome. This experience will sound all too familiar to public health practitioners, as well as anyone working in contexts with multiple stakeholders: the best laid plans, carefully constructed over months or years, can often be thrown out by factors completely beyond our control, like a mouse’s nest felled by a farmer’s plow (or Robert Burns’ pen).

A pragmatic pivot

While we were unable to carry out a rigorous evaluation of this intervention, we were nonetheless determined to generate meaningful evidence on the potential for our family planning integration model, even without an RCT. So we pivoted from a quantitative impact evaluation to qualitative insights harvesting. This participatory research method allows the people directly involved with an intervention to report on the realities of implementation and identify strengths and weaknesses to government decision-makers.

To this end, we distributed open-ended journaux de bord diaries to health workers, the tutors who trained them, and regional health officials involved with our project, which prompted them to share their candid thoughts on the intervention. We then organized four workshops, during which nearly a hundred health workers and local government officials shared their experiences with the intervention (some are featured in the video above).

These journals and workshops culminated in a set of recommendations, such as optimizing access to our IVR messages in areas with low cell network coverage, and improving the reach of our monitoring tools.

But these qualitative tools also critically created a feedback channel between the front lines of health service delivery and the central government, allowing the health workers and regional officials to convey their thoughts on how to sustain this model on family planning integration after the strike concluded. And while local stakeholders shared opportunities for strengthening the intervention and continuing the implementation, national stakeholders also reflected on what kinds of evidence would be persuasive for the Ministry of Health to ultimately scale up the intervention, allowing us to gauge both what could be done and what needed to be done to return to the field with an even stronger intervention.

A new mountain arises

With the restoration of service delivery in public health facilities in November and the conclusion of national elections in February, we seized the opportunity to return to the field this spring. Propelled by the enthusiastic feedback from local stakeholders and the continued support of the Hewlett Foundation, we are launching a two-month pilot test this summer to demonstrate the feasibility of our model in new settings. With this second pilot test, we want to gauge if the referral card system is feasible and if our re-developed IVR system can works at various health posts in the Ziguinchor region, which is a new context from where we first tried to pilot the intervention.

It is hard to know to what degree this revival would have been possible had we not leveraged the journals and workshops to engage stakeholders during a prolonged period of uncertainty. As we discovered, the strike may have been a blessing in disguise, as it pushed us to engage with our stakeholders in new ways that allowed us to receive useful and candid feedback. An RCT may never have uncovered the same nuances. Using these qualitative methods in a moment of uncertainty allowed our intervention to continually evolve – even when implementation was impossible – and allowed us to forge and sustain connections with key decision-makers that ultimately allowed us to return to the field and implement the next iteration of our integrated service model.

While one goal of this renewed implementation is to generate support for an eventual impact evaluation, an important realization during our experience in Senegal was that multiple forms of evidence can be sufficient, and that a fixation on RCTs can obscure other approaches to evaluation that can achieve goals beyond evidence generation.

At ideas42, we always design our interventions with the high standard of an RCT in mind, but in the challenging contexts in which we work, we should keep ourselves open to pragmatic alternatives – be they quasi-experimental quantitative techniques or qualitative methods similar to those we pursued in Senegal.

Ultimately, switching to this pragmatic approach allowed our intervention to rise anew: a mouse that emerges from the ruins of its nest to build a new and improved home – one that might once again turn into a mountain. We are hopeful that this opportunity to return to the field will give us another chance to implement and evaluate an even stronger program that can provide thousands of Senegalese women with access to more opportunities to choose the right family planning service for them.

ideas42 Seminar Series: A Talk with Christina Roberto

With the ideas42 Seminar Series, we invite leading scholars to share their insights and what inspires their exploration into human behavior.

Our New York office was pleased to host Christina Roberto, an Assistant Professor of Medical Ethics & Health Policy at the Perelman School of Medicine at the University of Pennsylvania and the Director of the Psychology of Eating And Consumer Health (PEACH) lab. Her research aims to identify and understand factors that promote unhealthy eating behaviors linked to obesity and eating disorders and design interventions to promote healthy eating. She completed her undergraduate degree at Princeton University and earned a joint-PhD in clinical psychology and chronic disease epidemiology from Yale University. After giving a talk to the ideas42 team, Christina was kind enough to share some of her thoughts on behavioral science:

What drew you to behavioral economics?

I became very interested in psychology as an undergraduate at Princeton. I became particularly interested in understanding the causes and treatments of eating disorders, and so I worked in an eating disorders research clinic after graduation. It wasn’t until graduate school, when I began studying food policies to promote healthy eating, that I became excited about merging behavioral science insights with a public health perspective. I remember when my PhD mentor presented the famous organ donation default effect for the first time in a large lecture course and the entire room gasped. That result got me really excited about the potential of behavioral economics to inform public health.

What would you say is one of the most surprising discoveries about human behavior?

Most important findings in behavioral economics and psychology don’t seem surprising after they’ve been discovered. Once you see the data on default options, the results make a lot of sense. In my experience, findings that are very surprising – so surprising that they make for good headlines – often rest on flimsy evidence, and wouldn’t replicate if investigated more rigorously. Maybe the most surprising realization in the last few years is that you can’t just accept that a published finding is true; you have to make sure the study was properly powered, and that the researchers didn’t observe the key effect only after trying to run many different analyses.

In your breadth of work, tell us about one of your favorite projects related to driving healthy behavior change.

Rather than a favorite project, I have a favorite approach to asking research questions. We call it Strategic Science. Usually scientists generate research questions based on what they are interested in and curious about. That can lead to important discoveries, but it often means that scientists miss out on the opportunity to provide policymakers with the answers to their most pressing questions. Strategic Science is about asking questions that policymakers need answered. To do this, I reach out to change agents—people who make or influence policy—to get their input into the types of research questions I’m asking. I then try to answer those questions in a rigorous, unbiased way, and, if all goes according to plan, publish it in a peer-reviewed journal. But publishing is not the only objective, as I’m also committed to communicating that work back to the same change agents who helped to formulate my research questions. Using this approach, my research has informed the design of food policies like national requirements to post calorie labels on chain restaurant menus and Philadelphia’s sodium warning label requirement for chain restaurants.

What have you learned in applying behavioral insights that has changed the way you work?

I think it is important to have theories that inform our choice of interventions, but if you are doing applied work and trying to really move the needle on behavior change, I think it’s critical to test those theories in the context you’re working in. You can’t just assume that interventions that have been effective in some contexts, such as implementation intentions or social norms messaging, will work in a different context. I’ve also come to appreciate just how large our sample sizes need to be to detect most of the small effects we are interested in. This is especially important when doing policy-relevant work in which a true null effect is informative; if your study is underpowered, you won’t know if the policy doesn’t work or if it will exert a small, but meaningful effect.

How do you use behavioral insights in your daily life (or recommend that people use behavioral insights in their daily lives)?

Behavioral science has taught me that it’s easier to sustain healthy habits if there are simple and concrete steps I can take that are easy to remember. So I try to adhere to a few simple rules-of-thumb when it comes to eating decisions. For example, rather than try to count calories, which takes a lot of effort or follow a fad diet that would be hard to maintain over the long-term, I have guidelines like: don’t drink sugary drinks, avoid red meat, make sure to have fruits or vegetables as part of lunch and dinner, and eat whole grains often.

Associated Materials


Sign up for updates: