An Intro to Rapid Feedback
Evidence about what actually works in improving education outcomes is lacking, as noted in a recent 3ie systematic review. We know a lot about what works in some countries, in specific contexts, some of the time, but we don’t know a lot about what works most of the time in any country, regardless of context.
An example of how this plays out in practice is in Early Grade Reading programs, where a critical barrier to real impact has been effective parent engagement. In this case, a key learning question may be: How might we encourage caregivers to read to their kids and, in turn, improve child literacy outcomes?
Activities from providing free books to trainings on the importance of literacy to creative content that appeal to both adults and children can empower caregivers to read to their kids. It’s when there is more than one way to approach a challenge that adaptive learning and experimentation on what’s working, what’s not, and why can make a significant difference in shaping a program to be as effective as possible. With this in mind Results for Development (R4D) developed a unique approach to monitoring and evaluation to help shed light on the best strategy to use when, for example, a program starts off with a few potential approaches but is uncertain about which works best.
This adaptive learning process builds experimentation—through rapid and iterative piloting, testing, and adapting—into program implementation to flexibly inform decision-making on an ongoing basis. The approach involves testing two or more alternative strategies in rapid cycles, building in more systematic and rigorous data collection and feedback as the program begins to uncover and refine promising strategies. This allows for timelier feedback, adaptation, and program improvement earlier in the design process while maintaining an appropriate level of rigor and evidence to drive organizational learning. We tailor the approach to the needs of the implementing partners and local contexts.
Applying Rapid Feedback to education programs
Our earlier example of an Early Grade Reading program is taken from our actual experience with a pilot program in India – Mobile Reading to Children. Existing evidence shows that when caregivers read to their children during their pre-literate, formative years, it has significant positive effects on their learning and social outcomes later in life, yet this is a rare practice in low income communities in Delhi. Recognizing this challenge in educational development, R4D partnered with Pearson and Worldreader to empower caregivers and parents in India to read to their preliterate children, and provide them with access to a mobile library of culturally relevant, quality books.
We originally assumed that the obstacles preventing parents from reading to their children were related to a lack of access to books, insufficient time, and low parental literacy rates. Our formative research proved, however, that improving parents’ access to books through a mobile app was insufficient on its own. There needed to be a component of the intervention focused on behavior change as parents did not know how to read to their children or why it was important.
Without the process of learning adaptively, which allowed us to test hypotheses and strategies before full-scale implementation, a program would have been created based on fundamentally flawed assumptions. Adaptive learning and experimentation provided the mechanism to better understand the target population, their challenges, and their needs.
Results for Development has been implementing this innovative approach to monitoring and evaluation in different contexts both within and outside of the education sector:
- Improving student literacy in junior secondary schools in Sierra Leone: R4D is working with Rising Academy Network to test different combinations of three interventions: peer-to-peer administered reading assessments, a reading board showing student progression, and a morning reading club
- Designing improved transparency and accountability interventions in Tanzania and Indonesia: In collaboration with the Harvard Kennedy School, R4D designed Transparency for Development (T4D) to help practitioners, donors, and health-focused civil society organizations strengthen citizen empowerment and maternal and newborn health outcomes.
Why does this matter?
At a basic level, we need to be thinking about how we can get better outcomes. Education initiatives are shifting toward more flexible programming that allows program needs to be identified and adjustments to be made to meet those needs. Especially with bigger projects, there are questions that come up in the early stages of developing and implementing programs that need to be answered. We need to start answering these questions thoughtfully and using data. We need a research mechanism that will inform our decision-making and guide our thinking so that we are approaching our questions in a systematic way, rather than making our best, most educated guesses about what works best. Adaptive learning and experimentation can fundamentally enhance and improve our program outcomes and add to our knowledge about what works in education.
For more information on Adaptive Learning, please visit here.
Isabel Krakoff joins Results for Development from Mathematica Policy Research where she was working as a program associate on international education projects. During her two years with Mathematica, she worked on both qualitative and quantitative program evaluations. Isabel received her BA from Hamilton College in 2014, and is currently working on her master’s of international affairs with a concentration in global gender policy from the George Washington University’s Elliott School.
Christina Synowiec is a Senior Program Officer on the learning and experimentation team at Results for Development. Christina joined R4D in September 2012 and has worked across the organization's health, education, early childhood development, and child protection initiatives. Previously, she was at the London School of Hygiene and Tropical Medicine, where she earned her MSc in Public Health as a Rotary Ambassadorial Scholar.