The full paper on which this is based can be found here.
In a payment by results (PbR) programme, a service provider may not be paid by the donor for what they spend - the provider has financial ‘skin in the game’. The central rationale of PbR is that this ‘skin’ should increase accountability and improve performance (their ‘level of play’). This could be either by encouraging the provider to innovate, or to prioritise their activities in a way to make them more successful in delivering the goals of the programme.
The £355 million DFID programme, the Girls’ Education Challenge (GEC), provides one of the largest applications of PbR in international development to date, particularly in the education sector and in work with non-government organisations (NGOs) who are delivering the programme’s 37 projects across 18 countries.
A PbR mechanism was applied for the 2013 to 2017 period for 15 out of these 37 projects. For the PbR projects around 10% of expenditure has been ‘at-risk’ by linking it to the performance on targets for improvements in outcomes, principally learning measured by literacy and numeracy tests. If projects achieve their targets they receive what they spend, if they over-achieve targets they can get a bonus (the PbR ‘upside’), and if they under-achieve, they can lose money (the PbR ‘downside’).
Figure 1: The PbR model applied on the GEC for 15/37 projects on the programme
We undertook research into the experience of PbR on the GEC through a two-stage design – a survey followed up by targeted interviews. There were 52 individuals responding to the survey, generally senior managers or project managers working for a mix of the PbR and non-PbR projects on the GEC. We then followed up with 24 interviews undertaken with 44 individuals from the head offices and country offices of GEC projects. DFID staff were also interviewed.
Whose skin is in the game?
When economic theory models PbR (see Clist and Verschoor, 2014 for example), the agency (NGO / service provider) is normally treated as though it were a single monolithic entity. The reality of course is that while the organisation has its corporate structure, it is made up of a collection of individuals and teams often working across countries as well as technical areas. Ultimately it is these people that will respond to any incentive from PbR, and they will be working within the culture of their employer organisation.
This begs the question of how an organisation’s financial risk is dealt with and interpreted by the individuals working within it. The organisation can lose money, but does this filter down through central office and governance structures to project managers and field staff?
We found through our survey and interviews with a range of GEC project staff that there was a large amount of variation in terms of how projects treated the financial risk of PbR.
We found that those working at central office level were very aware of the risks and incentive from PbR, and understood the details of the mechanism. This differed for local office staff, usually project managers, who knew about PbR but much less about the details.
The disparate understanding at different levels of the organisation also transposed into how individuals viewed PbR. Central office and senior staff experienced a lot of stress from the PbR downside risk, with this being particularly acute for those working for smaller NGOs where the financial risk represented a much larger share of their turnover. But larger NGOs also found it difficult to cope with the risk, particularly for those that do not have large pools of ‘unrestricted’ reserves.
We found that the vast majority of projects on PbR did not share any financial incentive with their staff. In addition, projects that involved a number of consortium partners often did not share the PbR risk with them, protecting smaller partners from the ‘downside’ financial risk. This meant that the level of PbR risk was magnified at the head office and centre of lead project organisations. We found that the NGOs on the GEC were not able to cope well with this risk.
Has the level of play improved?
The biggest test of a PbR mechanism, in terms of determining if the benefits outweigh costs, is whether organisations adapt and innovate in response to its presence.
On the GEC, with just less than half of projects contracted with the PbR mechanism linked to outcomes, and the other projects without this mechanism, we were able to look at whether the behaviour of projects under the two types of contracts differed.
We found little evidence that PbR motivated more change and innovation. Instead we found some respondents from PbR projects stating that it may have constrained innovation by making them more risk averse. They were much more concerned about not losing the PbR downside, than innovating in ways that might gain them the PbR upside.
In addition, respondents pointed to perverse incentives from PbR - for example to work with less disadvantaged children, or to prioritise short-term performance over longer-term sustainable and systemic change. There were some individuals who cited PbR as a motivation to change when they had interim results showing that their activities were not leading to literacy or numeracy improvements. But many others, including non-PbR projects, also made changes to improve their focus on outcomes. We found this was strongly linked to staff members’ professional (‘instrinsic’) motivation. More so than the financial incentive ('extrinsic motivation').
As Barder et al. (2014) put it: “the main purpose of PbR is to enable autonomy for local implementers, which is essential to find solutions to complex problems.” We found that autonomy was there and many projects did make changes as they went along, but there were some bureaucratic hurdles. The programme did not operate a ‘pure’ PbR approach, but rather a ‘hybrid’ approach, in which the majority of projects’ payments were linked to output milestones (or entirely in the case of non-PbR projects). Making changes to activity milestones and to budgets could sometimes be time-consuming and cumbersome. This may have hindered the capability of projects to adapt alongside their implementation.
Sharpening minds and labelling success
While we found a lack of evidence that the NGOs on the GEC were able to cope with their skin being in the game in terms of financial risk, and limited evidence of innovation or changes as a result of the incentive, there were some other benefits that weigh in favour of PbR.
The presence of the PbR on the programme strongly led the development of the rigorous approach to evaluation that the GEC has applied across all projects (including those not on PbR). This includes the use of best practice measurement tools, comparison groups, tracking a cohort of beneficiaries over time, and seeking statistically significant results, as well as the use of independent, external evaluators to deliver each project evaluation. The result is that PbR has had the effect of ‘sharpening minds’ in the sense of providing much clearer evidence of whether interventions are genuinely leading to the core programme goal of the GEC – the learning of marginalised girls. In this way, PbR, via the improved evaluation methods, has increased accountability for results on the programme.
In addition, we found evidence that the NGOs on the GEC largely saw the rigorous evaluation as a ‘step change’ in terms of what they were used to, and a big opportunity to prove that their models and organisations were capable of generating robust results. The fact that the evaluation ‘labelled success’ very clearly to all projects whether on PbR or not, was highly motivational for project staff, who wanted to succeed and have this recognised through the evaluation.
Recommendations from our research
Several recommendations emerge from the findings of our research in this light, which may be applicable to the design of PbR approaches in other programme contexts, particularly in donor contracting of NGOs as service providers:
- It is critical to consider whether a PbR downside will have more negative than positive consequences. In particular, whether smaller organisations may potentially fold due to lack of funds, and both small and larger organisations may face a significant opportunity cost from PbR losses particularly if they lack unrestricted reserves. Further, the potential that PbR downside may actively discourage or limit innovation needs to be taken into account.
- Maintaining a PbR mechanism can provide strong incentives for rigorous results measurement. By having a section of contracts with service providers linked directly to outcomes, programme managers and the donor have a clear priority to ensure rigour and robust measurement.
- A smaller PbR upside can be used in order to maintain the incentive for rigorous results measurement without having the costs and risks associated with PbR downside. Allowing the PbR upside to be channelled to unrestricted reserves can provide a strong motivation for smaller NGOs.
- It is critical to ensure the bureaucracy and time associated with changing project design during implementation is minimised to enable adaptive management to focus on outcomes, particularly in hybrid PbR models (where payments are both linked to outcomes and outputs/activities).
- A PbR incentive payment which is structured around longer term outcomes (measured after the end of intervention funding) could reward projects which are more genuinely sustainable and encourage programming to be more long-term, systemic and sustainable.
John Patch is an education adviser on the GEC Fund Manager team. John has worked on GEC since 2013, providing technical inputs on project monitoring, review and support, as well as thematic lesson learning and analysis. This has included issues such as EdTech, private sector engagement, school governance, and teacher training.
Joseph Holden is an independent consultant and Director of Foresight Development Associates, with evaluation experience on DFID programmes in a range or areas from financial services to education. He has been on the M&E team of the GEC Fund Manager since 2013 and has developed much of the guidance on evaluation used by the GEC's projects, including on the design of PbR.
Barder, Owen, Rita Perakis, William Savedoff, Theodore Talbot (2014) “12 Principles for Payment by Results (PbR) in the Real World.” Center for Global Development
Clist, Paul & Arjan Verschoor (2014) 'The Conceptual Basis of Payment by Results.' University of East Anglia, Norwich, UK. Research for Development, DFID.