Format

P12. Evaluating Performance-based Financing in Health

POLICY DESIGN, POLICY ANALYSIS, EXPERTISE AND EVALUATION
PANEL CHAIR(S)
Holly Nazar
McGill University Institute for Health and Social Policy
Main chair
Marnie Davidson
CARE Canada
Second chair
Arijit Nandi
McGill University
Third chair
CATEGORISATION
POLICY TOPIC
POLICY DESIGN, POLICY ANALYSIS, EXPERTISE AND EVALUATION
SECTOR
—
KEYWORDS
—
GENERAL OBJECTIVES, RESEARCH QUESTIONS AND SCIENTIFIC RELEVANCE

Performance-based financing (PBF) continues to grow in popularity as a method to finance health care systems and development aid projects. Both the World Bank and World Health Organization offer guidance on designing and implementing PBF programs, and the World Bank-managed Health Results Innovation Trust Fund has committed US$385.6 million to results-based financing programs, in addition to US$2 billion in loan financing.[i] Performance-based financing is a term for supply-side results-based financing: “a form of incentive where health providers are, at least partially, funded on the basis of their performance to meet targets or undertake specific actions.”[ii]

 

Proponents argue that PBF can achieve fast improvements in service coverage and efficiency, and some reviews suggest that overall this is the case.[iii] However, the quality of evidence is still weak.[iv] Few studies of PBF health programs have rigorously evaluated impacts on health and equity outcomes for affected populations. Significant research gaps exist: what are the impacts of PBF on aspects of health care quality that are difficult to measure with short-term quantitative indicators? What perverse incentives can be created by PBF schemes and how can they be monitored and mitigated? How does PBF measure up compared to other health care improvement measures also associated with PBF, such as increased funding, training and/or flexibility for providers? What is the evidence for the long-term effects of PBF in health programs?[v] A wider question that may also be informed by population-level impact evaluation is how PBF programs are interacting with efforts to implement universal health coverage in developing countries.[vi]

 

Research Question:

· How can the effectiveness of performance-based financing models for health outcomes be evaluated?

 

Objectives:

· Bring together contributions from researchers who have or are carrying out evaluations of the health outcomes of PBF programs in the health sector, in developing or high-income countries, using quantitative, qualitative or mixed methods.

· Provide a forum to discuss work in progress in this area and proposals for future research directions

· Establish where data collection needs to be improved to make impact evaluation possible

· Overall, develop consensus on the quality of evaluation still needed in the area of PBF and on action needed from program implementers to facilitate rigorous evaluation



[i] RBF Health. (2014) “Mission.” https://www.rbfhealth.org/mission
[ii] World Health Organization. (2018). “Performance-based Financing.” http://www.who.int/health_financing/topics/performance-based-financing/en/
[iii] Suthar et al. “Performance-based financing for improving HIV/AIDS service delivery: a systematic review.” BMC Health Services Research (2017) 17:6; Grittner A. “Results-based financing: Evidence from performance-based financing in the health sector.” German Development Institute discussion paper, June 2013. 
[iv] Turcotte-Tremblay et al. “Does performance-based financing increase value for money in low- and middle- income countries? A systematic review.” Health Economics Review (2016) 6:30; Grittner A. “Results-based financing: Evidence from performance-based financing in the health sector.” German Development Institute discussion paper, June 2013. 
[v] Grittner A. “Results-based financing: Evidence from performance-based financing in the health sector.” German Development Institute discussion paper, June 2013. 
[vi] Soucat et al. “Pay-for-Performance Debate: Not Seeing the Forest for the Trees.” Health Systems & Reform, 3:2, 74-79.

CALL FOR PAPERS

This panel invites papers evaluating the effectiveness of performance-based financing models for improving health outcomes. Research projects can be either in-progress or complete, and papers by non-academics with experience of performance-based financing are welcome. Study methods can be quantitative, qualitative or mixed. The panel chairs will also accept discussion papers outlining how implementation methods (data collection, etc.) could better facilitate evaluation.

 

Performance-based financing (PBF) continues to grow in popularity as a method to finance health care systems and development aid projects. Performance-based financing is a term for supply-side results-based financing: “a form of incentive where health providers are, at least partially, funded on the basis of their performance to meet targets or undertake specific actions” (World Health Organization).

 

Proponents argue that PBF can achieve fast improvements in service coverage and efficiency, and some reviews suggest that overall this is the case. However, the quality of evidence is still weak.  Only a minority of studies of PBF health programs have rigorously evaluated impacts on health and equity outcomes for affected populations using a randomized controlled trial design or quasi-experimental observational methods. Significant research gaps exist: what are the impacts of PBF on aspects of health care quality that are difficult to measure with short-term quantitative indicators? What perverse incentives can be created by PBF schemes and how can they be monitored and mitigated? How does PBF measure up compared to other health care improvement measures also associated with PBF, such as increased funding, training and/or flexibility for providers? What is the evidence for the long-term effects of PBF in health programs? A wider question that may also be informed by population-level impact evaluation is how PBF programs are interacting with efforts to implement universal health coverage in developing countries.

 

Objectives:

· Bring together contributions from researchers who have or are carrying out rigorous impact evaluations of PBF programs in the health sector, in developing or high-income countries

· Provide a forum to discuss work in progress in this area and proposals for future research directions

· Establish where data collection needs to be improved to make impact evaluation possible

· Overall, develop consensus on the quality of evaluation still needed in the area of PBF and on action needed from program implementers to facilitate rigorous evaluation

ACCEPTED
Format
01 - Determinants of sustainability: a case study of a PBF pilot project in Burkina Faso
Mathieu Seppey (Institut de recherche en santé publique)
CO-AUTHORS
—
ABSTRACT

Many sub-Saharan countries are reforming their health system through performance-based financing (PBF) programs, aiming for more and better quality services. Pilot projects are therefore implemented to set the basis of these reforms. After a pre-pilot in 3 health districts, such pilot was implemented in 15 districts across 3 regions in Burkina Faso, involving more than 550 health centres. However, after more than 3 years of implementation, this pilot’s sustainability is still not guaranteed.

 

The objective of this study is to assess the level of sustainability of this project by identifying the presence/absence of the different determinants of sustainability in accordance to a conceptual framework made to assess sustainability. The research design is a case study and involved in-depth interviews with a variety of actors: implementers (N=7), decision-makers (N=6), health practitioners (N=8), managers (N=6), evaluators (N=2), external experts (N=3) and other actors linked to the project (N=5) from primary health care centres, district/regional hospitals or government agencies and directions. A thematic analysis based on the conceptual framework was conducted, as well as a more inductive analysis.

 

The results show that the PBF project’s sustainability level was weak according to an unequal presence of sustainability’s determinants; some activities are being maintained but not fully routinized. Discrepancies between the project’s values and those at the individual, organisational or governmental levels were noted to be an important barrier towards sustainability. Project’s ownership by key stakeholders also seemed superficial despite the implementers’ leadership and actions towards the project’s success. The pilot project’s objective aiming for greater autonomy for health centres was also in direct confrontation with the Burkinabe’s hierarchical health system.

 

Therefore, PBF’s sustainability goes further than financial aspects and must account for other determinants such as stakeholders’ ownership and motivation. PBF’s motivational theory (e.g.: primes, supervision) need to be put back into the different contexts where it has been implemented. Many arguments about its ineffectiveness are already identified and can undermine the project’s core mechanisms. Therefore, a more contextual “motivational cocktail” should be elaborated, taking into account the different stakeholders (ex.: decision makers, health practitioners, or managers) to facilitate the project’s sustainability. These lessons should thus be applied to other contexts such as in sub-Saharan countries, maybe more specifically in West-African countries.

ACCEPTED
Format
02 - An assessment of efficiency across health facilities before and after the implementation of performance-based financing in Tanzania: A Data Envelopment Analysis
Binyaruka Peter (Ifakara Health Institute)
CO-AUTHORS
Peter Binyaruka (Ifakara Institute)
ABSTRACT

Background

Emerging evidence shows that most health facilities in low and middle-income countries could improve their technically efficiency, which implies that the use of significant share of limited resources could be more effective. Performance-based financing (PBF), which provides financial incentives for health workers and/or facilities reaching pre-defined targets, are gaining popularity as a means of improving facility performance and health system strengthening by incentivizing the achievement of pre-determined targets. PBF would therefore be expected to enhance technical efficiency across facilities by promoting an output oriented financing system. However, there is no study which has systematically assessed efficiency scores across facilities before and after the introduction of PBF. This paper seeks to fill this knowledge gap by analyzing a PBF scheme implemented in Tanzania.

 

Methods

We build on previous impact evaluation and use data on health care inputs and outputs from 75 health facilities (6 hospitals, 16 health centres and 53 dispensaries) in the intervention region of Pwani and 75 selected facilities from control districts. Data were collected in two rounds –January 2012 and 13 months later through facility survey. We measure technical efficiency using Data Envelope Analysis (DEA) technique relating facility inputs (number of staff, beds, drugs and equipment) and outputs (outpatient visits, antenatal care visits, normal deliveries), representing facility resources and output profiles respectively. We also obtain an efficiency score for each facility before and after the implementation of PBF. We then analyze which factors influence technical efficiency by regressing the efficiency scores over a number of contextual factors. We also test the impact of PBF on efficiency through a difference-in-differences analysis. Sensitivity analysis will be carried out to account for multiple outputs and quality of outputs.

 

Results

Efficiency scores will be presented to show the degree of technical inefficiency across facilities and the factors affecting it. Preliminary results from the difference-in-difference analysis suggest that efficiency did not increase in PBF facilities compared to non PBF ones.

 

Discussion

Preliminary results indicate that the positive impact of PBF on a number of outcomes is a reflection of increased inputs, rather than a significant change in the way that health facilities use resources to reach outcomes. Understanding the level of technical inefficiencies across facilities before and after PBF is crucial to inform better design of PBF or scale-up for improving overall health system’s performance. Identifying potential factors and/or challenges influencing facility efficiency is an important step for policy makers to address them, and enhance optimal use of limited resources.

ACCEPTED
Format
03 - A comprehensive mixed-method policy framework to evaluate how the re-design and implementation of a P4P program can effectively affect health system strengthening: and applying it to the Brazilian case (PMAQ)
Stephen Peckham (Centre for Health Services Studies, University of Kent)
CO-AUTHORS
Fabiana SADDI (Universidade de Brasília)
Germano Araujo Coelho (Universidade Federal de Goiás)
ABSTRACT

Background

This paper presents the detailed design of a mixed-method political framework to evaluate and compare the re-formulation and implementation process of a P4P/PBF program and explore how those policy process can effectively affect system strengthening. The framework design and forms of analysis is systematically presented in a general form, as well as in specific form - related to the Brazilian program(PMAQ). The aim is to explore if/how those inter-related processes can affect leadership and workforce in health unities with diverse management/leadership arrangements and organizational capacities. Being based on general and specific policy drivers, broadly adopted by Public Policy and Health System, the framework can be employed to uncover the policy dynamics and effectiveness of P4P/PBF in HIC and LMIC.

 

Questions

Who were the main actors, ideas and interests involved in the reformulation of P4P/PMAQ and to what extent the reformulation has been characterised by policy learning, feedback and communicative strategies with front liners? To what extent front liners health workers know the program and how participated in the implementation of the program? How the main performance drivers (policies, structures, behaviour) produced change in the leadership process of work at the front line? These questions and related assumptions will be presented and analysed in a general format, as well as related to the Brazilian case.

 

Methodology

The framework employs a comprehensive literature review, analysis of documents, interviews with key stakeholders and secondary analysis of data. Interviews will be made with policymakers and assessors and with policy managers and front liners in the cities. Data from each literature/transcript will be coded and analysed under determined policy and performance drivers. Combinations of policy drivers will be associated with effective levels of integration(ELIs) between implementation and re-formulation, while combinations of performance drivers will be classified as levels of perceived impact on performance(LPIPs). In the case and comparative analyses, we will explore the relationship between ELIs and LPIPs.

 

Also, secondary data will be organised and classified into levels of result-based performance(LRBPs). We will then triangulate/integrate the data between ELIs and LRBPs, exploring the extent to which qualitative process indicators(ELIs) are associated with result-based indicators(LRBPs) in the case and comparative analyses. Our main assumption is that integrative policy drivers are capable of strengthening HSSBB. Using framework analysis will enable us to better understand how/in what conditions integrative policy processes(ELIs) have, and can affect process(LPIPs) and result-based(LRBPs) performance indicators, highlighting changes that strengthen leadership and workforce. We will present the general framework and its assumptions/hypothesis and operationalization to the Brazilian case as well. This project is developed under the British Academy Newton Advanced Fellowship.

 

Fits with the panel: It is directly connected with the question: How can the effectiveness of performance-based financing models for health outcomes be evaluated?

ACCEPTED
Format
04 - How Performance-Based Financing Can Improve Equity of Access to Health Services in Rural Areas in Mozambique?
Octavio de Sousa (CARE International in Mozambique)
CO-AUTHORS
—
ABSTRACT

Performance-based Incentives (PBIs) is increasingly recognized as an effective approach to improve healthcare delivery to vulnerable populations. In recent years, different International NGOs have started to implement PBIs initiatives in order to increase the uptake and quality of health services in Mozambique, where over 60% of population lives in rural areas with limited access to health services.     

 

CARE International in Mozambique, in a partnership with Cornell University was one of the organization that have pilot the impact of Performance-Based Incentives on the quantity of services to prevent vertical transmission of HIV and evaluate the effects of PBIs on the pathway to service delivery for different health worker cadres. We selected two districts (implementation and control), and provide performance-based incentives for health centers workers and community volunteers.

To evaluate the initiative, we conducted 24 semi-structured interviews with nurses, volunteers, community health workers, and traditional birth attendants about the barriers and promoters they experienced delivering PVT services. We then explored emergent themes in subsequent focus group discussions.

 

The evaluation shows that all health worker cadres were motivated but at the same time, they were dissatisfied with several factors that ultimately affected how their “performance” was measured. For example, may of the factors were outside their direct control, such as lack of medicines (stock-outs); high patient volumes at times, poor infrastructure such as lack of privacy for counseling and giving birth. Furthermore, much of these challenges were encountered by pregnant women and mothers living with HIV in accessing health services e.g., long distances between communities and health facilities continue to affect the accessibility and quality of services provided by health workers.

 

PBIs have showed an increase and improvement of coordination and collaboration between the different health workers cadres, including community-based volunteers, responsible by home visits and referrals to the health facilities, and traditional birth attendants. This increased motivation, coordination and service quality, shows an enormous potential of the PBIs to improve the provision of health services, but become critical to link the PBIs initiatives with different strategies that can help to improve the administrative system and the collaboration between the health centers and different stakeholders at the community level.

 

Therefore, accessibility and quality of health services cannot be ensured only through the provision of incentives. Many health workers are overburdened and under-resourced to carry out their work and these facts are conditioning the quality of the services provided. PBIs represents a great potentiality (motivated coordinating and increase structured supervision) but insufficient to impact on accessibility.

 

Authors:

Octávio de Sousa & Delphine Pinault – CARE International

Roseanne C. Schuster & Sera Young – Cornell University

CO-AUTHORS
—
ABSTRACT

Many sub-Saharan countries are reforming their health system through performance-based financing (PBF) programs, aiming for more and better quality services. Pilot projects are therefore implemented to set the basis of these reforms. After a pre-pilot in 3 health districts, such pilot was implemented in 15 districts across 3 regions in Burkina Faso, involving more than 550 health centres. However, after more than 3 years of implementation, this pilot’s sustainability is still not guaranteed.

 

The objective of this study is to assess the level of sustainability of this project by identifying the presence/absence of the different determinants of sustainability in accordance to a conceptual framework made to assess sustainability. The research design is a case study and involved in-depth interviews with a variety of actors: implementers (N=7), decision-makers (N=6), health practitioners (N=8), managers (N=6), evaluators (N=2), external experts (N=3) and other actors linked to the project (N=5) from primary health care centres, district/regional hospitals or government agencies and directions. A thematic analysis based on the conceptual framework was conducted, as well as a more inductive analysis.

 

The results show that the PBF project’s sustainability level was weak according to an unequal presence of sustainability’s determinants; some activities are being maintained but not fully routinized. Discrepancies between the project’s values and those at the individual, organisational or governmental levels were noted to be an important barrier towards sustainability. Project’s ownership by key stakeholders also seemed superficial despite the implementers’ leadership and actions towards the project’s success. The pilot project’s objective aiming for greater autonomy for health centres was also in direct confrontation with the Burkinabe’s hierarchical health system.

 

Therefore, PBF’s sustainability goes further than financial aspects and must account for other determinants such as stakeholders’ ownership and motivation. PBF’s motivational theory (e.g.: primes, supervision) need to be put back into the different contexts where it has been implemented. Many arguments about its ineffectiveness are already identified and can undermine the project’s core mechanisms. Therefore, a more contextual “motivational cocktail” should be elaborated, taking into account the different stakeholders (ex.: decision makers, health practitioners, or managers) to facilitate the project’s sustainability. These lessons should thus be applied to other contexts such as in sub-Saharan countries, maybe more specifically in West-African countries.

CO-AUTHORS
Peter Binyaruka (Ifakara Institute)
ABSTRACT

Background

Emerging evidence shows that most health facilities in low and middle-income countries could improve their technically efficiency, which implies that the use of significant share of limited resources could be more effective. Performance-based financing (PBF), which provides financial incentives for health workers and/or facilities reaching pre-defined targets, are gaining popularity as a means of improving facility performance and health system strengthening by incentivizing the achievement of pre-determined targets. PBF would therefore be expected to enhance technical efficiency across facilities by promoting an output oriented financing system. However, there is no study which has systematically assessed efficiency scores across facilities before and after the introduction of PBF. This paper seeks to fill this knowledge gap by analyzing a PBF scheme implemented in Tanzania.

 

Methods

We build on previous impact evaluation and use data on health care inputs and outputs from 75 health facilities (6 hospitals, 16 health centres and 53 dispensaries) in the intervention region of Pwani and 75 selected facilities from control districts. Data were collected in two rounds –January 2012 and 13 months later through facility survey. We measure technical efficiency using Data Envelope Analysis (DEA) technique relating facility inputs (number of staff, beds, drugs and equipment) and outputs (outpatient visits, antenatal care visits, normal deliveries), representing facility resources and output profiles respectively. We also obtain an efficiency score for each facility before and after the implementation of PBF. We then analyze which factors influence technical efficiency by regressing the efficiency scores over a number of contextual factors. We also test the impact of PBF on efficiency through a difference-in-differences analysis. Sensitivity analysis will be carried out to account for multiple outputs and quality of outputs.

 

Results

Efficiency scores will be presented to show the degree of technical inefficiency across facilities and the factors affecting it. Preliminary results from the difference-in-difference analysis suggest that efficiency did not increase in PBF facilities compared to non PBF ones.

 

Discussion

Preliminary results indicate that the positive impact of PBF on a number of outcomes is a reflection of increased inputs, rather than a significant change in the way that health facilities use resources to reach outcomes. Understanding the level of technical inefficiencies across facilities before and after PBF is crucial to inform better design of PBF or scale-up for improving overall health system’s performance. Identifying potential factors and/or challenges influencing facility efficiency is an important step for policy makers to address them, and enhance optimal use of limited resources.

CO-AUTHORS
Fabiana SADDI (Universidade de Brasília)
Germano Araujo Coelho (Universidade Federal de Goiás)
ABSTRACT

Background

This paper presents the detailed design of a mixed-method political framework to evaluate and compare the re-formulation and implementation process of a P4P/PBF program and explore how those policy process can effectively affect system strengthening. The framework design and forms of analysis is systematically presented in a general form, as well as in specific form - related to the Brazilian program(PMAQ). The aim is to explore if/how those inter-related processes can affect leadership and workforce in health unities with diverse management/leadership arrangements and organizational capacities. Being based on general and specific policy drivers, broadly adopted by Public Policy and Health System, the framework can be employed to uncover the policy dynamics and effectiveness of P4P/PBF in HIC and LMIC.

 

Questions

Who were the main actors, ideas and interests involved in the reformulation of P4P/PMAQ and to what extent the reformulation has been characterised by policy learning, feedback and communicative strategies with front liners? To what extent front liners health workers know the program and how participated in the implementation of the program? How the main performance drivers (policies, structures, behaviour) produced change in the leadership process of work at the front line? These questions and related assumptions will be presented and analysed in a general format, as well as related to the Brazilian case.

 

Methodology

The framework employs a comprehensive literature review, analysis of documents, interviews with key stakeholders and secondary analysis of data. Interviews will be made with policymakers and assessors and with policy managers and front liners in the cities. Data from each literature/transcript will be coded and analysed under determined policy and performance drivers. Combinations of policy drivers will be associated with effective levels of integration(ELIs) between implementation and re-formulation, while combinations of performance drivers will be classified as levels of perceived impact on performance(LPIPs). In the case and comparative analyses, we will explore the relationship between ELIs and LPIPs.

 

Also, secondary data will be organised and classified into levels of result-based performance(LRBPs). We will then triangulate/integrate the data between ELIs and LRBPs, exploring the extent to which qualitative process indicators(ELIs) are associated with result-based indicators(LRBPs) in the case and comparative analyses. Our main assumption is that integrative policy drivers are capable of strengthening HSSBB. Using framework analysis will enable us to better understand how/in what conditions integrative policy processes(ELIs) have, and can affect process(LPIPs) and result-based(LRBPs) performance indicators, highlighting changes that strengthen leadership and workforce. We will present the general framework and its assumptions/hypothesis and operationalization to the Brazilian case as well. This project is developed under the British Academy Newton Advanced Fellowship.

 

Fits with the panel: It is directly connected with the question: How can the effectiveness of performance-based financing models for health outcomes be evaluated?

CO-AUTHORS
—
ABSTRACT

Performance-based Incentives (PBIs) is increasingly recognized as an effective approach to improve healthcare delivery to vulnerable populations. In recent years, different International NGOs have started to implement PBIs initiatives in order to increase the uptake and quality of health services in Mozambique, where over 60% of population lives in rural areas with limited access to health services.     

 

CARE International in Mozambique, in a partnership with Cornell University was one of the organization that have pilot the impact of Performance-Based Incentives on the quantity of services to prevent vertical transmission of HIV and evaluate the effects of PBIs on the pathway to service delivery for different health worker cadres. We selected two districts (implementation and control), and provide performance-based incentives for health centers workers and community volunteers.

To evaluate the initiative, we conducted 24 semi-structured interviews with nurses, volunteers, community health workers, and traditional birth attendants about the barriers and promoters they experienced delivering PVT services. We then explored emergent themes in subsequent focus group discussions.

 

The evaluation shows that all health worker cadres were motivated but at the same time, they were dissatisfied with several factors that ultimately affected how their “performance” was measured. For example, may of the factors were outside their direct control, such as lack of medicines (stock-outs); high patient volumes at times, poor infrastructure such as lack of privacy for counseling and giving birth. Furthermore, much of these challenges were encountered by pregnant women and mothers living with HIV in accessing health services e.g., long distances between communities and health facilities continue to affect the accessibility and quality of services provided by health workers.

 

PBIs have showed an increase and improvement of coordination and collaboration between the different health workers cadres, including community-based volunteers, responsible by home visits and referrals to the health facilities, and traditional birth attendants. This increased motivation, coordination and service quality, shows an enormous potential of the PBIs to improve the provision of health services, but become critical to link the PBIs initiatives with different strategies that can help to improve the administrative system and the collaboration between the health centers and different stakeholders at the community level.

 

Therefore, accessibility and quality of health services cannot be ensured only through the provision of incentives. Many health workers are overburdened and under-resourced to carry out their work and these facts are conditioning the quality of the services provided. PBIs represents a great potentiality (motivated coordinating and increase structured supervision) but insufficient to impact on accessibility.

 

Authors:

Octávio de Sousa & Delphine Pinault – CARE International

Roseanne C. Schuster & Sera Young – Cornell University