Effective evaluation practices are essential to ensuring that the policies and programs delivered to Aboriginal and Torres Strait Islander people reflect best practice.  Not only is evaluation crucial to increasing evidence by providing positive exemplars, it also serves to identify and ameliorate or avoid unintended negative consequences of policies, programs or practices. Part of the role evaluation plays in increasing transparency is through making the goals, objectives and theorised pathways between these and the actions undertaken explicit.  However, increasing health equity through the practice of evaluation is rarely an explicitly stated goal of evaluation.

There is widespread agreement that evaluation in Aboriginal and Torres Strait contexts needs to incorporate ethical principles, including being culturally appropriate, but there is less consensus regarding what this looks like in practice (AHURI 2002).   The incorporation of Aboriginal and Torres Strait Islander perspectives in evaluation is imperative to reframe evaluation in order to reflect the priorities and expectations that Aboriginal and Torres Strait Islander communities have regarding the policies, programs and services that affect their health.  This necessitates research conduct in evaluation that has at its centre strong partnerships with Aboriginal and Torres Strait Islander communities and an emphasis on capacity-building.  Evaluation of health policy, programs and services also needs to consider the role of Aboriginal and Torres Strait Islander people and communities in the planning, development and implementation of the evaluation targets.  The presence or absence and quality of governance structures within programs, policies and services, including accountability mechanisms and decision-making processes must therefore be central to the evaluation framework. 

The current paper describes the development of a coherent framework that guides the evaluation of policies, programs and services to improve the health of Aboriginal and Torres Strait Islander people.  The project aims to ensure that delivering benefit to community is an explicit of goal of the conduct of evaluations as well as a standard element of evaluations in addition to the policies, programs and services that are being evaluated. This will ensure a greater focus on both Aboriginal and Torres Strait Islander engagement in governance, agenda-setting and capacity building.  The approach incorporates the work of Pratt and Loff’s Research for Health Justice framework (Pratt and Loff 2012) to orient research conduct in the evaluation of Aboriginal and Torres Strait Islander health policy, programs and services towards a health equity perspective, focusing on the involvement of local stakeholders in priority setting, community benefits before and after the research, capacity-building and research partnerships.  The project serves to improve the development of policies and programs that incorporate the viewpoints and values of Aboriginal and Torres Strait Islander communities, thereby supporting the creation of programs and services that are more relevant to and appropriate for the communities they serve.

In recent years, there has been a remarkable increase in the number of consensus statements used to inform policy and guidelines in public health and medicine. This stems primarily from the need to assist policymakers, clinicians and managers to understand and incorporate an increasingly complex body of biomedical research into practice. Further impetus for this trend comes from broader issues such as the by emergence of evidence-based medicine (Priest et al. 2014) and rising health care costs (Crengle et al. 2014; Kelaher et al. 2014). This phenomenon has simultaneously coincided with an exponential increase in the number of stakeholders in health and medicine. This has likely contributed to the increase in the number of agencies commissioning and interested in consensus statements. There is also an increasing trend for consensus statements to be used as a way not only to inform policy practice but as a way to provide public accountability for policy and clinical decisions. As a result of this, the quality of consensus statements has been criticised in terms of the process of their development, the nature of the consensus they achieve, and their impact on public and policy perceptions of the role of science. (Sarewitz 2011) The expertise, ideological and/or financial interests of relevant stakeholders participating in the generation of consensus statements has come into question. In addition, where power inequalities exist between stakeholders, it is likely that deliberative processes will merely reinforce the status quo, giving an effective voice only to those stakeholders with considerable power and resources. (Young 2000) As such, this paper presents explores the utility, value and risks associated with consensus statements. It presents the findings of qualitative interviews undertaken with commissioners, policy makers, practitioners and consumers of consensus statements in a multi-region comparison that looks at the US, Australia, the UK and Europe. Four key areas of consensus generation are examined in depth. These areas include: genetic testing, obesity, depression and medical imaging, specifically focusing on cardiovascular imaging.  It explores key issues such as deliberative structures, participation of stakeholders, the perceptions of the value and risks associated with consensus statements; reasons for choosing consensus statements over other approaches; selection of participants; consideration of conflicts of interest and relationships; level of participation in consensus statements by policy makers, researchers, practitioners and the public; processes of generating consensus; processes for ensuring qualitative equality; the nature of consensus achieved; concerns about the process of generating consensus; as well as positive and negative exemplars of experiences with consensus statements.

Deliberative and participatory forms of governance have been widely theorized to improve health service responsiveness, community empowerment, and political and administrative accountability, with an eventual impact on better health care access. In India, health was transferred to Panchayati Raj Institutions (PRI) as one of the subjects under the 73rd constitutional amendment, to facilitate decentralization. The study explicates the pathways through which institutions of local participatory governance (LPG) influence access to health services for the poor and vulnerable, through a case study of the Panchayati Raj Institution (PRI)- local governance system in Kerala state, India. We use actor-centred approaches for policy implementation to bring out the interactions and negotiations between groups of actors within complex social and organizational contexts. We followed principle of framework analysis for qualitative data analysis for applied policy research.

In India, LPG is synonymous with Panchayati Raj Institutions (PRI) locally elected bodies operating at village, sub-district and district levels with financial and administrative powers over social services. The state of Kerala has extensively implemented democratic decentralised reforms over two decades. The rich history of social movements and people’s campaign for decentralized planning adds to social capital and builds the context in the state. The emergent platforms for participation, formal and non-formal increased avenues for community participation. On one hand formal institutions extended support in form of mobilising funds, medicines, services from the health department, thus assuring the society about the legitimacy and public credence of such initiatives.  On other hand participation of non-formal structures in form of community volunteers, felt need identification and beneficiary identification gave a community ownership. These collective agencies enabled mobilization of different cultural, regional and religious characteristics of community. Being more local in nature, the nodes of decision making moved closer to people, thus creating more spaces that are transparent and promotes co-creation and interaction. Leadership by individuals and organizations has also emerged as pathway for better responsiveness. During the study there were numerous individuals’ names quoted who were instrumental in getting an initiative started or expanded.

Despite being able to introduce many innovations, initiatives and projects to respond to local people’s health needs there remains caveats. Challenges were closely linked to design of decentralization. Health sector was not completely devolved, administratively health personnel were accountable to their line department while functionally they were devolved to the LPGs. Structural factors also inhibited local governments to address issues of resource allocation between geographies, location of health centres and addressing staff vacancies.

Participatory governance is not a panacea for addressing issues of health inequity. Carefully created institutional mechanisms are required in the design of LPG to address the issues of marginalised sections and to promote equity. The big challenge is to restructure governance system to make it appropriate for decentralization and participation without losing its core essence of being accountable and transparent.

The impact of public performance reporting on quality of care: A multiple stakeholders’ perspectives

Khic-Houy Prang1, Rachel Canaway1, Marie Bismark1, David Dunt1 and Margaret Kelaher1

1 Centre for Health Policy, Melbourne School of Population and Global Health, The University of Melbourne, Melbourne, Australia

Background

Mandatory public performance reporting (PPR) of hospitals has come to be seen as a key tool in improving hospital quality internationally.  PPR is hypothesised to improve quality of care through two pathways: 1) leading consumers to select high quality healthcare providers and services; 2) eliciting organisational response to improve quality by identifying areas in which they underperform (1). 

In Australia, national mandatory PPR for public hospitals was introduced in 2011 with the launch of the MyHospitals website (2).  However, most states have their own websites and report on a greater range of performance variables (3, 4).  PPR is not mandatory for private hospitals, although some participate on a voluntary basis. 

Despite a shift to mandatory PPR in hospitals in Australia and around the world, evidence of its impacts on quality of care is mixed (5).  To date there has been no study of the impacts of PPR in Australia.

Aim

The aim of the project is to identify strategies to improve the impact of Australian PPR on quality of care in hospitals by examining the perspectives of multiple stakeholders.

Methods

Semi-structured interviews were conducted with 98 stakeholders in Australia (face to face or via telephone).  This included 7 healthcare consumer advocates, 19 purchasers (public and private funders of healthcare services), 15 providers (clinicians and administrators), 17 senior hospital executives (hospital chief medical officers and directors of medical service) and 40 general practitioners (GPs). All interviews were audio recorded and transcribed verbatim.  The data were analysed using thematic analysis.

Results

Stakeholders shared similar perceptions regarding the role of PPR of hospitals such as increase transparency, accountability and quality improvement activities.  However, it was unclear who the primary intended audience of PPR was (e.g. consumers, purchasers or providers).  Additional perceived barriers of PPR included: 1) conceptual (reporting framework); 2) systems-level (lack of consumer choice, lack of consumer and clinician involvement, jurisdictional barriers, lack of mandate for private sector reporting); 3) technical and resource related (data complexity, lack of data relevance consistency, rigour); and 4) socio-cultural (provider resistance to PPR, poor consumer health literacy, lack of consumer empowerment).  Suggested strategies for improving PPR included: 1) conceptual (definition of ‘public’); 2) framework development (primary audience and objective, desired outcomes and impacts, relevant data needs, data collection, reporting and feedback mechanisms); 3) systems-level considerations (public and private sector reporting, institutional reporting cultures, leverage reputational incentives, involve all relevant stakeholders).    

Discussion

Whist stakeholders were generally supportive of PPR; current PPR of hospital data appeared unlikely to influence consumer choice as it is unclear who the primary intended audience is.  Stakeholders highlighted the need to tailor and align the objective of PPR with its relevant audience (consumers, providers or purchasers) and audience needs in order to increase PPR awareness and usage, and to strengthen its impact on quality of care.  Multiple PPR frameworks may be required to suit different audiences. 

References

1.            Berwick DM, James B, Coye MJ. Connections between quality measurement and improvement. Medical Care. 2003;41(1):I-30-I-8.

2.            Australian Institute of Health and Welfare. MyHospitals 2017 [16 January 2017]. Available from: http://www.myhospitals.gov.au/.

3.            Government of Western Australia Department of Health. Our performance 2017 [16 January 2017]. Available from: http://ww2.health.wa.gov.au/Our-performance.

4.            Bureau of Health Information. Hospital quartely 2017 [16 January 2017]. Available from: http://www.bhi.nsw.gov.au/BHI_reports/hospital_quarterly/HQ26.

5.            Chen J. Public reporting of health system performance: A rapid review of evidence on impact on patients, providers and healthcare organisations. Evidence Check. 2010.

Over the past few decades, interest has grown in the decentralisation of governing across fields as diverse as education, health, and justice (Atkinson, Medeiros, Oliveira, & de Almeida, 2000; Morgan, 2005; Stenson, 2005), enacted through a reduction in rigid hierarchical structures, increasing partnerships with civil society and localised decision-making processes—an impulse that has gone hand-in-hand with the rising discourse of community participation in health. Community participation in health is particularly relevant for Indigenous peoples, who are often excluded from decision-making and priority-setting in the development and implementation of health services. Indigenous community participation in health is not only a response to systemic barriers in accessing biomedical health care or a lack of culturally appropriate health care, but is underpinned by the principles of self-determination, autonomy, social justice and cultural retention.

Indigenous community participation in health in Chile has led to the development and implementation of a number of health services administered and governed by Indigenous community organisations. However, these decentralised governance structures, established to enable localised influence over policies, processes and services, are nevertheless contradictory in that they are strongly shaped by State power. This paper examines how staff from Chilean Indigenous community health care services perceive the concept and purpose of Indigenous community participation in relation to State policies and frameworks (Foucault, 1991; Swyngedouw, 2005; Taylor, 2007).

In examining the perceived role of Indigenous community participation in health, there were apparent tensions between the themes of community ‘empowerment’ and ‘responsibilisation’ (Clarke, 2005; Taylor, 2007). Taken as a whole, interviews with staff from community controlled health centres highlighted a dual discourse around community participation and the relationship between community organisations and the State: interviewees spoke about the responsibilisation of Indigenous communities on the one hand—that is, devolution to communities of the responsibility for service provision and attendant risks that would otherwise lie with the State—and on the other, a desire to increase Indigenous autonomy in health and more generally through the assumption of roles that allow for engagement and representation within the Chilean State. Corresponding processes are those by which the State shifts responsibility for service provision to communities and community organisations, but retains control of the ways service is provided through imposition of regulations and obligations, normally tied to financial arrangements (Taylor, 2007). Here, interviewees articulated the utilisation of participatory processes to increase Indigenous independence while at the same time being mindful of the necessity of compliance with such regulations imposed by the State in order to maintain this position.

The imposition of State-centred regulation onto Indigenous governance structures as a necessary part of the development of participatory models has profound implications for the capacity of Indigenous communities to realise their desire for independence and autonomy through these mechanisms. Consideration therefore needs to be given to how best to balance the ethical considerations such processes present.

Vivian Lin, lead author in international organizations and technologies of governance, and chair of cross-cutting thematic group on health 

Since 2015, some 300 social scientists have been involved in reviewing the state of social progress in the world, ie what the evidence is about what gives rise to social and economic inequalities, what measures improve them, and what is the prospect for social and economic transformations. This process of review and deliberation has now produced preliminary findings and policy recommendations. In nearly all sectoral policy areas, the dynamics between collective action and institutions, interconnected at local, national and global levels.

This presentation will reflect on the processes of the project as well as report on 2 areas of work within the IPSP (the influence of international organizations and technologies of governance in shaping social progress, and global health), framed by the overall project. International organizations commonly use a mix of four technologies of governance – legal instruments, accountability processes, indicators, and participatory mechanisms. In the health sphere, as with other arenas, the interactions with non-state actors inform the use of all governance strategies, although power differentials across non-state actors add complexities to the dynamics of policy-making and governance. Nonetheless, if gains to date in social progress are to be safeguarded, as well as further advances be made, stronger participatory and deliberative mechanisms will be necessary.