Format

W01. Defining Policy Success and Failure as Evaluation Challenges. Focusing on outcomes: Blessing or a curse for policymakers?

Policy Design, Evaluation, Policy Analysis
PANEL CHAIR(S)
Erica Melloni
Universitas Mercatorum
Main chair
Robert Shepherd
Carleton University
Second chair
Giancarlo Vecchi
Politecnico di Milano
Third chair
Dominika Wojtowicz
Kozminski University
Fourth chair
CATEGORISATION
POLICY TOPIC
Policy Design, Evaluation, Policy Analysis
SECTOR
—
KEYWORDS
—
GENERAL OBJECTIVES, RESEARCH QUESTIONS AND SCIENTIFIC RELEVANCE

Introduction

Outcome measures are designed to assess the contribution of public policies and programs to the improvement of social issues. Outcomes identification and measurement serve to assess the policy and program’s effectiveness and to improve its implementation and accountability. As far as effectiveness, public interventions are scrutinized not only for performance but also in their capacity to ‘make a difference’ or ‘drive a change’ concerning the issues to be tackled (Chen, 1990; Davies, 2004). As far as accountability, evaluation has been widely acknowledged as a means for empowering citizens (Dahlher-Larson, 2011), when it makes available to decision-makers and citizens reliable knowledge on which to build their judgments and choices.

The focus on the ‘end’ results of public policy and programs challenges policy makers and civil servants to understand the achievement of performance results to the value and purposiveness of public action. In other words, the outcomes perspective entails a judgement of success or failure of public programs (Bovens et al. 2001; Compton & t’Hart, eds., 2019; McConnel et al. 2020). This perspective requires greater clarity regarding the intervention’s logic, design, conduct and implementation as well as the goals to be achieved. The payment by results are examples of instruments, especially popular in Anglo-Saxon countries, that seek to armour the achievement of social outcomes, with the application (e.g., to health and judicial policies) (Farrar et al., 2009; Fox and Albertson, 2011). In Europe, both the Cohesion Funds and the post-Covid ‘Next Generation EU’ program, have intensified the linkages between funded programs and well-defined outcomes to be achieved: the NGEU will reimburse the member states’ expenses only after the realisation of specific outcomes. Within this framework, the appropriate identification of the results to be achieved and of the method of measurement becomes of paramount importance.

The transition towards outcomes has been welcomed by the evaluation community. Notwithstanding, shortcomings emerge in the day-to-day evaluation practice, making the role of policy advice challenging, as well as communication with stakeholder groups. The appreciation of outcomes initially established could be unsatisfactory or even misplaced, for instance: in case of poor or conflicting policy design (Compton et al. 2021); uneven design and implementation processes (Shepherd, 2011); change in the political mood; unexpected conflicts, or external events (Covid pandemic, Russia-Ukraine war, etc) that substantially changed the salience of the intervention or the resources available at the beginning (McConnell, 2010). The process of determining and measuring policy success or failure is moreover challenged when public interventions are either inconsistent or complex (e.g., in the case of multilevel governance, where responsibility for policy planning and implementation is dispersed between different levels of administration). These may include different policies to solve crosscutting issues or wicked problems (Bailie et al., 2022; Paparini et al., 2020; Daviter, 2017).

Various authors have noted that a linear and entirely predictable policy design is not suitable for the more complex and challenging policy domains. Theory-based evaluations (Chen, 1990; Weiss, 1997; Chen, 2012; Mayne, 2015) and realist designs based on the CMO or the CIMO configuration (Pawson & and Tilley, 1997; Pawson, 2006; Pedersen and Rieper, 2008; Punton et al., 2020), suffer from the tendency to represent oversimplified policy processes, without adequately considering the complexity of these processes and positive and negative, expected and unexpected outcomes. The advocates of ‘formative’ evaluations (Scriven, 1991, 1996; Stame, 2004) have underlined the shortcomings of evaluating complex policies and programs, and the consequent need for more open-ended evaluation designs, that in turn could be less able to synthesize clear-cut judgements about the program’s ultimate results. On their side, the quasi-experimental and formal evaluation advocates – or as they have been called, the ‘truth-seeking’ or positivist stream of evaluation theory (Shadish et al., 1991) - struggle to take into consideration the implementation process and its reasons for particular designs (Cook et al., 1979), thus risking an adequate interpretative framework for evaluation findings, or sometimes not focusing on the most relevant outcomes in the eyes of beneficiaries. In the case of payment by results tools, the push on outcomes could trigger a ‘creaming and parking mechanism’ (the process in which providers select job-ready clients for services and neglect clients more distant from the achievement of the goals (Greer et al., 2018). In this case, an evaluation design too fixated on pre-defined outcomes could provide opaque and incomplete findings, and not serve the scope of empowering citizens' decisions – at least not in an acceptable way.

Starting from the public policy debate on policy success and failure, we underline the need to bridge policy evaluation with policy process studies that tend to focus almost exclusively on goals.

 

Aims of the workshop & main questions

 

The workshop aims to gather contributions and experiences of public policy evaluation scholars who have addressed critical issues about the relationships between programmes, the evaluation of policy success and failures, and the capacity of evaluation to provide relevant policy advice with useful information to citizens and stakeholder groups.

In particular, the panel will raise the following research questions:

·       How does policy coherence (or the lack thereof) affect policy implementation in complex governance structures?

·       What are the risks inherent in understanding policy outcomes that focus mainly on policy aims at the expense of implementation?

·       What are the consequences of evaluation-policy process dysfunctions in terms of methodological choices, evaluation policy process, and relevance of evaluation findings for citizens and other stakeholder groups?

·       Are there epistemological questions, approaches and solutions to cope with these challenges?

·       Are there evaluation strategies and tools, including AI-based analytical instruments, able to mitigate or strengthen these relationships and challenges?

·       How do policy makers and policy advisors take into account and overcome these challenges? Are there systemic approaches being considered over episodic ones?

 

Expected Impact of the Workshop

 

The workshop findings would allow not only the evaluation scholars but the larger ICPP community to understand the main challenges in the way policy and programs are designed (i.e., goals, implementation features, incentives and other mechanisms, communication strategies); the capabilities of determining policy success and failure; and, in providing citizens and other stakeholder groups with valuable information deriving from the evaluation process.

The workshop’s findings would be suitable for a special journal issue or for an edited book.

 

Possible articles:

·      Giancarlo Vecchi and Erica Melloni (Politecnico di Milano). Was it a success or a failure? Methodological and conceptual challenges of evaluating justice sector innovation programmes. The case of justice reform in Italy was funded by the NGEU.

·      Simone Busetti (Università di Teramo). Discovering policy outcomes: effectiveness, extrapolation and policy transfer.

·      Dominika Wojtowicz (Kozminski University), Challenges of evaluation of incoherent policy. Assessing policy of Polish cities toward integration of refugees from Ukraine.

·      Robert Shepherd (Carleton). Moving the Canadian, UK and Australian federal evaluation systems from focusing on expenditure management to informing sound policy choices.

 

References

Bailie, J.; Cunningham, F.; Abimbola, S.; Laycock, A.; Bainbridge, R.; Bailie, R.; Conte, K.; Passey, M. & Peiris, D. (2022) Methodological pluralism for better evaluations of complex interventions: lessons from evaluating an innovation platform in Australia. Health Research Policy & Systems. Vol. 20(1): 1-14.

Bovens M., P. ‘T Hart & G.B. Peters. Eds.(2001). Success and Failure in Public Governance. A Comparative Analysis. Cheltenham: Edward Elgar.

Chen H.-T. (1990) Theory-driven evaluations. Sage, Newbury Park, Calif.

Chen H. T. (2012) Theory-driven evaluation: Conceptual framework, application and advancement. Springer.

Compton M.E., J. Luetjens & P. ‘t Hart (2019). "Designing for Policy Success." International Review of Public Policy 1(2): 119-146.

Compton M.E. & P. ‘t Hart. eds. (2019). Great Policy Success. Oxford: Oxford University Press.

Cook T. D., Campbell D. T. and Day A. (1979) Quasi-experimentation: Design & analysis issues for field settings. Houghton Mifflin, Boston.

Dahler-Larsen P. (2011) The evaluation society. Stanford University Press.

Farrar S., Yi D., Sutton M., Chalkley M., Sussex J. And Scott A. (2009) Has payment by results affected the way that English hospitals provide care? Difference-in-differences analysis, Bmj 339.

Fox C. And Albertson K. (2011) Payment by results and social impact bonds in the criminal justice sector: New challenges for the concept of evidence-based policy?, Criminology & Criminal Justice 11, 395-413.

Greer I., Schulte L. and Symon G. (2018) Creaming and parking in marketized employment services: An Anglo-German comparison, human relations 71, 1427-53.

Mayne J. (2015) Useful theory of change models, Canadian Journal of Program Evaluation 30.

Pawson R. (2006) Evidence-Based Policy : A Realist Perspective. Sage, London.

Pawson R. and Tilley N. (1997) Realistic evaluation. Sage.

Pedersen L. H. and Rieper O. (2008) Is realist evaluation a realistic approach for complex reforms?, Evaluation 14, 271-93.

Punton M., Isabel V., Leavy J., Michaelis C. and Boydell E. (2020) Reality bites: Making realist evaluation useful in the real world, CDI Practice Paper 22.

Scriven M. (1991) Prose and cons about goal-free evaluation, Evaluation Practice 12, 55-62.

Scriven M. (1996) Types of evaluation and types of evaluator, Evaluation Practice 17, 151-61.

Shadish W. R., Cook T. D. And Leviton L. C. (1991) Foundations of program evaluation: Theories of practice. Sage.

Shepherd, R. (2011). “In search of a balanced Canadian federal evaluation function: Getting to relevance.” Canadian Journal of Program Evaluation, 26(2), 1-45.

Stame N. (2004) Theory-Based Evaluation and Types of Complexity, Evaluation 10, 58-76.

Weiss C. H. (1997) How can theory-based evaluation make greater headway?, Evaluation review 21, 501-24.

CALL FOR PAPERS

The workshop aims to gather contributions and experiences of public policy evaluation scholars who have addressed critical issues about the relationship between programmes, the evaluation of policy success and failures and the capacity of evaluation as a field to provide relevant policy advice with useful information to citizens and stakeholder groups. In particular, the workshop would receive papers discussing the following research questions:

- How does policy coherence (or the lack thereof) affect policy implementation in complex governance structures?

- What are the risks inherent in understanding policy outcomes that focus mainly on policy aims at the expense of implementation?

- What are the consequences of evaluation-policy process dysfunctions in terms of methodological choices, evaluation policy process, and relevance of evaluation findings for citizens and other stakeholder groups?

- Are there epistemological questions, approaches and solutions to cope with these challenges?

- Are there evaluation strategies and tools, including AI-based analytical instruments, able to mitigate or strengthen these relationships and challenges?

- How do policy makers and policy advisors take into account and overcome these challenges? Are there systemic approaches being considered over episodic ones?

ABSTRACT

Outcome measures are required to assess a policy and program’s effectiveness and to improve its accountability. The focus on the ‘end’ results challenges policy makers and civil servants from the achievement of performance results to purposiveness of public action. In other words, the outcomes’ perspective entails a judgement of success or failure of action. The shift towards outcomes has been welcomed by the evaluation community. Notwithstanding, shortcomings emerge in day-to-day evaluation practice, making the role of policy advice challenging, as well as the communication with the stakeholder groups. Starting from the public policy debate on policy success and failure, the workshop underlines the need to bridge policy evaluation with the findings of policy process studies. The workshop aims to gather contributions and experiences of public policy evaluation scholars who have addressed critical issues about the relationship between programmes, the evaluation of policy success and failures, and the capacity of evaluation to provide relevant policy advice with useful information for citizens and stakeholder groups.

BIOGRAPHICAL PRESENTATION

Erica Melloni is a researcher at the Politecnico di Milano. Past president of the Italian Evaluation Society (2017-20), member of the Italian Political Science Association, she is also a member of the evaluation units of the Ministry of Economics and Finance and of the University of Naples. Her main scientific interests are on quality of public administration and territorial policies.

 

Robert Shepherd is full professor and graduate Supervisor of the online Diploma in Public Policy & Program Evaluation (DPPE) at the School of Public Policy & Administration at Carleton University (Ottawa, Canada). He is a former Canadian federal public servant and management consultant in the areas of audit, evaluation, and performance measurement. He is the current co-chair of the Consortium of Evaluators in Evaluation Education (CUEE). His current research focuses on the digitalization of government, creating the conditions for effective spending and strategic reviews in comparative perspective, and generating youth leadership in Indigenous communities.

 

Giancarlo Vecchi is associate professor of Public Policy at the School of Management of the Politecnico di Milano. He teaches “Public Analysis” and “Policy Design & Evaluation”. He is Past president of the Italian Evaluation Society (2002-2003). Recent research: Cybersecurity as a Public Policy. The Institutionalization of platform and network security in the EU and Italy; the modernization of the Italian Judicial system; development of digitization in the Italian & European public sector; evaluation of protected areas policy. His key qualifications are in the field of Public Policy Analysis, applied to the following aspects: institutional and capacity building; public sector reforms (with special attention to ICT-driven innovations in the public sector); policy design & public policy evaluation.

 

Dominika Wojtowicz is an instructor in the Department of Economics. She received scholarships from the Department of Social Sciences at the University of Siena, Italy, and the Department of Management at the Polytechnic University of Cartagena, Spain. Her scientific interests focus on the effectiveness of public interventions aimed at strengthening regional and local development.

ACCEPTED
Format
01 - CHALLENGES AND SOLUTIONS IN IMPLEMENTING PAKISTAN’S NATIONAL SCIENCE, TECHNOLOGY AND INNOVATION POLICY
Khurram Shahzad (University of Seoul, South Korea)
CO-AUTHORS
—
ABSTRACT

This policy paper examines the challenges faced in the implementation of Pakistan's National STI Policies and presents a set of comprehensive solutions to address these issues for the effective implementation of Pakistan's National STI Policy-2022. The effective execution of this policy is crucial for Pakistan's socio-economic development and global competitiveness. Moreover, a study has been conducted on the current STI policy to investigate if the heightened recognition is accompanied by novel and more dynamic strategies. The empirical data, acquired through a survey involving 172 influential figures consisting of government authorities, scientists and researchers, reveals that Pakistan has indeed instituted public entities and establishments to bolster STI, including research councils, research institutes, national plans, and an array of policy tools. Despite these efforts, the accessible science and technology metrics indicate modest outcomes. This manuscript delineates the obstacles and difficulties encountered by these administrations in the execution of STI policies. 159 (92.4%) stakeholders believe that there is a missing link between industries and universities in Pakistan, even though it is essential for unearthing knowledge and steering industrialization amid the rapidly advancing global competition and technological transformation. 81 (47.1%) believe that weak research infrastructure and 77 (44.7%) believe that inadequate funding for R&D is the most critical factor for non-implementation of the Pakistan's National STI Policy-2022.  At the outset of the examination, it became evident that significant deficiencies exist within Pakistan's STI system. A long-term STI policy is deemed necessary to enhance the effectiveness of the STI system. Consequently, the study concludes with the presentation of policy recommendations and implications derived from the results of the survey and capacity of the STI indicators. These suggestions hold substantial significance for policymakers and planners involved in shaping STI policies in Pakistan.

ACCEPTED
Format
02 - Success of Risk Management and Reduction Policies: A Comparative Perspective between Mexico City and Quito
Jonathan Menoscal (FLACSO Ecuador)
CO-AUTHORS
—
ABSTRACT

More than 200 million people are affected by disasters that increase in magnitude due to factors such as climate change, but also because of socio-economic conditions of the vulnerable population, thus constituting disaster risk as a developmental challenge for society. From this concern, the public agenda on disaster risk and its impact has expanded over the past 30 years, shifting from a focus on biophysical studies to a more complex systemic vision that considers risk as a social product, i.e., stemming from the threats and vulnerabilities of an exposed society. In this way, disaster risk-related policies have evolved, moving from notions that were focused on responding to an event to processes related to the corrective and prospective management of risk in order to mitigate and reduce it. This is in collaboration with international organizations, which set the guidelines for policy generation in the region. In recent years, due to the Covid-19 pandemic, the notion has expanded to the concept of systemic risk.

 

Despite advances in risk knowledge and accompanying policies, the impacts caused by natural events are increasing, especially in the Latin America and the Caribbean region. This is due to various factors such as weak institutional capacity, lack of regulatory frameworks, insufficient planning that considers the territory, and, above all, the poverty conditions of the majority of its inhabitants, who are pushed towards threatened and informal areas for habitation. Therefore, this research aims to answer the question: What factors determine the effectiveness of risk management and reduction policies? The hypothesis suggests that a greater coordination of state and non-state actors in the design processes of public policies generates instruments that are more coherent with the stated political objectives, thus leading to better territorial planning framed in effective policies to manage and reduce disaster risk. For this purpose, a methodological analytical framework is proposed, including the analysis of public policy for risk management and reduction through the coherence and consistency of the design of its instruments, in relation to the stated objectives, and how governance modes at the time of designing them can be the explanatory factor of their effectiveness. A comparative method of more similar cases is proposed between Mexico City and Quito.

 

As preliminary results, initially there were policies focused on addressing emergency situations and characterizing threats. Currently, there is a transition towards policies centered on risk governance, as well as characterizing multi-threats in dynamic and complex territorial contexts, where a series of factors interact to make a community more or less vulnerable. Thus, under territorial planning and development policies, as well as policies aimed at addressing root causes of risk such as poverty, inequality, lack of access to goods and services, among others, the goal is to achieve effective risk reduction. This is observed with better outcomes in Mexico City, attributed to the coherence and consistency between the objectives and means of its policy, in contrast to Quito, where despite having a consistent regulatory framework and policy instruments, they are not aligned with the stated objectives.

ACCEPTED
Format
03 - Assessing Performance Management Policies: A Comparative Perspective
Robert Shepherd (Carleton University)
CO-AUTHORS
Nadini Persaud (University of the West Indies, Cave Hill)
ABSTRACT

Context: Governments in most Western countries have struggled to implement their policy promises. Advocating and committing to multiple policy priorities and prioritizing all as equally important culminates in ineffective and untimely responses to the challenges of the day, since Governments are often constrained by financial and human resources. The problem of poor implementation can be linked in part to a bureaucracy weighed down in red tape with too many layers of management, financial constraints in many countries, inadequate trained personnel to manage and execute, and to low clarity on articulating accountability of the public service for service results.

Another important reason is a system that is obsessed with compliance to rules, making the system risk averse (and to some extent more resources are diverted to managing scarcity rather than productivity) manifesting in a public service less concerned with innovation and service delivery and more concerned with the appearance of being error‐free (Shepherd, 2022). This view aligns with the literature that asserts performance polices have not delivered on their promises to understand results (Martin, et.al., 2016). Additionally, the exercise of control has tightened leading to low innovation on how to better deliver services under conditions of turbulence (Caron, 2020). We assert that performance management policies (i.e., administrative policy) has been incoherent in meeting performance expectations of government.

Objective: In Canada and other comparator countries (e.g., CARICOM, Australia, New Zealand), administrative policy efforts have been made to better connect performance management (PM) regimes with myriad performance targets whether these are larger public policy aims, internal process aims, or programmatic outputs and outcomes. The challenge on the performance management side is that regimes and repertoires have been designed for multiple purposes or reform efforts (Schick, 1966) leading to significant confusion for determining the value proposition of the performance management function. Instead, in countries like Canada, PM has been largely discounted as a useful contributor to corporate results reporting. Also, other functions such audit and evaluation are dependent on performance information to generate policy or program findings. Because performance information is gathered for multiple purposes, establishing validity as it relates to supporting audit and evaluation findings is weak at best.

Approach: This paper is based on a preliminary study that examines the rationale of performance management and measurement policies in Canada and CARICOM (Caribbean). It addresses: 1. the extent to which federal approaches to performance are coherent and relevant as they support other functions; and, 2. the extent to which governmental performance reporting with their emphasis on several purposes is adequately supported to drive government-level results reporting. Several semi-structured interviews and focus groups were conducted in Canada between 2021 and 2022 that show a narrative of control that was amplified by the pandemic with no signs of relenting. The paper is concerned with evaluating the Canadian performance management policy with an eye to effective results reporting. Although other papers in this workshop will be concerned with public policy evaluation, this paper attempts to make a connection with administrative policy arrangements that contribute to sound public policy management and assurance systems that support accurate reporting on results.

A key question for our paper is: although these mechanisms and functions are useful to maintain some focus and momentum on results, are they the best suited as designed to our current public sector environment where decisions have to be made quickly and executed in a timely fashion?

Caron, Isabelle. 2020. “L'évolution de l'exercice du contrôle dans l'administration publique fédérale canadienne.” Canadian Public Administration, 63(1), 34–52.

Schick, Allen. 1966. “The Road to PPB: The Stages of Budget Reform.” Public Administration Review, 26(4), 243–58.

Shepherd, Robert. 2022. "Internal governmental performance and accountability in Canada: Insights and lessons for post-pandemic improvement." Canadian Public Administration, 65(2), 516-537.

ACCEPTED
Format
04 - Evaluation Methods and Strategies for Policy Integration: A Case Study of the Department of Planning of the Ministry of Education and Bilingual 2030 in Taiwan
Yi-Hua Lai (Soochow University)
CO-AUTHORS
—
ABSTRACT

     The significance of policy integration has grown notably as a means to amplify the synergy between policy design and organizational operation. However, in practical application, realizing the anticipated efficacy of policy integration often proves challenging, with potential conflicts or complications arising. Despite Taiwan's establishment of various dedicated departments or agencies for policy integration as part of governmental organizational reform efforts, such as the creation of general planning units within ministries, the effectiveness of these initiatives has fallen short of expectations. Given practical and research needs and existing literature constraints, several critical research questions are underscored: Is current policy integration effective? How can the achievement of policy integration be gauged? What pragmatic approaches facilitate policy integration? This study explores challenges and issues related to policy integration at both policy and organizational levels, developing evaluation methods and proposing strategies in the process. The specific research objectives include: 1. Constructing the concrete content of policy integration and devising an evaluation framework and indicators. 2. Evaluating the efficacy of policy integration within (1) the Bilingual 2030 policy[1] and (2) the Department of Planning of the Ministry of Education, utilizing the evaluation framework and indicators established in this study. 3. Formulating strategies to fortify policy integration and assessing their effectiveness through small-scale pilot projects.

 

     A three-year research plan targeting the Ministry of Education is proposed to fulfill these objectives. Employing a mixed-methods approach encompassing literature analysis, secondary data examination, questionnaire surveys, and focus group discussions, the initial year will focus on developing evaluation frameworks and indicators for organizational and policy integration. Subsequently, the second year will see the application of these frameworks and indicators to assess policy integration in Bilingual 2030 and the Ministry of Education's Department of Planning in Taiwan, thereby scrutinizing the current state, issues, and challenges of policy integration at both levels. In the third year, the study will refine strategies to reinforce policy integration and conduct small-scale pilot projects, culminating in research conclusions and recommendations.

 

     The research is funded by the Taiwan National Science and Technology Council. In this workshop, I would like to present the results of the first-year research, encompassing the delineation of concrete policy integration content and the development of evaluation indicators and methods for organizational and policy integration, aimed at assessing policy integration effectiveness. Methodologically, the study relies primarily on literature analysis and interviews (with a purposive sample of 4 academic and practitioner experts), initially constructing evaluation frameworks and indicators for policy integration, focusing on local contexts to mitigate implementation gaps. Through iterative literature review and interviews, the evaluation frameworks and indicators were finalized via focus group discussions (with a purposive sample of 6 academic and practitioner experts), seeking consensus and conclusions. Anticipated research outcomes are poised to contribute to defining methods for assessing policy integration attainment and evaluating existing policy integration effectiveness, thereby advancing practical approaches to achieving policy integration.

[1] The policy, currently one of the most important integrative policies implemented by the Taiwan government, has been initiated since 2020. Its vision is to make Taiwan a bilingual country by 2030, focusing on six major axes to implement related policies: accelerating the bilingualization of higher education, balancing and enhancing the conditions for bilingualization in pre-high school education, digital learning, expanding English proficiency testing capabilities, enhancing the English proficiency of public servants, and establishing an executive agency dedicated to promotion. The executing units involved in this policy primarily include the National Development Council, various ministries led by the Ministry of Education, and local governments.

ACCEPTED
Format
05 - Who gets what? Revisiting Policy Choices and Implications in Social Benefit Distribution
Ana de Alba (Tecnologico de Monterrey, Mexico)
CO-AUTHORS
—
ABSTRACT

It is generally understood that targeting concerns itself with “who should get what and how should they get it” (Marchal and Van Lancker 2019). While seemingly simple, determining who is included (and excluded) from public programs is a delicate and strenuous task and requires ensuring that benefits reach those for whom the program is intended while at (or below) the budget. Although there is a vast literature on targeting, including the methodologies and impacts,[1] it is generally agreed that targeting should be evaluated while measuring a program’s impact (Ravallion 2009). However, designing a cash transfer program for rural women embeds a set of assumptions that narrow the scope of the program, potentially beyond the process of choosing the individuals (e.g., rural women) and the modality (cash transfers). Yet why were these choices made? What do these choices mean for the program's impact? The decision to craft a program to address this specific group and not others is not casual, and still the literature on targeting -and for the most part, policy evaluation- obviates this initial set of choices. I argue the distribution of social benefits hinges first and foremost on societal values. Who gets what? Summarizes perhaps the most fundamental question of policy design, and the response is not random. Based on an extensive literature review, my article highlights the inherently political nature of targeting and revisits the set of key choices embedded in targeting design, and their implications for both beneficiaries and policymakers throughout the policy cycle. It introduces a comprehensive framework to evaluate targeting strategies independently from, and interdependent with, policy outcomes and program results. I draw on the Progresa-Oportunidades-Prospera (POP) Mexican Cash Transfer (CCT) Program to illustrate the advantages of the framework. Despite the program’s targeting design, program implementation, and welfare outcomes being extensively evaluated, these elements were studied separately. This led to the literature overlooking by and large that the program targeting design implied that during POP implementation two distinct targeting strategies were carried out: (1) being administered door-to-door in small rural areas (barridos), and (2) accepting applications in designated fixed locations in larger rural areas and urban areas (mesas de atención). By observing targeting beyond the traditional targeting design stage, my framework draws light on decisions have gone unnoticed yet resulted in differentiated entry costs for beneficiaries and may have ultimately affected program impacts of the most significant anti-poverty effort implemented by the country between 1997and 2018. In sum, this paper proposes a comprehensive framework that revisits the set of key policy choices embedded in targeting, facilitating systemic policy coherence assessments. It enables alternative scenarios to be easily contrasted, potentially improving a program's success at a low cost, as well as a more nuanced interpretation of outcome evaluation findings.



[1] For example, an electronic database search carried out by Devereux and colleagues, merely for documents on "targeting social transfer programs," exclusively in developing countries, and only from 2000 onwards, generated 1406 documents (Devereux et al. 2017).

ACCEPTED
Format
06 - The Effect of Responsiveness on Government Evaluation: Mediated through Fairness Perception and Moderated by Democratic Preferences
Ming LI
CO-AUTHORS
Wanrong LI
ABSTRACT

The Effect of Responsiveness on Government Evaluation:Mediated through Fairness Perception and Moderated by Democratic Preferences

Abstract

This study investigates the relationship of government responsiveness on citizens' evaluation of government, focusing on the mediating role of fairness perception and the moderating effect of democratic preference. Anchored in Social Identity Theory, we aim to elucidate the mechanisms through which identity emotions in a political context influence governmental assessments, offering insights into their potential to inform policy-making and enhance governance.

  Utilizing questionnaires and surveys from The Asian Barometer Survey (2023 wave 5) of mainland China, we focus on the research questions as follows : 1) What is the relationship between government responsiveness and citizens’ evaluation of the government? 2) How does the perception of fairness in distribution influence the relationship between government response and government evaluation? 3) To what extent does the preference for democracy influence the relationship between fairness perception and government evaluation?

  Preliminary findings within the dataset indicate that government response has a modest direct effect on government evaluation (coefficient=0.1636, p<0.05), with fairness perception partially mediating this relationship (coefficient=0.0962, p<0.05), and preference for democracy moderating the impact of fairness perception on government evaluation (coefficient=-0.2401, p<0.05). This underscores the significance of emotional factors, specifically, fairness perception and democratic preference, in relation to government evaluation.

  This study contributes by illustrating that perceptions of fair distribution positively impact government evaluations. Noticeably, the negative moderating effects of democratic preference among citizens considered “against the norm” is one of the contributions of this study. Meanwhile, this study explores the contextual impact of democratic preference, aiming to enhance comprehension of political attitudes and behaviors.

Key words: Responsiveness, Government Evaluation, Fairness Perception, Democratic Preferences

ACCEPTED
Format
07 - Evaluating local food security policy: what is the impact?
Simone Busetti (University of Teramo)
CO-AUTHORS
Alessandra Orsini (Universitá degli Studi di Teramo)
Lorenzo Pagni (University of Teramo)
ABSTRACT

Since the Milan 2015 Universal Exposition, increasing attention has been paid to food policy in Italy, where food issues forcefully entered the agenda of the national and local governments (Busetti & Dente, 2018). Furthermore, the COVID-19 pandemic (Pereira & Oliveira, 2020) and the rise in food prices due to the Ukraine war (ben Hassen & el Bilali, 2022) keep all governments’ attention high on food policy and—unexpectedly for rich industrialised countries—on food security (Lang, 2021).

Food policy and food security involve a wide range of actors, intertwining multiple dimensions of impact, and are still experimental when implemented at the local level. Food policy covers a vast array of activities (production, manufacturing, consumption, waste management) and intersects a wide range of policy problems (nutrition, health, environment, social welfare) (FAO, 2016; Lang et al., 2009; Lang & Heasman, 2015). Food security entails a multi-dimensional understanding of impacts: i) availability of food, ii) access to food, iii) utilization of food, and iv) stability, which is the capacity of households to withstand risks and shocks that erode any of the other three dimensions (Webb et al., 2006). Further, local governments also add social inclusion and environmental sustainability as expected results of their food security programs.

Such an ambitious set of impacts may pose notable challenges to evaluators. Local food security interventions assume this holistic view in their program objectives, although without a clear idea of which specific impacts are to be measured in each program and for which beneficiaries. The point of contention of this paper is that when impacts are ambiguous, evaluators need process evaluation in order to understand the impacts to be measured. If not, the evaluation will rest on a theoretical view of the program, increasing the likelihood of false positives (when the evaluator finds an impact not supported by the implementation process) and false negatives (when the evaluator misses an impact implemented but not previously considered among the ones expected). This connection between process and impact evaluation is not new but particularly dramatic in the case of local food policies, whose implementation is still in their infancy if not experimental.

The paper will present an evaluation study of a local food security program implemented in one city in Italy. We will use a mixed-method process evaluation to uncover what impacts are plausibly produced, how they differ from stated goals and how studying the process can inform the understanding of impacts and avoid false positives and negatives. The specific food security interventions will be selected from an ongoing national research program on the evaluation of food donations, food stamps, and farmers' markets.

ACCEPTED
Format
08 - The Limits of Impact Evaluation in Multi-level Governance Contexts: The Case of Justice Reform in Italy Post-Covid-19
Erica Melloni (Universitas Mercatorum )
CO-AUTHORS
Giancarlo Vecchi (Politecnico di Milano)
ABSTRACT

This paper examines the challenges and limitations of impact evaluation in multi-level governance contexts, using Italy's post-Covid-19 justice reform as a case study. In particular, it questions the efficacy of summative evaluations in accurately reflecting the complexities of policy sectors characterised by multi-level governance, and in providing relevant policy recommendations.  Summative impact evaluation essentially aims at verifying the achievement of the anticipated outcomes, also providing the stakeholders with clear evidence about the respect of the collective commitments. The question this article seeks to answer is: what side effects does it produce on policy sectors characterized by multilevel governance? The analysis reveals that while summative evaluations provide certain insights, they often overlook the nuanced progress and challenges in complex governance systems, potentially leading to misleading conclusions and hindering innovation. The paper concludes with recommendations for adapting evaluation methodologies in multi-level governance contexts to foster innovation and informed policy-making.

The Italian judicial sector, entangled in multilevel governance structures, provides a pertinent example. The sector is governed by multiple levels and structures (the Supreme Council of the Judiciary, the Ministry of Justice, the Judicial Offices), as well as by diverse actors (e.g., judges and clerks) in various networks that are more or less active at the territorial level in support of judicial policies. In this complex framework, justice represents one of the most critical areas of the Italian public system. For these reasons, the sector has been selected as one of the main focuses of the NRRP (National Recovery and Resilience Plan), the extraordinary national plan funded by the European Next Generation EU program to counteract the effects of Covid-19 in Europe. The NRRP has dedicated an investment of 2.827 billion euros to the justice sector, tied to the achievement of certain precise targets, including the reduction of civil trial times and the elimination of the backlog. These targets appear indisputable by the judicial sector itself and its stakeholders. However, the strategy for using the funds, which is essentially based on a massive injection of fixed-term personnel distributed across various Italian judicial offices, has appeared optimistic. Despite the magnitude of the efforts, both in terms of economic resources and the commitment of the actors involved, the expected targets will be difficult to achieve, complicating the relationship between the Italian government and Europe.

A summative evaluation, based on the comparison between outcomes and pre-established targets, would, and will, lead to a negative judgment on the justice reform policy implemented by the NRRP. The effects in terms of accountability towards both Europe and the Italian citizens will lead to a worsening perception of the functioning of justice, confirming the idea that the sector represents the most serious illness of the Italian Public Administration. However, a synthetic reading of this type, within a system of high complexity in terms of types and levels of actors involved, risks underestimating the real starting conditions, the progress made, and also the final results, effectively discouraging innovation coalitions. It also does not allow for deriving operational indications on how to direct a justice reform policy so that it can achieve the desired outcomes, even in the presence of constraints and criticalities in implementation processes.

Starting from the case of the Italian justice reform post-Covid, the article intends to reflect on the importance of adapting evaluation systems in contexts characterized by multilevel governance, so that they are capable of supporting processes of innovation and learning, encouraging innovation coalitions, within an approach clearly based on the adequate and transparent collection of empirical evidence.

ACCEPTED
Format
09 - Systematic literature review of public policy evaluation and monitoring structures adopted in the Federal and States Comptrollerships
Tiago Araujo (Universidade Federal da Bahia)
CO-AUTHORS
LUIZA TEIXEIRA (Universidade Federal da Bahia)
ABSTRACT

Based on the guidelines of the Courts of Auditors and the actions of the Controllerships in Brazil, a concrete institutional alignment between Internal and External Control can be seen to improve the structures for Evaluation and Monitoring of Public Policies, especially to consider not only the technical elements of effectiveness, efficiency and economy, as well as to give greater importance to the conditions of insertion of society, thereby materializing an institutional process of relations between political actors, which requires the Public Administration to maintain a channel of accountability and evaluation /monitoring of implemented actions, which, in the sense of Gussi (2015), would function as a multilateral access mechanism between the State and organized society. Therefore, considering the institutional and federative multiplicity of controllerships in Brazil, the research raises the problem of what are the characteristics for structuring a system for evaluating and monitoring public policies in Controllerships in Brazil? To complete the research, the specific objective was to identify the models for evaluating and monitoring public policies adopted in the Federal and States Comptrollerships.    In view of this, the work brings as an approach research of an applied nature, with exploratory-descriptive objectives, where the use of qualitative methods is proposed, through Systematic Review (PRISMA), through data collection (CAPES Periodicals, Web of Science, scielo and scopus), to identify the characteristics of the evaluation and monitoring structures of public policies adopted in the Federal and States Controllerships and through an exploratory bibliographic and documentary field research, we seek to survey participation actions of the Union and States Comptrollerships. With this step, it is assumed that it will be possible to categorize, quantify, summarize and present a qualitative synthesis of the Evaluation and Monitoring structures of the Union and States Comptrollerships, as well as studies related to policy evaluation and monitoring models of public policies in these adopted, especially to highlight mechanisms of social participation related to the monitoring and evaluation of public policies.     The work establishes as a premise the understanding established by Rodrigues (2008), which, when dealing with a public policy or program linked to it, implies considering that its evaluation will only do justice to the term if the analytical scope operates beyond the policy itself, its legal framework and its content, and beyond the empirical approach, so that more general inferences can be made from localized results. To this end, the notions of context, process, trajectory, plurality, interaction and multidimensionality. The presentation of the work contributes to the workshop as it provides an updated overview of the measures adopted in Brazil by the Controllerships in structuring the Evaluation and Monitoring of public Policies, highlighting social participation measures.

ACCEPTED
Format
10 - THE ROLE OF NATIONAL INFORMATION & COMMUNICATION TECHNOLOGY POLICY ON POVERTY ALLEVIATION AMONG WOMEN IN RURAL SETUP IN KENYA: THE CASE OF SIAYA COUNTY
CHRISTINE MUKHONGO (University of Nairobi)
CO-AUTHORS
—
ABSTRACT

 

ABSTRACT

In Kenya, poverty reduction has been a challenge since independence as several strategies have been put in place all the way from 1965 with no significant outcome, based on the fact that poverty is still widespread. The policy objective of the session paper No. 10 of 1965 which aimed at rapid economic development and social progress for all citizens has been reverberated in the Vision 2030 drive which intends to make Kenya a newly industrialized income country by 2030.

 

The Kenyan government has been actively promoting the use of ICT to achieve its development goals. Kenya's Vision 2030 outlines the importance of leveraging technology to drive economic growth and reduce poverty. In line with this vision, the Kenyan government has formulated a national ICT policy to guide the country's ICT-related initiatives. The policy aims to improve digital infrastructure, foster innovation, and promote digital literacy and skills development among various segments of the population, including women.

 

While Kenya's national ICT policy may hold the promise of transforming the lives of citizens, the envisioned development goals may end up not being achieved if implementation of the policy is not monitored and evaluated. The purpose of this study is to evaluate the policy’s implementation and effectiveness in achieving poverty alleviation goals. It is essential to assess whether the policy has been successful in providing targeted poverty alleviation measures, improving access to ICT resources and technology, and creating opportunities for socio-economic development.

 

The general research objective of this study is to examine the level of implementation and effectiveness of Kenya's national ICT policy in empowering women and promoting their economic and social development.

 

The study will attempt to answer the following research questions:

i.      To what extent has Kenya's national ICT policy been implemented and how has it impacted poverty alleviation in Siaya County with a specific focus on poverty alleviation measures targeted at women?

ii.     What are the main challenges and barriers faced by women in Siaya County in benefiting from the ICT policy, and how do these challenges influence poverty alleviation?

iii.   Based on the research findings, what evidence-based policy adjustments and interventions can be recommended to maximize the positive impact of Kenya's national ICT policy on poverty alleviation among women in Siaya County?

 

The study will employ a mixed method research methodology and the research design will be a case study. This is aligned with the workshop’s research question since it provides an opportunity for in-depth evaluation of the ICT policy outcomes to establish whether the policy was a success or a failure.

ACCEPTED
Format
11 - Brazilian rural credit: evaluation outcomes under the Policy Design perspective
Ademir Vilaca Junior (National School of Public Administration (ENAP))
CO-AUTHORS
Isis Moreira (Universidade de Brasília)
ABSTRACT

The Brazilian rural credit policy generates approximately R$200 billion annually, benefiting rural producers throughout the country, encompassing over 30 crops and owners of all sizes. The policy includes one of the world's largest programs of subsidized credit specifically for small family farmers, o Programa de Fortalecimento da Agricultura Familiar (PRONAF). Nevertheless, it is important to note that financing targeted at medium and large-scale farmers is responsible for absorbing more than half of the total resources, as observed in official documents of the Planos Safra during the last government period. Thus, rural credit is a fundamental tool to boost agricultural production.

 

When reviewing the literature on the subject, three main distinct and complementary approaches can be identified. The first approach is a set of descriptive studies that analyze the concentration of resources from different perspectives, both territorially and in terms of the benefited products. These studies indicate a concentration of financing in the Central-South region of the country and a tendency to strengthen the production of export commodities. The second group focuses on quantitative analyses of measuring the impact of rural credit policy, examining the relationship between credit provision and macroeconomic variables such as municipal GDP, agricultural production, employment, and income. The results show that the impact of rural credit varies across the national territory. A third strand is related to qualitative analyses from case studies, applying questionnaires that capture the specificities of the territories. Such studies can capture microeconomic variables that contribute to the strengthening of agricultural production.

These perspectives offer different insights into the policy results and factors that are important to enhance the effects of rural credit on Brazilian agricultural production and productivity. However, these results are often not substantiated with indications for improving the policy design.

 

In this sense, from the perspective of policy design, the work seeks to answer what changes in the policy design can be suggested based on the analysis of its concrete impacts on Brazilian producers, with specific emphasis on the effects on producers of different sizes. To do so, the work adopts an analysis of existing literature along with a quantitative approach using the quantile regression method to assess the effects of rural credit according to the size of the owners.

 

From this perspective, the IPPA Workshop - "Defining Policy Success and Failure as Evaluation Challenges. Focusing on outcomes: Blessing or a curse for policymakers?" is an ideal forum to discuss and understand how the results observed in the literature can indeed contribute valuable insights to the (re)design of the policy, enhancing its effectiveness and efficiency.

 

We state that if approved, the authors will attend the workshop in person.

ACCEPTED
Format
12 - Bridging the Gap: Exploring Synergies Between Serious Games and Policy Evaluation in Food Policy
Dominika Wojtowicz (Kozminski University)
CO-AUTHORS
Tomasz Kupiec (University of Warsaw)
Karol Olejniczak (SWPS University)
ABSTRACT

Serious games are increasingly popular and versatile, utilised in various contexts to achieve goals such as education, behaviour change, and stakeholder engagement. Their adaptability and effectiveness in diverse settings underscore their potential to support public policy objectives. However, as the use of serious games in policy contexts is relatively novel, there is a need to explore how evaluation can assess their impact and improve their design and implementation.

The paper aims to identify potential synergies and opportunities for cooperation between the serious games community and policy evaluators, thereby enhancing both the effectiveness of serious games and the robustness of policy evaluations. Our research focuses on two primary areas: (i) the integration of evaluation methods to assess serious games used for public policy purposes and (ii) the potential for serious games themselves to function as evaluation tools. By examining the current state of evaluations of serious games and exploring innovative methodologies, we aim to provide insights that can enhance their effectiveness in achieving policy goals.

Despite the proliferation of serious games designed for public policy purposes, the evaluation of these games remains limited. This gap presents an opportunity for evaluators to contribute significantly to the development of more effective serious games by providing knowledge on their actual impact. New approaches and methods may be required to accurately assess the unique aspects of serious games, posing a challenge for evaluators. Another significant area of intersection between the serious games community and policy evaluation lies in the potential of serious games as evaluation and research tools. These games can measure specific outcomes or engage stakeholders in the policy design and implementation processes, offering a dynamic and interactive means of assessment.

To explore these opportunities, we conducted a systematic literature review focusing on serious games related to food policy issues. Our review concentrated on two main areas: how these games are currently being evaluated and how they can be utilised within the broader framework of public policy evaluations.

This paper contributes to the growing field of public policy evaluation by demonstrating the potential of serious games to improve policy coherence, enhance implementation effectiveness, and provide valuable insights. Our findings highlight the need for innovative evaluation strategies tailored to the unique characteristics of serious games, emphasising their role in fostering effective and engaging public policies. By identifying synergies and opportunities for cooperation, we aim to bridge the gap between the serious games community and policy evaluators, ultimately advancing the field of public policy.

 

 

ACCEPTED
Format
13 - Discussion and prospective projects
Erica Melloni (Universitas Mercatorum )
CO-AUTHORS
Dominika Wojtowicz (Kozminski University)
ABSTRACT

Conclusions

CO-AUTHORS
—
ABSTRACT

This policy paper examines the challenges faced in the implementation of Pakistan's National STI Policies and presents a set of comprehensive solutions to address these issues for the effective implementation of Pakistan's National STI Policy-2022. The effective execution of this policy is crucial for Pakistan's socio-economic development and global competitiveness. Moreover, a study has been conducted on the current STI policy to investigate if the heightened recognition is accompanied by novel and more dynamic strategies. The empirical data, acquired through a survey involving 172 influential figures consisting of government authorities, scientists and researchers, reveals that Pakistan has indeed instituted public entities and establishments to bolster STI, including research councils, research institutes, national plans, and an array of policy tools. Despite these efforts, the accessible science and technology metrics indicate modest outcomes. This manuscript delineates the obstacles and difficulties encountered by these administrations in the execution of STI policies. 159 (92.4%) stakeholders believe that there is a missing link between industries and universities in Pakistan, even though it is essential for unearthing knowledge and steering industrialization amid the rapidly advancing global competition and technological transformation. 81 (47.1%) believe that weak research infrastructure and 77 (44.7%) believe that inadequate funding for R&D is the most critical factor for non-implementation of the Pakistan's National STI Policy-2022.  At the outset of the examination, it became evident that significant deficiencies exist within Pakistan's STI system. A long-term STI policy is deemed necessary to enhance the effectiveness of the STI system. Consequently, the study concludes with the presentation of policy recommendations and implications derived from the results of the survey and capacity of the STI indicators. These suggestions hold substantial significance for policymakers and planners involved in shaping STI policies in Pakistan.

CO-AUTHORS
—
ABSTRACT

More than 200 million people are affected by disasters that increase in magnitude due to factors such as climate change, but also because of socio-economic conditions of the vulnerable population, thus constituting disaster risk as a developmental challenge for society. From this concern, the public agenda on disaster risk and its impact has expanded over the past 30 years, shifting from a focus on biophysical studies to a more complex systemic vision that considers risk as a social product, i.e., stemming from the threats and vulnerabilities of an exposed society. In this way, disaster risk-related policies have evolved, moving from notions that were focused on responding to an event to processes related to the corrective and prospective management of risk in order to mitigate and reduce it. This is in collaboration with international organizations, which set the guidelines for policy generation in the region. In recent years, due to the Covid-19 pandemic, the notion has expanded to the concept of systemic risk.

 

Despite advances in risk knowledge and accompanying policies, the impacts caused by natural events are increasing, especially in the Latin America and the Caribbean region. This is due to various factors such as weak institutional capacity, lack of regulatory frameworks, insufficient planning that considers the territory, and, above all, the poverty conditions of the majority of its inhabitants, who are pushed towards threatened and informal areas for habitation. Therefore, this research aims to answer the question: What factors determine the effectiveness of risk management and reduction policies? The hypothesis suggests that a greater coordination of state and non-state actors in the design processes of public policies generates instruments that are more coherent with the stated political objectives, thus leading to better territorial planning framed in effective policies to manage and reduce disaster risk. For this purpose, a methodological analytical framework is proposed, including the analysis of public policy for risk management and reduction through the coherence and consistency of the design of its instruments, in relation to the stated objectives, and how governance modes at the time of designing them can be the explanatory factor of their effectiveness. A comparative method of more similar cases is proposed between Mexico City and Quito.

 

As preliminary results, initially there were policies focused on addressing emergency situations and characterizing threats. Currently, there is a transition towards policies centered on risk governance, as well as characterizing multi-threats in dynamic and complex territorial contexts, where a series of factors interact to make a community more or less vulnerable. Thus, under territorial planning and development policies, as well as policies aimed at addressing root causes of risk such as poverty, inequality, lack of access to goods and services, among others, the goal is to achieve effective risk reduction. This is observed with better outcomes in Mexico City, attributed to the coherence and consistency between the objectives and means of its policy, in contrast to Quito, where despite having a consistent regulatory framework and policy instruments, they are not aligned with the stated objectives.

CO-AUTHORS
Nadini Persaud (University of the West Indies, Cave Hill)
ABSTRACT

Context: Governments in most Western countries have struggled to implement their policy promises. Advocating and committing to multiple policy priorities and prioritizing all as equally important culminates in ineffective and untimely responses to the challenges of the day, since Governments are often constrained by financial and human resources. The problem of poor implementation can be linked in part to a bureaucracy weighed down in red tape with too many layers of management, financial constraints in many countries, inadequate trained personnel to manage and execute, and to low clarity on articulating accountability of the public service for service results.

Another important reason is a system that is obsessed with compliance to rules, making the system risk averse (and to some extent more resources are diverted to managing scarcity rather than productivity) manifesting in a public service less concerned with innovation and service delivery and more concerned with the appearance of being error‐free (Shepherd, 2022). This view aligns with the literature that asserts performance polices have not delivered on their promises to understand results (Martin, et.al., 2016). Additionally, the exercise of control has tightened leading to low innovation on how to better deliver services under conditions of turbulence (Caron, 2020). We assert that performance management policies (i.e., administrative policy) has been incoherent in meeting performance expectations of government.

Objective: In Canada and other comparator countries (e.g., CARICOM, Australia, New Zealand), administrative policy efforts have been made to better connect performance management (PM) regimes with myriad performance targets whether these are larger public policy aims, internal process aims, or programmatic outputs and outcomes. The challenge on the performance management side is that regimes and repertoires have been designed for multiple purposes or reform efforts (Schick, 1966) leading to significant confusion for determining the value proposition of the performance management function. Instead, in countries like Canada, PM has been largely discounted as a useful contributor to corporate results reporting. Also, other functions such audit and evaluation are dependent on performance information to generate policy or program findings. Because performance information is gathered for multiple purposes, establishing validity as it relates to supporting audit and evaluation findings is weak at best.

Approach: This paper is based on a preliminary study that examines the rationale of performance management and measurement policies in Canada and CARICOM (Caribbean). It addresses: 1. the extent to which federal approaches to performance are coherent and relevant as they support other functions; and, 2. the extent to which governmental performance reporting with their emphasis on several purposes is adequately supported to drive government-level results reporting. Several semi-structured interviews and focus groups were conducted in Canada between 2021 and 2022 that show a narrative of control that was amplified by the pandemic with no signs of relenting. The paper is concerned with evaluating the Canadian performance management policy with an eye to effective results reporting. Although other papers in this workshop will be concerned with public policy evaluation, this paper attempts to make a connection with administrative policy arrangements that contribute to sound public policy management and assurance systems that support accurate reporting on results.

A key question for our paper is: although these mechanisms and functions are useful to maintain some focus and momentum on results, are they the best suited as designed to our current public sector environment where decisions have to be made quickly and executed in a timely fashion?

Caron, Isabelle. 2020. “L'évolution de l'exercice du contrôle dans l'administration publique fédérale canadienne.” Canadian Public Administration, 63(1), 34–52.

Schick, Allen. 1966. “The Road to PPB: The Stages of Budget Reform.” Public Administration Review, 26(4), 243–58.

Shepherd, Robert. 2022. "Internal governmental performance and accountability in Canada: Insights and lessons for post-pandemic improvement." Canadian Public Administration, 65(2), 516-537.

CO-AUTHORS
—
ABSTRACT

     The significance of policy integration has grown notably as a means to amplify the synergy between policy design and organizational operation. However, in practical application, realizing the anticipated efficacy of policy integration often proves challenging, with potential conflicts or complications arising. Despite Taiwan's establishment of various dedicated departments or agencies for policy integration as part of governmental organizational reform efforts, such as the creation of general planning units within ministries, the effectiveness of these initiatives has fallen short of expectations. Given practical and research needs and existing literature constraints, several critical research questions are underscored: Is current policy integration effective? How can the achievement of policy integration be gauged? What pragmatic approaches facilitate policy integration? This study explores challenges and issues related to policy integration at both policy and organizational levels, developing evaluation methods and proposing strategies in the process. The specific research objectives include: 1. Constructing the concrete content of policy integration and devising an evaluation framework and indicators. 2. Evaluating the efficacy of policy integration within (1) the Bilingual 2030 policy[1] and (2) the Department of Planning of the Ministry of Education, utilizing the evaluation framework and indicators established in this study. 3. Formulating strategies to fortify policy integration and assessing their effectiveness through small-scale pilot projects.

 

     A three-year research plan targeting the Ministry of Education is proposed to fulfill these objectives. Employing a mixed-methods approach encompassing literature analysis, secondary data examination, questionnaire surveys, and focus group discussions, the initial year will focus on developing evaluation frameworks and indicators for organizational and policy integration. Subsequently, the second year will see the application of these frameworks and indicators to assess policy integration in Bilingual 2030 and the Ministry of Education's Department of Planning in Taiwan, thereby scrutinizing the current state, issues, and challenges of policy integration at both levels. In the third year, the study will refine strategies to reinforce policy integration and conduct small-scale pilot projects, culminating in research conclusions and recommendations.

 

     The research is funded by the Taiwan National Science and Technology Council. In this workshop, I would like to present the results of the first-year research, encompassing the delineation of concrete policy integration content and the development of evaluation indicators and methods for organizational and policy integration, aimed at assessing policy integration effectiveness. Methodologically, the study relies primarily on literature analysis and interviews (with a purposive sample of 4 academic and practitioner experts), initially constructing evaluation frameworks and indicators for policy integration, focusing on local contexts to mitigate implementation gaps. Through iterative literature review and interviews, the evaluation frameworks and indicators were finalized via focus group discussions (with a purposive sample of 6 academic and practitioner experts), seeking consensus and conclusions. Anticipated research outcomes are poised to contribute to defining methods for assessing policy integration attainment and evaluating existing policy integration effectiveness, thereby advancing practical approaches to achieving policy integration.

[1] The policy, currently one of the most important integrative policies implemented by the Taiwan government, has been initiated since 2020. Its vision is to make Taiwan a bilingual country by 2030, focusing on six major axes to implement related policies: accelerating the bilingualization of higher education, balancing and enhancing the conditions for bilingualization in pre-high school education, digital learning, expanding English proficiency testing capabilities, enhancing the English proficiency of public servants, and establishing an executive agency dedicated to promotion. The executing units involved in this policy primarily include the National Development Council, various ministries led by the Ministry of Education, and local governments.

CO-AUTHORS
—
ABSTRACT

It is generally understood that targeting concerns itself with “who should get what and how should they get it” (Marchal and Van Lancker 2019). While seemingly simple, determining who is included (and excluded) from public programs is a delicate and strenuous task and requires ensuring that benefits reach those for whom the program is intended while at (or below) the budget. Although there is a vast literature on targeting, including the methodologies and impacts,[1] it is generally agreed that targeting should be evaluated while measuring a program’s impact (Ravallion 2009). However, designing a cash transfer program for rural women embeds a set of assumptions that narrow the scope of the program, potentially beyond the process of choosing the individuals (e.g., rural women) and the modality (cash transfers). Yet why were these choices made? What do these choices mean for the program's impact? The decision to craft a program to address this specific group and not others is not casual, and still the literature on targeting -and for the most part, policy evaluation- obviates this initial set of choices. I argue the distribution of social benefits hinges first and foremost on societal values. Who gets what? Summarizes perhaps the most fundamental question of policy design, and the response is not random. Based on an extensive literature review, my article highlights the inherently political nature of targeting and revisits the set of key choices embedded in targeting design, and their implications for both beneficiaries and policymakers throughout the policy cycle. It introduces a comprehensive framework to evaluate targeting strategies independently from, and interdependent with, policy outcomes and program results. I draw on the Progresa-Oportunidades-Prospera (POP) Mexican Cash Transfer (CCT) Program to illustrate the advantages of the framework. Despite the program’s targeting design, program implementation, and welfare outcomes being extensively evaluated, these elements were studied separately. This led to the literature overlooking by and large that the program targeting design implied that during POP implementation two distinct targeting strategies were carried out: (1) being administered door-to-door in small rural areas (barridos), and (2) accepting applications in designated fixed locations in larger rural areas and urban areas (mesas de atención). By observing targeting beyond the traditional targeting design stage, my framework draws light on decisions have gone unnoticed yet resulted in differentiated entry costs for beneficiaries and may have ultimately affected program impacts of the most significant anti-poverty effort implemented by the country between 1997and 2018. In sum, this paper proposes a comprehensive framework that revisits the set of key policy choices embedded in targeting, facilitating systemic policy coherence assessments. It enables alternative scenarios to be easily contrasted, potentially improving a program's success at a low cost, as well as a more nuanced interpretation of outcome evaluation findings.



[1] For example, an electronic database search carried out by Devereux and colleagues, merely for documents on "targeting social transfer programs," exclusively in developing countries, and only from 2000 onwards, generated 1406 documents (Devereux et al. 2017).

CO-AUTHORS
Wanrong LI
ABSTRACT

The Effect of Responsiveness on Government Evaluation:Mediated through Fairness Perception and Moderated by Democratic Preferences

Abstract

This study investigates the relationship of government responsiveness on citizens' evaluation of government, focusing on the mediating role of fairness perception and the moderating effect of democratic preference. Anchored in Social Identity Theory, we aim to elucidate the mechanisms through which identity emotions in a political context influence governmental assessments, offering insights into their potential to inform policy-making and enhance governance.

  Utilizing questionnaires and surveys from The Asian Barometer Survey (2023 wave 5) of mainland China, we focus on the research questions as follows : 1) What is the relationship between government responsiveness and citizens’ evaluation of the government? 2) How does the perception of fairness in distribution influence the relationship between government response and government evaluation? 3) To what extent does the preference for democracy influence the relationship between fairness perception and government evaluation?

  Preliminary findings within the dataset indicate that government response has a modest direct effect on government evaluation (coefficient=0.1636, p<0.05), with fairness perception partially mediating this relationship (coefficient=0.0962, p<0.05), and preference for democracy moderating the impact of fairness perception on government evaluation (coefficient=-0.2401, p<0.05). This underscores the significance of emotional factors, specifically, fairness perception and democratic preference, in relation to government evaluation.

  This study contributes by illustrating that perceptions of fair distribution positively impact government evaluations. Noticeably, the negative moderating effects of democratic preference among citizens considered “against the norm” is one of the contributions of this study. Meanwhile, this study explores the contextual impact of democratic preference, aiming to enhance comprehension of political attitudes and behaviors.

Key words: Responsiveness, Government Evaluation, Fairness Perception, Democratic Preferences

CO-AUTHORS
Alessandra Orsini (Universitá degli Studi di Teramo)
Lorenzo Pagni (University of Teramo)
ABSTRACT

Since the Milan 2015 Universal Exposition, increasing attention has been paid to food policy in Italy, where food issues forcefully entered the agenda of the national and local governments (Busetti & Dente, 2018). Furthermore, the COVID-19 pandemic (Pereira & Oliveira, 2020) and the rise in food prices due to the Ukraine war (ben Hassen & el Bilali, 2022) keep all governments’ attention high on food policy and—unexpectedly for rich industrialised countries—on food security (Lang, 2021).

Food policy and food security involve a wide range of actors, intertwining multiple dimensions of impact, and are still experimental when implemented at the local level. Food policy covers a vast array of activities (production, manufacturing, consumption, waste management) and intersects a wide range of policy problems (nutrition, health, environment, social welfare) (FAO, 2016; Lang et al., 2009; Lang & Heasman, 2015). Food security entails a multi-dimensional understanding of impacts: i) availability of food, ii) access to food, iii) utilization of food, and iv) stability, which is the capacity of households to withstand risks and shocks that erode any of the other three dimensions (Webb et al., 2006). Further, local governments also add social inclusion and environmental sustainability as expected results of their food security programs.

Such an ambitious set of impacts may pose notable challenges to evaluators. Local food security interventions assume this holistic view in their program objectives, although without a clear idea of which specific impacts are to be measured in each program and for which beneficiaries. The point of contention of this paper is that when impacts are ambiguous, evaluators need process evaluation in order to understand the impacts to be measured. If not, the evaluation will rest on a theoretical view of the program, increasing the likelihood of false positives (when the evaluator finds an impact not supported by the implementation process) and false negatives (when the evaluator misses an impact implemented but not previously considered among the ones expected). This connection between process and impact evaluation is not new but particularly dramatic in the case of local food policies, whose implementation is still in their infancy if not experimental.

The paper will present an evaluation study of a local food security program implemented in one city in Italy. We will use a mixed-method process evaluation to uncover what impacts are plausibly produced, how they differ from stated goals and how studying the process can inform the understanding of impacts and avoid false positives and negatives. The specific food security interventions will be selected from an ongoing national research program on the evaluation of food donations, food stamps, and farmers' markets.

CO-AUTHORS
Giancarlo Vecchi (Politecnico di Milano)
ABSTRACT

This paper examines the challenges and limitations of impact evaluation in multi-level governance contexts, using Italy's post-Covid-19 justice reform as a case study. In particular, it questions the efficacy of summative evaluations in accurately reflecting the complexities of policy sectors characterised by multi-level governance, and in providing relevant policy recommendations.  Summative impact evaluation essentially aims at verifying the achievement of the anticipated outcomes, also providing the stakeholders with clear evidence about the respect of the collective commitments. The question this article seeks to answer is: what side effects does it produce on policy sectors characterized by multilevel governance? The analysis reveals that while summative evaluations provide certain insights, they often overlook the nuanced progress and challenges in complex governance systems, potentially leading to misleading conclusions and hindering innovation. The paper concludes with recommendations for adapting evaluation methodologies in multi-level governance contexts to foster innovation and informed policy-making.

The Italian judicial sector, entangled in multilevel governance structures, provides a pertinent example. The sector is governed by multiple levels and structures (the Supreme Council of the Judiciary, the Ministry of Justice, the Judicial Offices), as well as by diverse actors (e.g., judges and clerks) in various networks that are more or less active at the territorial level in support of judicial policies. In this complex framework, justice represents one of the most critical areas of the Italian public system. For these reasons, the sector has been selected as one of the main focuses of the NRRP (National Recovery and Resilience Plan), the extraordinary national plan funded by the European Next Generation EU program to counteract the effects of Covid-19 in Europe. The NRRP has dedicated an investment of 2.827 billion euros to the justice sector, tied to the achievement of certain precise targets, including the reduction of civil trial times and the elimination of the backlog. These targets appear indisputable by the judicial sector itself and its stakeholders. However, the strategy for using the funds, which is essentially based on a massive injection of fixed-term personnel distributed across various Italian judicial offices, has appeared optimistic. Despite the magnitude of the efforts, both in terms of economic resources and the commitment of the actors involved, the expected targets will be difficult to achieve, complicating the relationship between the Italian government and Europe.

A summative evaluation, based on the comparison between outcomes and pre-established targets, would, and will, lead to a negative judgment on the justice reform policy implemented by the NRRP. The effects in terms of accountability towards both Europe and the Italian citizens will lead to a worsening perception of the functioning of justice, confirming the idea that the sector represents the most serious illness of the Italian Public Administration. However, a synthetic reading of this type, within a system of high complexity in terms of types and levels of actors involved, risks underestimating the real starting conditions, the progress made, and also the final results, effectively discouraging innovation coalitions. It also does not allow for deriving operational indications on how to direct a justice reform policy so that it can achieve the desired outcomes, even in the presence of constraints and criticalities in implementation processes.

Starting from the case of the Italian justice reform post-Covid, the article intends to reflect on the importance of adapting evaluation systems in contexts characterized by multilevel governance, so that they are capable of supporting processes of innovation and learning, encouraging innovation coalitions, within an approach clearly based on the adequate and transparent collection of empirical evidence.

CO-AUTHORS
LUIZA TEIXEIRA (Universidade Federal da Bahia)
ABSTRACT

Based on the guidelines of the Courts of Auditors and the actions of the Controllerships in Brazil, a concrete institutional alignment between Internal and External Control can be seen to improve the structures for Evaluation and Monitoring of Public Policies, especially to consider not only the technical elements of effectiveness, efficiency and economy, as well as to give greater importance to the conditions of insertion of society, thereby materializing an institutional process of relations between political actors, which requires the Public Administration to maintain a channel of accountability and evaluation /monitoring of implemented actions, which, in the sense of Gussi (2015), would function as a multilateral access mechanism between the State and organized society. Therefore, considering the institutional and federative multiplicity of controllerships in Brazil, the research raises the problem of what are the characteristics for structuring a system for evaluating and monitoring public policies in Controllerships in Brazil? To complete the research, the specific objective was to identify the models for evaluating and monitoring public policies adopted in the Federal and States Comptrollerships.    In view of this, the work brings as an approach research of an applied nature, with exploratory-descriptive objectives, where the use of qualitative methods is proposed, through Systematic Review (PRISMA), through data collection (CAPES Periodicals, Web of Science, scielo and scopus), to identify the characteristics of the evaluation and monitoring structures of public policies adopted in the Federal and States Controllerships and through an exploratory bibliographic and documentary field research, we seek to survey participation actions of the Union and States Comptrollerships. With this step, it is assumed that it will be possible to categorize, quantify, summarize and present a qualitative synthesis of the Evaluation and Monitoring structures of the Union and States Comptrollerships, as well as studies related to policy evaluation and monitoring models of public policies in these adopted, especially to highlight mechanisms of social participation related to the monitoring and evaluation of public policies.     The work establishes as a premise the understanding established by Rodrigues (2008), which, when dealing with a public policy or program linked to it, implies considering that its evaluation will only do justice to the term if the analytical scope operates beyond the policy itself, its legal framework and its content, and beyond the empirical approach, so that more general inferences can be made from localized results. To this end, the notions of context, process, trajectory, plurality, interaction and multidimensionality. The presentation of the work contributes to the workshop as it provides an updated overview of the measures adopted in Brazil by the Controllerships in structuring the Evaluation and Monitoring of public Policies, highlighting social participation measures.

CO-AUTHORS
—
ABSTRACT

 

ABSTRACT

In Kenya, poverty reduction has been a challenge since independence as several strategies have been put in place all the way from 1965 with no significant outcome, based on the fact that poverty is still widespread. The policy objective of the session paper No. 10 of 1965 which aimed at rapid economic development and social progress for all citizens has been reverberated in the Vision 2030 drive which intends to make Kenya a newly industrialized income country by 2030.

 

The Kenyan government has been actively promoting the use of ICT to achieve its development goals. Kenya's Vision 2030 outlines the importance of leveraging technology to drive economic growth and reduce poverty. In line with this vision, the Kenyan government has formulated a national ICT policy to guide the country's ICT-related initiatives. The policy aims to improve digital infrastructure, foster innovation, and promote digital literacy and skills development among various segments of the population, including women.

 

While Kenya's national ICT policy may hold the promise of transforming the lives of citizens, the envisioned development goals may end up not being achieved if implementation of the policy is not monitored and evaluated. The purpose of this study is to evaluate the policy’s implementation and effectiveness in achieving poverty alleviation goals. It is essential to assess whether the policy has been successful in providing targeted poverty alleviation measures, improving access to ICT resources and technology, and creating opportunities for socio-economic development.

 

The general research objective of this study is to examine the level of implementation and effectiveness of Kenya's national ICT policy in empowering women and promoting their economic and social development.

 

The study will attempt to answer the following research questions:

i.      To what extent has Kenya's national ICT policy been implemented and how has it impacted poverty alleviation in Siaya County with a specific focus on poverty alleviation measures targeted at women?

ii.     What are the main challenges and barriers faced by women in Siaya County in benefiting from the ICT policy, and how do these challenges influence poverty alleviation?

iii.   Based on the research findings, what evidence-based policy adjustments and interventions can be recommended to maximize the positive impact of Kenya's national ICT policy on poverty alleviation among women in Siaya County?

 

The study will employ a mixed method research methodology and the research design will be a case study. This is aligned with the workshop’s research question since it provides an opportunity for in-depth evaluation of the ICT policy outcomes to establish whether the policy was a success or a failure.

CO-AUTHORS
Isis Moreira (Universidade de Brasília)
ABSTRACT

The Brazilian rural credit policy generates approximately R$200 billion annually, benefiting rural producers throughout the country, encompassing over 30 crops and owners of all sizes. The policy includes one of the world's largest programs of subsidized credit specifically for small family farmers, o Programa de Fortalecimento da Agricultura Familiar (PRONAF). Nevertheless, it is important to note that financing targeted at medium and large-scale farmers is responsible for absorbing more than half of the total resources, as observed in official documents of the Planos Safra during the last government period. Thus, rural credit is a fundamental tool to boost agricultural production.

 

When reviewing the literature on the subject, three main distinct and complementary approaches can be identified. The first approach is a set of descriptive studies that analyze the concentration of resources from different perspectives, both territorially and in terms of the benefited products. These studies indicate a concentration of financing in the Central-South region of the country and a tendency to strengthen the production of export commodities. The second group focuses on quantitative analyses of measuring the impact of rural credit policy, examining the relationship between credit provision and macroeconomic variables such as municipal GDP, agricultural production, employment, and income. The results show that the impact of rural credit varies across the national territory. A third strand is related to qualitative analyses from case studies, applying questionnaires that capture the specificities of the territories. Such studies can capture microeconomic variables that contribute to the strengthening of agricultural production.

These perspectives offer different insights into the policy results and factors that are important to enhance the effects of rural credit on Brazilian agricultural production and productivity. However, these results are often not substantiated with indications for improving the policy design.

 

In this sense, from the perspective of policy design, the work seeks to answer what changes in the policy design can be suggested based on the analysis of its concrete impacts on Brazilian producers, with specific emphasis on the effects on producers of different sizes. To do so, the work adopts an analysis of existing literature along with a quantitative approach using the quantile regression method to assess the effects of rural credit according to the size of the owners.

 

From this perspective, the IPPA Workshop - "Defining Policy Success and Failure as Evaluation Challenges. Focusing on outcomes: Blessing or a curse for policymakers?" is an ideal forum to discuss and understand how the results observed in the literature can indeed contribute valuable insights to the (re)design of the policy, enhancing its effectiveness and efficiency.

 

We state that if approved, the authors will attend the workshop in person.

CO-AUTHORS
Tomasz Kupiec (University of Warsaw)
Karol Olejniczak (SWPS University)
ABSTRACT

Serious games are increasingly popular and versatile, utilised in various contexts to achieve goals such as education, behaviour change, and stakeholder engagement. Their adaptability and effectiveness in diverse settings underscore their potential to support public policy objectives. However, as the use of serious games in policy contexts is relatively novel, there is a need to explore how evaluation can assess their impact and improve their design and implementation.

The paper aims to identify potential synergies and opportunities for cooperation between the serious games community and policy evaluators, thereby enhancing both the effectiveness of serious games and the robustness of policy evaluations. Our research focuses on two primary areas: (i) the integration of evaluation methods to assess serious games used for public policy purposes and (ii) the potential for serious games themselves to function as evaluation tools. By examining the current state of evaluations of serious games and exploring innovative methodologies, we aim to provide insights that can enhance their effectiveness in achieving policy goals.

Despite the proliferation of serious games designed for public policy purposes, the evaluation of these games remains limited. This gap presents an opportunity for evaluators to contribute significantly to the development of more effective serious games by providing knowledge on their actual impact. New approaches and methods may be required to accurately assess the unique aspects of serious games, posing a challenge for evaluators. Another significant area of intersection between the serious games community and policy evaluation lies in the potential of serious games as evaluation and research tools. These games can measure specific outcomes or engage stakeholders in the policy design and implementation processes, offering a dynamic and interactive means of assessment.

To explore these opportunities, we conducted a systematic literature review focusing on serious games related to food policy issues. Our review concentrated on two main areas: how these games are currently being evaluated and how they can be utilised within the broader framework of public policy evaluations.

This paper contributes to the growing field of public policy evaluation by demonstrating the potential of serious games to improve policy coherence, enhance implementation effectiveness, and provide valuable insights. Our findings highlight the need for innovative evaluation strategies tailored to the unique characteristics of serious games, emphasising their role in fostering effective and engaging public policies. By identifying synergies and opportunities for cooperation, we aim to bridge the gap between the serious games community and policy evaluators, ultimately advancing the field of public policy.

 

 

CO-AUTHORS
Dominika Wojtowicz (Kozminski University)
ABSTRACT

Conclusions