LIST OF WORKSHOPS

Introduction

Outcome measures are designed to assess the contribution of public policies and programs to the improvement of social issues. Outcomes identification and measurement serve to assess the policy and program’s effectiveness and to improve its implementation and accountability. As far as effectiveness, public interventions are scrutinized not only for performance but also in their capacity to ‘make a difference’ or ‘drive a change’ concerning the issues to be tackled (Chen, 1990; Davies, 2004). As far as accountability, evaluation has been widely acknowledged as a means for empowering citizens (Dahlher-Larson, 2011), when it makes available to decision-makers and citizens reliable knowledge on which to build their judgments and choices.

The focus on the ‘end’ results of public policy and programs challenges policy makers and civil servants to understand the achievement of performance results to the value and purposiveness of public action. In other words, the outcomes perspective entails a judgement of success or failure of public programs (Bovens et al. 2001; Compton & t’Hart, eds., 2019; McConnel et al. 2020). This perspective requires greater clarity regarding the intervention’s logic, design, conduct and implementation as well as the goals to be achieved. The payment by results are examples of instruments, especially popular in Anglo-Saxon countries, that seek to armour the achievement of social outcomes, with the application (e.g., to health and judicial policies) (Farrar et al., 2009; Fox and Albertson, 2011). In Europe, both the Cohesion Funds and the post-Covid ‘Next Generation EU’ program, have intensified the linkages between funded programs and well-defined outcomes to be achieved: the NGEU will reimburse the member states’ expenses only after the realisation of specific outcomes. Within this framework, the appropriate identification of the results to be achieved and of the method of measurement becomes of paramount importance.

The transition towards outcomes has been welcomed by the evaluation community. Notwithstanding, shortcomings emerge in the day-to-day evaluation practice, making the role of policy advice challenging, as well as communication with stakeholder groups. The appreciation of outcomes initially established could be unsatisfactory or even misplaced, for instance: in case of poor or conflicting policy design (Compton et al. 2021); uneven design and implementation processes (Shepherd, 2011); change in the political mood; unexpected conflicts, or external events (Covid pandemic, Russia-Ukraine war, etc) that substantially changed the salience of the intervention or the resources available at the beginning (McConnell, 2010). The process of determining and measuring policy success or failure is moreover challenged when public interventions are either inconsistent or complex (e.g., in the case of multilevel governance, where responsibility for policy planning and implementation is dispersed between different levels of administration). These may include different policies to solve crosscutting issues or wicked problems (Bailie et al., 2022; Paparini et al., 2020; Daviter, 2017).

Various authors have noted that a linear and entirely predictable policy design is not suitable for the more complex and challenging policy domains. Theory-based evaluations (Chen, 1990; Weiss, 1997; Chen, 2012; Mayne, 2015) and realist designs based on the CMO or the CIMO configuration (Pawson & and Tilley, 1997; Pawson, 2006; Pedersen and Rieper, 2008; Punton et al., 2020), suffer from the tendency to represent oversimplified policy processes, without adequately considering the complexity of these processes and positive and negative, expected and unexpected outcomes. The advocates of ‘formative’ evaluations (Scriven, 1991, 1996; Stame, 2004) have underlined the shortcomings of evaluating complex policies and programs, and the consequent need for more open-ended evaluation designs, that in turn could be less able to synthesize clear-cut judgements about the program’s ultimate results. On their side, the quasi-experimental and formal evaluation advocates – or as they have been called, the ‘truth-seeking’ or positivist stream of evaluation theory (Shadish et al., 1991) - struggle to take into consideration the implementation process and its reasons for particular designs (Cook et al., 1979), thus risking an adequate interpretative framework for evaluation findings, or sometimes not focusing on the most relevant outcomes in the eyes of beneficiaries. In the case of payment by results tools, the push on outcomes could trigger a ‘creaming and parking mechanism’ (the process in which providers select job-ready clients for services and neglect clients more distant from the achievement of the goals (Greer et al., 2018). In this case, an evaluation design too fixated on pre-defined outcomes could provide opaque and incomplete findings, and not serve the scope of empowering citizens' decisions – at least not in an acceptable way.

Starting from the public policy debate on policy success and failure, we underline the need to bridge policy evaluation with policy process studies that tend to focus almost exclusively on goals.

Aims of the workshop & main questions

The workshop aims to gather contributions and experiences of public policy evaluation scholars who have addressed critical issues about the relationships between programmes, the evaluation of policy success and failures, and the capacity of evaluation to provide relevant policy advice with useful information to citizens and stakeholder groups.

In particular, the panel will raise the following research questions:

·       How does policy coherence (or the lack thereof) affect policy implementation in complex governance structures?

·       What are the risks inherent in understanding policy outcomes that focus mainly on policy aims at the expense of implementation?

·       What are the consequences of evaluation-policy process dysfunctions in terms of methodological choices, evaluation policy process, and relevance of evaluation findings for citizens and other stakeholder groups?

·       Are there epistemological questions, approaches and solutions to cope with these challenges?

·       Are there evaluation strategies and tools, including AI-based analytical instruments, able to mitigate or strengthen these relationships and challenges?

·       How do policy makers and policy advisors take into account and overcome these challenges? Are there systemic approaches being considered over episodic ones?

Expected Impact of the Workshop

The workshop findings would allow not only the evaluation scholars but the larger ICPP community to understand the main challenges in the way policy and programs are designed (i.e., goals, implementation features, incentives and other mechanisms, communication strategies); the capabilities of determining policy success and failure; and, in providing citizens and other stakeholder groups with valuable information deriving from the evaluation process.

The workshop’s findings would be suitable for a special journal issue or for an edited book.

Possible articles:

·      Giancarlo Vecchi and Erica Melloni (Politecnico di Milano). Was it a success or a failure? Methodological and conceptual challenges of evaluating justice sector innovation programmes. The case of justice reform in Italy was funded by the NGEU.

·      Simone Busetti (Università di Teramo). Discovering policy outcomes: effectiveness, extrapolation and policy transfer.

·      Dominika Wojtowicz (Kozminski University), Challenges of evaluation of incoherent policy. Assessing policy of Polish cities toward integration of refugees from Ukraine.

·      Robert Shepherd (Carleton). Moving the Canadian, UK and Australian federal evaluation systems from focusing on expenditure management to informing sound policy choices.

References

Bailie, J.; Cunningham, F.; Abimbola, S.; Laycock, A.; Bainbridge, R.; Bailie, R.; Conte, K.; Passey, M. & Peiris, D. (2022) Methodological pluralism for better evaluations of complex interventions: lessons from evaluating an innovation platform in Australia. Health Research Policy & Systems. Vol. 20(1): 1-14.

Bovens M., P. ‘T Hart & G.B. Peters. Eds.(2001). Success and Failure in Public Governance. A Comparative Analysis. Cheltenham: Edward Elgar.

Chen H.-T. (1990) Theory-driven evaluations. Sage, Newbury Park, Calif.

Chen H. T. (2012) Theory-driven evaluation: Conceptual framework, application and advancement. Springer.

Compton M.E., J. Luetjens & P. ‘t Hart (2019). "Designing for Policy Success." International Review of Public Policy 1(2): 119-146.

Compton M.E. & P. ‘t Hart. eds. (2019). Great Policy Success. Oxford: Oxford University Press.

Cook T. D., Campbell D. T. and Day A. (1979) Quasi-experimentation: Design & analysis issues for field settings. Houghton Mifflin, Boston.

Dahler-Larsen P. (2011) The evaluation society. Stanford University Press.

Farrar S., Yi D., Sutton M., Chalkley M., Sussex J. And Scott A. (2009) Has payment by results affected the way that English hospitals provide care? Difference-in-differences analysis, Bmj 339.

Fox C. And Albertson K. (2011) Payment by results and social impact bonds in the criminal justice sector: New challenges for the concept of evidence-based policy?, Criminology & Criminal Justice 11, 395-413.

Greer I., Schulte L. and Symon G. (2018) Creaming and parking in marketized employment services: An Anglo-German comparison, human relations 71, 1427-53.

Mayne J. (2015) Useful theory of change models, Canadian Journal of Program Evaluation 30.

Pawson R. (2006) Evidence-Based Policy : A Realist Perspective. Sage, London.

Pawson R. and Tilley N. (1997) Realistic evaluation. Sage.

Pedersen L. H. and Rieper O. (2008) Is realist evaluation a realistic approach for complex reforms?, Evaluation 14, 271-93.

Punton M., Isabel V., Leavy J., Michaelis C. and Boydell E. (2020) Reality bites: Making realist evaluation useful in the real world, CDI Practice Paper 22.

Scriven M. (1991) Prose and cons about goal-free evaluation, Evaluation Practice 12, 55-62.

Scriven M. (1996) Types of evaluation and types of evaluator, Evaluation Practice 17, 151-61.

Shadish W. R., Cook T. D. And Leviton L. C. (1991) Foundations of program evaluation: Theories of practice. Sage.

Shepherd, R. (2011). “In search of a balanced Canadian federal evaluation function: Getting to relevance.” Canadian Journal of Program Evaluation, 26(2), 1-45.

Stame N. (2004) Theory-Based Evaluation and Types of Complexity, Evaluation 10, 58-76.

Weiss C. H. (1997) How can theory-based evaluation make greater headway?, Evaluation review 21, 501-24.

The workshop aims to gather contributions and experiences of public policy evaluation scholars who have addressed critical issues about the relationship between programmes, the evaluation of policy success and failures and the capacity of evaluation as a field to provide relevant policy advice with useful information to citizens and stakeholder groups. In particular, the workshop would receive papers discussing the following research questions:

- How does policy coherence (or the lack thereof) affect policy implementation in complex governance structures?

- What are the risks inherent in understanding policy outcomes that focus mainly on policy aims at the expense of implementation?

- What are the consequences of evaluation-policy process dysfunctions in terms of methodological choices, evaluation policy process, and relevance of evaluation findings for citizens and other stakeholder groups?

- Are there epistemological questions, approaches and solutions to cope with these challenges?

- Are there evaluation strategies and tools, including AI-based analytical instruments, able to mitigate or strengthen these relationships and challenges?

- How do policy makers and policy advisors take into account and overcome these challenges? Are there systemic approaches being considered over episodic ones?

Outcome measures are required to assess a policy and program’s effectiveness and to improve its accountability. The focus on the ‘end’ results challenges policy makers and civil servants from the achievement of performance results to purposiveness of public action. In other words, the outcomes’ perspective entails a judgement of success or failure of action. The shift towards outcomes has been welcomed by the evaluation community. Notwithstanding, shortcomings emerge in day-to-day evaluation practice, making the role of policy advice challenging, as well as the communication with the stakeholder groups. Starting from the public policy debate on policy success and failure, the workshop underlines the need to bridge policy evaluation with the findings of policy process studies. The workshop aims to gather contributions and experiences of public policy evaluation scholars who have addressed critical issues about the relationship between programmes, the evaluation of policy success and failures, and the capacity of evaluation to provide relevant policy advice with useful information for citizens and stakeholder groups.

Erica Melloni is a researcher at the Politecnico di Milano. Past president of the Italian Evaluation Society (2017-20), member of the Italian Political Science Association, she is also a member of the evaluation units of the Ministry of Economics and Finance and of the University of Naples. Her main scientific interests are on quality of public administration and territorial policies.

Robert Shepherd is full professor and graduate Supervisor of the online Diploma in Public Policy & Program Evaluation (DPPE) at the School of Public Policy & Administration at Carleton University (Ottawa, Canada). He is a former Canadian federal public servant and management consultant in the areas of audit, evaluation, and performance measurement. He is the current co-chair of the Consortium of Evaluators in Evaluation Education (CUEE). His current research focuses on the digitalization of government, creating the conditions for effective spending and strategic reviews in comparative perspective, and generating youth leadership in Indigenous communities.

Giancarlo Vecchi is associate professor of Public Policy at the School of Management of the Politecnico di Milano. He teaches “Public Analysis” and “Policy Design & Evaluation”. He is Past president of the Italian Evaluation Society (2002-2003). Recent research: Cybersecurity as a Public Policy. The Institutionalization of platform and network security in the EU and Italy; the modernization of the Italian Judicial system; development of digitization in the Italian & European public sector; evaluation of protected areas policy. His key qualifications are in the field of Public Policy Analysis, applied to the following aspects: institutional and capacity building; public sector reforms (with special attention to ICT-driven innovations in the public sector); policy design & public policy evaluation.

Dominika Wojtowicz is an instructor in the Department of Economics. She received scholarships from the Department of Social Sciences at the University of Siena, Italy, and the Department of Management at the Polytechnic University of Cartagena, Spain. Her scientific interests focus on the effectiveness of public interventions aimed at strengthening regional and local development.