T07P06 - Evidence-based Policy Making and Policy Evaluation

Topic : Policy Design, Policy Analysis, Expertise and Evaluation

Panel Chair : Giancarlo Vecchi - giancarlo.vecchi@polimi.it

Panel Second Chair : Hiroko Kudo - hirokokd@tamacc.chuo-u.ac.jp

Panel Third Chair : mita marra - mimarra@unisa.it

Objectives and Scientific Relevance of the panel

Call for papers

Session 1 Session 1 - Evidence-Based Policy Making: Evaluation and Methodological Challenges

Discussants

Giancarlo Vecchi - giancarlo.vecchi@polimi.it - Politecnico di Milano - Italy

Toward better use of the evaluation evidences in public polices: The MORE project

dominika wojtowicz - dominika.wojtowicz@kozminski.edu.pl - Kozminski University - Poland

Despite the extensive number of evaluation research, the use of evaluation results for the formulation and implementation of public policies is still quite limited (Olejniczak, 2013; Wojtowicz and Kupiec, 2016). This may derive from the fact that many of evaluation studies are of a poor quality or they do not cover the actual decision makers’ knowledge needs. However, delivering credible and rigorous evidence to decision makers seems not to be sufficient either.  Hence, an opportunity to improve effectiveness of public policies with the use of evidence remains a great challenge.

The implementation of a holistic approach of the provision of evidence to the policymakers may be the answer to this challenge. The approach should ensure - on one hand - reliable evidence, consistent with the needs and expectations of decision makers, and – on the other - provide that this evidence will be delivered in a proper manner, understandable to the recipient, or, to put it in different words, that the knowledge will be “broked” (Meyer, 2010; Olejniczak et al. 2016).

We designed and implemented a pilot project, which aimed at introduction of such approach to delivering useful and practical evidence for the Polish Ministries’ decision makers.  Within a tree-year MORE project we focused on the solutions implemented or planned for implementation in the four areas of public policies in Poland: special economic zones, public transport in non-urban areas, safety of nutrition in educational institutions and employment policy. In each case we concentrated on a specific problem or question, which may have been limiting the effectiveness of adopted solutions. 
We elaborated a four step process, which aimed at (1) provision of useful and practical evidence, obtained as a result of adaptation of research designs corresponding to the knowledge needs (Stern et. al 2012) and (2) increase the likelihood of actual use of the results by decision makers, through the application of knowledge brokering approach (Turnhout et al., 2013).

The effectiveness of the proposed process was proved by actual changes, which were adopted by the decision makers to the 3 out of 4 policies, which were the subject of the MORE project.

Making Policy while you do not have any Evidences – How to deal with scientific and technical uncertainty

Hiroko Kudo - hirokokd@tamacc.chuo-u.ac.jp - Chuo University, Faculty of Law - Japan

Research question and focus:

This exploratory study investigates into a rather common question in a specific field. The research question is; to which extent do we (academia, government agencies, lawmakers, etc.) have evidences which contribute to design and implement public policies? And the field of research is; how to promote, support and regulate the development and/or use of AI in the society with public policy.

When the issue is addressed from the viewpoint of scientific uncertainty, of course we know that there are limitations of “evidences” on these quickly-developing high-tech fields, mainly because of their nature. At the same time, however, there are strong attentions and expectations, especially from lawmakers and the industry, because of their potentials. The technology-driven field, thus, has always been controversial. Past cases and researches show that the policy design and implementation in these areas have often been driven by political ambitions and business interests, not necessary through evidences and rational analyses, resulting in wrong investment and policy failure (Callon, 1995; Kudo, 2015).

Since it is difficult to predict the scientific and technological development, these policies often lack policy objectives and goals, making them difficult to evaluate. This lack of clear objectives and goals turn out to be an opportunity to some lawmakers and bureaucrats, who like to aim higher and put goals like “number one in the world” without clear definition. These ambitious but ambiguous goals survived among science and technology related policies until the era of austerity budget, but then strongly criticized by lawmakers themselves as well as the taxpayers (Kudo, 2015).

The paper analyses two cases, Japanese smart city policy (Granier and Kudo, 2016) and on-going effort to introduce AI related policy in Japan, and tries to understand the issues and to draw some solutions.

 

Methodology:

The research adopts qualitative analysis, including analysis of literature and primary documents, semi-directive interviews and direct observation. Although the cases are limited, they represent a first attempt to provide preliminary investigation and to generate hypotheses for further studies. The selection of the cases has been conducted based on the relevance to the subject under investigation.

The case of Japanese smart city policy (2010-2014) will be analysed to identify the issues and at that time solutions to tackle the lack of evidences in designing and implementing the policy and then evaluating the results. The AI policy case is based on participatory observation of the author, as she has been participating in various working groups with government agencies, the industry, and academia on the issue. The collection of data is carried out through different sources, in order to allow a better accuracy and to ensure the triangulation of the resources used (Strauss, Corbin, 1990; Lee, 1999). In particular, data will be collected through semi-structured interviews to some actors and participatory observation for enriching the data and better understanding the links between the object of investigation and the context.

Can evaluations really contribute to evidence-based policy making at government level? - the case of the French Government Modernisation Evaluations of public policies (2012-2016).

Thomas DELAHAIS - tdelahais@quadrant-conseil.fr - Quadrant Conseil - France

Clément Lacouette-Fougere - clement.lacouette@sciencespo.fr - SGMAP Prime minister's office - France

There is a longstanding debate over the circumstances in which evaluations, or any scientific production of evidence, may play a significant part in policy making. Theoretical (see Weiss, Patton, Lindblom…) and empirical works (in the form of meta-evaluations, see e.g. CDI, 2015 or OECD, 2016) have contributed to defining the right environment and conditions for evidence-based policy making.

 

Political decisions do not always have to follow what evidence dictates. Yet the continuing debate also proves that even the most consequential decisions may not be « evidence aware » (Nutley et al, 2000). Some even wonder about the pointlessness of evaluation (Spenlehauer, 2016). Institutionalisation of evaluation over the last 20 years may have, to some extent, embedded evaluation into administrative processes (Jacob et al, 2015 and, for France, Lascoumes, Lacouette-Fougère, 2013), but can evaluation still contribute to political decisions at a higher level?

 

To answer this question, we build upon an evaluation commissioned by the French SGMAP, which stands for Secretariat-General pour la Modernisation de l’Action Publique (Prime minister’s office). The rationale behind MAP was to use evaluation technologies to improve public policies at a governmental and inter-departmental level, over a wide range of topics such as the national housing policy, waste, education, etc. We used a meta-evaluation (Stufflebeam, 1974) to assess the quality of 65 evaluations launched between 2013 and 2016, as well as in-depth case studies of 8 evaluations to appraise their contribution to policy making. Our results show that these evaluations can (and did) contribute to taking evidence-aware decisions under certain circumstances, despite quite widespread quality issues. Using a Theory of change model (Weiss, 1997), we demonstrate that actual contributions owe in large part to how evaluation was embedded into decision-making mechanisms and ecosystems within the French national administration, and more marginally to the evaluation rationale itself.

 

Indicative Bibliography


JACOB, S., SPEER, S., & FURUBO, J.-E. (2015). The institutionalization of evaluation matters: Updating the International Atlas of Evaluation 10 years later. Evaluation, 21(1), 631. https://doi.org/10.1177/1356389014564248

 

OECD, “What works: Towards Evidence Informed Policy Making”, Public Governance Seminar, Issue Paper, Nov. 2016

Sandra M NUTLEY, Peter C SMITH et H. T. O DAVIES, « Introducing evidence-based policy and practice in public services », dans What works?: evidence-based policy and practice in public services, The Policy Press, 2000, p. 11.

 

LACOUETTE-FOUGERE, C., & LASCOUMES, P. (2013). Les scènes multiples de l’évaluation-les problèmes récurrents de son institutionnalisation. SciencesPo, LIEPP.

 

LLOYD, R., & SCHATZ, F. (2015). Improving Quality: Current Evidence on What Affects the Quality of Commissioned Evaluations. Centre for Development Impact (CDI) Working Paper.

 

SPENLEHAUER, V. (2016). La (f) utilité gouvernementale de l’évaluation des politiques publiques, quelques leçons américaines et françaises. Consulté à l’adresse https://spire.sciencespo.fr/hdl:/2441/32ipedmmo59m5qkc28gtneb4do/resources/wp49-spenlehauer.pdf

 

STUFFLEBEAM, D. L. (1974). Toward a Technology for Evaluating Evaluation. Consulté à l’adresse http://eric.ed.gov/?id=ED090319

WEISS, C. H. (1997). Theory-based evaluation: Past, present, and future. New directions for evaluation, 1997(76), 41–55.

Evaluating personalisation programmes: methodological challenges

Chris Fox - c.fox@mmu.ac.uk - Manchester Metropolitan University - United Kingdom

This paper reports on an evaluation that draws on quantitative and qualitative methods to assess the impact of a complex personalisation programme implemented in the criminal justice system. 


The ‘personalisation’ of public services is a key theme in current social policy reform in the UK. The Policy Evaluation and Research Unit at Manchester Metropolitan University has been working in collaboration with Interserve Ltd to develop and evaluate a model of personalised offender rehabilitation in 5 of the 21 newly formed Community Rehabilitation Companies (CRCs) that deliver probation services in England and Wales. We have identified learning from the design and implementation of personalisation in the UK social care sector and used this to specify a project to pilot personalisation in the English probation sector. However, a short-coming of personalisation in social care has been the lack rigorous impact evaluation. The aim of this paper is to discuss innovative evaluation methods that address the lack of rigorous impact evaluation of personalised public services.

 

A multi-phase personalisation pilot has been commenced. Too often, pilots in the criminal justice system have been implemented prematurely with insufficient time and resource put into first developing a sound theory of change (Weiss, 1997) and then testing key elements prior to a larger pilot. We are therefore following an adapted version of the model of piloting set out by the Education Endowment Foundation in its guidance (Education Endowment Foundation, 2015). This specifies three types of trial conducted in sequence in the development and testing of new interventions. These are pilot, efficacy and effectiveness trials. Pilot trials are early stage studies conducted in a small number of sites with the objective to ‘develop and refine the approach and test [an interventions] feasibility’ (Education Endowment Foundation, 2015: 3). This paper reports on the pilot trial. 

 


Five distinct personalisation concepts were implemented and evaluated during the pilot. 

Each concept will be piloted with a small number of cases (defined variously as service users, case managers or probation teams) selected purposively to allow testing of key theoretical constructs. Emphasis was placed on implementation evaluation, including the issue of implementing models of personalisation in a criminal justice setting where culture change is key. Qualitative research has been the predominant data collection methodology during the pilot, but in order to start to explore early impact we have used ‘small n’ impact evaluation designs, specifically process tracing (White and Phillips 2012). 

 

The paper concludes by outlining the next phase of the evaluation: an efficacy trial to test whether the personalisation model works at a larger scale in circumstances where the developers still have a degree of control over implementation and where (quasi)experimental impact designs are employed.

Session 2 Session 2 - Evaluation and Evidences in Public Policies

Discussants

Hiroko Kudo - hirokokd@tamacc.chuo-u.ac.jp - Chuo University, Faculty of Law - Japan

Maximising Evidence-Informed Change in Complex Policy Systems: Lessons from Africa and Asia

Fanie Cloete - gsc@sun.ac.za - University of Stellenbosch - South Africa

Paper focus: The evidence-informed approach to policy decisions is a fast-growing global industry, stimulated by a number of international initiatives like 3ie, DFID, IDRC and other policy scholars. It has its roots in the scientific approach to social science, combined with the exponential growth of the information society. It is a scientifically well-grounded methodological approach that has huge potential for improving policy decisions, processes and outcomes, if policy decision makers can be persuaded to take more rational, evidence-informed decisions based on sound relevant policy-related research findings and conclusions. This, however, is difficult to achieve in most policy contexts which are highly complex, and where policy decisions are traditionally influenced by the cumulative impact of a diverse number of considerations of which rational evidence of good or bad practices and of historically successful or failed interventions constitute only a small and sometimes even a negligible part.

 

Research question: The question that is addressed in this paper is how evidence-informed policy decisions can be maximized in the complex policy environments existing in developing country contexts.

 

Methodology: A brief assessment is made of the current international knowledge base on this topic, focussing especially on what policy change strategies have so far proven to work best in specific contexts. Examples of more and less successful policy change strategies in selected African and Asian countries are summarised to illustrate different issues. Practical illustrations also include first hand experiences by the author in a number of policy interventions.

 

Conclusions: The conclusions of the paper are that the nature of complex policy contexts precludes the identification of simple, linear, direct causal relationships between evidence adduced and the nature of policy decisions that the evidence relate to. Many competing interrelationships among individual, network, organizational and systemic variables in each unique policy context, determine the degree of successful transformation of policy-related evidence into improved policy processes, content and outcomes. A number of generic guidelines that have so far emanated from different policy programme design, implementation and evaluation interventions to maximize the impact of good quality policy-related evidence on policy decisions on changes in policy processes, outputs and outcomes in a number of African and Asian countries, are also identified and summarized.

 

Panel Fit: According to this panel “..the objective of the panel is to improve the impact of programme evaluation on the decision makers, avoiding the risk of policy-based evidences, developing a better conceptualization of collective problems and the feasible paths of solution”. The paper covers elements of all three panel themes for the conference: the integration between programme evaluation and policy analysis studies, approaches dealing with causal and transfer problems and lessons learnt from systematic programme and project evaluations in different policy domains.

Ontologies, Theories and Methodology in the Evaluation Research Debate. Dealing with the evidence-based policy making challenges

Giancarlo Vecchi - giancarlo.vecchi@polimi.it - Politecnico di Milano - Italy

The debate on ‘Evidence-based policy-making’ underlines the issue concerning the role of social science in supporting the design of public programmes and policy decisions. The reflection of many scholars is often translated in methodological terms, and the conclusion is oriented to the adoption of approaches that improve the reliability of the causal inferences that sustain the attribution of an outcome to a programme; and the preference of the RCT designs is quite often the result.  Indeed, we sustain a wider pluralism and expansion in the use of social science techniques in evaluation (Stoker and Evans; Bastow et al.). For example, if it is important to collect evidences on ‘what works’ about public programmes, at the same time policy makers and citizens are interest in the transferability issue, i.e. the effectiveness of a programme if implemented in another site; in that case we need to add to the ‘what works?’ analysis, answers regarding ‘what works, for whom, in what context’, the mechanism at the base of the observed changes, the reasons for the success of the winners and the failure of the losers, the implementation gaps, etc.. In other situations, the capacity to use participatory evaluation techniques can be fruitful to understand the positions of different actors and to help policy makers in overcoming conflicting setting or to empower some categories of actors in representing their point of view.

The paper will reconstruct, firstly, the ‘evidence-based policy making’ debate from the point of view of policy evaluation, discussing the (explicit or implicit) role of the ‘ontology’ issue.

After that, we’ll sustain that the methodological issue needs to be faced considering both the ontology of reference, and the theories used to explain the characteristics of the policy under evaluation. In doing that, the development of better connections between evaluation and policy analysis seems relevant. We will develop our thesis using examples derived from researches regarding the Italian case, especially from the area of public sector reforms.

Evidence-based Policy and Classifying Public Policy

Kazuya Sugitani - turnx2@gmail.com - Kyoto University - Japan

  Evidence-based policy is an important principle in many developed countries. Several books on the subject have been published. In addition, many related articles have appeared. Randomized Controlled Trials (RCTs) and systematic reviews have been especially highlighted as important in public policy-making.

  However, when the term “policy” is used, its meaning can vary substantially from one instance to the next. In truth, there are many different types of policies and it is difficult to argue that RCTs and systematic reviews are suitable for all types.

  Many prior studies have focused on specific types of policies. For example, policies on health care, education, medical services have been addressed in previous research dealing with evidence-based policy. These policy types appear suitable for evidence-based policy-making supported by RCTs and systematic reviews. However, it is important to understand the types of policies that are not suitable for RCTs in order to better understand the type of evidence required for each of the various policy fields.  In this pursuit, it is necessary to consider evidence-based policy more specifically.

  Public policy studies and policy sciences provide some tools for classifying the various types of policies. For example, one method is to classify policies based on “policy areas.” Public policy includes several areas of policy such as social security, culture, environment, and the economy. Each area requires different evidence from the others. In my presentation, I will try to identify the relationship between policy areas and evidence.

  Another method of classification is “Policy Typologies.” Policy typologies are complex concepts but can be used to classify public policies. Using policy typologies enables us to understand the use of power in policy-making and implementation. It can be applied in different policy studies. For example, in Japan, one researcher has suggested that policy evaluation should reflect policy typologies. I believe that this is applicable for evidence-based policy as well.

  In the theory of evidence-based policy, the term “evidence” has diverse meanings. Its definition is often vague. Hence, the meaning of evidence should be made clearer and more specific. Using “policy areas” and “policy typology,” I will attempt to identify the type of evidence needed based on the type of policy involved.

  It should be recognized, however, that the existing policy classification schemes were not specifically designed for evidence-based policy, but for other purposes. Consequently, in this presentation, I will suggest a policy classification that is appropriate for evidence-based policy. I will reconsider the two major policy classification schemes and examine whether they can be used in a discussion of evidence-based policy. The aim is to make evidence-based policy clearer and to provide guidelines for its use in devising specific plans.

A Systematic Review of Empirical Studies of Essential Medicines Policy in China: Implications for Evidence-Based Policy Making (EBPM) in Developing Countries

Xun Wu - wuxun@ust.hk - Hong Kong University of Science and Technology - Hong Kong, (China)

LILI LI - lililienviron@gmail.com - National University of Singapore - Singapore

Qian Jiwei - jiwei.qian@nus.edu.sg - East Asian Institute, National University of Singapore - Singapore

Although evidence-based policy making (EBPM) in health care has gained currency in developin g countries, their analytical capacity for producing sound evidence for policy making cannot be taken for granted. The meta-analysis presented here reviews 79 empirical studies published in Chinese academic journals 2010–2015 concerning the country’s essential medicines policy, a measure introduced beginning in 2009 to control cost escalation in health expenditures. Results show that most of these studies rely on simple before-and-after comparisons to assess the impacts of policy measures, and that few used control groups to deal with validity concerns. Our conclusions point to the critical importance of adequate analytical capacity in producing reliable evidence for policy making.

DECONSTRUCTING SOCIAL POLICY INNOVATION AND THE NEED TO GO BEYOND LINEAR WELFARE SYSTEMS: A PLURALISTIC QUEST FOR EVIDENCE-BASED POLICY MAKING

GIULIO PASI - giulio.pasi@ec.europa.eu - European Commission - Joint Research Centre - Spain

Gianluca Misuraca - gianluca.misuraca@ec.europa.eu - European Commission, JRC-IPTS - Spain

The financial crisis, along with the changing nature of work and the dramatic socio-demographic challenges current societies are facing, has put the economy and society of several European Union Member States under enormous pressure, at a time when the demand for social services is growing due to the ageing of societies. State budgets were decreased during the crisis, and considerable cuts in social services were made, especially after the adoption of policy approaches based on austerity. This situation calls for the adoption of innovative, long-term social policy strategies and for modernized welfare systems, which foster more efficient, responsive and appropriate social services. As pointed out by several scholars from different fields, a general rethinking on how economies and societies work and interact with each other is needed. However, though many initiatives have been launched and funds allocated, as well as many debates have spread in the epistemic community, evidence on the results obtained is as yet lacking, and one of the more relevant reason coming from a systematic literature review is the resistance to go beyond linear approaches and still well established assumptions on economic, political and institutional behaviours.

This paper presents a proposal of a methodological framework to analyze the economic and social return on investment of initiatives in social policies and services, defined as social policy innovations. This framework – called i-FRAME – proposes a novel approach that mainly combines System Dynamics and Agent-Based Modelling Simulation in a hybrid model. The inspiration for this approach lies in different strands of the literature, such as the debate on the crisis of evidence-based policy-making, complexity theory and system thinking and is informed by a critical appraisal of traditional economic modelling approaches. After a brief overview of the policy context and research background, the chapter illustrates the method, sources and key findings of the literature and practice review, identifying the knowledge gaps, with a specific emphasis on the need to go beyond a mere linear approach to the understanding of social policy innovations, especially under the present circumstances. On the one hand, this will help in grounding and identifying the rationale underpinning the design of the i-FRAME modelling and the simulation approach proposed; on the other hand, also the pluralistic quest for evidence-based policy making will emerge as a promising opportunity for a new and closer role of social sciences within the policy process. The proposed innovative toolkit for policy modelling (and informatics which underpins the i-FRAME) is then outlined, arguing on its possible use for ex-ante, in-itinere, and ex-post simulation of the impact and the functioning of a given policy initiative in social services planning, design or provision. The chapter concludes with suggestions as to possible directions for further research and outlines the policy implications.

Session 3 Session 3 - Policy Sectors, Evaluation and Evidences

Discussants

Giancarlo Vecchi - giancarlo.vecchi@polimi.it - Politecnico di Milano - Italy

Hiroko Kudo - hirokokd@tamacc.chuo-u.ac.jp - Chuo University, Faculty of Law - Japan

The Affordable Care Act’s Excise Taxes: Impact on Medical Device Manufacturers

Ngoc Dao - ngdao@indiana.edu - Indiana University - United States

 

The Affordable Care Act (ACA) enacted in 2010 levies an excise tax on the sale of medical devices (Medical Device Tax) that are consumed domestically in the United State. The Medical Device tax became effective January 1st 2013.  The tax applies to sales of medical devices in the United States, regardless of whether the device was manufactured in the U.S. or whether it was imported for use in the U.S.  The tax is equal to 2.3% of the manufacturer’s price, and is reported by manufacturers or importers on IRS Form 720 – Quarterly Federal Excise Tax Return.  

This paper examines the impact of this tax on industry-level total employment, number of production workers, hours worked, average wages, value of shipments (sales), exports, and tax liability. In order to conduct our estimation, we utilize the synthetic control method of Abadie et al. (2010), where the treatment group consists of industries involved in medical device manufacturing, and the donor pool consists of other manufacturing industries. The synthetic control group is created by assigning weights to potential control industries so that the weighted average of some descriptive statistics matches as closely as possible to those in the treatment industry in the years before the medical device excise tax was implemented.  This method allows us construct the most relevant comparison groups while potentially reduce the uncertainty about the parallel trend assumption under the difference in difference approach. Using the Annual Survey of Manufactures (ASM) and U.S International Trade data during the period 2007 and 2014, we have found a statistically significant increase in and significant declines in employment, production workers and hours worked.  Though declines are found in average wages and the value of shipments, the estimates are not significant.

This research project fits the panel in the way that it provides empirical findings using causal inference framework when evaluating public policy. The evidence is of importance to advice policy makers in designing health policy in order to mitigate negative effects that might have.

Embedding impact assessment in policy making. The case of Flanders-Belgium: developments, difficulties and challenges.

Jan De Mulder - jan.demulder@kb.vlaanderen.be - PermRep Belgium/Flanders to EU / Public Governance department - Belgium

Impact assessment (IA) is not a general requirement in the policy making process of the Flemish region (North Belgium).

Environmental and Strategic Impact Assessments are mandatory. Both instruments are based on supranational legislation (inter alia European Union directives). But their transposition and implementation tracks reveal several difficulties, including court cases and burdensome legislative processes, also in Flanders.

Furthermore – and inspired by EU and OECD work - Regulatory Impact Assessment (RIA) was introduced by a Governmental decision in 2005 but is not a legal requirement. Evaluations of the RIA practice concluded that RIA is yet not firmly established. Next to SEA and RIA, a kind of children and youth IA exists and is linked to the RIA approach. During the past legislature of the Flemish Government (2009-2014) some initiatives regarding additional impact assessment approaches were introduced. They concern different policy issues: poverty, inclusion (equal opportunities), Brussels, local governments and sustainable development

Improving RIA was one of the objectives of the new Regulatory Management Strategy adopted in 2010. This strategy stipulates that RIA is the central impact assessment instrument. In 2011 an initiative to elaborate a flexible approach to align a Sustainability Impact Assessment (SIA) approach to the RIA practice was agreed between all concerned policy domains of the Flemish government. This initiative was enhanced by the fact that existing impact assessment practices/requirements regarding youth and gender favoured a more clearly embedded SIA. A new poverty impact assessment requirement was adopted in 2014 as an element of the integrated SIA/RIA. The RIA manual refers to a SIA matrix or checklist. Its application remains however limited but the revision of the IA Guidelines of the European Commission in 2015 might trigger further actions. The refugee and migration developments in the EU did also raise the potential of Migration Impact Assessment.

 

The paper will frame this regional regulatory and policy development within the implementation of several EU policies and strategies (directives, better regulation policy including IA and evaluation). It will also assess the impact of the experiences with mandatory EIA on the introduction and use of other IA tools. Evidence based policymaking as favoured by the different policy layers in a multilevel governance perspective creates also complexities. Next to addressing new societal problems public administrations need at the same time to simplify procedures and processes…  This paper will look with a practitioners view into the institutionalization of impact assessment in Flanders with a particular focus on interactions (administration, experts, stakeholders, politics) and progress. The current situation will be assessed in the light of the latest European policy objectives (Council conclusions) regarding impact assessments, evaluation and public governance improvements.

The Impact of a Universal Free School Lunch Program on Students' Health and School Life

Jung Haeil - hijk@korea.ac.kr - Department of Public Administration, Korea University - Korea, (South) Republic of

Dahye Kim - moon4145@korea.ac.kr - Korea, (South) Republic of

This study investigates whether a universal free school lunch program affects students’ health and school life. Although the positive impact of providing free meals to poor students on health and nutrition level has been well demonstrated, it has been debated whether the stigma and limited food quality related to such programs for disadvantaged students discourage their registration and even harm their school life with their peers. On the other hand, providing free school meals to all students seems to be too costly and its impact may be limited because it provides free meals even to those students who do not need it. Therefore, studying the impact of a universal free school meals program is meaningful because it is one of many fronts of serious debates related to whether the central or local governments should provide public programs to everyone regardless of any pre-conditions or only to eligible individuals based on their income and circumstances. It has been challenging to study the causal impact of a universal free school lunch program because observational studies are limited due to selection bias and the large-scale social experiments are rarely possible. Also, there were few studies which looked at the impact of such programs on students’ academic performance and social life in school beyond their physical health and nutrition levels. In order to estimate the unbiased causal impact of the program, we take advantage of a natural experiment driven by the policy expansion of the universal free school lunch program in South Korea that varied across school districts since 2011. For example, all elementary students in Seoul (largest city in S Korea) received the free school lunch starting in 2011 while those in Pusan (second largest city in S Korea) received the free school lunch starting in 2014. Aligned with this research strategy, this study uses the panel data from the Korean Children & Youth Panel Survey (KCYPS) that collected the individual, family characteristics and school records of the first, fourth and seventh-grade students in 2010. This panel is ideal for this study because the KCYPS followed around 7,000 students every year from 2010 to 2013 and surveyed the school districts, students’ physical and mental health, their family income and parents’ characteristics, their academic records, and their relationship with peers at school. Using this data and a natural experiment of the expansion of the free school lunch program in South Korea, this study implements the Difference-in-Differences method. This study is expected to make a considerable contribution to the related literature by showing whether the universal free school meal programs are effective on students’ academic performance and social life in school beyond their physical health and nutrition levels.

LOW CARBON URBAN STRATEGIES: AN INVESTIGATION OF 124 EUROPEAN CITIES

Benedetta Lucchitta - benedetta.lucchitta@unibocconi.it - Bocconi University - Italy

Edoardo Croci - edoardo.croci@unibocconi.it - IEFE - Università Bocconi - Italy

Local governments play a crucial role in reducing CO2 emissions. In order to endorse and support the efforts of local authorities, the European Commission launched the Covenant of Mayors (CoM) in 2008: by November 2016 more than 7,000 local authorities had signed the CoM, accounting for 212 million inhabitants. Covenant signatories commit to reduce CO2 by at least 20 per cent in 2020 (in respect to the level in a baseline year), to prepare a Baseline Emission Inventory (BEI) and to submit a Sustainable Energy Action Plan (SEAP). The SEAP is the key document in which each Covenant signatory outlines how it intends to reach its CO2 reduction target. It defines the activities and measures set up to achieve the target, together with time frames and assigned responsibilities. Each SEAP includes a list of actions for reducing emissions and the quantification of their intended emission reductions.

The aim of this paper is to assess the distribution of intended emission reductions by cities among sectors, and to evaluate the relevance of the policies and actions adopted by cities to achieve these reductions. The analysis is based on data provided by a subset of cities participating in the European Commission CoM initiative: all European cities with more than 100,000 inhabitants and with an accepted SEAP by February 2014 are included in the study. The sample is composed of 124 cities implementing more than 5,500 actions. Each action has been classified based on the official description into a “category of action” (area of intervention targeted by an action) and into a “policy lever” (instrument used by the local authority to implement the action).

Cities in the sample account for a total of 370 megatons of CO2 emissions in selected baseline years and 94 megatons of intended emission reductions per year. The total emission reduction planned by cities corresponds to 25 per cent of baseline emissions, beyond the minimum target of 20 per cent required by the CoM. Emission reductions are concentrated in the Building and Transport sectors. Furthermore, sectoral analysis points out that the largest portion of intended emission reductions derives from actions implemented by private actors. However, actions in sectors under the direct control of cities’ administrations are planned to deliver higher reductions with respect to the correspondent baseline emissions.

In the Building sector, the category of action delivering the highest share of CO2 emission reduction is integrated actions, which combines several types of intervention to maximise the energy efficiency of buildings; in the Transport sector, it is modal shift, which implies a transition from private transport to public and cleaner transport modes. In both the Building and Transport sectors, cities plan to reduce the major amount of CO2 emissions through the implementation of management and organisation, infrastructure construction, and awareness-raising infrastructure policy levers.

The results can be used by policy makers at different government levels to facilitate the implementation of planned actions, to reshape their mitigation strategies in coherence with cities’ commitments and plans and to replicate policies and actions on the basis of CoM signatories’ experiences.

Who are Beneficiaries of Public Assistance?——The Performance of the Targeting Mechanism of China's Rural DiBao program

fengqin liu - chunzeevaluation@vip.163.com - China

The objective of this paper is to explore who are beneficiaries of DiBao program in rural China and to examine why some poor families haven't received any DiBao benefits,whereas some non-poor families have been benefiting from public assistance. A quantitative study,involving 9107 households and 34132 individuals from five China provinces,was conducted in this article. The analysis,based on micro-level data,suggests that income poverty families only accounts for 31.86 percent of total DiBao benefits and a large quantity of DiBao benefits are allocated to families in expenditure poverty. Community targeting is the main reason of the observed inconsistency between real beneficiaries and beneficiaries stipulated by policy. Drawing on the empirical results,this study posed that a hybrid targeting mechanism should be employed to identify the beneficiaries of rural DiBao program. The hybrid targeting mechanism consists of proxy means test and community targeting. In the practical application, community targeting will be used to determine the beneficiaries list firstly.Then proxy means test can be used to avoid elite capture by identifying unqualified families. Existed literatures about the performance of DiBao targeting is focusing on the income poverty and could not present the real distribution of DiBao benefits.

Export PDF