T07P14 - Policy Evaluation in Performance Regimes: A Comparative Perspective

Topic : Policy Design, Policy Analysis, Expertise and Evaluation

Chair : Liang Ma - ken0821@gmail.com

Second Chair : Bo Yan - yanbosir@xjtu.edu.cn

General Objectives, Research Questions and Scientific Relevance

Call for papers

Session 1

Friday, June 30th 10:30 to 12:30 (Block B 5 - 2 )


Elaine Yi Lu - anny_luyi@yahoo.com - City University of New York - United States

Impact of Performance Regime on Local Government Policy Evaluation in Indonesia

Meita Ahadiyati Kartikaningsih - meitakartika@yahoo.com - National Institute of Public Administration - Indonesia

Aldhino Niki Mancer - aldhino.niki.m@gmail.com - National Institute of Public Administration Republic of Indonesia - Indonesia

Local government policy evaluation is needed to understand local government capacity in administrating local autonomy. This process require measurement and benchmarking of the results that have been achieved with the results should be achieved. It is a process that is about the production of information of success (and failure).

Based on analysis of documents and literatures review, this paper attempts to examine local government policy evaluations in Indonesia. Key research question of this paper includes how does performance regime affect the construction of instrument for local government policy evaluation? In addition, this paper attempts to analyze several mechanisms in local government policy evaluation, how these seem to be overlapping and competing in creating definition of ‘success’ of local government, to understand  the contributing factors to the construction of such competing mechanism. Competing evaluation approach of Local Government policy in Indonesia may create different versions of ‘success’ (and multiple interests in being successful). This situation is not so effective in achieving objective of policy evaluation, policy coordination and the ways to improve local government performance.


Key words: competing evaluation, local government policy evaluation, performance




JASMINE AHMAD - jasmi661@salam.uitm.edu.my - UNIVERSITI TEKNOLOGI MARA (UiTM) - Malaysia

Download the paper




Rafidah Mohamed Hashim

FSPPP Universiti Teknologi MARA, 40450 Shah Alam, Malaysia



Jasmine Ahmad

FSPPP Universiti Teknologi MARA, 40450 Shah Alam, Malaysia






Policy evaluation is fundamental to any public policy management due to the rising concern of public on ‘what works’ about public programs. Although evaluation on public policies and programs are mainly championed by government entities and organizations, recent trend witnesses the rising roles of third party entities in evaluation activities, which subsequently acts as a mechanism of check and balance as well as creating competitive conditions. The non-governmental entities such as community groups, media, non-profit organization, research institutions and international organizations begin to take roles in evaluation activities. Apart from enhancing accountability and transparency, the involvement of third party entities such as international organizations also helps in building the capacity for conducting evaluation among donor-recipient countries. The paper seeks to examine to what extent participatory approach and capacity building in policy evaluation are enhanced with the roles of third party entities in Malaysia. Do these third party entities help in creating an enabling environment that facilitates policy evaluation? It is therefore interesting to uncover these aspects with some comparison from selected developing countries in the ASEAN region. The scope of the paper is limited to the roles played by selected third party entities such as media, evaluation societies, research institutions, and international organizations with the focus on participatory approach and capacity building aspects. The paper is written mainly based on available academic sources of information, which also includes international organization reports, annual reports, and conference proceedings. The paper concludes that although the presence of these third party entities has somehow helped in facilitating evaluation activities in the Malaysian public sector policy framework, there are still lots of improvement works need to be done especially in terms of enhancing transparency and accountability through the management of evaluation results. The evaluation capacity building efforts especially at the second and third level of government is another aspect that requires attention. The paper will contribute to additional information on the roles of third party entities in evaluation. Future studies may look deeper into how third parties entities enable independent evaluation that helps building reliable national evaluation systems.



Keywords      : Third party entities, policy evaluation, participatory evaluation/ approach, and evaluation capacity building

Sub-theme    : T07 – Policy Design, Policy Analysis, Expertise and Evaluation

                      P14 – Policy Evaluation in Performance Regimes: A Comparative Perspective

Evaluation for accountability or improvement?: A cross-country comparison of performance regimes

Liang Ma - ken0821@gmail.com - Renmin University of China - China

Policy evaluation and performance measurement have been extensively adopted in public sector, and evaluations by external entities (e.g., academia, consultancy firm, media, and auditor) has been burgeoning in various fields. It is intriguing that so many countries adopted external performance evaluation/review albeit for different purposes and by distinct approaches. Why do central/federal governments implement external evaluation? What are the key approaches and attributes of external evaluations across countries? What are the primary similarities and differences in external evaluation across countries? In this study we develop a theoretical framework to analyze external evaluations in multiple countries. The framework focuses on two dimensions, namely purposes and focuses, of external evaluation. The purpose of external evaluation is about the primary aims of evaluation, e.g., political/administrative accountability or organizational performance improvement. The focus of external evaluation refers to the level of analysis, including the whole of the government, agency, and program/project. We report empirical findings of comparative case studies in four countries (the US, China, Singapore, and New Zealand). The findings help understand the underlining rationales and key success factors of external evaluation, and suggest policy implications for implementing and improving external evaluation.

Session 2

Friday, June 30th 13:45 to 15:45 (Block B 5 - 2 )


Elaine Yi Lu - anny_luyi@yahoo.com - City University of New York - United States

Does the Local Government Follow the Strategic Intention of the Central Government in China? A Comparative Social Network Analysis of the Implementation of the Mass Innovation and Entrepreneurship Policies

Bo Yan - yanbosir@xjtu.edu.cn - Xi'an Jiaotong University - China

WEI LI - liwei@cuhk.edu.hk - Chinese University of Hong Kong - Hong Kong, (China)

China has initiated the national policy of Mass Innovation and Entrepreneurship and then a series of policies in a top-down approach in recent years. However, few is known about how those policies are implemented and whether the local government would conform to the strategic intention of the central government in the authoritarian system. This paper attempts to explore the differences between the central and local governments in policy implementation. We compare the policies of supporting Mass Innovation and Entrepreneurship by the state council and the Beijing municipal government. The findings show that the local government has adopted old strategies in implementing the policy of Mass Innovation and Entrepreneurship by the central government., From the path-dependency perspective, we demonstrate that the local government is under the performance pressure and prefer a familiar approach to implement the policy. In future research, we will explore structural and normative factors which explain such preferences.


Can the US Keep the PACE? A Natural Experiment in Accelerating the Growth of Solar Electricity

Nadia Ameli - nd.ameli@gmail.com - University College London, Institute for Sustainable Resources - United Kingdom

Download the paper

Paper project

Governments' efforts to expand solar generation base and integrate it into municipal, regional, and national energy systems, have spawned several programs that require rigorous policy evaluations to assess their effectiveness, costs and contribution to Paris Agreement's goals. In this study, we exploit a natural experiment in northern California to test the capacity of Property Assessed Clean Energy (PACE) to promote PV investment.

PACE is an innovative energy scheme used in certain areas of the US to support renewable energy deployment. The installation of clean energy technology through PACE is financed by local governments, by issuing bonds whose proceeds are used to finance loans to homeowners for PV installations. Residential property owners pay back the loan through an increment on their property tax bill over a 20-year period. If the property is sold before the end of the repayment period, the new owner takes over the remaining debt.


Research question

Has been the PACE program an effective policy to boost the solar PV market in California?



This study assesses PACE’s effectiveness on new solar installations using a regression discontinuity (RD) exploiting the geographical discontinuity of the program. Under the RD design, a geographic or administrative boundary allows the investigator to select units into treated and control areas. Indeed, the unique characteristic of this design is the method by which research units are assigned to program or comparison groups as the units’ placement depend solely on the basis of county border. This allows the investigator to control for unobserved confounding factors, which if uncontrolled will result in biased estimates. Making causal inference in policy evaluation exercises is challenging as it requires constructing a credible counterfactual, i.e. what the outcome of interest (PV installations) would have been in the absence of the policy intervention (PACE program). The RD approach permits to do just that.

Given that PACE was implemented only in Sonoma County, the county boundary determines whether households are eligible for the PACE financing program, thus allowing us to draw arbitrarily the treated (cities eligible for the program) and control groups (cities not eligible for the program).


How this paper fits the panel chosen

This study is an example of a rigorous policy evaluation based on an experimental framework. This approach is still quite rare in the energy and environment policy field compared to other areas of social science probably because of scientists’ lack of familiarity with this technique and specific issues linked to energy policy evaluations (such as missing baselines, long time lag between intervention and response, high outcome variability, lack of sufficiently detailed geographical data).

From a methodological point of view, this paper advances our understanding about how to assess energy and environmental policies, by providing evidence on what types of interventions work and under what conditions. I believe the methodology used in this analysis is broadly applicable to other programs/policies and should become part of the toolbox of empirical studies in the energy and environment field to lead to better policy evaluation.



Granting Urban Residency to Rural Migrant Workers in China:Who Wins and Who Loses?

Wei Yifang - yifang_wei@sina.com - Institute of Social Development, National Development and Reform Commission - China

Gu Yan - guyanruc@126.com - Institute of Social Development, National Development and Reform Commission - China

Download the paper

China has experienced a rapid development of urbanization. The urbanization rate raised from17.9% in 1978 to 56.1% in 2015. There are 277 million rural migrant workers who have made great contributions to China’s urbanization, while many of them have worked and lived in cites and towns for a long time but can not enjoy the same basic public services as urban residencies.


In 2014, China issued the “National Plan on New Urbanization (2014-2020)” which aims to improve its quality by taking people-oriented urbanization as an essential value. The plan has set a goal of granting urban residency to around 100 million people with rural household registration living in urban areas and other permanent urban residents. A new round of household registration system reform has been carried out shortly after in order to adjust household registration transfer policies. Many have pointed out that the household registration system was the main obstacle of the new urbanization process. Behind this, the equalization of basic public services and its fiscal guarantee are the key issues.


What are the costs and benefits of the policy of granting urban residency to rural migrant workers? Who bears those costs and who shares the benefits? Will the total benefits cover all the costs? We try to answer those questions by making a benefit-cost analysis of the new urbanization policy.


This paper includes four parts. The first part makes an introduction of the research background and a literature review. Secondly, a model of benefit-cost analysis on the policy of granting urban residency to rural migrant workers is set up and the benefit items and cost items are derived from the model. The third part makes a quantitative analyze on the policy. Quantitative benefits, costs and the net benefits are calculated separately. Fiscal responsibilities between central and local governments as well as immigrant receiving and outflow governments are  discussed in this part. Finally, conclusions and policy suggestions are given based on the empirical study in order to promote China’s new urbanization.


Key words: China’s new urbanization, benefit-cost analysis, intergovernmental fiscal responsibilities, incentive compatibility

Export PDF