T07P14 - Policy Evaluation in Performance Regimes: A Comparative Perspective

Topic : Policy Design, Policy Analysis, Expertise and Evaluation

Panel Chair : Liang Ma - ken0821@gmail.com

Objectives and Scientific Relevance of the panel

Call for papers

Session 1

Impact of Performance Regime on Local Government Policy Evaluation in Indonesia

Meita Ahadiyati Kartikaningsih - meitakartika@yahoo.com - National Institute of Public Administration - Indonesia

Aldhino Niki Mancer - aldhino.niki.m@gmail.com - National Institute of Public Administration Republic of Indonesia - Indonesia

Local government policy evaluation is needed to understand local government capacity in administrating local autonomy. This process require measurement and benchmarking of the results that have been achieved with the results should be achieved. It is a process that is about the production of information of success (and failure).

Based on analysis of documents and literatures review, this paper attempts to examine local government policy evaluations in Indonesia. Key research question of this paper includes how does performance regime affect the construction of instrument for local government policy evaluation? In addition, this paper attempts to analyze several mechanisms in local government policy evaluation, how these seem to be overlapping and competing in creating definition of ‘success’ of local government, to understand  the contributing factors to the construction of such competing mechanism. Competing evaluation approach of Local Government policy in Indonesia may create different versions of ‘success’ (and multiple interests in being successful). This situation is not so effective in achieving objective of policy evaluation, policy coordination and the ways to improve local government performance.


Key words: competing evaluation, local government policy evaluation, performance

National Performance Frameworks and the Good State and Governance Report

Gabor Bozso - BOZSOGABORPHD@GMAIL.COM - National University of Public Service - Hungary

One of the key aspects of governance capacities is to enhance the political community to create its public policy strategies and targets, based on their common values and vision. It also aims to coordinate, drive and operate evident-based planning and performance evaluation and assessment processes which draw the scheduled outline of government actions preformed and results, outcomes achieved understandably giving regular feedback on the progress towards the goals set through performance measurements. National Performance Frameworks (NPFs) enable government to drive, monitor and assess progress towards achieving their overarching national objectives. NPFs also provide an accountability framework through which parliaments and civil society can measure the effectiveness of government action. The Good State and Governance Report (GSGR) is a public policy tool to measure changes in governance capabilities at specified time intervals with scientific methodology and well-structured statistical indicators. The philosophy of this innovative governance tool is that the government performance is inseparable from the given country’s socio-economic background, its special attributes and problems, as well as from the targets set by the government, developing the need for a set of indicators that would be applicable in national context. The paper aims to compare national indicator methodologies with the GSGR by analysing extensive data and highlighting international examples of national indicator systems measuring and evaluating governance performance. One of the highlighted examples are Scotland Performs is a national strategic framework developed by the Scottish Government setting and measuring the progress of public policy goals and results in accordance with strategies and cardinal Acts of the government.

The research question of the paper is whether GSGR coincide in their characteristics with the international trends of national performance frameworks. Second and third question focuses on what are the main characteristics of national performance measurement systems and what are the lessons to learn for policy experts. According to the panel description Policy Evaluation in Performance Regimes: A Comparative Perspective panel focuses on distinct approaches of policy evaluation developed in public sectors in different countries and based on different cultural, political, and social contexts. These approaches are rather different, but may share similar underpinning values and rationales. It is thus meaningful to examine and compare policy evaluations in different countries and regions. Thus, this paper can add to the research question: What policy experts can learn from policy evaluation practices in other countries and regions?



JASMINE AHMAD - jasmi661@salam.uitm.edu.my - UNIVERSITI TEKNOLOGI MARA (UiTM) - Malaysia




Rafidah Mohamed Hashim

FSPPP Universiti Teknologi MARA, 40450 Shah Alam, Malaysia



Jasmine Ahmad

FSPPP Universiti Teknologi MARA, 40450 Shah Alam, Malaysia






Policy evaluation is fundamental to any public policy management due to the rising concern of public on ‘what works’ about public programs. Although evaluation on public policies and programs are mainly championed by government entities and organizations, recent trend witnesses the rising roles of third party entities in evaluation activities, which subsequently acts as a mechanism of check and balance as well as creating competitive conditions. The non-governmental entities such as community groups, media, non-profit organization, research institutions and international organizations begin to take roles in evaluation activities. Apart from enhancing accountability and transparency, the involvement of third party entities such as international organizations also helps in building the capacity for conducting evaluation among donor-recipient countries. The paper seeks to examine to what extent participatory approach and capacity building in policy evaluation are enhanced with the roles of third party entities in Malaysia. Do these third party entities help in creating an enabling environment that facilitates policy evaluation? It is therefore interesting to uncover these aspects with some comparison from selected developing countries in the ASEAN region. The scope of the paper is limited to the roles played by selected third party entities such as media, evaluation societies, research institutions, and international organizations with the focus on participatory approach and capacity building aspects. The paper is written mainly based on available academic sources of information, which also includes international organization reports, annual reports, and conference proceedings. The paper concludes that although the presence of these third party entities has somehow helped in facilitating evaluation activities in the Malaysian public sector policy framework, there are still lots of improvement works need to be done especially in terms of enhancing transparency and accountability through the management of evaluation results. The evaluation capacity building efforts especially at the second and third level of government is another aspect that requires attention. The paper will contribute to additional information on the roles of third party entities in evaluation. Future studies may look deeper into how third parties entities enable independent evaluation that helps building reliable national evaluation systems.



Keywords      : Third party entities, policy evaluation, participatory evaluation/ approach, and evaluation capacity building

Sub-theme    : T07 – Policy Design, Policy Analysis, Expertise and Evaluation

                      P14 – Policy Evaluation in Performance Regimes: A Comparative Perspective

The Implementation of Performance Audit as a Baseline of Policy Evaluation in Democratic Country (Case Study in Supreme Audit Institution in Indonesia)

Windhu Wibisono - windhu.wibisono@gmail.com - University of Indonesia - Indonesia

Various public policies in democratic countries need the oversight from external to obtain the public trust and policy legitimacy of the government policy itself. Public in the democratic countries are also watched the public funds that used. In Indonesia, the number of public money that represent in the budget is increasing from year to year. However, the results from budget are not optimal felt by the public. Then, Indonesia’s government needs to improve the quality of the evaluation, including policy evaluation. Public policies are not only evaluated by the management but also by the external, in this case is the Supreme Audit Institution.

Policy evaluation does not stand-alone but interacts with other aspects such as the institutional and governance systems. In this case, policy evaluation used to achieve a good performance in the perspective as a part of performance regime. Moynihan et. al. (2011) examines the role of performance regimes in contemporary governance and concludes that performance regimes are and will remain an essential aspect of contemporary governance arrangements even their effectiveness depends on their capacity and context. Talbot (2011) also states "by setting standards or making specific recommendation these bodies (audit bodies) can effectively impose performance measures or targets." The SAI as audit body become important to support the performance regime. Wollmann (2006) in Fischer et.al. (2006) explained that policy evaluation has two primary tasks which are related to the goal attainment and the causal question.

The main question of this paper is what is the role of SAI in the policy evaluation, especially in the context of a democratic state. This paper uses qualitative approach based on in-depth interviews with SAI’s officials and the secondary data from the SAI’s documents such as audit report and SAI strategic planning.

This paper discuss how the SAI secure their role from the Constitution 1945 and Act No.15/2004 and how the SAI performs its role. SAI’s conduct policy evaluation by performing the audit performance that using the interaction from many stakeholders in the audit stage. This paper specifically discusses how the SAI use conformity strategy that linier with government strategy in a medium term development plan. This paper concludes that SAI’s performance audit can be regarded as a good baseline in the policy evaluation. The results of the performance audit are not only problem findings related to goal attainment but also recommendations related causal question. This paper propose SAI to use a model of democratic governance related to their role in the policy evaluation in democratic countries and society as a further step to increase the level of performance regimes that adaptive and democratic.

Evaluation for accountability or improvement?: A cross-country comparison of performance regimes

Liang Ma - ken0821@gmail.com - Renmin University of China - China

Policy evaluation and performance measurement have been extensively adopted in public sector, and evaluations by external entities (e.g., academia, consultancy firm, media, and auditor) has been burgeoning in various fields. It is intriguing that so many countries adopted external performance evaluation/review albeit for different purposes and by distinct approaches. Why do central/federal governments implement external evaluation? What are the key approaches and attributes of external evaluations across countries? What are the primary similarities and differences in external evaluation across countries? In this study we develop a theoretical framework to analyze external evaluations in multiple countries. The framework focuses on two dimensions, namely purposes and focuses, of external evaluation. The purpose of external evaluation is about the primary aims of evaluation, e.g., political/administrative accountability or organizational performance improvement. The focus of external evaluation refers to the level of analysis, including the whole of the government, agency, and program/project. We report empirical findings of comparative case studies in four countries (the US, China, Singapore, and New Zealand). The findings help understand the underlining rationales and key success factors of external evaluation, and suggest policy implications for implementing and improving external evaluation.

Do political factors influence performance of Public Sector Enterprises? The Indian Disinvestment Experience

Ritika Jain - ritika@cds.ac.in - Centre For Development Studies - India

India had adopted disinvestment, as part of broader reforms, in 1991 to improve performance of public sector enterprises (PSEs) owned by the Central government of India. The current study captures the e ffect of disinvestment policy on the performance of Central Public Sector Enterprises (CPSEs) with special emphasis on the local political and economic environments in which these enterprises operate. Using fi rm efficiency to capture performance of all CPSEs between 1991-92 and 2010-11, the study employs instrument variable regression and di fference-in-di fference estimation models. The results suggest that performance of CPSEs is driven by a harmonious union of internal ( firm specifi c) and external factors. Experience, large fi rm size and low debts are among the internal factors that boost performance of CPSEs. Among external factors, disinvestment, as a policy intervention, has a positive impact on performance. Political and economic factors at the state level also have a signi ficant impact on firm performance. Importantly, the e ffect of disinvestment is stronger if the enterprise is located in a state that is right winged and is ideologically similar to the centre.

Session 2

Does the Local Government Follow the Strategic Intention of the Central Government in China? A Comparative Social Network Analysis of the Implementation of the Mass Innovation and Entrepreneurship Policies

Bo Yan - yanbosir@xjtu.edu.cn - Xi'an Jiaotong University - China

WEI LI - liwei@cuhk.edu.hk - Chinese University of Hong Kong - Hong Kong, (China)

China has initiated the national policy of Mass Innovation and Entrepreneurship and then a series of policies in a top-down approach in recent years. However, few is known about how those policies are implemented and whether the local government would conform to the strategic intention of the central government in the authoritarian system. This paper attempts to explore the differences between the central and local governments in policy implementation. We compare the policies of supporting Mass Innovation and Entrepreneurship by the state council and the Beijing municipal government. The findings show that the local government has adopted old strategies in implementing the policy of Mass Innovation and Entrepreneurship by the central government., From the path-dependency perspective, we demonstrate that the local government is under the performance pressure and prefer a familiar approach to implement the policy. In future research, we will explore structural and normative factors which explain such preferences.

Can the US Keep the PACE? A Natural Experiment in Accelerating the Growth of Solar Electricity

Nadia Ameli - nd.ameli@gmail.com - University College London, Institute for Sustainable Resources - United Kingdom

Paper project

Governments' efforts to expand solar generation base and integrate it into municipal, regional, and national energy systems, have spawned several programs that require rigorous policy evaluations to assess their effectiveness, costs and contribution to Paris Agreement's goals. In this study, we exploit a natural experiment in northern California to test the capacity of Property Assessed Clean Energy (PACE) to promote PV investment.

PACE is an innovative energy scheme used in certain areas of the US to support renewable energy deployment. The installation of clean energy technology through PACE is financed by local governments, by issuing bonds whose proceeds are used to finance loans to homeowners for PV installations. Residential property owners pay back the loan through an increment on their property tax bill over a 20-year period. If the property is sold before the end of the repayment period, the new owner takes over the remaining debt.


Research question

Has been the PACE program an effective policy to boost the solar PV market in California?



This study assesses PACE’s effectiveness on new solar installations using a regression discontinuity (RD) exploiting the geographical discontinuity of the program. Under the RD design, a geographic or administrative boundary allows the investigator to select units into treated and control areas. Indeed, the unique characteristic of this design is the method by which research units are assigned to program or comparison groups as the units’ placement depend solely on the basis of county border. This allows the investigator to control for unobserved confounding factors, which if uncontrolled will result in biased estimates. Making causal inference in policy evaluation exercises is challenging as it requires constructing a credible counterfactual, i.e. what the outcome of interest (PV installations) would have been in the absence of the policy intervention (PACE program). The RD approach permits to do just that.

Given that PACE was implemented only in Sonoma County, the county boundary determines whether households are eligible for the PACE financing program, thus allowing us to draw arbitrarily the treated (cities eligible for the program) and control groups (cities not eligible for the program).


How this paper fits the panel chosen

This study is an example of a rigorous policy evaluation based on an experimental framework. This approach is still quite rare in the energy and environment policy field compared to other areas of social science probably because of scientists’ lack of familiarity with this technique and specific issues linked to energy policy evaluations (such as missing baselines, long time lag between intervention and response, high outcome variability, lack of sufficiently detailed geographical data).

From a methodological point of view, this paper advances our understanding about how to assess energy and environmental policies, by providing evidence on what types of interventions work and under what conditions. I believe the methodology used in this analysis is broadly applicable to other programs/policies and should become part of the toolbox of empirical studies in the energy and environment field to lead to better policy evaluation.


On paradox of managing government financed enterprise projects in China

Lu Yangfan - yfanlu@scut.edu.cn - South China University of Technology - China

Yang Xuejuan - 861390482@qq.com - South China Normal University - China

Purpose - The Purpose of this paper is to explore a paradox in managing government financed enterprise projects in China, which is procedure compliance against implementation effectiveness, and how it differentiates among enterprise types. This paper, by adopting a theoretical perspective of performance evaluation, enriches the empirical research on government financed enterprise projects’ management.

Methodology/approach - This paper collects a sample of 687 government financed enterprise projects within a southern Chinese province during 2005~2012. Descriptive statistics is used to show the structural feature of the paradox, and multiple regression analysis to examine the possible impacting factors like background of enterprises. Indicators to measure procedure compliance and implementation effectiveness are invented by a third-party research institute independent from government, which also gives the evaluation data and results.

Findings - For projects do well in procedure compliance, they usually do worse in implementation effectiveness, and vice versa. This paradox is significantly weaker in enterprises of larger scale, gaining more government finances, affiliated to higher administrative level or located in better developed areas. One possible explanation is due to the real political system, what lies behind government financing scale, areas of location and administrative level is actually the enterprises’ bargaining power and social resources. That is, enterprises superior on these aspects are more likely to challenge or even disobey procedures to achieve effectiveness. It is compared to one traditional Chinese saying that a swordsman violates the rules owing to his best kungfu.

Originality/value – This paper benefits from a unique database created by the authors in performance evaluation of government financed enterprise projects in a third-party position. It provides a micro-perspective and in-depth observation into the relationship between government and enterprises under real Chinese circumstances, and may bring operational implications to such projects’ management.

An international comparative study of competitive advantages in convergent science fields

FANG XU - xufang@casipm.ac.cn - Institute of Science and Development - China

At present, the world is confronted with a series of global major challenges, such as climate change, shortage of resources, ecological destruction, and environmental pollution and so on. The traditional single-disciplinary, interdisciplinary, transdisciplinary research is not enough to solve such increasingly complex economic and social problems. In this context, convergent research, which directly focus on major economic and social issues is gradually risen and attract much attention. Such studies are firstly seen in life science areas within the international science community. In China, the Chinese Academy of Sciences has proposed 8+2 layout for future research priorities in its 13th five-year plan. The researchers and science and technology managers are facing a major problem that how to organize and manage convergent research activities. In this study, we attempt to carry out an international comparative study of the competitive advantages of CAS’s 8 + 2 research porosities through case studies in Energy, Life and Health and Agriculture three congregant science fields. Besides traditional methods, we are interested to investigate the impact of convergent research results and so as to adopt methods which can evaluate the convergent research result and impact from the perspective of the whole innovation chain.

This work are expected to help CAS to identify its advantages in these research properties, and provide evidences for future resources decision-making of science and technology layouts in CAS.

Granting Urban Residency to Rural Migrant Workers in China:Who Wins and Who Loses?

Wei Yifang - yifang_wei@sina.com - Institute of Social Development, National Development and Reform Commission - China

Gu Yan - guyanruc@126.com - Institute of Social Development, National Development and Reform Commission - China

China has experienced a rapid development of urbanization. The urbanization rate raised from17.9% in 1978 to 56.1% in 2015. There are 277 million rural migrant workers who have made great contributions to China’s urbanization, while many of them have worked and lived in cites and towns for a long time but can not enjoy the same basic public services as urban residencies.


In 2014, China issued the “National Plan on New Urbanization (2014-2020)” which aims to improve its quality by taking people-oriented urbanization as an essential value. The plan has set a goal of granting urban residency to around 100 million people with rural household registration living in urban areas and other permanent urban residents. A new round of household registration system reform has been carried out shortly after in order to adjust household registration transfer policies. Many have pointed out that the household registration system was the main obstacle of the new urbanization process. Behind this, the equalization of basic public services and its fiscal guarantee are the key issues.


What are the costs and benefits of the policy of granting urban residency to rural migrant workers? Who bears those costs and who shares the benefits? Will the total benefits cover all the costs? We try to answer those questions by making a benefit-cost analysis of the new urbanization policy.


This paper includes four parts. The first part makes an introduction of the research background and a literature review. Secondly, a model of benefit-cost analysis on the policy of granting urban residency to rural migrant workers is set up and the benefit items and cost items are derived from the model. The third part makes a quantitative analyze on the policy. Quantitative benefits, costs and the net benefits are calculated separately. Fiscal responsibilities between central and local governments as well as immigrant receiving and outflow governments are  discussed in this part. Finally, conclusions and policy suggestions are given based on the empirical study in order to promote China’s new urbanization.


Key words: China’s new urbanization, benefit-cost analysis, intergovernmental fiscal responsibilities, incentive compatibility

Dilemma of “Emission Reduction Just in Statistics” in Chinese Environmental Regulation ——Yang DONG

Yang Dong - dongyang11@mails.ucas.ac.cn - National Academy of Innovation Strategy - China

Dilemma of “Emission Reduction Just in Statistics” in environmental regulation, means that environmental regulation and performance, promulgated by government, is becoming better and better, however, public perception about environmental quality tends to deteriorate. To solve this problem, take G20 Countries as analysis sample, and respectively set environmental technology patent diffusion and environmental tax percentage of total taxes, PM2.5 quality concentration and public’s mortality of lung cancer as independent variables and dependent variables. We can use comparative study method, nested analysis method and institutional analysis method, to study general problems of G20 and special problems of China. Based on regression model, we can find that, both in all of G20 and in China, air pollution have significant negative impact on male and female’s lung cancer mortality. In another word, there is not public feelings bias about environmental quality in China. But, it is quite mismatch between environmental regulation and environmental quality, and it means that there is a phenomenon called environmental regulation failure. The reason why this phenomenon appears is that, 1. The effect of Chinese environmental regulation technology will lag behind, caused by weak innovation capacity; 2. Because of limited total scale of environmental tax, it can’t affect effectively; 3. Contribution of pollution sources from urbanization, industry structure and energy consumption structure, is so large that exceeds environmental regulation capacity.

Export PDF