T10P03 - How to Create Quantitatively Comparable Policy Measures

Topic : Methodologies

Panel Chair : Jody Heymann - jody.heymann@ph.ucla.edu

Panel Second Chair : DAVID Godfrey - dgodfrey@ph.ucla.edu

Panel Third Chair : Arijit Nandi - arijit.nandi@mcgill.ca

Objectives and Scientific Relevance of the panel

Call for papers

Session 1 How to Create Quantitatively Comparable Policy Measures


Jody Heymann - jody.heymann@ph.ucla.edu - Fielding School of Public Health; WORLD Policy Analysis Center - University of California, Los Angeles - United States

DAVID Godfrey - dgodfrey@ph.ucla.edu - WORLD Policy Analysis Center, UCLA - United States

Measuring and comparing state capacities to achieve the Sustainable Development Goals (SDGs) in an emerging economy: challenges and research proposal.

FLAVIO FERNANDES - f.cireno@gmail.com - Escola Nacional de Administração pública - ENAP - Brazil

Pedro Palotti - pedropalotti@gmail.com - National School of Public Administration - ENAP - Brazil

CIRO FERNANDES - ciro.fernandes@uol.com.br - National School of Public Administration - Brazil


The field of Research on state capabilities has been a recurring theme since the late 1980’s when Peter Evans, Dietrich Rueschemeyer and Theda Skocpol's publhished the book “Bringing the State back in”. Since then, most of the studies have focused on the definition of the concept of state capabilities, the different strategies to measure it and its relation with variables of results, such as economic growth and anti-corruption measures.

The research design for the measurement of state capabilities must let space for comparing these capabilities amongcountries. Because of it, the works of Evans, Rausch and most recently Ramesh will be the guidelines of the methodology of the study.

Besides that, the progress of the Millennium Development Goals (MDGs) agenda of the United Nations is dedicated to promoting the Sustainable Development Goals (SDGs). A new set of commitments aims to conduct public policies for greater social justice and the extension of human rights. The challenge is articulate to people, markets and governments to work together to promote human development.

The present paper discusses the concept of state capacity as the administrative (professional bureaucracy, stable and valued careers) and relational capacities (interlocution in and out of state organizations) of public servants. In this context, autonomy of the bureaucracy is a previous condition for the development of capacities, and is subsequently driven by them.

The Brazilian federal bureaucracy operates in all 27 states, divided into more than 200 agencies and comprises more than 600 thousand civil servants. There are careers dedicated to the formulation and implementation of public policies, as well as those dedicated to administrative services and the provision of citizen services.

The research design will be applied to develop an extensive agenda on state capacities by the National School of Public Administration. From the federal case, it is expected to compare the results in the international level and adapt this methodology to the sub-national public administrations.

In this way, the measurement of state capacities starts from two previous definitions: a) it focuses on careers directly related to policy-making; and b) it is restricted to government agencies and programs that directly contribute to the achievement of SDGs. Owing to the absence of information about state capabilities and autonomy, it will be necessary to collect primary data. Due to the financial and logistical difficulties for an on-site search with the federal servants, a survey will be applied throughout the internet.

Estimating the effect of compulsory and tuition-free education policies on financial inclusion in Asian countries

Ni Luh Putu Satyaning Pradnya Paramita - ni.paramita@un.or.id - Pulse Lab Jakarta - Indonesia

Dikara Alkarisya - dikara.alkarisya@un.or.id - Pulse Lab Jakarta - Indonesia

Education has a key role to play in moving towards environmentally sustainable and inclusive economic growth. Inclusive economic growth is not only about expanding national economies but also about ensuring that we reach the most vulnerable people of societies. The “equality of opportunity” and “participation in growth by all” with special focus on the working poor and the unemployed are the very basis of inclusive growth. In the meantime, a recent IMF and NBER Working Paper (Era Dabla-Norris, Yan Ji, Robert Townsend, and D. Filiz Unsal, 2015) shows that elimination of blockages to financial inclusion has significant and unambiguous direct impacts on GDP growth and productivity through smarter allocation of resources and more efficient financial contracting; resulting in stronger entrepreneurial activities and new business start-ups that increase aggregate output.


This paper examines how (i) compulsory and tuition-free education policies affect financial inclusion in Asian countries and (ii) the effects differ across different economic setting based on UN definition of low-income, middle-income and high-income country. We used country-level policies data collected by WORLD Policy Analysis Center, contains the information whether a country has constitutions on making certain levels of education compulsory and/or tuition-free. As a measurement of financial inclusion, we referred to the Global Findex database, the world’s most comprehensive database on financial inclusion, provides in-depth data on how individuals save, borrow, make payments, and manage risks.

However, the causal relationship of compulsory and tuition-free education and financial inclusion is not that simple. There are some unobservable ‘latent’ variables that affects the relationship, includes enrollment rate, literacy rate, household income, financial literacy. We proposed to use Structural Equation Modelling (SEM) to address this complexity and reveal how those variables interact each other. SEM provides a very general and convenient framework for statistical analysis that includes several traditional multivariate procedures, for example factor analysis, regression analysis, discriminant analysis, and canonical correlation, as special cases.


This paper shall contribute on comparison analysis of how education policies and under which condition can affect financial inclusion in low-income, middle-income, and higher-income Asian countries.

Measuring political symbolism for large-n comparative policy studies

Andrew Tanabe - andrew.tanabe@mcgill.ca - McGill University - Canada

Biesbroek Robbert - robbert.biesbroek@wur.nl - Wageningen University & Research - Netherlands

Symbolism plays a pivotal role in the politics and decision making around complex societal issues such as sustainable development. Politicians and policy makers use and create, in both instrumental and expressive sense, symbolic policies to achieve their (own) goals. Many examples of political symbolism are part of the political repertoire (“less government”, “tough on crime”, “the 99%”) and both positive or negative connotations are associated with symbolism. In the context of sustainable development symbolic actions include relabeling existing policies and instruments to fit the prevalent public discourse on sustainability – the mechanism of greenwashing. Symbolism of political rhetoric, symbolic politics, as well as recent symbolism of policy instruments have been widely discussed in the political science and public policy literatures, see for example the works of Edelman, Gusfield, Bludhorn and others. Symbolic policy is the inevitable part of the ‘dynamic interplay’ between symbolic and substantive policy making.


Demarcating symbolic from substantive elements of policies is important as symbolism can create significant measurement error when comparing policy outputs across contexts and over time. Many large-n comparative studies are insensitive to the dynamic interplay of symbolism and substance in policies and fail to adequately address this issue. As yet, no measurement frameworks exist that demarcate symbolic from substantive policy instruments for the purpose of tracking or evaluating policy output (over time and across contexts). 


The aim of this paper is to provide such a measurement framework. We review the main strands of the literature on political and policy symbolism. We propose a framework for measuring the symbolism of a specific policy instrument as a tool to measure how policy has changed and provide insight into the likely impact of a policy portfolio. To measure the symbolism of a policy portfolio, the policy objectives, instrument setting and calibration, are identified. Next, these are compared with a taxonomy of the expected use of governing resources to assess a level of dissonance between the two. When the level of dissonance between expected and empirically identified tools and calibrations reaches a critical level, the policy can be identified as having a high level of symbolism.


We illustrate the applicability of our framework to the example of climate change adaptation policy instruments related to one of the key SDG goals (SDG13) which calls for integrating climate change measures into national policies, strategies and planning. In addition to this SDG goal, the need for systematic approaches for tracking climate change adaptation progress following the Paris Agreement on Climate Change has been highlighted by Ford et al. 2015. The framework can provide more precision to comparative policy studies involved both in measuring changes in policy portfolios as well as comparing policies across space or time. The proposed framework for estimating levels of symbolism advances the current metrics for measuring policy output by allowing for a more nuanced estimation of the potential impact of policy instruments on the policy outcome.

Marginalisation, deprivation and development among rural disabled member households in different regions of India: A study based on Socio-economic and Caste Census

Srei Chanda - srei1988@gmail.com - International Institute for Population Sciences - India

One of the major goal of the sustainable development is the development and increased wellbeing of the marginalised population, that includes disabled. Development is largely measured by materials wellbeing, poverty and multidimensional deprivation in the literature and present context. Disability is a multidimensional area which not only includes health, but also the development regarding livelihood, living arrangement, earning etc. These are intensified by the exclusion, deprivation from fulfilling the demands of basic needs, material ownership and services accessed in social and economic arena. It needs deeper understanding to measure the fathom. The study primarily finds out the intensity of unequal distribution of material ownership for exclusion, economic situation and deprivation among the households and its relationship with the selected background characters. Socio-Economic and Caste Census in India (2011) is used. Composite index has been constructed for exclusion from poverty and deprivation, which later ranked by districts; and relationship with the background characters in different regions of India has been found. Households deprived are almost 80% and excluded from poverty measures is 24% - 68% among them. Less income, cultivation is negatively correlated, and owning of material assets, irrigated and non irrigated land etc. has positive correlation for more proportion of the household (with disability) not deprived or excluded from facing poverty. Eastern, central and north-eastern region shows more inequality compared to developed region like south or west. Complex structure of unequal access, owning and participation is strong determining factor for neglect, marginalization and regional inequality in the development. The policy must demands a need based assessment at different regions of India, not only similar supply to every demand to maintain a sustainable progress.

Creating Quantitatively Comparable Policy Measures to Strengthen Equal Rights and Opportunities Worldwide: New Global Public Use Data Being Created on Equal Rights at Work

DAVID Godfrey - dgodfrey@ph.ucla.edu - WORLD Policy Analysis Center, UCLA - United States

Jody Heymann - jody.heymann@ph.ucla.edu - Fielding School of Public Health; WORLD Policy Analysis Center - University of California, Los Angeles - United States

Comparative data can help leaders understand the range of options other countries have taken to address some of the world’s most pressing challenges, and help civil society to urge their lawmakers to adopt similar effective approaches. Furthermore, periodically updating these datasets allows us to track individual countries’ progress towards fulfilling targets like the Sustainable Development Goals (SDGs). 

This paper examines our approach to building a database of quantitatively comparable policy measures relevant to SDGs 5 and 10, which aim to achieve gender equality and empower women and girls and reduce inequalities based on sex, age, disability, race, ethnicity, origin, religion or economic or other status, respectively. We chose to focus on workplace discrimination and sexual harassment laws because workplace equality is fundamental to ensuring everyone has the opportunity to earn a living income in decent working conditions, laying the foundation for health, well-being, and economic growth.


This paper details every step of building a comprehensive quantitative policy database, from conceptualization, to creating variables, to gathering and analyzing legislation in a systematic way that ensures consistency across countries.  We first created a conceptual framework of quantitative indicators based on international agreements and research evidence on this topic and with input from civil society.  Our multilingual team then collected and analyzed thousands of primary sources including laws and policies that make discrimination illegal and prohibit sexual harassment in the workplace from 193 UN countries.  All questions were answered independently by two different researchers who then compared their answers to minimize human errors.  The resulting dataset can then be transformed into accessible, user-friendly resources, including interactive maps, tables, and downloadable datasets. Periodic updates to this dataset over the coming decade will provide a detailed picture of the steps countries have taken to realize these development goals, and will serve to quickly identify leaders and laggards in reducing inequalities.


By sharing some of our lessons learned in gathering and evaluating quantitatively comparable policy data, we hope to encourage and support the work of other organizations in this field.

Export PDF