T10P03. How to Create Quantitatively Comparable Policy Measures

Topic : Methodologies

Chair : Jody Heymann (Fielding School of Public Health; WORLD Policy Analysis Center - University of California, Los Angeles)

Second Chair : DAVID Godfrey (WORLD Policy Analysis Center, UCLA)

Third Chair : Arijit Nandi (McGill University)

Share to Facebook Share to Twitter Share to Linkedin Share by mail

General Objectives, Research Questions and Scientific Relevance

The proposed panel seeks to achieve three key objectives: 

  • Identify innovative approaches to measuring quantitatively comparative law and policy data;
  • Explore the merits of different methodologies to capture law and policy data in a comparative and accessible manner; and
  • Highlight the challenges and promising practices in developing comparative quantitative databases on key law and policy issues.

To date, there have been few studies that have rigorously addressed the impact of national legislation on individual outcomes.  The select studies that have ventured to explore policy have often relied on qualitative sources of data, which are rich but do not allow for rigorous quantitative research. One of the greatest impediments to this type of research that would enable a data-driven approach to improving outcomes has been the lack of comparable policy data. The inaccessibility of comparative legal and policy data has hampered our ability to systemically measure a) gaps in laws and policies that we know work to improve outcomes, b) progress over time in strengthening legal rights and protections, c) what policies have been feasible and effective in different economic settings, and d) which policies are most effective at improving individual and population outcomes.

When countries’ laws and policies are captured in a quantitative and comparable format that can be used for analyses and is easily accessible to the public, it increases the transparency of countries’ actions, or lack thereof, on issues that have been shown to impact outcomes. The research community also stands to benefit greatly from the proliferation of quantifiable comparable measures of comparative policy. Equipped with newly created rich datasets, researchers can undertake rigorous analyses linking policies to outcomes in order to determine which policies matter and which work best. This panel thus encourages researchers to present innovative efforts to create large quantitative databases of laws and policies at the sub-national, regional, and global levels.

Given its experience with pioneering an approach to creating quantitatively comparable policy data, the WORLD Policy Analysis Center (WORLD), at the University of California Los Angeles, is well placed to chair this panel.  WORLD strives to improve the quantity and quality of globally comparative data on policies affecting human health, development, well-being, and equity.  In this pursuit, WORLD has developed quantitatively comparable indicators measuring over 1,500 laws and policies on adult labor and working conditions, poverty, gender, health, education, equal rights and non-discrimination, children, and family in 193 countries. 

The approach taken by WORLD is only one possible approach. WORLD is committed to working with groups who are creating public use data and bringing together those who are advancing innovative methodologies to capture quantitatively comparable policy data. In this spirit, this panel aims to highlight promising practices in developing comparative quantitative databases on key law and policy issues.

Call for papers

The inaccessibility of globally comparative legal and policy data has hampered our ability to measure countries’ progress toward fulfilling their global commitments, evaluate progress over time, and determine what works. Quantitatively comparable data that enables us to monitor national actions around key international commitments such as the Paris climate change agreement can help strengthen accountability. Citizens are empowered to hold their leaders accountable for their actions or lack thereof in addressing the urgent problem of climate change.  Policymakers are able to learn from the experiences of other countries to understand what policies have been feasible and effective in different economic or social settings and effective.  Alongside the existing rich qualitative monitoring of human rights committees, quantitative measures of policy allow civil society and international organizations to quickly see global progress, which countries are leading, and which countries are lagging behind. Similarly, quantitative data can be used to accelerate progress on the Sustainable Development Goals by identifying which policies are most effective at improving outcomes and where countries need to strengthen implementation of existing policies.

This panel invites researchers from a range of disciplines to explore new approaches to measuring policy data. Questions of particular interest are: What are the different ways in which we can translate the enormous amounts of qualitative law and policy data into quantitatively comparable databases? What are the constraints to developing such data?  How do the challenges differ between creating sub-national, regional, and global comparative data? Submitted papers should present either innovative methodologies for measuring law and policy data or critical reflections on the process of translating qualitative policy data into comparable and actionable quantitative data. 

ROOM
Block B 3 - 5
Wed 28th
14:00
Session 1

Discussants : Jody Heymann (Fielding School of Public Health; WORLD Policy Analysis Center - University of California, Los Angeles)

Discussants : DAVID Godfrey (WORLD Policy Analysis Center, UCLA)

Responsive image
PDF
Estimating the effect of compulsory and tuition-free education policies on financial inclusion in Asian countries

Ni Luh Putu Satyaning Pradnya Paramita

Dikara Alkarisya (Universitas Gadjah Mada)

Education has a key role to play in moving towards environmentally sustainable and inclusive economic growth. Inclusive economic growth is not only about expanding national economies but also about ensuring that we reach the most vulnerable people of societies. The “equality of opportunity” and “participation in growth by all” with special focus on the working poor and the unemployed are the very basis of inclusive growth. In the meantime, a recent IMF and NBER Working Paper (Era Dabla-Norris, Yan Ji, Robert Townsend, and D. Filiz Unsal, 2015) shows that elimination of blockages to financial inclusion has significant and unambiguous direct impacts on GDP growth and productivity through smarter allocation of resources and more efficient financial contracting; resulting in stronger entrepreneurial activities and new business start-ups that increase aggregate output.

 

This paper examines how (i) compulsory and tuition-free education policies affect financial inclusion in Asian countries and (ii) the effects differ across different economic setting based on UN definition of low-income, middle-income and high-income country. We used country-level policies data collected by WORLD Policy Analysis Center, contains the information whether a country has constitutions on making certain levels of education compulsory and/or tuition-free. As a measurement of financial inclusion, we referred to the Global Findex database, the world’s most comprehensive database on financial inclusion, provides in-depth data on how individuals save, borrow, make payments, and manage risks.


However, the causal relationship of compulsory and tuition-free education and financial inclusion is not that simple. There are some unobservable ‘latent’ variables that affects the relationship, includes enrollment rate, literacy rate, household income, financial literacy. We proposed to use Structural Equation Modelling (SEM) to address this complexity and reveal how those variables interact each other. SEM provides a very general and convenient framework for statistical analysis that includes several traditional multivariate procedures, for example factor analysis, regression analysis, discriminant analysis, and canonical correlation, as special cases.

 

This paper shall contribute on comparison analysis of how education policies and under which condition can affect financial inclusion in low-income, middle-income, and higher-income Asian countries.

Measuring political symbolism for large-n comparative policy studies

Andrew Tanabe (McGill University)

Robbert Biesbroek (Wageningen University)

Symbolism plays a pivotal role in the politics and decision making around complex societal issues such as sustainable development. Politicians and policy makers use and create, in both instrumental and expressive sense, symbolic policies to achieve their (own) goals. Many examples of political symbolism are part of the political repertoire (“less government”, “tough on crime”, “the 99%”) and both positive or negative connotations are associated with symbolism. In the context of sustainable development symbolic actions include relabeling existing policies and instruments to fit the prevalent public discourse on sustainability – the mechanism of greenwashing. Symbolism of political rhetoric, symbolic politics, as well as recent symbolism of policy instruments have been widely discussed in the political science and public policy literatures, see for example the works of Edelman, Gusfield, Bludhorn and others. Symbolic policy is the inevitable part of the ‘dynamic interplay’ between symbolic and substantive policy making.

 

Demarcating symbolic from substantive elements of policies is important as symbolism can create significant measurement error when comparing policy outputs across contexts and over time. Many large-n comparative studies are insensitive to the dynamic interplay of symbolism and substance in policies and fail to adequately address this issue. As yet, no measurement frameworks exist that demarcate symbolic from substantive policy instruments for the purpose of tracking or evaluating policy output (over time and across contexts). 

 

The aim of this paper is to provide such a measurement framework. We review the main strands of the literature on political and policy symbolism. We propose a framework for measuring the symbolism of a specific policy instrument as a tool to measure how policy has changed and provide insight into the likely impact of a policy portfolio. To measure the symbolism of a policy portfolio, the policy objectives, instrument setting and calibration, are identified. Next, these are compared with a taxonomy of the expected use of governing resources to assess a level of dissonance between the two. When the level of dissonance between expected and empirically identified tools and calibrations reaches a critical level, the policy can be identified as having a high level of symbolism.

 

We illustrate the applicability of our framework to the example of climate change adaptation policy instruments related to one of the key SDG goals (SDG13) which calls for integrating climate change measures into national policies, strategies and planning. In addition to this SDG goal, the need for systematic approaches for tracking climate change adaptation progress following the Paris Agreement on Climate Change has been highlighted by Ford et al. 2015. The framework can provide more precision to comparative policy studies involved both in measuring changes in policy portfolios as well as comparing policies across space or time. The proposed framework for estimating levels of symbolism advances the current metrics for measuring policy output by allowing for a more nuanced estimation of the potential impact of policy instruments on the policy outcome.

Creating Quantitatively Comparable Policy Measures to Strengthen Equal Rights and Opportunities Worldwide: New Global Public Use Data Being Created on Equal Rights at Work

DAVID Godfrey (WORLD Policy Analysis Center, UCLA)

Jody Heymann (Fielding School of Public Health; WORLD Policy Analysis Center - University of California, Los Angeles)

Comparative data can help leaders understand the range of options other countries have taken to address some of the world’s most pressing challenges, and help civil society to urge their lawmakers to adopt similar effective approaches. Furthermore, periodically updating these datasets allows us to track individual countries’ progress towards fulfilling targets like the Sustainable Development Goals (SDGs). 

This paper examines our approach to building a database of quantitatively comparable policy measures relevant to SDGs 5 and 10, which aim to achieve gender equality and empower women and girls and reduce inequalities based on sex, age, disability, race, ethnicity, origin, religion or economic or other status, respectively. We chose to focus on workplace discrimination and sexual harassment laws because workplace equality is fundamental to ensuring everyone has the opportunity to earn a living income in decent working conditions, laying the foundation for health, well-being, and economic growth.

 

This paper details every step of building a comprehensive quantitative policy database, from conceptualization, to creating variables, to gathering and analyzing legislation in a systematic way that ensures consistency across countries.  We first created a conceptual framework of quantitative indicators based on international agreements and research evidence on this topic and with input from civil society.  Our multilingual team then collected and analyzed thousands of primary sources including laws and policies that make discrimination illegal and prohibit sexual harassment in the workplace from 193 UN countries.  All questions were answered independently by two different researchers who then compared their answers to minimize human errors.  The resulting dataset can then be transformed into accessible, user-friendly resources, including interactive maps, tables, and downloadable datasets. Periodic updates to this dataset over the coming decade will provide a detailed picture of the steps countries have taken to realize these development goals, and will serve to quickly identify leaders and laggards in reducing inequalities.

 

By sharing some of our lessons learned in gathering and evaluating quantitatively comparable policy data, we hope to encourage and support the work of other organizations in this field.