T02P04. Methodology for Comparative Policy Analysis

Topic : Comparative Public Policy sponsored by Journal of Comparative Policy Analysis

Chair : B. Guy Peters (University of Pittsburgh)

Second Chair : Guillaume Fontaine (FLACSO Ecuador)

Share to Facebook Share to Twitter Share to Linkedin Share by mail

General Objectives, Research Questions and Scientific Relevance

This panel will integrate methodological issues in comparative politics and comparative sociology with those used in policy analysis. We argue that if the promise of comparative policy analysis is to be fulfilled, then greater attention must be given to ways of linking comparative methods with policy, and likewise how to link methods associated with policy analysis to comparative cases. Too often these connections of research traditions are not made clearly and the resulting research may not contribute as much as it might, either to comparative studies or policy studies. As well as the substantive differences among these research traditions, there are marked differences between large-N and small-N traditions of research, especially in policy analysis. In this panel we would want to concentrate on the utility of the methods themselves more that on the application of the methods. That said, the methods should always be considered in light of how they can address policy problems.

As well as the intellectual developments we expect from this panel, we intend to use the papers as a major component of a Handbook for Methodology for Comparative Policy Analysis.

Call for papers

We envisage five sessions for this panel:

The first session will focus on the linkage of issues in comparative analysis, such as case selection and the relative utility of Most-Similar and Most-Different Systems Designs with policy analysis. Likewise, we will discuss the utility of policy analytic methods such as cost-benefit analysis and evaluation research in different cultural and political settings. This session will also include some general survey of existing research in comparative public policy to identify not only the accomplishments within this field but also the research agenda for enhancing the contributions of this field.

The second session will consider the role of qualitative methods such as case studies, discourse analysis, and combinatory analysis in comparative policy analysis. The complexity of the policy process and the numerous factors affecting policy outcomes may make qualitative analysis more viable and more effective than quantitative approaches. This session will explore those methods relevance for policy, as well as their constraints. And we will attempt to link these methods with some of dominant theoretical approaches in policy studies.

The third session will deal with the comparative method in political analysis. Special attention will be given to problems of causation and measurement. We expect to discuss the contribution of mix methods to comparison, in particular regarding different approaches of process tracing.

The fourth session will examine the use of standard quantitative techniques such as regression and events-history for policy work, as well as examine some methods such as risk-benefit that tend to be directly associated with public policy. We would also consider papers discussing the role of multiple methods in policy analysis for this panel. There are a number of significant strands of policy research using quantitative methodology, such as the political economy of the welfare state and the comparative agendas project, and we will want to assess how the choice of methods have influenced the research outcomes.

The fifth session will be dedicated to multi-methods and experimental designs. Experimental designs are less commonly used in policy analysis but are becoming a more significant part of the armamentarium of economics and political science. We will examine their role –both laboratory and field experiments– as means of strengthening policy analysis and providing new insights into relationships among the many factors involved in public policy. Some aspects of impact analysis for programs can be included in this session of the panel.

ROOM
Manasseh Meyer MM 2 - 2
Fri 30th
08:15
Session 1 Methodologies for CPA - Session 1/3

Discussants : Adrian Kay (The Australian National University)

Discussants : Guillaume Fontaine (FLACSO Ecuador)

Responsive image
PDF
The Comparative Method and Comparartive Public Policy

B. Guy Peters (University of Pittsburgh)

In his seminal paper on methodology for comparative politics Arend Lijphart identifies a particular “comparative method”.  This method depends heavily on careful case selection to provide control over extraneous variance that may be done through other means in the other three fundamental methodologies mentioned by Lijphart.  Although scholars often pay obeisance to this method, it is honored more in the breach than the observance.  This paper will explore the utility of the comparative method, strictly defined, in the study of public policy, pointing to the pitfalls and promises of this methodological approach.

 

Why we need more transparent and rigorous inductive research design in comparative public policy scholarship, and what to do about it

Caner Bakir (Koc University)

Understanding how and why policy outcomes occur is at the heart of public policy research. Thus current research on public policy research relies mostly on qualitative methods. However, the research design, especially data collection, analysis and presentation, which could have enhanced qualitative rigor in inductive interpretive research and theorizing, is weak in public policy research. Specifically, in the absence of transparent and rigorous inductive research design, we cannot see how the research progressed from raw data to analyses which precede the theorizing process. There are two main methodological weaknesses in public policy research. First, researchers limit themselves to explain and/or explore why and how a certain outcome occurs, ignoring inductive methods that aim to generate theory from data. Second, although multiple causes of policy outcomes are understood in terms of interactions among structures, institutions and agents, weaknesses in data collection, presentation, and analysis in the current research allow for only limited progress towards qualitative rigour achieving a balance between data and its interpretation in explaining agency behaviour that generate policy outcome. This paper argues that blending case study methods with grounded theory logic, offers a useful avenue towards explaining and exploring policy outcomes and theory building with clear constructs and their relationships in dynamic causal processes.

 

Applying Qualitative Methods to Comparative Public Policy Analysis: Insights from Multi-Site Ethnographies of Informal Garbage Governance

Raul Pacheco-Vega (Facultad Latinoamericana de Ciencias Sociales (FLACSO) Mexico)

While quantitative methods have remained a fixture of traditional public policy analysis, comparative approaches to policy analysis have privileged qualitative methods. In this paper, I present a novel application of the anthropological approach to multi-site ethnography and apply it to case studies in four different cities across three countries. For the past five years I have been studying the relationships between informal waste pickers and local governments in Mexico, Canada and Spain. In this paper, I present empirical results from a four-city comparison of multisite ethnographies of informal garbage governance in Aguascalientes and Leon (Mexico), Vancouver (Canada) and Madrid (Spain). I show the potential of multisite ethnographic approaches, as well as potential drawbacks, particularly when dealing with sensitive subjects and very vulnerable populations. Insights gained from these case studies can be readily applied to other attempts to engage in comparative public policy analysis using qualitative methods.

ROOM
Manasseh Meyer MM 2 - 2
Fri 30th
10:30
Session 2 Methodology for CPA 2/3

Discussants : B. Guy Peters (University of Pittsburgh)

Discussants : Guillaume Fontaine (FLACSO Ecuador)

Does policy matter? A quali-quantitative approach to historical transport policy processes in Paris and London.

Charlotte Halpern (Sciences Po Paris - Centre d'Etudes Européennes (CEE))

This paper contributes to debates about comparative methods in political analysis by discussing the added value of a quali-quantitative approach to understanding policy processes and their outcomes. Drawing on an original qualitative dataset about long-term transport policy developments in European cities (CREATE project, funded under Horizon 2020), this paper critically examines the extent to which policy processes have contributed to the reduction of car use in Paris and London (OECD, 2012; CEC, 2016). The main outcome of interest, car use, having peaked in both cities and each case sharing some common features, the research design can be considered a “most similar systems” one (Mill 1843). However, it has been extremely difficult to conclude with absolute certainty what specific variable(s) explain this similar outcome. A vast share of the social sciences literature refers to macro-economic, social and cultural trends observable throughout industrialized societies (Kaufmann et al., 2004; Banister and Hickman, 2013). Yet there is a growing tendency to account for car reduction and changing mobility patterns in cities by focusing on policy offer and initiatives seeking to promote alternatives to car transportation across levels of government. Notwithstanding the vast number of Large N comparative policy evaluation studies, most policy studies seeking to examine whether or not policy matters refer to context specific initiatives or projects, and less so to policy processes (Banister, 2005; Goodwin 2012).

 

By contrast, this paper argues that the relationship between the reduction of car use in both cities and long-term policy developments in transport can be explored by developing a comparative analysis of historical policy processes (Haydu 1998; Howlett 2009).

This is particularly true of large metropolitan areas such as London and Paris, which are reputed ungovernable (Travers, 2002; Lefèvre, 2003) and characterized by specific forms of governance (Le Galès and Vitale 2013; Peters and Pierre, 2016). Insofar as it seeks to achieve a qualitative analysis of policy processes over time, a process tracing method was applied in order to combine within-case comparisons with small-n analysis as a way to identify intervening causal mechanisms between policy outcomes and independent variable (George and Bennett, 2005, 180; Blatter and Blume, 2008). The paper first discusses problems of causation and the extent to which the process tracing method helps addressing them. Second it introduces the quali-quantitative research design that was used in order to analyze historical transport policy processes in both Paris and London since the 1060s, and more specifically the qualitative database that was developed as part of the CREATE project in order to collect a longitudinal and a comparative dataset about dimensions of policy processes (objectives, resources, instruments etc.) and transport governance (issue salience, degrees of local autonomy, etc.). Third it examines their relationships with changes in transport behavior – and the reduction of car use – over time by identifying co-variations and beyond, some causal mechanisms. Finally, the paper finds general support for the idea that “policy matters” but details of the processes vary considerably from one city to another with important spatial and temporal variations.

Qualitative Comparative Analysis (QCA) as an approach to Comparative Policy Analysis

Eva Thomann (University of Exeter)

This paper outlines how different approaches to Qualitative Comparative Analysis (QCA) pose particular promises and challenges for public policy analysis. Both in terms of research design and the actual goals and needs of policy-oriented research, there seems to be a preferential connection between QCA and public policy analysis. While this has led to an increased popularity of QCA in this field, recent years have also witnessed a diversification of research approaches with QCA. Being situated in-between “quantitative” and “qualitative” methodologies, both small-N and large-N, exploratory and theory-led variants of QCA have developed, each with their particular strengths, limitations and practical issues. This development not only broadens our understanding of what QCA is and what it can be employed for; it also easily creates misunderstandings about the nature, purpose and usefulness of this method. Recently, it was even argued that QCA is an entirely unsuitable method for policy analysis.

 

The paper rejects such global convictions, but also views criticism as a welcome opportunity for self-reflection and improvement. Correspondingly, it asks: what are the different promises, challenges and tools posed by diverse approaches to QCA for various purposes of policy analysis? To this end, it reviews and links recent contributions and good practice examples on using QCA for public policy analysis, approaches to QCA, and problematic aspects of the method and its contemporary use. A core message is that QCA poses particular assumptions about causation / explanation, which in turn set out the possibilities for its use in small-N and large-N, theory-developing and theory-testing research designs. By discussing the implications for important questions of public policy research along the different stages of the research cycle, the paper identifies a hands-on map of situations in which different variants of QCA may be more or less useful for public policy analysts, and pitfalls that can arise.

 

USING Q-METHODOLOGY FOR (COMPARATIVE) POLICY ANALYSIS

Astrid Molenveld (Erasmus University Rotterdam)

Numerous scholars and practitioners have stressed that objective policy measures - free from political preferences - do not exist (Howlett, 2009; Peters & Hoornbeek, 2005; Scharpf, 2000; Wlezien, 2004). Authors explain for example that ideas about the nature of knowledge, the approach, envisioned outcomes, spending and politics associated with policy solutions differ among stakeholders, politicians, civil servants and citizens. These preferences and views are hard to comprehend, and are ‘perceptual lenses’ (Allison, 1971), ideas, frames or policy stories (Thompson, 2006) around a certain issue, which can influence public policy decision-making and solutions.

 

It seems that Q-methodology is gaining ground in Public administration, and more specifically in studying public policy. Considering the ideas put forward in the first paragraph this method appears to be a way to comprehend policy problem frames. Q-methodology (Brown, 1993; Van Exel & De Graaf, 2005) is an approach to study perceptions that exist within a given population. Built on the debate within this population, the researcher designs statements which respondents have to sort in a quasi-normal distribution. On the basis of how the respondents sort the statements, the researcher can recognize patterns. People who have sorted the statements in a similar fashion, have similar propositions and are clustered. Afterwards, researchers try to interpret these clusters: what do these respondents think about this subject?

 

Q-methodology applications in public policy analysis vary and one can distinguish many different styles, for instance: an inductive or deductive approach on the subject. A researcher can formulate statements on the basis of theory, case knowledge, or interviews/focus group, and can decide whether one links the clusters back to a theoretical frame, or not. Besides that, the research questions, theories,

fields and sectors differ. On the basis of a literature review of about 50 articles and book chapters, this paper will distinguish different dimensions on which Q-methodology applications in public policy research vary. Furthermore, it will show different conceptual ideas and approaches which might be of interest to researchers planning a study based on Q-methodology.

 

Bibliography

Allison, G. T. (1971). Essence of decision: Explaining the Cuban missile crisis. e-libro, Corp.

Brown, S. R. (1993). A primer on Q methodology. Operant Subjectivity. Operant Subjectivity, 16(3/4), 91–138.

Howlett, M. (2009). Governance modes, policy regimes and operational plans: A multi-level nested model of policy instrument choice and policy design. Policy Sciences, 42(1), 73–89. http://doi.org/10.1007/s11077-009- 9079-1

Peters, G. B., & Hoornbeek, J. (2005). The Problem of Policy Problems. Journal of Comparative Policy Analysis: Research and Practice, 7(4), 349–370. http://doi.org/10.1080/13876980500319204

Scharpf, F. W. (2000). Institutions in Comparative Policy Research. Comparative Political Studies, 33(6-7), 762–790. http://doi.org/10.1177/001041400003300604

Thompson, M. (2006). Cultural Theory, Climate Change and Clumsiness. Clumsy Solutions for a Complex World: Governance, Politics and Plural Perceptions, 216–237. Retrieved from projects.chass.utoronto.ca/semiotics/cyber/douglas4.pdf\nhttp://www.palgrave.com/products/title.aspx?pid=275720

Van Exel, J., & De Graaf, G. (2005). Q methodology : A sneak preview. Social Sciences, 2, 1 – 30. Retrieved from http://qmethod.org/articles/vanExel.pdf

Wlezien, C. (2004). Patterns of Representation: Dynamics of Public Preferences and Policy. Journal of Politics, 66(1), 1–24.

http://doi.org/10.1046/j.1468-2508.2004.00139.x

ROOM
Manasseh Meyer MM 2 - 2
Fri 30th
13:45
Session 3 Methodology for CPA 3/3

Discussants : Jon Pierre (Dept of Political Science, University of Gothenburg)

Discussants : B. Guy Peters (University of Pittsburgh)

Responsive image
PDF
The importance of instruments mix in policy design

Guillaume Fontaine (FLACSO Ecuador)

A central question in comparative policy design is: what is to be compared? If policy design is context-dependent, then what is the lowest common denominator between different policies addressing similar problems or similar policies addressing different problems? We argue that comparative policy design should be conducted as a court trial rather than a medical experiment, for it assesses our degree of confidence in the existence of a causal mechanism linking a policy problem to a policy outcome. The inferential logic of policy design is neither probabilistic in the quantitive way, nor systematic in the semi-experimental sense given by Lijphart to the comparative method.

This paper discusses the use of policy instruments as evidence of a causal process, either for within-case or cross-case studies. We distinguish the moment when a policy is being designed (within-case study) and the moment when a policy is being compared to another (cross-case study). Yet in both cases, policy instruments offer clear-cut sources of information compatible with parsimonious theories, either to explain policy success or failure, policy change or continuity, policy innovation or diffusion, and so forth. 

Examples are taken from the policy of accountability in nine oil and gas exporting countries from Latin America.

 

Comparative Public Policy and Comparative Public Administration: “Never the Twain Shall Meet”?

Jon Pierre (Dept of Political Science, University of Gothenburg)

This paper reviews the linkages between comparative public policy and comparative public administration. Although the two research fields clearly have the potential to inform each other, very little effort has been made to bridge the gap between them. The paper outlines different analytical models that would help provide linkage between comparative analyses of public policy and public administration: institutional analysis; administrative traditions; and a model in which dependent and independent variable alternate, thus allowing for the same cluster of variables to serves as explanandum and explanans.

Numbers and Comparative Policy Analysis: Measurements Issues in Global Indicators

Tero Erkkilä (University of Helsinki)

Global indicators have become influential as policy instruments, effecting national policies and serving as assessment criteria when making decisions on countries’ eligibility for loans and development funding. This has also raised interest in the composition of these measurements and their methodology. Global country rankings and indicators have faced criticism for their normative character and methodology. In particular, the global rankings have been criticized for using blunt aggregate figures in establishing rank orders. Because of this, there have been attempts at creating so-called actionable governance indicators that provide more detailed and reform-oriented measurements of governance. My paper analyses measurement issues that arise in developing indices for policy in an international context. Particular attention is given to the rise of actionable governance indicators and related methodological debates and changes in the production and use of indicators. I use measurements of transparency as a case for analyzing the above shifts in global governance indicators.