T10P05. Methodological Advances in Policy Studies and Comparative Public Policy

Topic : Methodologies

Chair : Christine Rothmayr Allison (Université de Montréal)

Second Chair : Isabelle Engeli (University of Exeter)

Third Chair : Eric Montpetit (Université de Montréal)

Share to Facebook Share to Twitter Share to Linkedin Share by mail

General Objectives, Research Questions and Scientific Relevance

Globalisation and regional integration have increased complexity and interdependence, waves of democratization and the growing number of emerging countries have enlarged possibilities of comparison. Parallel to these developments, the debate on methods in public policy analysis has evolved too.

  • First of all, case studies have occupied a prominent place in comparative policy studies. Instead of considering case studies as a ‘by default’ comparison when research resources for a large-N are lacking, process tracing offers a meaningful tool to enhance theory building and, to a certain extent, theory testing in policy studies. Comparative research designs have also integrated innovations such as compound research designs. Compound research designs, proposed by Levi-Faur (2004, 2006), aim at maximizing the explanatory capacity of qualitative comparative analysis. As the number of available ‘real’ cases is limited, comparative policy analysis may take advantage of combining two or more comparative lines of inquiry in order to strengthen the generalization capacity of their explanation: cross-country comparison, comparison across policy domains and comparison across time.
  • From single and small-N comparisons, comparative policy studies have moved on to the issue of intermediate-N comparison with the developing of qualitative comparative analysis and fuzzy sets (QCA, Ragin 1978, 2008, Rihoux et al. 2013) in order to benefit from the complexity of each case while enhancing the generalization across cases. Based on algorithms derived from Boolean algebra, QCA simplifies the interaction of explanatory factors in order to identify configurations of causality valid across cases. Thus it allows for the development of parsimonious qualitative explanations with a strong capacity for middle-range generalization.
  • With the increased availability and accessibility of policy-relevant quantitative data, large-N studies are also becoming more prominent in comparative policy analysis. Quantitative comparison aims at explaining and predicting patterns of policy-making processes and policy outcomes across cases and attempts to broaden the generalization of research findings.

This panel aims at discussing the current state of development in comparative policy studies from a methodological point of view by focusing on these and other recent methodological innovations and their application to policy studies, and in particular in comparative policy studies. All these methods are confronted with a number of challenges specific to comparative public policy, such as case selection, the definition of the dependent variable, the availability of data, concept stretching and comparability of available, often aggregate, data. The paper in this panel will identify the challenges specific to recent methodological advances for comparative policy studies and offer an assessment of the current state of application of recent methodological innovations in the field. Furthermore, the panel aims at discussing how to address these challenges and contribute to further strengthen research designs in policy studies and in comparative public policy.

Call for papers

This panel aims at discussing the current state of development in policy studies  -- in particular in comparative public policy -- from a methodological point of view by focusing on recent methodological innovations and their application in policy studies and comparative policy studies. Globalisation and regional integration have increased complexity and interdependence, waves of democratization and the growing number of emerging countries have enlarged possibilities of comparison. Parallel to these developments, the debate on methods in public policy analysis has evolved too. First of all, case studies have occupied a prominent place in comparative policy studies. Instead of considering case studies as a ‘by default’ comparison when research resources for a large-N are lacking, process tracing offers a meaningful tool to enhance theory building and, to a certain extent, theory testing in policy studies. From single and small-N comparisons, comparative policy studies have moved on to the issue of intermediate-N comparison with the development of qualitative comparative analysis and fuzzy sets in order to benefit from the complexity of each case while enhancing the generalization across cases. With the increased availability and accessibility of policy-relevant quantitative data, large-N studies are also becoming more prominent in comparative policy analysis. All these methods are confronted with a number of challenges specific to policy studies and comparative public policy, such as case selection, the definition of the dependent variable, the availability of data, concept stretching and comparability of available, often aggregate, data. The paper in this panel will identify the challenges specific to recent methodological advances for policy studies and comparative policy analysis and offer an assessment of the current state of application of recent methodological advances in the field. Furthermore, the panel aims at discussing how to address these challenges and intends to contribute to further strengthening research designs in comparative public policy.

ROOM
Block B 2 - 2
Thu 29th
10:30
Session 1

Discussants : Isabelle Engeli (University of Exeter)

Discussants : Christine Rothmayr Allison (Université de Montréal)

The Comparative Method and Comparartive Public Policy

B. Guy Peters (University of Pittsburgh)

In his seminal paper on methodology for comparative politics Arend Lijphart identifies a particular “comparative method”.  This method depends heavily on careful case selection to provide control over extraneous variance that may be done through other means in the other three fundamental methodologies mentioned by Lijphart.  Although scholars often pay obeisance to this method, it is honored more in the breach than the observance.  This paper will explore the utility of the comparative method, strictly defined, in the study of public policy, pointing to the pitfalls and promises of this methodological approach.

 

Comparative Public Policy at Forty: Taking the (preliminary) Stock

Christine Rothmayr Allison (Université de Montréal)

Isabelle Engeli (University of Exeter)

Eric Montpetit (Université de Montréal)

This paper assesses the evolution of comparative policy analysis since the 1980s through a systematic analysis of the entire body of research articles published in five preeminent journals in the field of public policy. Our analysis aims at better understanding how the field of comparative policy analysis has responded to a number of theoretical and methodological challenges. Research designs in comparative policy studies still have to respond to a number of historical barriers such as conceptual challenges and lack of accessibility and comparability of data. In parallel, the debate on methods in public policy analysis has evolved too. New methodological tools are advocated for enhancing theory-building and greater generalisability of research results. This paper assesses whether comparative research designs have integrated innovations such as process tracing to comparative case-study research, configurational comparative methods to intermediate-N comparison and move beyond the classic country-level comparison. By asking who publishes, about what, and what kind of comparative research from a methodological and theoretical point of view, our analysis provides for a systematic assessment of the evolution of the field over the last four decades and shows where we stand today, compared to the 1980s and 1990s, when comparative policy studies were still emerging as a turning point in policy analysis.

Responsive image
PDF
A New Approach to Case Selection: Conceptualizing Positive, Instrumental Case Studies for Qualitative Public Policy Research

Philipp Pechmann (Department of Political Science, Aarhus University)

In this paper, I develop a new approach to systematic case selection that public policy researchers can follow when formulating concepts and building theory.

 

The need for this approach is derived from the insight that the existing methodological literature on case study research gives ample, increasingly sophisticated advice on how to select cases once we have formulated a specific research question, developed clear concepts, and spelled out precise hypotheses. That is, for the “context of justification”, in which theoretical assumptions are tested and verified (or falsified), we find a well-developed toolkit of strategies for selecting and comparing cases. Common case study designs are, e.g., most-likely and least-likely case studies, typical case studies, and deviant cases studies.

 

By contrast, the literature is silent on how we can systematically identify and select cases in the phase of concept formulation and theory-building in both comparative and more single case-oriented research. This is particularly concerning since we make consequential decisions during these early stages of our research. We confront initial, vague concepts and “proto-theories” with empirical observations in order to adapt, refine, or reformulate these concepts and develop precise hypotheses for subsequent tests. Yet, we cannot draw on established guidelines on how to select cases and empirical material in this phase of our research, even though the early engagement with the empirics shapes the concepts and theories we develop and later test. In brief, this means that, in the “context of discovery”, in which theoretical assumptions and concepts are developed, we stand in the dark.

 

In this paper, I tackle this shortcoming in the methodological literature and lay out a strategy for systematically selecting cases when developing concepts and theory in public policy research and comparative public policy analysis. In developing my approach, I draw on the growing literature on abductive reasoning and bring in the concept of instrumental cases studies. In general terms, I suggest the theory-guided identification and selection of positive, instrumental cases that are particularly well-suited for empirical investigations in the phase of theory-building phase and concept formulation. In concrete terms, I illustrate this strategy in the context of my own research on long-term, strategic policy-making and the intentional design of policy feedback: First, I introduce the theoretical concept of political architecture that guides my research. Second, I theorize a list of indicators that helps me evaluate the suitability of potential cases and distinguish between suitable, promising and ideal cases of policy-making for the study of political architecture.

 

The contribution of my work is twofold: First, I outline a concrete, systematic strategy for case selection that public policy researchers can follow in early phases of their research, when they formulate concepts and build theory. Second, I improve our ability to develop new theories, frameworks, and concepts and emphasize that abduction means, first and foremost, that we need to sharpen our self-awareness in early phases of our research, when consequential decision are made,  in order to develop concrete, systematic guidelines that we can follow in the context of discovery.

Responsive image
PDF
Does policy design predict a policy mix’s future outlook? A new approach to analyzing path-dependency

Blair Bateson (ETH Zurich)

Tobias Schmidt (ETH Zurich)

Sebastian Sewerin (Delft University of Technology)

Prediction has been described as a goal of quantitative policy analysis (Breunig and Ahlquist 2014). So far researchers engage primarily with the prediction of policy outcomes (e.g. in the form of public spending as such or public spending for specific purposes) but not with predicting future policy output. However, path-dependency literature (e.g., Pierson 2001) identifies a number of elements that constrain the future policy choices of decision-makers, including the creation of interest group networks, the adaptation of actor expectations and the presence of learning effects. Similarly, policy design studies have turned towards the question of designing ‘sticky’ policies (e.g., Jordan and Matt 2014), arguing that policy design choices impact the durability of policy instruments or overall policy approaches.

 

In this paper, we bring together these two perspectives in order to theorize about the linkage between the existence and strength of policy design characteristics and the strength and stability (or ‘stickiness’) of policy instruments and policy mixes in subsequent time periods. It is theorized that the mechanism driving this relationship is the degree to which path-dependency (or ‘stickiness’) is induced by the design of a policy, that is to say policy design characteristics also matter for the temporal development of complex policy mixes.

 

We derive three hypotheses: First, on an individual policy level, the design of a policy influences the likelihood that that policy will persist; the characteristics that matter in this regard are hypothesized to be the ones that directly link to path dependency. Second, this can be generalized to the policy mix level as well; mixes that contain the relevant characteristics in abundance will have greater stability overall. Third, we address the stringency of a policy mix, investigating whether the same set of design characteristics makes for a stringent policy mix as well as a stable one.

 

Crucially, our theory is tested in panel regression analyses using a detailed and comparable dataset of policy output across nine countries (Australia, Austria, Canada, Germany, Ireland, New Zealand, Spain, Switzerland, United Kingdom) over a time period of seventeen years (1998-2014). We focus on the policy field of renewable energy production and base our empirical assessment of policy output (i.e., the dependent variable in our study) on Schaffrin et al.’s (2014, 2015) Index of Policy Activity (IPA) that uses, among others, Howlett and Cashore’s (2009) matrix of policy elements as a reference point. In addition to the six policy design characteristics covered with the standard IPA approach (scope, objectives, integration, budget, implementation, and monitoring) we also systematically assess a further design criteria, namely a policy’s technology specificity. This design characteristic has been identified as being crucial for the successful deployment of renewable energy technologies (Schmidt et al. 2016). By systematically analyzing which policy design characteristics drive the future outlook of complex policy mixes, we contribute both to empirical, quantitative and comparable policy analysis as well as, potentially, to the theorization of temporal relationships between current policy choices and future policy mixes. 

 

Breunig, C.; Ahlquist, J. (2014). Quantitative Methodologies in Public Policy. In: Comparative Policy Studies. Conceptual and Methodological Challenges. Eds.: Engeli, Rothmayr Allison.

Howlett, M.; Cashore, B. (2009). Journal of Comparative Policy Analysis, 11(1): 33–46.

Jordan, A.; Matt, E. (2014). Policy Sciences, 47(3): 227-247.

Pierson, P. (2001). The new politics of the welfare state.

Schaffrin, A.; Sewerin, S.; Seubert, S. (2014). Environmental Politics, 23(5): 860–883.

Schaffrin, A.; Sewerin, S.; Seubert, S. (2015). Policy Studies Journal, 43(2): 257–282.

Schmidt, T. et al. (2016). Research Policy, 45(10): 1965–1983.