In his seminal paper on methodology for comparative politics Arend Lijphart identifies a particular “comparative method”.  This method depends heavily on careful case selection to provide control over extraneous variance that may be done through other means in the other three fundamental methodologies mentioned by Lijphart.  Although scholars often pay obeisance to this method, it is honored more in the breach than the observance.  This paper will explore the utility of the comparative method, strictly defined, in the study of public policy, pointing to the pitfalls and promises of this methodological approach.

This paper assesses the evolution of comparative policy analysis since the 1980s through a systematic analysis of the entire body of research articles published in five preeminent journals in the field of public policy. Our analysis aims at better understanding how the field of comparative policy analysis has responded to a number of theoretical and methodological challenges. Research designs in comparative policy studies still have to respond to a number of historical barriers such as conceptual challenges and lack of accessibility and comparability of data. In parallel, the debate on methods in public policy analysis has evolved too. New methodological tools are advocated for enhancing theory-building and greater generalisability of research results. This paper assesses whether comparative research designs have integrated innovations such as process tracing to comparative case-study research, configurational comparative methods to intermediate-N comparison and move beyond the classic country-level comparison. By asking who publishes, about what, and what kind of comparative research from a methodological and theoretical point of view, our analysis provides for a systematic assessment of the evolution of the field over the last four decades and shows where we stand today, compared to the 1980s and 1990s, when comparative policy studies were still emerging as a turning point in policy analysis.

In this paper, I develop a new approach to systematic case selection that public policy researchers can follow when formulating concepts and building theory.

The need for this approach is derived from the insight that the existing methodological literature on case study research gives ample, increasingly sophisticated advice on how to select cases once we have formulated a specific research question, developed clear concepts, and spelled out precise hypotheses. That is, for the “context of justification”, in which theoretical assumptions are tested and verified (or falsified), we find a well-developed toolkit of strategies for selecting and comparing cases. Common case study designs are, e.g., most-likely and least-likely case studies, typical case studies, and deviant cases studies.

By contrast, the literature is silent on how we can systematically identify and select cases in the phase of concept formulation and theory-building in both comparative and more single case-oriented research. This is particularly concerning since we make consequential decisions during these early stages of our research. We confront initial, vague concepts and “proto-theories” with empirical observations in order to adapt, refine, or reformulate these concepts and develop precise hypotheses for subsequent tests. Yet, we cannot draw on established guidelines on how to select cases and empirical material in this phase of our research, even though the early engagement with the empirics shapes the concepts and theories we develop and later test. In brief, this means that, in the “context of discovery”, in which theoretical assumptions and concepts are developed, we stand in the dark.

In this paper, I tackle this shortcoming in the methodological literature and lay out a strategy for systematically selecting cases when developing concepts and theory in public policy research and comparative public policy analysis. In developing my approach, I draw on the growing literature on abductive reasoning and bring in the concept of instrumental cases studies. In general terms, I suggest the theory-guided identification and selection of positive, instrumental cases that are particularly well-suited for empirical investigations in the phase of theory-building phase and concept formulation. In concrete terms, I illustrate this strategy in the context of my own research on long-term, strategic policy-making and the intentional design of policy feedback: First, I introduce the theoretical concept of political architecture that guides my research. Second, I theorize a list of indicators that helps me evaluate the suitability of potential cases and distinguish between suitable, promising and ideal cases of policy-making for the study of political architecture.

The contribution of my work is twofold: First, I outline a concrete, systematic strategy for case selection that public policy researchers can follow in early phases of their research, when they formulate concepts and build theory. Second, I improve our ability to develop new theories, frameworks, and concepts and emphasize that abduction means, first and foremost, that we need to sharpen our self-awareness in early phases of our research, when consequential decision are made,  in order to develop concrete, systematic guidelines that we can follow in the context of discovery.

Prediction has been described as a goal of quantitative policy analysis (Breunig and Ahlquist 2014). So far researchers engage primarily with the prediction of policy outcomes (e.g. in the form of public spending as such or public spending for specific purposes) but not with predicting future policy output. However, path-dependency literature (e.g., Pierson 2001) identifies a number of elements that constrain the future policy choices of decision-makers, including the creation of interest group networks, the adaptation of actor expectations and the presence of learning effects. Similarly, policy design studies have turned towards the question of designing ‘sticky’ policies (e.g., Jordan and Matt 2014), arguing that policy design choices impact the durability of policy instruments or overall policy approaches.

In this paper, we bring together these two perspectives in order to theorize about the linkage between the existence and strength of policy design characteristics and the strength and stability (or ‘stickiness’) of policy instruments and policy mixes in subsequent time periods. It is theorized that the mechanism driving this relationship is the degree to which path-dependency (or ‘stickiness’) is induced by the design of a policy, that is to say policy design characteristics also matter for the temporal development of complex policy mixes.

We derive three hypotheses: First, on an individual policy level, the design of a policy influences the likelihood that that policy will persist; the characteristics that matter in this regard are hypothesized to be the ones that directly link to path dependency. Second, this can be generalized to the policy mix level as well; mixes that contain the relevant characteristics in abundance will have greater stability overall. Third, we address the stringency of a policy mix, investigating whether the same set of design characteristics makes for a stringent policy mix as well as a stable one.

Crucially, our theory is tested in panel regression analyses using a detailed and comparable dataset of policy output across nine countries (Australia, Austria, Canada, Germany, Ireland, New Zealand, Spain, Switzerland, United Kingdom) over a time period of seventeen years (1998-2014). We focus on the policy field of renewable energy production and base our empirical assessment of policy output (i.e., the dependent variable in our study) on Schaffrin et al.’s (2014, 2015) Index of Policy Activity (IPA) that uses, among others, Howlett and Cashore’s (2009) matrix of policy elements as a reference point. In addition to the six policy design characteristics covered with the standard IPA approach (scope, objectives, integration, budget, implementation, and monitoring) we also systematically assess a further design criteria, namely a policy’s technology specificity. This design characteristic has been identified as being crucial for the successful deployment of renewable energy technologies (Schmidt et al. 2016). By systematically analyzing which policy design characteristics drive the future outlook of complex policy mixes, we contribute both to empirical, quantitative and comparable policy analysis as well as, potentially, to the theorization of temporal relationships between current policy choices and future policy mixes. 

Breunig, C.; Ahlquist, J. (2014). Quantitative Methodologies in Public Policy. In: Comparative Policy Studies. Conceptual and Methodological Challenges. Eds.: Engeli, Rothmayr Allison.

Howlett, M.; Cashore, B. (2009). Journal of Comparative Policy Analysis, 11(1): 33–46.

Jordan, A.; Matt, E. (2014). Policy Sciences, 47(3): 227-247.

Pierson, P. (2001). The new politics of the welfare state.

Schaffrin, A.; Sewerin, S.; Seubert, S. (2014). Environmental Politics, 23(5): 860–883.

Schaffrin, A.; Sewerin, S.; Seubert, S. (2015). Policy Studies Journal, 43(2): 257–282.

Schmidt, T. et al. (2016). Research Policy, 45(10): 1965–1983.