T02P04 - Methodology for Comparative Policy Analysis

Topic : Comparative Public Policy

Panel Chair : B. Guy Peters - bgpeters@pitt.edu

Panel Second Chair : Guillaume Fontaine - gfontaine@flacso.edu.ec

Objectives and Scientific Relevance of the panel

Call for papers

Session 1 Methodologies for CPA - Session 1/3


Adrian Kay - adrian.kay@anu.edu.au - The Australian National University - Australia

Guillaume Fontaine - gfontaine@flacso.edu.ec - FLACSO - Ecuador


Astrid Molenveld - molenveld@fsw.eur.nl - Netherlands

Numerous scholars and practitioners have stressed that objective policy measures - free from political preferences - do not exist (Howlett, 2009; Peters & Hoornbeek, 2005; Scharpf, 2000; Wlezien, 2004). Authors explain for example that ideas about the nature of knowledge, the approach, envisioned outcomes, spending and politics associated with policy solutions differ among stakeholders, politicians, civil servants and citizens. These preferences and views are hard to comprehend, and are ‘perceptual lenses’ (Allison, 1971), ideas, frames or policy stories (Thompson, 2006) around a certain issue, which can influence public policy decision-making and solutions.


It seems that Q-methodology is gaining ground in Public administration, and more specifically in studying public policy. Considering the ideas put forward in the first paragraph this method appears to be a way to comprehend policy problem frames. Q-methodology (Brown, 1993; Van Exel & De Graaf, 2005) is an approach to study perceptions that exist within a given population. Built on the debate within this population, the researcher designs statements which respondents have to sort in a quasi-normal distribution. On the basis of how the respondents sort the statements, the researcher can recognize patterns. People who have sorted the statements in a similar fashion, have similar propositions and are clustered. Afterwards, researchers try to interpret these clusters: what do these respondents think about this subject?


Q-methodology applications in public policy analysis vary and one can distinguish many different styles, for instance: an inductive or deductive approach on the subject. A researcher can formulate statements on the basis of theory, case knowledge, or interviews/focus group, and can decide whether one links the clusters back to a theoretical frame, or not. Besides that, the research questions, theories,

fields and sectors differ. On the basis of a literature review of about 50 articles and book chapters, this paper will distinguish different dimensions on which Q-methodology applications in public policy research vary. Furthermore, it will show different conceptual ideas and approaches which might be of interest to researchers planning a study based on Q-methodology.



Allison, G. T. (1971). Essence of decision: Explaining the Cuban missile crisis. e-libro, Corp.

Brown, S. R. (1993). A primer on Q methodology. Operant Subjectivity. Operant Subjectivity, 16(3/4), 91–138.

Howlett, M. (2009). Governance modes, policy regimes and operational plans: A multi-level nested model of policy instrument choice and policy design. Policy Sciences, 42(1), 73–89. http://doi.org/10.1007/s11077-009- 9079-1

Peters, G. B., & Hoornbeek, J. (2005). The Problem of Policy Problems. Journal of Comparative Policy Analysis: Research and Practice, 7(4), 349–370. http://doi.org/10.1080/13876980500319204

Scharpf, F. W. (2000). Institutions in Comparative Policy Research. Comparative Political Studies, 33(6-7), 762–790. http://doi.org/10.1177/001041400003300604

Thompson, M. (2006). Cultural Theory, Climate Change and Clumsiness. Clumsy Solutions for a Complex World: Governance, Politics and Plural Perceptions, 216–237. Retrieved from projects.chass.utoronto.ca/semiotics/cyber/douglas4.pdf\nhttp://www.palgrave.com/products/title.aspx?pid=275720

Van Exel, J., & De Graaf, G. (2005). Q methodology : A sneak preview. Social Sciences, 2, 1 – 30. Retrieved from http://qmethod.org/articles/vanExel.pdf

Wlezien, C. (2004). Patterns of Representation: Dynamics of Public Preferences and Policy. Journal of Politics, 66(1), 1–24.


The Comparative Method and Comparartive Public Policy

B. Guy Peters - bgpeters@pitt.edu - University of Pittsburgh - United States

In his seminal paper on methodology for comparative politics Arend Lijphart identifies a particular “comparative method”.  This method depends heavily on careful case selection to provide control over extraneous variance that may be done through other means in the other three fundamental methodologies mentioned by Lijphart.  Although scholars often pay obeisance to this method, it is honored more in the breach than the observance.  This paper will explore the utility of the comparative method, strictly defined, in the study of public policy, pointing to the pitfalls and promises of this methodological approach.


Why we need more transparent and rigorous inductive research design in comparative public policy scholarship, and what to do about it

Caner Bakir - cbakir@ku.edu.tr - Koc University - Turkey

Understanding how and why policy outcomes occur is at the heart of public policy research. Thus current research on public policy research relies mostly on qualitative methods. However, the research design, especially data collection, analysis and presentation, which could have enhanced qualitative rigor in inductive interpretive research and theorizing, is weak in public policy research. Specifically, in the absence of transparent and rigorous inductive research design, we cannot see how the research progressed from raw data to analyses which precede the theorizing process. There are two main methodological weaknesses in public policy research. First, researchers limit themselves to explain and/or explore why and how a certain outcome occurs, ignoring inductive methods that aim to generate theory from data. Second, although multiple causes of policy outcomes are understood in terms of interactions among structures, institutions and agents, weaknesses in data collection, presentation, and analysis in the current research allow for only limited progress towards qualitative rigour achieving a balance between data and its interpretation in explaining agency behaviour that generate policy outcome. This paper argues that blending case study methods with grounded theory logic, offers a useful avenue towards explaining and exploring policy outcomes and theory building with clear constructs and their relationships in dynamic causal processes.


Comparative Public Policy and Comparative Public Administration: “Never the Twain Shall Meet”?

Jon Pierre - jon.pierre@pol.gu.se - Dept of Political Science, University of Gothenburg - Sweden

This paper reviews the linkages between comparative public policy and comparative public administration. Although the two research fields clearly have the potential to inform each other, very little effort has been made to bridge the gap between them. The paper outlines different analytical models that would help provide linkage between comparative analyses of public policy and public administration: institutional analysis; administrative traditions; and a model in which dependent and independent variable alternate, thus allowing for the same cluster of variables to serves as explanandum and explanans.

Session 2 Methodology for CPA 2/3


B. Guy Peters - bgpeters@pitt.edu - University of Pittsburgh - United States

Guillaume Fontaine - gfontaine@flacso.edu.ec - FLACSO - Ecuador

Does policy matter? A quali-quantitative approach to historical transport policy processes in Paris and London.

HALPERN Charlotte - charlotte.halpern@sciencespo.fr - Sciences Po, Centre d'Etudes Européennes - France

This paper contributes to debates about comparative methods in political analysis by discussing the added value of a quali-quantitative approach to understanding policy processes and their outcomes. Drawing on an original qualitative dataset about long-term transport policy developments in European cities (CREATE project, funded under Horizon 2020), this paper critically examines the extent to which policy processes have contributed to the reduction of car use in Paris and London (OECD, 2012; CEC, 2016). The main outcome of interest, car use, having peaked in both cities and each case sharing some common features, the research design can be considered a “most similar systems” one (Mill 1843). However, it has been extremely difficult to conclude with absolute certainty what specific variable(s) explain this similar outcome. A vast share of the social sciences literature refers to macro-economic, social and cultural trends observable throughout industrialized societies (Kaufmann et al., 2004; Banister and Hickman, 2013). Yet there is a growing tendency to account for car reduction and changing mobility patterns in cities by focusing on policy offer and initiatives seeking to promote alternatives to car transportation across levels of government. Notwithstanding the vast number of Large N comparative policy evaluation studies, most policy studies seeking to examine whether or not policy matters refer to context specific initiatives or projects, and less so to policy processes (Banister, 2005; Goodwin 2012).


By contrast, this paper argues that the relationship between the reduction of car use in both cities and long-term policy developments in transport can be explored by developing a comparative analysis of historical policy processes (Haydu 1998; Howlett 2009).

This is particularly true of large metropolitan areas such as London and Paris, which are reputed ungovernable (Travers, 2002; Lefèvre, 2003) and characterized by specific forms of governance (Le Galès and Vitale 2013; Peters and Pierre, 2016). Insofar as it seeks to achieve a qualitative analysis of policy processes over time, a process tracing method was applied in order to combine within-case comparisons with small-n analysis as a way to identify intervening causal mechanisms between policy outcomes and independent variable (George and Bennett, 2005, 180; Blatter and Blume, 2008). The paper first discusses problems of causation and the extent to which the process tracing method helps addressing them. Second it introduces the quali-quantitative research design that was used in order to analyze historical transport policy processes in both Paris and London since the 1060s, and more specifically the qualitative database that was developed as part of the CREATE project in order to collect a longitudinal and a comparative dataset about dimensions of policy processes (objectives, resources, instruments etc.) and transport governance (issue salience, degrees of local autonomy, etc.). Third it examines their relationships with changes in transport behavior – and the reduction of car use – over time by identifying co-variations and beyond, some causal mechanisms. Finally, the paper finds general support for the idea that “policy matters” but details of the processes vary considerably from one city to another with important spatial and temporal variations.

Qualitative Comparative Analysis (QCA) as an approach to Comparative Policy Analysis

Eva Thomann - escriba@hotmail.ch - Heidelberg University - Germany

This paper outlines how different approaches to Qualitative Comparative Analysis (QCA) pose particular promises and challenges for public policy analysis. Both in terms of research design and the actual goals and needs of policy-oriented research, there seems to be a preferential connection between QCA and public policy analysis. While this has led to an increased popularity of QCA in this field, recent years have also witnessed a diversification of research approaches with QCA. Being situated in-between “quantitative” and “qualitative” methodologies, both small-N and large-N, exploratory and theory-led variants of QCA have developed, each with their particular strengths, limitations and practical issues. This development not only broadens our understanding of what QCA is and what it can be employed for; it also easily creates misunderstandings about the nature, purpose and usefulness of this method. Recently, it was even argued that QCA is an entirely unsuitable method for policy analysis.


The paper rejects such global convictions, but also views criticism as a welcome opportunity for self-reflection and improvement. Correspondingly, it asks: what are the different promises, challenges and tools posed by diverse approaches to QCA for various purposes of policy analysis? To this end, it reviews and links recent contributions and good practice examples on using QCA for public policy analysis, approaches to QCA, and problematic aspects of the method and its contemporary use. A core message is that QCA poses particular assumptions about causation / explanation, which in turn set out the possibilities for its use in small-N and large-N, theory-developing and theory-testing research designs. By discussing the implications for important questions of public policy research along the different stages of the research cycle, the paper identifies a hands-on map of situations in which different variants of QCA may be more or less useful for public policy analysts, and pitfalls that can arise.


Probing Complexity and Sustainability: Beyond System and Structure in South Asia

Preeti Raghunath - preetimalaraghunath@gmail.com - University of Hyderabad - India

Comparative studies of policies have often been system-centric, drawing on systems and the structures that come to occupy them as the ground to draw out similarities and differences. The post-2015 world is one that is characterised by complexity, uncertainty and grapples with questions of sustainability, all of which are at the heart of the comparative study of policies. Drawing on empirical work on policymaking for community radio in four countries of South Asia, the paper reflects on methodological considerations in conducting such a comparative study. This it does, by drawing on policy literature dealing with competing ideas of methodological nationalism versus connected histories, diversified policy actors like nation-states, epistemic communities, transnational advocacy coalitions and networks, and considerations of structure and post-structure. Further, this paper seeks to go beyond the systemic approach, to open up the methodological stipulations of comparative policy research to 'uncertainty' and externalities', allowing for theoretical engagement with Critical Theory, as espoused by the Frankfurt School. The researcher, synthesising empirical research on South Asia and Critical Theory, aims to present a theoretical framework for a comparative study of policies. 

Configurations, action situations, and policy performance

Alessia Damonte - alessia.damonte@unimi.it - Dept. Social and Political Sciences - Università degli Studi di Milano - Italy

It is already known from environmental governance that Qualitative Comparative Analysis especially suits the Institutional Analysis and Development framework. In IAD's terms, tragedies of the commons follow when people's strategies unfold within ill-shaped action situations -- due to misfits in the setting and interplay of the rules in use. QCA renders each rule setting as a variable, and their interplay as Boolean conjunctions; solutions from cross-case comparisons cast new light on the institutional source of tragedies, and indicate remedies. This contribution widens the scope of such matching. It argues that the IAD with QCA applies to the link between governance design and effectiveness beyond common pool issues. 


Noluthando Matsiliza - Shirleymats2@GMAIL.COM - Durban University of Technology - South Africa


NS Matsiliza

Durban University of Technology

South Africa, Shirleymats2@gmail.com


Taxonomies for policy research have been dominated by a disciplinary system that demarcates domains of specialized inquiry. More desires to conduct policy research is informed by rapid societal needs that necessitate government interventions. However, policy research has received less attention from scholars from policy studies and public administration, while it is an important component of finding out whether performance outcomes have been achieved in a planned period through assessing results at various policy phases. This article’s intent is to examine the taxonomies that will be effective for conducting policy research. This study argues that a succinctly planned policy research can contribute towards policy development in the South African public sector. This study will draw literature from various studies on housing policy and explore the effectiveness of taxonomic methodologies applied in policy research. This review highlights conceptualisation and theories in policy research, taxonomies for policy research, methodological implications, conclusion and recommendations. Findings from this study will contribute towards specific guidelines for conducting policy research where researchers continue to sort out meanings and opportunities in interdisciplinary experience.

Key words: Empiricism, evaluation, evidence-based, genre, Housing, methodology, policy, research, taxonomy.


Session 3 Methodology for CPA 3/3


Jon Pierre - jon.pierre@pol.gu.se - Dept of Political Science, University of Gothenburg - Sweden

B. Guy Peters - bgpeters@pitt.edu - University of Pittsburgh - United States

The importance of instruments mix in policy design

Guillaume Fontaine - gfontaine@flacso.edu.ec - FLACSO - Ecuador

A central question in comparative policy design is: what is to be compared? If policy design is context-dependent, then what is the lowest common denominator between different policies addressing similar problems or similar policies addressing different problems? We argue that comparative policy design should be conducted as a court trial rather than a medical experiment, for it assesses our degree of confidence in the existence of a causal mechanism linking a policy problem to a policy outcome. The inferential logic of policy design is neither probabilistic in the quantitive way, nor systematic in the semi-experimental sense given by Lijphart to the comparative method.

This paper discusses the use of policy instruments as evidence of a causal process, either for within-case or cross-case studies. We distinguish the moment when a policy is being designed (within-case study) and the moment when a policy is being compared to another (cross-case study). Yet in both cases, policy instruments offer clear-cut sources of information compatible with parsimonious theories, either to explain policy success or failure, policy change or continuity, policy innovation or diffusion, and so forth. 

Examples are taken from the policy of accountability in nine oil and gas exporting countries from Latin America.


Numbers and Comparative Policy Analysis: Measurements Issues in Global Indicators

Tero Erkkilä - tero.erkkila@helsinki.fi - University of Helsinki - Finland

Global indicators have become influential as policy instruments, effecting national policies and serving as assessment criteria when making decisions on countries’ eligibility for loans and development funding. This has also raised interest in the composition of these measurements and their methodology. Global country rankings and indicators have faced criticism for their normative character and methodology. In particular, the global rankings have been criticized for using blunt aggregate figures in establishing rank orders. Because of this, there have been attempts at creating so-called actionable governance indicators that provide more detailed and reform-oriented measurements of governance. My paper analyses measurement issues that arise in developing indices for policy in an international context. Particular attention is given to the rise of actionable governance indicators and related methodological debates and changes in the production and use of indicators. I use measurements of transparency as a case for analyzing the above shifts in global governance indicators.

Applying Qualitative Methods to Comparative Public Policy Analysis: Insights from Multi-Site Ethnographies of Informal Garbage Governance

Raul Pacheco-Vega - raul.pacheco-vega@cide.edu - Centro de Investigación y Docencia Económicas (CIDE) - Mexico

While quantitative methods have remained a fixture of traditional public policy analysis, comparative approaches to policy analysis have privileged qualitative methods. In this paper, I present a novel application of the anthropological approach to multi-site ethnography and apply it to case studies in four different cities across three countries. For the past five years I have been studying the relationships between informal waste pickers and local governments in Mexico, Canada and Spain. In this paper, I present empirical results from a four-city comparison of multisite ethnographies of informal garbage governance in Aguascalientes and Leon (Mexico), Vancouver (Canada) and Madrid (Spain). I show the potential of multisite ethnographic approaches, as well as potential drawbacks, particularly when dealing with sensitive subjects and very vulnerable populations. Insights gained from these case studies can be readily applied to other attempts to engage in comparative public policy analysis using qualitative methods.

Causal mechanisms in comparative policy analysis: methodological reflections on research designs for explanations-by-consequence

Adrian Kay - adrian.kay@ubd.edu.bn - Universiti Brunei Darussalam - Brunei Darussalam

Causal mechanisms are well established in the philosophy of social science, with lively, long-standing and productive debates about their role in strategies of explanation and of inference. However, both methodologically and in terms of actual research designs for comparative policy analysis work on causal mechanisms remains relatively under developed. This paper considers one particular set of debates about researching causal mechanisms in policy studies: explanation by consequence.


The prospects for evolutionary theory in policy studies have attracted recent attention. This work investigates the potential to explain policy phenomena through appeal to their actual consequences. However, the extant and long recognised divide between analysis for and analysis of policy continues to limit contributions from the study of policy consequences (prospectively or retrospectively) in research on causal mechanisms in multi-case designs for examining the process of policy-making

This paper advances the argument that bridging this divide would contribute to research about causal mechanisms that might support explanation-by-consequences. In particular, recent realist and theory-based policy evaluation work in the area of contribution analysis offer a suite of plausible mechanisms by which the consequences of a policy may uphold or undermine the policy process wants to explain.


A functional explanation (FE) is a particular class of consequence explanation where the consequences of a policy or institution are favourable or functional, for some agent or group who maintain that institution or behaviour. Explaining a cause, a functional policy X, in terms of its consequences, the functions Y it carries out, reverses the temporal sequence of causes preceding consequences. This is tricky in policy analysis requiring an emphasis on historical contingency, accident or random variation in terms of why an X occurs at some time; then focus on feedback loops from Y to X that explains why X continues to perform the function. The separation of analysis between the creation of institutions and what sustains them is a characteristic of historical institutionalism in policy studies.


There are many reasons why functionalism is a good strategy in biology by presenting the famous variation-and-selection model, which says that functional structures will lead to greater survival and replication of the genes that produce them. However, within the policy sciences, appeal is more often made to the ‘missing mechanism’ argument used generally to deny ‘biological analogies’ in the social world. There is simply no equivalent of natural selection operating in the social sphere.

However, policy studies could look to its own long literature on policy evaluation for causal mechanisms to support explanations by consequence. Bridging gaps between literatures on realist and theory-based policy evaluation studies and policy learning can contribute to claims about causal mechanisms in, for example, the development of policy knowledge through experimentation, trial and error learning and selection or research on the relationships between actor’s belief structures and policy logic models. Without such connections, some plausible causal mechanisms in the comparative policy analysis are being missed.

Export PDF