In his seminal paper on methodology for comparative politics Arend Lijphart identifies a particular “comparative method”.  This method depends heavily on careful case selection to provide control over extraneous variance that may be done through other means in the other three fundamental methodologies mentioned by Lijphart.  Although scholars often pay obeisance to this method, it is honored more in the breach than the observance.  This paper will explore the utility of the comparative method, strictly defined, in the study of public policy, pointing to the pitfalls and promises of this methodological approach.

Understanding how and why policy outcomes occur is at the heart of public policy research. Thus current research on public policy research relies mostly on qualitative methods. However, the research design, especially data collection, analysis and presentation, which could have enhanced qualitative rigor in inductive interpretive research and theorizing, is weak in public policy research. Specifically, in the absence of transparent and rigorous inductive research design, we cannot see how the research progressed from raw data to analyses which precede the theorizing process. There are two main methodological weaknesses in public policy research. First, researchers limit themselves to explain and/or explore why and how a certain outcome occurs, ignoring inductive methods that aim to generate theory from data. Second, although multiple causes of policy outcomes are understood in terms of interactions among structures, institutions and agents, weaknesses in data collection, presentation, and analysis in the current research allow for only limited progress towards qualitative rigour achieving a balance between data and its interpretation in explaining agency behaviour that generate policy outcome. This paper argues that blending case study methods with grounded theory logic, offers a useful avenue towards explaining and exploring policy outcomes and theory building with clear constructs and their relationships in dynamic causal processes.

While quantitative methods have remained a fixture of traditional public policy analysis, comparative approaches to policy analysis have privileged qualitative methods. In this paper, I present a novel application of the anthropological approach to multi-site ethnography and apply it to case studies in four different cities across three countries. For the past five years I have been studying the relationships between informal waste pickers and local governments in Mexico, Canada and Spain. In this paper, I present empirical results from a four-city comparison of multisite ethnographies of informal garbage governance in Aguascalientes and Leon (Mexico), Vancouver (Canada) and Madrid (Spain). I show the potential of multisite ethnographic approaches, as well as potential drawbacks, particularly when dealing with sensitive subjects and very vulnerable populations. Insights gained from these case studies can be readily applied to other attempts to engage in comparative public policy analysis using qualitative methods.

This paper contributes to debates about comparative methods in political analysis by discussing the added value of a quali-quantitative approach to understanding policy processes and their outcomes. Drawing on an original qualitative dataset about long-term transport policy developments in European cities (CREATE project, funded under Horizon 2020), this paper critically examines the extent to which policy processes have contributed to the reduction of car use in Paris and London (OECD, 2012; CEC, 2016). The main outcome of interest, car use, having peaked in both cities and each case sharing some common features, the research design can be considered a “most similar systems” one (Mill 1843). However, it has been extremely difficult to conclude with absolute certainty what specific variable(s) explain this similar outcome. A vast share of the social sciences literature refers to macro-economic, social and cultural trends observable throughout industrialized societies (Kaufmann et al., 2004; Banister and Hickman, 2013). Yet there is a growing tendency to account for car reduction and changing mobility patterns in cities by focusing on policy offer and initiatives seeking to promote alternatives to car transportation across levels of government. Notwithstanding the vast number of Large N comparative policy evaluation studies, most policy studies seeking to examine whether or not policy matters refer to context specific initiatives or projects, and less so to policy processes (Banister, 2005; Goodwin 2012).

By contrast, this paper argues that the relationship between the reduction of car use in both cities and long-term policy developments in transport can be explored by developing a comparative analysis of historical policy processes (Haydu 1998; Howlett 2009).

This is particularly true of large metropolitan areas such as London and Paris, which are reputed ungovernable (Travers, 2002; Lefèvre, 2003) and characterized by specific forms of governance (Le Galès and Vitale 2013; Peters and Pierre, 2016). Insofar as it seeks to achieve a qualitative analysis of policy processes over time, a process tracing method was applied in order to combine within-case comparisons with small-n analysis as a way to identify intervening causal mechanisms between policy outcomes and independent variable (George and Bennett, 2005, 180; Blatter and Blume, 2008). The paper first discusses problems of causation and the extent to which the process tracing method helps addressing them. Second it introduces the quali-quantitative research design that was used in order to analyze historical transport policy processes in both Paris and London since the 1060s, and more specifically the qualitative database that was developed as part of the CREATE project in order to collect a longitudinal and a comparative dataset about dimensions of policy processes (objectives, resources, instruments etc.) and transport governance (issue salience, degrees of local autonomy, etc.). Third it examines their relationships with changes in transport behavior – and the reduction of car use – over time by identifying co-variations and beyond, some causal mechanisms. Finally, the paper finds general support for the idea that “policy matters” but details of the processes vary considerably from one city to another with important spatial and temporal variations.

This paper outlines how different approaches to Qualitative Comparative Analysis (QCA) pose particular promises and challenges for public policy analysis. Both in terms of research design and the actual goals and needs of policy-oriented research, there seems to be a preferential connection between QCA and public policy analysis. While this has led to an increased popularity of QCA in this field, recent years have also witnessed a diversification of research approaches with QCA. Being situated in-between “quantitative” and “qualitative” methodologies, both small-N and large-N, exploratory and theory-led variants of QCA have developed, each with their particular strengths, limitations and practical issues. This development not only broadens our understanding of what QCA is and what it can be employed for; it also easily creates misunderstandings about the nature, purpose and usefulness of this method. Recently, it was even argued that QCA is an entirely unsuitable method for policy analysis.

The paper rejects such global convictions, but also views criticism as a welcome opportunity for self-reflection and improvement. Correspondingly, it asks: what are the different promises, challenges and tools posed by diverse approaches to QCA for various purposes of policy analysis? To this end, it reviews and links recent contributions and good practice examples on using QCA for public policy analysis, approaches to QCA, and problematic aspects of the method and its contemporary use. A core message is that QCA poses particular assumptions about causation / explanation, which in turn set out the possibilities for its use in small-N and large-N, theory-developing and theory-testing research designs. By discussing the implications for important questions of public policy research along the different stages of the research cycle, the paper identifies a hands-on map of situations in which different variants of QCA may be more or less useful for public policy analysts, and pitfalls that can arise.

Numerous scholars and practitioners have stressed that objective policy measures - free from political preferences - do not exist (Howlett, 2009; Peters & Hoornbeek, 2005; Scharpf, 2000; Wlezien, 2004). Authors explain for example that ideas about the nature of knowledge, the approach, envisioned outcomes, spending and politics associated with policy solutions differ among stakeholders, politicians, civil servants and citizens. These preferences and views are hard to comprehend, and are ‘perceptual lenses’ (Allison, 1971), ideas, frames or policy stories (Thompson, 2006) around a certain issue, which can influence public policy decision-making and solutions.

It seems that Q-methodology is gaining ground in Public administration, and more specifically in studying public policy. Considering the ideas put forward in the first paragraph this method appears to be a way to comprehend policy problem frames. Q-methodology (Brown, 1993; Van Exel & De Graaf, 2005) is an approach to study perceptions that exist within a given population. Built on the debate within this population, the researcher designs statements which respondents have to sort in a quasi-normal distribution. On the basis of how the respondents sort the statements, the researcher can recognize patterns. People who have sorted the statements in a similar fashion, have similar propositions and are clustered. Afterwards, researchers try to interpret these clusters: what do these respondents think about this subject?

Q-methodology applications in public policy analysis vary and one can distinguish many different styles, for instance: an inductive or deductive approach on the subject. A researcher can formulate statements on the basis of theory, case knowledge, or interviews/focus group, and can decide whether one links the clusters back to a theoretical frame, or not. Besides that, the research questions, theories,

fields and sectors differ. On the basis of a literature review of about 50 articles and book chapters, this paper will distinguish different dimensions on which Q-methodology applications in public policy research vary. Furthermore, it will show different conceptual ideas and approaches which might be of interest to researchers planning a study based on Q-methodology.

Bibliography

Allison, G. T. (1971). Essence of decision: Explaining the Cuban missile crisis. e-libro, Corp.

Brown, S. R. (1993). A primer on Q methodology. Operant Subjectivity. Operant Subjectivity, 16(3/4), 91–138.

Howlett, M. (2009). Governance modes, policy regimes and operational plans: A multi-level nested model of policy instrument choice and policy design. Policy Sciences, 42(1), 73–89. http://doi.org/10.1007/s11077-009- 9079-1

Peters, G. B., & Hoornbeek, J. (2005). The Problem of Policy Problems. Journal of Comparative Policy Analysis: Research and Practice, 7(4), 349–370. http://doi.org/10.1080/13876980500319204

Scharpf, F. W. (2000). Institutions in Comparative Policy Research. Comparative Political Studies, 33(6-7), 762–790. http://doi.org/10.1177/001041400003300604

Thompson, M. (2006). Cultural Theory, Climate Change and Clumsiness. Clumsy Solutions for a Complex World: Governance, Politics and Plural Perceptions, 216–237. Retrieved from projects.chass.utoronto.ca/semiotics/cyber/douglas4.pdf\nhttp://www.palgrave.com/products/title.aspx?pid=275720

Van Exel, J., & De Graaf, G. (2005). Q methodology : A sneak preview. Social Sciences, 2, 1 – 30. Retrieved from http://qmethod.org/articles/vanExel.pdf

Wlezien, C. (2004). Patterns of Representation: Dynamics of Public Preferences and Policy. Journal of Politics, 66(1), 1–24.

http://doi.org/10.1046/j.1468-2508.2004.00139.x

A central question in comparative policy design is: what is to be compared? If policy design is context-dependent, then what is the lowest common denominator between different policies addressing similar problems or similar policies addressing different problems? We argue that comparative policy design should be conducted as a court trial rather than a medical experiment, for it assesses our degree of confidence in the existence of a causal mechanism linking a policy problem to a policy outcome. The inferential logic of policy design is neither probabilistic in the quantitive way, nor systematic in the semi-experimental sense given by Lijphart to the comparative method.

This paper discusses the use of policy instruments as evidence of a causal process, either for within-case or cross-case studies. We distinguish the moment when a policy is being designed (within-case study) and the moment when a policy is being compared to another (cross-case study). Yet in both cases, policy instruments offer clear-cut sources of information compatible with parsimonious theories, either to explain policy success or failure, policy change or continuity, policy innovation or diffusion, and so forth. 

Examples are taken from the policy of accountability in nine oil and gas exporting countries from Latin America.

This paper reviews the linkages between comparative public policy and comparative public administration. Although the two research fields clearly have the potential to inform each other, very little effort has been made to bridge the gap between them. The paper outlines different analytical models that would help provide linkage between comparative analyses of public policy and public administration: institutional analysis; administrative traditions; and a model in which dependent and independent variable alternate, thus allowing for the same cluster of variables to serves as explanandum and explanans.

Global indicators have become influential as policy instruments, effecting national policies and serving as assessment criteria when making decisions on countries’ eligibility for loans and development funding. This has also raised interest in the composition of these measurements and their methodology. Global country rankings and indicators have faced criticism for their normative character and methodology. In particular, the global rankings have been criticized for using blunt aggregate figures in establishing rank orders. Because of this, there have been attempts at creating so-called actionable governance indicators that provide more detailed and reform-oriented measurements of governance. My paper analyses measurement issues that arise in developing indices for policy in an international context. Particular attention is given to the rise of actionable governance indicators and related methodological debates and changes in the production and use of indicators. I use measurements of transparency as a case for analyzing the above shifts in global governance indicators.