Format

P09. Citizens and Evidence-Based Policy Making: Perfect Couple or Strange Bedfellows?

POLICY DESIGN, POLICY ANALYSIS, EXPERTISE AND EVALUATION
PANEL CHAIR(S)
Valerie Pattyn
Institute of Public Administration - Leiden University
Main chair
Pirmin Bundi
University of Lausanne
Second chair
CATEGORISATION
POLICY TOPIC
POLICY DESIGN, POLICY ANALYSIS, EXPERTISE AND EVALUATION
SECTOR
—
KEYWORDS
—
GENERAL OBJECTIVES, RESEARCH QUESTIONS AND SCIENTIFIC RELEVANCE

Policy evaluations and other research activities generate information that can be used in order to improve public policies, which is commonly captured under the (rational) label of evidence-based policy making (EBP). However, there is ample empirical evidence that the use of such information is not systematic and often politicized. Moreover, the generation of evidence is dominated by elites, such as experts, academics, evaluators or public servants, and rarely includes the involvement of citizens. This is rather surprising as citizens are often the main beneficiary group of public policy and their beliefs and opinions define the political framework of public measures. It can even be argued that the involvement of citizens has the potential of overcoming the classic tension between democracy and technocracy in public policy making.

 

In research on evaluation, the involvement of citizens has been discussed in frameworks coined as Democratic Evaluation (MacDonald 1993), Transformative Participatory Evaluation (Cousins and Whitmore 1998) or Deliberative Democratic Evaluation (House/Howe 1999), which promise to be more inclusive, and foster participation and collaboration. Still, despite the existence of such frameworks, the systematic involvement of citizens in evidence creation seems to be rather exception than rule (Halachmi & Holzer 2010). How can we explain this discrepancy, and what are strategies to make research and evaluation processes more democratic?

 

This panel gathers theoretical, conceptual, and empirical contributions that deal with the relationship between citizens, evidence and public policy, with the aim to enhance our understanding of how citizens’ claims are mobilized in evidence and how such evidence is utilized in public policy settings. We conceive the panel as a forum to exchange comparative research experience and case studies about this issue, and welcome contributions relating to different policy domains, institutional settings and countries. In the panel, we approach ‘evidence’ in a comprehensive way: papers can focus on any kind of evidence, such as policy evaluations, research reports, etc.

 

Research questions can tackle any aspect of the ‘evidence lifecycle’ and can relate to the agenda-setting of involving citizens in evidence generation; the actual methods to involve citizens in evidence creation; and the (mis)use and influence of citizen input in evidence generation. Potential questions of interest can include (but are not limited to):

 

1. Which institutional settings foster or impede the involvement of citizens in evidence generation? Do we see differences across administrative cultures, or across policy fields?

2. How can scholars institutionalize democratic processes in policy evaluations and research so that evidence is based on a valid and reliable judgement of public policies with evaluation and research methods that rely on citizen input?

3. What are the potential and limitations of evidence that relies on citizen input for public policy making? To what extent can citizen-based evidence approaches overcome the technocracy-democracy tension that characterizes the evaluation field and can inclusive evidence approaches serve accountability and policy learning purposes?

CALL FOR PAPERS

This panel gathers theoretical, conceptual, and empirical contributions that deal with the relationship between citizens, evidence and public policy, with the aim to enhance our understanding of how citizens’ claims are mobilized in evidence and how such evidence is utilized in public policy settings. We conceive the panel as a forum to exchange comparative research experience and case studies about this issue, and welcome contributions relating to different policy domains, institutional settings and countries.

 

In the panel, we approach ‘evidence’ in a comprehensive way: papers can focus on any kind of evidence, such as policy evaluations, research reports, etc. Research questions can tackle any aspect of the ‘evidence lifecycle’ and can relate to the agenda-setting of involving citizens in evidence generation; the actual methods to involve citizens in evidence creation; and the (mis)use and influence of citizen input in evidence generation.

 

Preference will be given to papers that systematically study the role and influence of using democratic approaches to evidence creation, and the eventual influence of such evidence on public policies; rather than applied evaluation or research studies.

 

Abstracts should include a clear puzzle, research question, outline of the approach, research design, and main contribution of the proposed paper.

ACCEPTED
Format
01 - Systematic Review of Mathematical Models of Citizens’ Deliberative Participation in Public Policy: Proposal of Mixed Deliberation Groups
Ryota Sakai (Chuogakuin University)
CO-AUTHORS
—
ABSTRACT

[Summary]

This study proposes that deliberation among mixed participants (i.e., citizens and experts) helps generate epistemically superior policy assessments. This study is based on a systematic review of mathematical deliberation models, and uses epistemic reason to argue that interpretations of evidence should not be left solely to experts; rather, these should be open to citizens with diverse perspectives. We propose that citizens and evidence-based policy make “the perfect couple.”

 

[Literature]

For decades, citizens have deliberatively participated in public policy, concurrent with academic research on democratic innovations (Bächtiger et al., 2018). From an epistemic perspective, political theorists have recently supported citizens’ deliberative participation in public policy; relatedly, political theorists such as Landemore (2013) cite the deductive results of mathematical and computational models.

    One of the most commonly cited models is the diversity trumps ability (DTA) theorem, which suggests that diverse groups of laypersons can outperform homogeneous groups of experts in terms of problem-solving (Hong and Page, 2004; Landemore, 2013; Gaus, 2016). Potentially, the theorem theoretically supports citizen-based evidence approaches.

 

[Puzzle]

However, concerns vis-à-vis the accuracy and applicability of the DTA theorem have been widely publicized (Estlund, 2008; Vermeule, 2009; Brennan, 2014, 2016; Thompson, 2014; Weymark, 2015; Holman et al., 2018). Although the DTA theorem has been cited more than 900 times in Google Scholar—and a number of related models have been proposed—many critics condemn the theorem’s use, given that the common conditions and results of the theorem and related models have not been systematically investigated.

 

[Research question]

What are the robust conditions of the epistemic benefits of citizens’ deliberation? Little of the literature systematically assesses the common conditions and results of the theorem and related models. Moreover, the epistemically appropriate composition of participants in deliberating in various policy categories has not been fully investigated.

 

[Approach taken]

In the context of science philosophy, robustness analysis has been proposed as a means of systematically reviewing mathematical models (Wimsatt, 2007; Weisberg, 2013). Robustness analysis has already been applied to economics, global warming, technology, and biology models. This supports the use of robustness analysis to study mathematical models of citizens’ deliberation.

 

[Research design]

This study applies across-model robustness analysis (Lisciandra, 2017) to 16 researcher-proposed mathematical models of citizens’ deliberative participation. Following the PRISMA systematic review protocol, this study shortlists 16 DTA-related model analyses from the 352 studies published between 2004 and 2018 that cite Hong and Page’s (2004) original DTA article, as found on the Web of Science database; it then closely examines each.

 

[Main contribution]

Based on this analysis and the policy uncertainty index of Baker et al. (2016), this study suggests that diverse citizens can outperform experts only with respect to highly unpredictable policy issues (i.e., economic policy containing multiple factors). For moderately predictable policies—which comprise most policies—results show that the deliberation of neither 100% laypersons (which is common in mini-publics) nor 100% experts are epistemically desirable. Our systematic results suggest that for modestly predictable policy issues (e.g., taxes), the best layperson–expert ratio within a deliberation group is 60:40; for highly predictable policy issues (e.g., financial regulation), reliance on experts and a small number of laypersons is optimal. These results differ from those of the original DTA model analysis, which suggest that a 100% laypersons group will outperform a 100% experts group.

    Although the robustness analysis results need to be confirmed through empirical investigation, this study offers theoretical predictions concerning citizens’ deliberative participation, and brings model analysis one step closer to real-world policy recommendations.

ACCEPTED
Format
02 - Evaluation and Citizens: Reflections on a Future Research Agenda
Pirmin Bundi (University of Lausanne)
CO-AUTHORS
Valerie Pattyn (Institute of Public Administration - Leiden University)
ABSTRACT

The very concept of evidence is commonly associated with knowledge produced by elites (such as experts, academics, or civil servants). Involving citizens in the production of evidence creation is rather rare. This is surprising as citizens are often the main beneficiary group of public policy and their beliefs and opinions define the political framework of public measures.

 

In the evaluation field, however, we can discern several approaches to evidence generation, which pay considerable attention to citizens. Democratic Evaluation, for instance, is a kind of information service that is provided to society with the aim of informing them about the characteristics of the programme being evaluated. In such approach, the evaluator takes the role of a broker in the exchange of information between various stakeholders and enables all actors to verify the use of their statements in the final report (MacDonald 1993). Transformative Participatory Evaluation, to name a second approach, particularly aims to empower the traditionally less influential segments of society (Cousins and Whitmore 1998).  Deliberative Democratic Evaluation has similar ambitions, as it relies on the assumption that evaluations can contribute to democracy if they allow the voices of the disadvantaged to flow appropriately into the evaluation process (House/Howe 1999).

 

Despite the existence of such frameworks, the actual involvement of citizens is still rather exception than rule, also in the evaluation field (Halachmi & Holzer 2010). The question is how this can be overcome, especially in view of stronger calls for citizen participation in the post-truth area. In this paper, we address this challenge, by presenting a systematic review of the role of citizens in different evaluation models. On this basis, we point out several avenues of future research in studying the relationship between evaluation and citizens.

ACCEPTED
Format
03 - Legislating for deliberation: Drivers, policies and prospects for the implementation of deliberative practices in local government by the State Government in Victoria, Australia.
Emanuela Savini (University of Technology Sydney - Institute for Public Policy and Governance)
CO-AUTHORS
—
ABSTRACT

Growing dissatisfaction with representative democracy, and a perceived ‘democracy deficit’, have seen government agencies increasingly required to relinquish control from the otherwise technocratic development of policy. Participatory forms of policy-making, such as deliberative practices like citizen juries and co-design models, are seen as a way of addressing distrust in elected representatives and building social capital.

 

In line with this, the proposed reforms to the Victorian Local Government Act signal an intention to embed participatory governance practice across the sector in Victoria. State Government (DELWP 2016: 60) has asserted ‘As the level of government closest to the people, councils have both the opportunity and arguably the responsibility to enable participatory democracy’. Accordingly, the first draft of the Act proposed that Councils undertake ‘deliberative community engagement processes before adopting four-year Council Plan and Budgets’ (DELWP 2017: 4). Whilst this was later changed to ‘in collaboration with’, rather than specifically prescribing deliberative practice, it nonetheless has signalled an expectation of community engaged practice to a larger degree than has been expected previously (for an account, see Grant and Drew 2017).

 

This paper explores what factors influenced this clear shift to deliberative practice by the State Government in Victoria, Australia. It will identify which actors, and what evidence was used to underpin this legislative change, as well as uncover how much citizen participation influenced these new expectations. At the time of revising the Local Government Act, a Victorian Auditor General’s Office report (VAGO 2017: viii) found that whilst councils typically had strong public participation frameworks, they did not have adequate processes for documenting and evaluating these activities. As such, the paper explores what other forms of evidence and evaluation were utilised to inform the legislative changes. It analyses reports and submissions and academic literature that supported this shift and findings from case studies that are likely to have influenced the draft legislation. Finally, given that ‘empowered participation may demand unrealistically high levels of popular commitment, especially in contemporary climates of civic and political disengagement’ (Fung & Wright 2003:33), the paper researches the level of involvement citizens had in shaping the legislation.

 

 

References:

 

Department of Environment, Land, Water and Planning (DELWP) (2016). Act for the future - Directions for a new Local Government Act https://s3-ap-southeast-2.amazonaws.com/ehq-production-australia/7aa05ea50976225e7ee45681e4520f5cdfa2e355/documents/attachments/000/037/297/original/Act_for_the_Future_-_Directions_for_a_new_Local_Government_Act.pdf?1465442287 accessed 28 August 2018

 

Department of Environment, Land, Water and Planning (DELWP) (2018). Local Government Bill Exposure Draft https://www.localgovernment.vic.gov.au/__data/assets/pdf_file/0022/91138/Local-Government-Bill-Exposure-Draft.pdf accessed 28 August 2018

 

Fung, A., & Wright, E. O. (2003). Deepening democracy: Institutional innovations in empowered participatory governance (Vol. 4). Verso.

 

Grant, B. and Drew, J. (2017). Local Government in Australia: History, Theory and Public Policy. Singapore: Springer.

 

Local Government Victoria (LGV) (2018) Local Government Act Review https://www.localgovernment.vic.gov.au/our-programs/local-government-act-review Accessed 23 January 2019.

 

Victoria Auditor-General’s Office (2017). Public Participation and Community Engagement: Local Government Sector Report May 2017, https://www.audit.vic.gov.au/sites/default/files/20170510-PP-Local-Gov.pdf accessed 2 October 2018

ACCEPTED
Format
04 - Public hearing testimonies as policy evidence and its influence over policy-makers: the case of Single-Payer issue in NYS
Yongjin Choi (London School of Hygiene & Tropical Medicine, University of London)
CO-AUTHORS
Ashley Fox (Rockefeller College, University at Albany)
ABSTRACT

Public hearings have faced criticism when subjected to scrutiny in the political science and public policy literatures that they have no significant influence over policy decisions, yet they remain a frequently used method for direct citizen participation in practice, especially in public health policy-making requiring the both of scientific knowledge and public opinion. This irony, the persistent use of a seemingly useless method, has been explained in the literature as resulting from either mandatory provisions that require public involvement or from deceptive strategies of policy-makers aimed at legitimizing foregone policy decisions, especially in advancing single-payer health coverage, a simultaneously populist and technocratic policy issue. However, this explanation may stand only with a strong assumption that all policy-makers are self-interested, failing to reveal the influence of the citizens’ inputs over policy-makers, no matter how such information contributes to producing democratic outcomes or not.

 

In this study, we develop a set of categories enabling us to analyze the evidence in citizens’ testimonies for public health policy from the perspective of Evidence-Based Policy and suggest a new perspective on the effectiveness of public hearings, by exploring the content of hearings on single-payer healthcare in New York State in 2014. We argue that although public opinion and information that flow into the policy process through public hearings have defects as policy evidence, it can affect decision-making and discourses by leading policy-makers to consider public interests before and after public hearings more than they would when there is no relevant hearing. This finding is in part consistent with the argument that public hearings have effects over policy-making as a ritual and a deliberative form. However, at the same time, it alerts government decision-makers to be vigilant on biases and populism caused by specious and overstated arguments. From this point of view, academic debates about public hearings should deal with scrutinizing the information that flows from citizens to policy-makers and designing better hearings allowing more useful evidence with fewer biases coming from public hearings.

 

This study involved the two following tasks. First, by using public hearing transcripts, we assess the invited speakers’ testimonies from the perspective of evidence-based policy assessing the types and quality of evidence presented at public hearings to consider their utility in informing the policy process. Second, from the policy-makers’ point of view, we look at how policy-makers recognize and use public hearings and those testimonies in the process of drafting legislation.

CO-AUTHORS
—
ABSTRACT

[Summary]

This study proposes that deliberation among mixed participants (i.e., citizens and experts) helps generate epistemically superior policy assessments. This study is based on a systematic review of mathematical deliberation models, and uses epistemic reason to argue that interpretations of evidence should not be left solely to experts; rather, these should be open to citizens with diverse perspectives. We propose that citizens and evidence-based policy make “the perfect couple.”

 

[Literature]

For decades, citizens have deliberatively participated in public policy, concurrent with academic research on democratic innovations (Bächtiger et al., 2018). From an epistemic perspective, political theorists have recently supported citizens’ deliberative participation in public policy; relatedly, political theorists such as Landemore (2013) cite the deductive results of mathematical and computational models.

    One of the most commonly cited models is the diversity trumps ability (DTA) theorem, which suggests that diverse groups of laypersons can outperform homogeneous groups of experts in terms of problem-solving (Hong and Page, 2004; Landemore, 2013; Gaus, 2016). Potentially, the theorem theoretically supports citizen-based evidence approaches.

 

[Puzzle]

However, concerns vis-à-vis the accuracy and applicability of the DTA theorem have been widely publicized (Estlund, 2008; Vermeule, 2009; Brennan, 2014, 2016; Thompson, 2014; Weymark, 2015; Holman et al., 2018). Although the DTA theorem has been cited more than 900 times in Google Scholar—and a number of related models have been proposed—many critics condemn the theorem’s use, given that the common conditions and results of the theorem and related models have not been systematically investigated.

 

[Research question]

What are the robust conditions of the epistemic benefits of citizens’ deliberation? Little of the literature systematically assesses the common conditions and results of the theorem and related models. Moreover, the epistemically appropriate composition of participants in deliberating in various policy categories has not been fully investigated.

 

[Approach taken]

In the context of science philosophy, robustness analysis has been proposed as a means of systematically reviewing mathematical models (Wimsatt, 2007; Weisberg, 2013). Robustness analysis has already been applied to economics, global warming, technology, and biology models. This supports the use of robustness analysis to study mathematical models of citizens’ deliberation.

 

[Research design]

This study applies across-model robustness analysis (Lisciandra, 2017) to 16 researcher-proposed mathematical models of citizens’ deliberative participation. Following the PRISMA systematic review protocol, this study shortlists 16 DTA-related model analyses from the 352 studies published between 2004 and 2018 that cite Hong and Page’s (2004) original DTA article, as found on the Web of Science database; it then closely examines each.

 

[Main contribution]

Based on this analysis and the policy uncertainty index of Baker et al. (2016), this study suggests that diverse citizens can outperform experts only with respect to highly unpredictable policy issues (i.e., economic policy containing multiple factors). For moderately predictable policies—which comprise most policies—results show that the deliberation of neither 100% laypersons (which is common in mini-publics) nor 100% experts are epistemically desirable. Our systematic results suggest that for modestly predictable policy issues (e.g., taxes), the best layperson–expert ratio within a deliberation group is 60:40; for highly predictable policy issues (e.g., financial regulation), reliance on experts and a small number of laypersons is optimal. These results differ from those of the original DTA model analysis, which suggest that a 100% laypersons group will outperform a 100% experts group.

    Although the robustness analysis results need to be confirmed through empirical investigation, this study offers theoretical predictions concerning citizens’ deliberative participation, and brings model analysis one step closer to real-world policy recommendations.

CO-AUTHORS
Valerie Pattyn (Institute of Public Administration - Leiden University)
ABSTRACT

The very concept of evidence is commonly associated with knowledge produced by elites (such as experts, academics, or civil servants). Involving citizens in the production of evidence creation is rather rare. This is surprising as citizens are often the main beneficiary group of public policy and their beliefs and opinions define the political framework of public measures.

 

In the evaluation field, however, we can discern several approaches to evidence generation, which pay considerable attention to citizens. Democratic Evaluation, for instance, is a kind of information service that is provided to society with the aim of informing them about the characteristics of the programme being evaluated. In such approach, the evaluator takes the role of a broker in the exchange of information between various stakeholders and enables all actors to verify the use of their statements in the final report (MacDonald 1993). Transformative Participatory Evaluation, to name a second approach, particularly aims to empower the traditionally less influential segments of society (Cousins and Whitmore 1998).  Deliberative Democratic Evaluation has similar ambitions, as it relies on the assumption that evaluations can contribute to democracy if they allow the voices of the disadvantaged to flow appropriately into the evaluation process (House/Howe 1999).

 

Despite the existence of such frameworks, the actual involvement of citizens is still rather exception than rule, also in the evaluation field (Halachmi & Holzer 2010). The question is how this can be overcome, especially in view of stronger calls for citizen participation in the post-truth area. In this paper, we address this challenge, by presenting a systematic review of the role of citizens in different evaluation models. On this basis, we point out several avenues of future research in studying the relationship between evaluation and citizens.

CO-AUTHORS
—
ABSTRACT

Growing dissatisfaction with representative democracy, and a perceived ‘democracy deficit’, have seen government agencies increasingly required to relinquish control from the otherwise technocratic development of policy. Participatory forms of policy-making, such as deliberative practices like citizen juries and co-design models, are seen as a way of addressing distrust in elected representatives and building social capital.

 

In line with this, the proposed reforms to the Victorian Local Government Act signal an intention to embed participatory governance practice across the sector in Victoria. State Government (DELWP 2016: 60) has asserted ‘As the level of government closest to the people, councils have both the opportunity and arguably the responsibility to enable participatory democracy’. Accordingly, the first draft of the Act proposed that Councils undertake ‘deliberative community engagement processes before adopting four-year Council Plan and Budgets’ (DELWP 2017: 4). Whilst this was later changed to ‘in collaboration with’, rather than specifically prescribing deliberative practice, it nonetheless has signalled an expectation of community engaged practice to a larger degree than has been expected previously (for an account, see Grant and Drew 2017).

 

This paper explores what factors influenced this clear shift to deliberative practice by the State Government in Victoria, Australia. It will identify which actors, and what evidence was used to underpin this legislative change, as well as uncover how much citizen participation influenced these new expectations. At the time of revising the Local Government Act, a Victorian Auditor General’s Office report (VAGO 2017: viii) found that whilst councils typically had strong public participation frameworks, they did not have adequate processes for documenting and evaluating these activities. As such, the paper explores what other forms of evidence and evaluation were utilised to inform the legislative changes. It analyses reports and submissions and academic literature that supported this shift and findings from case studies that are likely to have influenced the draft legislation. Finally, given that ‘empowered participation may demand unrealistically high levels of popular commitment, especially in contemporary climates of civic and political disengagement’ (Fung & Wright 2003:33), the paper researches the level of involvement citizens had in shaping the legislation.

 

 

References:

 

Department of Environment, Land, Water and Planning (DELWP) (2016). Act for the future - Directions for a new Local Government Act https://s3-ap-southeast-2.amazonaws.com/ehq-production-australia/7aa05ea50976225e7ee45681e4520f5cdfa2e355/documents/attachments/000/037/297/original/Act_for_the_Future_-_Directions_for_a_new_Local_Government_Act.pdf?1465442287 accessed 28 August 2018

 

Department of Environment, Land, Water and Planning (DELWP) (2018). Local Government Bill Exposure Draft https://www.localgovernment.vic.gov.au/__data/assets/pdf_file/0022/91138/Local-Government-Bill-Exposure-Draft.pdf accessed 28 August 2018

 

Fung, A., & Wright, E. O. (2003). Deepening democracy: Institutional innovations in empowered participatory governance (Vol. 4). Verso.

 

Grant, B. and Drew, J. (2017). Local Government in Australia: History, Theory and Public Policy. Singapore: Springer.

 

Local Government Victoria (LGV) (2018) Local Government Act Review https://www.localgovernment.vic.gov.au/our-programs/local-government-act-review Accessed 23 January 2019.

 

Victoria Auditor-General’s Office (2017). Public Participation and Community Engagement: Local Government Sector Report May 2017, https://www.audit.vic.gov.au/sites/default/files/20170510-PP-Local-Gov.pdf accessed 2 October 2018

CO-AUTHORS
Ashley Fox (Rockefeller College, University at Albany)
ABSTRACT

Public hearings have faced criticism when subjected to scrutiny in the political science and public policy literatures that they have no significant influence over policy decisions, yet they remain a frequently used method for direct citizen participation in practice, especially in public health policy-making requiring the both of scientific knowledge and public opinion. This irony, the persistent use of a seemingly useless method, has been explained in the literature as resulting from either mandatory provisions that require public involvement or from deceptive strategies of policy-makers aimed at legitimizing foregone policy decisions, especially in advancing single-payer health coverage, a simultaneously populist and technocratic policy issue. However, this explanation may stand only with a strong assumption that all policy-makers are self-interested, failing to reveal the influence of the citizens’ inputs over policy-makers, no matter how such information contributes to producing democratic outcomes or not.

 

In this study, we develop a set of categories enabling us to analyze the evidence in citizens’ testimonies for public health policy from the perspective of Evidence-Based Policy and suggest a new perspective on the effectiveness of public hearings, by exploring the content of hearings on single-payer healthcare in New York State in 2014. We argue that although public opinion and information that flow into the policy process through public hearings have defects as policy evidence, it can affect decision-making and discourses by leading policy-makers to consider public interests before and after public hearings more than they would when there is no relevant hearing. This finding is in part consistent with the argument that public hearings have effects over policy-making as a ritual and a deliberative form. However, at the same time, it alerts government decision-makers to be vigilant on biases and populism caused by specious and overstated arguments. From this point of view, academic debates about public hearings should deal with scrutinizing the information that flows from citizens to policy-makers and designing better hearings allowing more useful evidence with fewer biases coming from public hearings.

 

This study involved the two following tasks. First, by using public hearing transcripts, we assess the invited speakers’ testimonies from the perspective of evidence-based policy assessing the types and quality of evidence presented at public hearings to consider their utility in informing the policy process. Second, from the policy-makers’ point of view, we look at how policy-makers recognize and use public hearings and those testimonies in the process of drafting legislation.