evaluation

More Criteria for Evaluating Participatory Processes

Jason Diceman — Wed, 19 Sep 2007 15:55:55 -0500

I found this “Practical Guide for Evaluating Participatory Processes” (PDF) by the International Observatory of Participatory Democracy (OIDP is the Spanish acronym). I'm not that impressed in the ambiguity of the criteria, the redundancy between criteria and the lack of reference to foundational research. Instead I would highly recommend “Public Participation Methods: A Framework for Evaluation” by Gene Rowe and Lynn J. Frewer, whose criteria I summarized in a previous blog post.

The OIDP guide was written for municipal governance, but can be applied to other levels of government and community organizations. I believe the original document was written in Spanish and the meaning of some terms may have been a bit confused in the English translation I read.

Below I copy the criteria titles directly from the guide, although the definitions were rewritten based on my interpretations of their descriptions.

A. Criteria related to the process coordination

Consensus: Agreement from political groups, social groups and government related technicians on the need and methodology of the process.
Transversality: Participation in planning and implementation from relevant political and technical bodies.
Initiative and leadership: Government leadership is required for a institutionally valid process. A diverse group of stakeholders and technicians should also be part of the leadership.
Integration in the [government] participatory system: The process should be within an established participatory system or at least be consistent and complimentary to any established participatory system.
Clarity of the objectives: Establish clear expectations of results and limits to their scope. Fulfilment of the objectives.
Planning and resources: Confirm a detailed plan including sensible objectives, stages, schedules, and sufficient provisioning of economic and personnel resources.

B. Participant-related criteria

Number of participants: Percentage of participants in relation to the reference population. Percentage of organized players versus the reference total.
Diversity: compare the demographics of those participating in the process, with the demographics of those in the reference society. Consideration for social groups that are usually under-represented (e.g. women, immigrants, disabled, low-income). Compare the profiles of the participating organizations (e.g. neighbourhood, corporate, union, cultural, sports, political) to the theme of process. Look for key stakeholders, players and organizations that were not included but should have been.
Representativity of the participants (specific to organization representatives): Where organization representatives play a role, what is the democratic quality of how they recognize and present the views of their constituents. E.g. information flow to and from representatives and their constituents, use of elections, consistency between the organizations' formal statements and that of the representative.
Openness of the process: The public is invited to participate and take part in the decision-making process.

C. Criteria related to the reason for participation

Relevance: The topic of the process is within the governments agenda and viewed as important and relevant to the citizens. Significance of the potential result on the government budget.
Capacity of intervention by the [government] administration: The topic of the process is within the governments administrative scope.

D. Criteria related to the type of participation

Participatory diagnosis: Start with a diagnosis that defines the main problems and matter for debate. Ideally this diagnosis would be participatory in nature.
Capacity to make proposals: Allows citizens to make proposals.
Level of participation: On an upward scale from Information to Communication, Query, Deliberation, and then Decision.
Quality of the information: How effective the information channels were at distributing key information to prepare and support participants. Plurality, objectivity, clarity and utility of the information provided.
Deliberation methods and techniques: techniques or mechanisms were used to avoid inequalities between participants in the deliberations. Each participant felt empowered to voice their opinion.

E. Criteria related to the consequences of the process:

Substantive results: The results respond to the needs recognized in the initial planning of the process.
Implementing the results: Verify the implementation of tangible results. The existence, or the study to create a body to follow-up on the outcomes.
Result feedback: The participants know the results, validate them and consider the process as ended.
Improvement of relationships among the players: Strengthen relationships among the participants, their organizations and the government administration.
Training: Trained participants in the field of citizen participation.
Building a political participatory culture: Participant satisfaction with the process and the willingness to participate once again.

Again, while you may find these criteria useful, I would recommend using the Rowe and Frewer criteria instead.

Evaluation Criteria for Public Participation Methods

Jason Diceman — Wed, 19 Sep 2007 14:57:38 -0500

In search of a solid frame of reference for comparing various international citizen assembly processes I am investigating, I discovered an excellent paper called “Public Participation Methods: A Framework for Evaluation” by Gene Rowe and Lynn J. Frewer, two Ph.D.s working at the Institute of Food Research in Norwich, United Kingdom. Below I copied and pasted together all nine key criteria defined in the paper and their related suggestions. You can use these criteria and suggestions to help plan and evaluate any stakeholder engagement processes you are involved with, such as members surveys, focus groups, public hearings and citizen juries. I think it is very useful.

The original paper was published in 2000 in the Science, Technology, & Human Values journal and has since been the most read article on the journal's web site. You can download a free copy of the PDF from socialsciences.wur.nl or see the attached here. The full paper is 27 pages long with 75 bibliographic entries. It is definitely a key paper in the emerging canon of research on public participation mothods.

A Summary of the Criteria from “Public Participation Methods: A Framework for Evaluation” by Gene Rowe and Lynn J. Frewer

Acceptance Criteria

Representativeness: The public participants should comprise a broadly representative sample of the population of the affected public.
Suggestions: select a random stratified sample of the affected population; involve the use of questionnaires to determine the spread of attitudes with regard to a certain issue, using this as a basis for the proportionate selection of members.
Independence: The participation process should be conducted in an independent, unbiased way.
Suggestions: steering committee or management team incorporates members from diverse bodies or neutral organizations, such as university academics; disclosure from participants of any relationship to the sponsoring body ; rhe use of a respected facilitator.
Early [public] involvement: The public should be involved as early as possible in the process as soon as value judgements become salient.
Suggestions: Public debate should thus be allowed on underlying assumptions and agenda setting and not just on narrow, predefined problems
Influence: The output of the procedure should have a genuine impact on policy.
Suggestions: ensure that there is a clear acceptance beforehand as to how the output will be used and how it might direct policy; use of the media to inform the general public about the specific ways in which the output has influenced policy.
Transparency: The process should be transparent so that the public can see what is going on and how decisions are being made.
Suggestions: releasing information on aspects of the procedure, varying from the manner of the selection of the public participants to the way in which a decision is reached to the minutes of meeting; if any information needs to be withheld from the public, for reasons of sensitivity or security, ... admit the nature of what is being withheld and why.

Process Criteria

Resource accessibility: Public participants should have access to the appropriate resources to enable them to successfully fulfil their brief.
Suggestions: information resources (summaries of the pertinent facts); human resources (e.g., access to scientists, witnesses, decision analysts); material resources (e.g., overhead projectors/whiteboards); and time resources (participants should have sufficient time to make decisions).
Task definition: The nature and scope of the participation task should be clearly defined.
Suggestions: [clearly define at the outset] the scope of a participation exercise, its expected output, and the mechanisms of the procedure.
Structured decision making: The participation exercise should use/provide appropriate mechanisms for structuring and displaying the decision-making process.
Suggestions: A variety of decision-aiding tools might be incorporated into a participation procedure, such as decision analysis, decision trees, multiattribute utility theory, and the Delphi technique; structure the decision process in groups; important to structure the decision process in [small] groups; ian ndependent decision analyst could be usefully involved; use of an [experienced] group facilitator to employ rules for effective group decision making.
Cost-effectiveness: The procedure should in some sense be cost-effective.
Suggestions: take account of the potential costs of the alternative methods, in both time and money, and to consider the extent to which they fulfill the other criteria.

If I find other evaluation frameworks I will post them to this blog.