Evaluation of a Stakeholder Workshop

I suggest the following key goal based criteria:

1. Representative
2. Knowledgeable
3. Shared Understanding
4. Fair and Effective

To measure these criteria, the following questions will be asked through participant and observer surveys at key points during and/or after each Working Group meeting:

1. Representative
Looking at the target versus participant stakeholder profiles, were the participants a fair representation of the affected stakeholders?

2. Knowledgeable
Did the participants comprehend the most important information to make knowledgeable recommendations on this topic?

3. Shared Understanding
Did participants understand the perspectives of other stakeholders?

4. Fair and Effective
Was the meeting process fair and effective at providing equitable opportunity for all in scope ideas and opinions to be considered and recorded?

Open-ended “Why” questions will also be included to gather insight into the reasoning for the answers provided. Results will be analyzed to inform the preparation of the next Working Group meeting.

Results of these survey questions will be published as part of each Working Group meeting documentation.

July 19, 2016

Did the consultation represent the community?

white crowd In public consultation, we often say we “consulted with the community”, but did we?

Any self-selecting public consultation process (e.g. online or paper surveys, public meetings) will not be statistically representative of the community. To be reliably representative, participants would need to be selected randomly (aka “sortition”) to avoid self-selection bias. Self selecting participants are more likely to be advocates with stronger opinions compared to the average resident.

That said, notwithstanding the self-selection bias, how representative a public survey is can be judged, in part, by comparing how well the demographic profile of the respondents matches the known demographics of a community, as recognized by the regional census, for example, looking at the gender and age of participants.

We can also look at the address of the participants – were they from the local community?

Specific to each consultation topic, we should also consider representation of stakeholder types. For example a consultation about a park should include all types of park users, including seniors, parents with children, organized sport players, people with and without dogs, etc. In the case of roadway reconfigurations, how people travel (walk, bike, bus, or drive) is a key stakeholder type to consider.

And importantly, is the rate of participation large enough to be significant. A survey of 50 people is likely less representative than one of 500 people.

September 19, 2007

More Criteria for Evaluating Participatory Processes

I found this “Practical Guide for Evaluating Participatory Processes” (PDF) by the International Observatory of Participatory Democracy (OIDP is the Spanish acronym). I’m not that impressed in the ambiguity of the criteria, the redundancy between criteria and the lack of reference to foundational research. Instead I would highly recommend “Public Participation Methods: A Framework for Evaluation” by Gene Rowe and Lynn J. Frewer, whose criteria I summarized in a previous blog post.

The OIDP guide was written for municipal governance, but can be applied to other levels of government and community organizations. I believe the original document was written in Spanish and the meaning of some terms may have been a bit confused in the English translation I read.

Below I copy the criteria titles directly from the guide, although the definitions were rewritten based on my interpretations of their descriptions.

A. Criteria related to the process coordination

Consensus: Agreement from political groups, social groups and government related technicians on the need and methodology of the process.
Transversality: Participation in planning and implementation from relevant political and technical bodies.
Initiative and leadership: Government leadership is required for a institutionally valid process. A diverse group of stakeholders and technicians should also be part of the leadership.
Integration in the [government] participatory system: The process should be within an established participatory system or at least be consistent and complimentary to any established participatory system.
Clarity of the objectives: Establish clear expectations of results and limits to their scope. Fulfilment of the objectives.
Planning and resources: Confirm a detailed plan including sensible objectives, stages, schedules, and sufficient provisioning of economic and personnel resources.

B. Participant-related criteria

Number of participants: Percentage of participants in relation to the reference population. Percentage of organized players versus the reference total.
Diversity: compare the demographics of those participating in the process, with the demographics of those in the reference society. Consideration for social groups that are usually under-represented (e.g. women, immigrants, disabled, low-income). Compare the profiles of the participating organizations (e.g. neighbourhood, corporate, union, cultural, sports, political) to the theme of process. Look for key stakeholders, players and organizations that were not included but should have been.
Representativity of the participants (specific to organization representatives): Where organization representatives play a role, what is the democratic quality of how they recognize and present the views of their constituents. E.g. information flow to and from representatives and their constituents, use of elections, consistency between the organizations’ formal statements and that of the representative.
Openness of the process: The public is invited to participate and take part in the decision-making process.

C. Criteria related to the reason for participation

Relevance: The topic of the process is within the governments agenda and viewed as important and relevant to the citizens. Significance of the potential result on the government budget.
Capacity of intervention by the [government] administration: The topic of the process is within the governments administrative scope.

D. Criteria related to the type of participation

Participatory diagnosis: Start with a diagnosis that defines the main problems and matter for debate. Ideally this diagnosis would be participatory in nature.
Capacity to make proposals: Allows citizens to make proposals.
Level of participation: On an upward scale from Information to Communication, Query, Deliberation, and then Decision.
Quality of the information: How effective the information channels were at distributing key information to prepare and support participants. Plurality, objectivity, clarity and utility of the information provided.
Deliberation methods and techniques: techniques or mechanisms were used to avoid inequalities between participants in the deliberations. Each participant felt empowered to voice their opinion.

E. Criteria related to the consequences of the process:

Substantive results: The results respond to the needs recognized in the initial planning of the process.
Implementing the results: Verify the implementation of tangible results. The existence, or the study to create a body to follow-up on the outcomes.
Result feedback: The participants know the results, validate them and consider the process as ended.
Improvement of relationships among the players: Strengthen relationships among the participants, their organizations and the government administration.
Training: Trained participants in the field of citizen participation.
Building a political participatory culture: Participant satisfaction with the process and the willingness to participate once again.

—

Again, while you may find these criteria useful, I would recommend using the Rowe and Frewer criteria instead.

September 19, 2007

Evaluation Criteria for Public Participation Methods

In search of a solid frame of reference for comparing various international citizen assembly processes I am investigating, I discovered an excellent paper called “Public Participation Methods: A Framework for Evaluation” by Gene Rowe and Lynn J. Frewer, two Ph.D.s working at the Institute of Food Research in Norwich, United Kingdom. Below I copied and pasted together all nine key criteria defined in the paper and their related suggestions. You can use these criteria and suggestions to help plan and evaluate any stakeholder engagement processes you are involved with, such as members surveys, focus groups, public hearings and citizen juries. I think it is very useful.

The original paper was published in 2000 in the Science, Technology, & Human Values journal and has since been the most read article on the journal’s web site. You can download a free copy of the PDF from socialsciences.wur.nl or see the attached here. The full paper is 27 pages long with 75 bibliographic entries. It is definitely a key paper in the emerging canon of research on public participation mothods.

A Summary of the Criteria from “Public Participation Methods: A Framework for Evaluation” by Gene Rowe and Lynn J. Frewer

Acceptance Criteria

Representativeness: The public participants should comprise a broadly representative sample of the population of the affected public.
Suggestions: select a random stratified sample of the affected population; involve the use of questionnaires to determine the spread of attitudes with regard to a certain issue, using this as a basis for the proportionate selection of members.
Independence: The participation process should be conducted in an independent, unbiased way.
Suggestions: steering committee or management team incorporates members from diverse bodies or neutral organizations, such as university academics; disclosure from participants of any relationship to the sponsoring body ; rhe use of a respected facilitator.
Early [public] involvement: The public should be involved as early as possible in the process as soon as value judgements become salient.
Suggestions: Public debate should thus be allowed on underlying assumptions and agenda setting and not just on narrow, predefined problems
Influence: The output of the procedure should have a genuine impact on policy.
Suggestions: ensure that there is a clear acceptance beforehand as to how the output will be used and how it might direct policy; use of the media to inform the general public about the specific ways in which the output has influenced policy.
Transparency: The process should be transparent so that the public can see what is going on and how decisions are being made.
Suggestions: releasing information on aspects of the procedure, varying from the manner of the selection of the public participants to the way in which a decision is reached to the minutes of meeting; if any information needs to be withheld from the public, for reasons of sensitivity or security, … admit the nature of what is being withheld and why.

Process Criteria

Resource accessibility: Public participants should have access to the appropriate resources to enable them to successfully fulfil their brief.
Suggestions: information resources (summaries of the pertinent facts); human resources (e.g., access to scientists, witnesses, decision analysts); material resources (e.g., overhead projectors/whiteboards); and time resources (participants should have sufficient time to make decisions).
Task definition: The nature and scope of the participation task should be clearly defined.
Suggestions: [clearly define at the outset] the scope of a participation exercise, its expected output, and the mechanisms of the procedure.
Structured decision making: The participation exercise should use/provide appropriate mechanisms for structuring and displaying the decision-making process.
Suggestions: A variety of decision-aiding tools might be incorporated into a participation procedure, such as decision analysis, decision trees, multiattribute utility theory, and the Delphi technique; structure the decision process in groups; important to structure the decision process in [small] groups; ian ndependent decision analyst could be usefully involved; use of an [experienced] group facilitator to employ rules for effective group decision making.
Cost-effectiveness: The procedure should in some sense be cost-effective.
Suggestions: take account of the potential costs of the alternative methods, in both time and money, and to consider the extent to which they fulfill the other criteria.

—

If I find other evaluation frameworks I will post them to this blog.

Attachment	Size
Rowe frewer public engagement.pdf	79.99 KB