Making Connections: Community Organizing, Empowerment Planning, and Participatory Research in Participatory Evaluation
Randy Stoecker
e-mail: rstoecker@wisc.edu
Prepublication draft of article published in Sociological Practice, 1999.
ABSTRACT
This paper shows the intersections of participatory research, popular education, empowerment planning, and community organizing with participatory evaluation. It argues that a truly successful participatory evaluation involves participants in guiding and even conducting the research, doing a process of self and program study, creating plans for change, and organizing themselves for implementing those plans. Next, the paper shows how these elements played out in a participatory evaluation of a community organizing training and technical assistance project in Toledo Ohio. The first year of the project was facilitated by participatory evaluation that helped identify early successes and problems so participants could make programmatic changes early in the process. The telling of the story also develops practices of participatory evaluation, including planning the evaluation, doing the research and adapting it to changing conditions, uncovering creative tensions, participatory validity checking, and linking the process to planning and action. The paper concludes with some lessons for participatory evaluation practice.
I learn most not from ideas developed within a field, but from the intersections and overlaps of schools of thought. And so it is with this new field of "empowerment" or "participatory" evaluation. There has recently been an explosion of interest in an evaluation methodology that is more participatory, more democratic, and more useful (Patton, 1997; Fetterman et al., 1996; 1997; Estrella and Gaventa, 1997, Park and Williams, forthcoming). Surprisingly, however, the proponents of this new evaluation method have been working in intellectual isolation from other frameworks that could help them better place, justify, and develop their practice. As a consequence, included but unexplored within the framework of empowerment or participatory evaluation are community-based planning, community organizing, participatory research, and popular education.
So while analysts like David Fetterman (1996b) assert that a participatory form of evaluation has developed as a solid model, others are not convinced (Patton, 1997b; Andrews, 1996). The conceptual and practical connections between this new evaluation method and its associated practices are still at best only implicit. Fetterman (1996), for example, cites the roots of "empowerment evaluation" as community psychology and action anthropology. Yet, his evaluation model is so close to the participatory research tradition to be difficult to distinguish from it. In addition, the overlaps with other fields become very clear when looking at how typical empowerment evaluation processes are conceptualized. Fetterman, (1994; 1994b) and Andrews (1996) see the steps as: assessing needs and resources; setting mission and objectives; developing plans and strategies; monitoring process and outcomes; communicating relevant information to relevant audiences; and promoting adaptation, renewal, and institutionalization. Fetterman (1996) later presents the steps as: taking stock, setting goals, developing strategies, and documenting progress. Stevenson et al (1996) portray the steps as: initial mobilization, establishing an organizational structure, building capacity for action, planning for action, implementation, refinement, and institutionalization. Dugan (1996) lists the steps as: organizing for action, building capacity, taking action, refining action, institutionalizing action.
Without connection to the fields of participatory research, popular education, empowerment planning, and community organizing, empowerment evaluators miss important steps in the total process, and their models seem linear rather than iterative--they all end with institutionalization, rather than feedback into continuous cycles of evaluation, planning, and action. This leads me to fear that the question of how evaluation is used in an iterative process has been neglected. Evaluations that are not designed from the outset with usability in mind will waste the evaluation's potential (Patton, 1997:3-10). Finally, by not connecting to popular education and community organizing practices, Fetterman (1996) can make the mistake of arguing for the evaluator as advocate--a role which ultimately contradicts empowerment, since it involves an expert speaking for a constituency rather than organizing them to speak for themselves (Beckwith and Lopez, 1997). As Gaventa (1993:33-34) charges, "To the extent that the research still remains in the hands of the researcher, a real transfer of ownership of knowledge may not have occurred. The dichotomy between those who produce knowledge and those who are most affected by it still exists."
This paper will show the utility of integrating participatory evaluation with participatory research, community organizing, popular education, and empowerment planning approaches in social change programs. I begin by developing an integrated model for participatory evaluation in the context of social change efforts. Next, I outline the participatory evaluation process for a community organizing mentoring program which develops this model. Finally, I discuss the lessons of this model in practice and its implications for future practice.
I distinguish participatory evaluation from "empowerment evaluation" as presented by Fetterman et al. (1996). Patton (1997:101) and I agree that empowerment evaluation may be biased toward individual empowerment rather than community or organizational empowerment. As Fetterman (1996) notes, the emphasis in empowerment evaluation is on "self-determination" rather than social change. Patton (1997:101) portrays empowerment evaluation as "most appropriate where the goals of the program include helping participants become more self-sufficient and personally effective." In my usage, participatory evaluation also focuses on empowerment, but emphasizes community self-determination as well as individual self-determination.
For Patton (1997:100), the principles of participatory evaluation include: 1) involving participants at every stage of research process; 2) making sure they own the evaluation; 3) focusing the process on the outcomes they think are important; 4) facilitating participants to work collectively; 5) organizing the evaluation to be understandable and meaningful to all; 6) using the evaluation to support participants' accountability to themselves and their community first, and outsiders second if at all; 7) developing the evaluator role as a facilitator, collaborator, and learning resource; 8) developing participants' roles as decision-makers and evaluators; 9) recognizing and valuing participants' expertise and helping them to do the same; and 10) minimizing status differences between the evaluation facilitator and participants.
Within this model there is allowance for variation--in how involved the evaluation facilitator is in doing the work; in whether the evaluation facilitator is an outsider, and in what specific tasks participants take on. I will address these issues later on. It is important here to emphasize that this notion of "participation," is about power and control, not just "involvement." That will be even more clear as we look at the overlaps between participatory evaluation and the other fields, especially participatory research.
The most important overlaps between participatory research and participatory evaluation are that both are research processes and both emphasize power-structure-transforming forms of participation. Yoland Wadsworth (1991) is explicit in saying that her evaluation process--a version of participatory evaluation--follows the same route as participatory research (which is referred to as "action research" in her Australian context). It begins with action that generates questions, then research, analysis, conclusions, and recommendations that lead to more action (Wadsworth, 1984:44). But what is participatory research anyway? Following the popular educator Paulo Freire (1982), John Gaventa (1991:121-122) argues that PR "is simultaneously a tool for the education and development of consciousness as well as mobilization for action." In practice, this means that people reappropriate knowledge that has been denied them, develop their knowledge skills, and participate in the social production of knowledge. Orlando Fals-Borda (1991) argues similarly that PR is collective research, critical recovery of history, valuing and applying folk culture, and producing and diffusing knowledge. Peter Park (1993) and Budd Hall (1993) also emphasize that PR combines research, education, and action; and is fundamentally about oppressed communities overcoming their oppression.
I combine these positions into two basic characteristics of participatory research: participation in the research process and making social change (Stoecker and Bonacich 1992; Stoecker, 1998). As communities conduct research on issues that matter to them, they are changing power relations both because they are controlling research processes that have historically been used to control them, and because the knowledge they generate can support broader social struggles. The participatory evaluation model derived from participatory research necessarily adopts a "conflict approach" to social relations and contrasts starkly with the mainstream "instrumental/technocratic" evaluation approach rooted in the functionalist tradition (Brown and Tandon, 1983; Marsden and Oakley, 1991). From a conflict approach, structural conflict is a fundamental organizing feature of society. This has two implications, which I will elaborate upon later. First, a participatory evaluation process looks for structural conflict within a group or organization, and if the conflicts are not uncovered strategically and handled with grace, the evaluation can do more harm and good. Second, the participatory research focus on structural conflict is always rooted in broader processes of social inequality, which helps attribute internal organizational conflict to external conditions (Grills et al., 1996; Andrews, 1996; Wadsworth, 1991; Marsden and Oakley, 1991).
The contemporary field of community organizing is most attributed to the work of Saul Alinsky (1969[1946]; 1971), but has strong roots in the Civil Rights Movement as well (Morris, 1984). In basic, community organizing is the process of organizing relationships, identifying issues, mobilizing around those issues, and building an enduring organization (Stoecker and Stall, 1996). An effective organization or group will have two important, fundamental qualities: strong mutually supporting relationships and effective strategy. No field has focused on developing relationships of solidarity and strategic effectiveness under conditions of adversity like community organizing. It is the community organizer's job to accomplish those objectives.
In practice, the community organizer's role is often combined with the researcher's role. While this can be dangerous if one person is doing both but is not good at both, their frequent overlap shows their combined importance. Ernie Stringer (1996) describes the "researcher" as a catalyst--to stimulate people rather than impose on them, emphasize process over product, enable people to do it themselves, start where people are, help people plan and act and evaluate, not advocate for people, and not focus just on solutions to problems but also on human development. Likewise, Peter Park's (1993:8) discussion of the need for a "researcher" makes it sound like that of an organizer: "This sense of the problem may not always be externalized as a consensually derived and objectified target of attack in the community, although there may be suffering, a sense of malaise and frustration, and anger. For this reason, the situation characteristically requires outside intervention in the guise of a researcher...to help formulate an identifiable problem to be tackled." Park even describes the process of how the researcher should enter the community almost exactly the way others describe how the organizer should enter a community, advocating that the "researcher has to learn everything about the community before entering and needs to be sponsored and accepted into community."
In community organizing, knowing when you've won, and maintaining self-sufficiency is part of an ongoing "research" process that often involves community self-evaluation research. And this ongoing evaluation can also be done as participatory research. Evaluation is the foundation of this process. In many organizing models, evaluation is conducted after every door-knocking session, every action, and even meetings. Called "debriefings," these semi-formal discussions try to better understand what worked, what didn't work, and what to do the next time. Grills et al. (1996:128) combined an organizer with an evaluator and conducted an organizing needs assessment. They used the evaluation data for "identifying issues, clarifying problems, prioritizing agendas for action, developing strategies with policymakers, defining goals, selecting targets for organizing, and critically assessing actions taken."
The only way to change strategy is to learn from the evaluation process, so it seems only natural to look at participatory forms of learning deriving from the "popular education" or "adult education" approach to learning. Popular education is a participatory approach to learning that makes participatory research a central part of the learning process. Research is done not just to generate facts, but to develop understanding of one's self and one's context. It is also about understanding how to learn, which allows people to then become self-sufficient learners and evaluate knowledge that others generate (Park, 1993, Heron, 1996). Paulo Freire and Myles Horton are most often cited as core influences of this approach (see Horton and Freire, 1990). And those working in the tradition of participatory research regularly draw from this tradition as part of their practice (Williams, 1996; Lynd, 1993).
"Learning" is consistently mentioned as an important concept in participatory evaluation. Fetterman (1996:15) talks about the importance of "illumination...new insight or understanding about roles, structures, and program dynamics." Estrella and Gaventa (1997) list learning as one of the important principles of participatory evaluation. Vargas (1991) explicitly combines participatory evaluation and popular education. Evaluation is fundamental to a popular education process, but because popular education is participatory, so must be the evaluation. And when the evaluation is participatory, it "is necessarily an educational process, because each step helps to bring more comprehension of what has been done, how and why. All conclusions are obtained from collective reflection which is part of a learning process on reality." (Vargas, 1991:269).
Hamilton (1992:87-100), following Dean and Dowling (1987), goes the furthest in developing a popular education model that brings all of the fields together and overlaps almost completely with the participatory evaluation models described in the introduction of this paper. The process begins with some threat arousing the community, and then invokes a learning process as organizers help community members explore the causes and ramifications of the threat. The community then conducts research and begins a planning/organizing process that leads to social action. After evaluating their efforts, they expand on their initial organizing to spin-off development activity.
For an evaluation to be useful, its results must become part of a planning process. That planning process must adopt the same kind of approach as the other aspects of participatory evaluation. Thus, transformative, community-based, or "empowerment" planning (Reardon, n.d.; ESLARP, 1996; Kennedy, 1996) (in contrast to strategic planning which is much more about meeting bureaucratic requirements than community needs) is the final field that intersects with this process.
There are three qualities of this planning approach that are readily adaptable to a participatory evaluation process. First, the planning takes into account all of the physical and social aspects of a whole neighborhood (or program, organization, etc.). Typically, participatory planners use an evaluation process known as SWOT--strengths, weaknesses, opportunities, and threats--to compile planning data (Jones 1990). The plan must understand the neighborhood (or organization, group, etc) as a community, taking into account relationships among groups in the neighborhood and between the neighborhood and its neighbors, and evaluating the possible effects of change on those relationships (Pyatok and Weber 1978). Second, the plan is community-based--focused on strengthening existing social relationships and building new ones. This necessitates creating conditions that support mutual-help networks, a spirit of caring, and an ethic of responsibility. This approach judges planning proposals on the basis of how they effect informal support networks (Friedman 1978) and connects with community organizing as a process to build and strengthen community bonds (Cassidy 1980; Jones 1990). Third, community-based planning is participatory, bringing residents (or program participants, organization members, etc) in to lead, control, and own the process (Hasell 1987:82; Jones, 1990). This involves accepting them as having unique expertise, conducting the planning process on the people's turf, making planning technology and language available and user-friendly, and putting the decision-making authority in the hands of the people (Jones 1990).
Participatory planning techniques have been surprisingly neglected by participatory evaluators. Butterfoss et al. (1996) are some of the few participatory evaluators to recognize the importance of the planning process, and they even develop an evaluation process to judge its quality.
I have had the thrill of facilitating a number of evaluation processes recently, which has given me the opportunity to develop a model of participatory evaluation informed by the four fields of participatory research, community organizing, popular education, and empowerment planning. This section provides a little background on the case I consider in this paper.
Toledo Community Organizing Training and Support Program
The "Toledo Project," as I will refer to it, developed as an idea in 1997 through a combination of discussions involving the Needmor Fund, the Toledo Community Foundation, and Dave Beckwith from the Center for Community Change. At issue was a concern that Toledo lacked a solid foundation of community organizing groups. The two foundations formed a project advisory group, and they shaped a two-tier program. The main focus of the program was to provide intensive community organizing training to three community development corporations who had submitted grant requests to Needmor. In addition, the program would fund a series of community organizing training events. Three CDCs would receive funds to support the salary of their own community organizer. The Needmor fund would provide the funds and the Toledo Community Foundation administered them. The goal, at the end of three years is to have strong city-wide community organizing in Toledo.
The three CDCs who were selected had not all developed proposals fitting the usual definitions of community organizing, so some careful discussion was necessary to change the emphasis of their original grant applications. The Lagrange Development Corporation (LDC), representing a multi-racial working class community, was doing the most community organizing, and had an informally separate organizing arm--the Lagrange Village Council. Organized Neighbors Yielding Excellence (ONYX), representing a mixed class African American neighborhood, had some false starts with organizing In the past, but was ready to try again. Toledo Central City Neighborhoods (TCCN) represented a working class African American neighborhood and was trying to coalesce with a neighborhood association in its service area--Neighbors In Action (NIA). This was a challenging period in the program, since the CDCs felt pressured to do what the foundations wanted. At the same time, the process of choosing a trainer was very open, with the CDCs ultimately holding sway in a split decision between a trainer from the region, or a trainer from the nationally famous ACORN--Association of Community Organizations for Reform Now (ACORN, 1998). Madeline Talbott, the ACORN national field director, got the nod.
The early phase of this program, from March to June of 1998, was built on a tripod of support. First, ACORN conducted two trainings. Both were an "introduction to organizing," with one day open to all and the other exclusively for members of the participating organizations. About two dozen people attended each training, which lasted from mid morning to mid afternoon. Second, Madeline focused on training the community organizers for each group. How, and how often, to do this was the subject of intense negotiations. In addition, much of the training would be done from Chicago over the phonewith the organizers together in one room and Madeline on the speakerphone. This would become an important focus of the evaluation effort as "how to train organizers from afar" was the most unusual aspect of the program. Third was the mentoring of individual organizations. Here ACORN assigned a staffperson (which included their St. Louis lead organizer, and their New York lead organizer in addition to Madeline) to mentor each of the three groups. Through regular phone conversations and monthly face to face meetings, the ACORN mentor's job was to help each organization build its member base and do better community organizing.
The Toledo Project participatory evaluation process completed its first evaluation cycle in June of 1998, and its second cycle in January of 1999. This paper reports on the evaluation process through that point.
THE FIRST CYCLE: MARCH-JUNE 1998
Step 1: Evaluating, Planning, and Planning the Evaluation
Evaluation, especially when it is focused on how well an organization or program is meeting its goals, can be quite turbulent (Patton, 1997:179-181). One of the best ways to prepare for this turbulence is to make the evaluation process part of an overall planning process. In a story of an initial failed evaluation process, the victims of the process told the evaluator they "want[ed] to be part of an easy-to-use process from the beginning." (Dugan, 1996:278). If the evaluation plan is based on the program plan, then people know what they want to achieve and the evaluation plan simply outlines how they will know if they achieved it.
Thus, the process to follow in designing a participatory evaluation is the same as that used in designing any participatory research project. The end users have to participate in making the design, and control what the ultimate design is either directly or through a strict accountability process of giving feedback to the evaluation facilitator. The process starts by generating the evaluation questions from those who will be effected by the evaluation (Patton, 1997:30). Users also need to help design the methodology. Participants' intended uses for the evaluation will effect their choice of methods, and their knowledge of resource and political limits will inform what tradeoffs to make. Through this process users gain skills and knowledge, and debates get settled before the data are collected (Patton, 1997:243). The latter is important, as the results will only be valid if people agree on the methods.
Designing the evaluation process for the Toledo Project was challenging in two important ways. First, participants had little understanding of what a participatory evaluation would entail. Second, I needed to submit an evaluation proposal to the project advisory committee before meeting with the groups. The advisory committee was comfortable, however, with considering the proposal an outline that would change as the CDCs participated in designing the ultimate evaluation process.
The project held an initial day and a half planning meeting in March of 1998, including the ACORN mentoring team, the project advisory group, and the CDC executive directors and community organizers. I facilitated a two-hour planning process, asking participants to list the qualities of a good evaluation, discuss what parts of this program they wanted to evaluate, and decide how to measure them. I was initially met by silence, though people eventually, tentatively, offered some thoughts. I wrote up the discussion and sent out a draft of the evaluation plan to everyone. Only a couple of participants offered any feedback at all, the most coming from the lead ACORN mentor, Madeline. The eventual outline is available in Appendix B.
In retrospect, the lack of participation is understandable. The CDCs hadn't bought in to the overall project yet, far less the evaluation of it. This was also a group for whom "research" was not a central mission, and so interest in research (and perceived need for it) was much lower than in other evaluation processes I had facilitated. It would be important to get that buy-in, as it is one of the necessary conditions for a useful evaluation (Patton, 1997).
Some believe that an outside evaluation facilitator to actually do the research as violating the principle of putting the community in charge of knowledge creation (Gaventa, 1993). Thus, my more central role in doing the research in this case may be suspect. However, I distinguish between working with organized groups and disorganized groups. When doing research with disorganized groups, part of the organizing process is for them to do their own collective research and writing. But for already organized groups whose resources are already stretched to the limit, and who know what they want to say, having an outsider do the research and writing is less contradictory (Stoecker, 1998). If the evaluator is really involved, and the process is really empowering, people can take over much of the evaluation process and use the evaluator more as a resource (Vargas, 1991; Grills et al. 1996).
We had decided, at the initial planning meeting in March, to fully evaluate the first four months of the program, rather than wait until the end of a year, so we could catch early problems and make course corrections. Wadsworth (1991) proposes an evaluation continuum from "daily informal personal reflection," to "weekly spans" to "special effort evaluations" on specific programs, to "monthly collective problem-pooling sessions" to "annual what-have-we-achieved and where-are-we-heading next year efforts" to "comprehensive programme 'stock-takes'" every 3-10 years or so. This evaluation was somewhere between the monthly and annual evaluation models.
We used a case study design (Stoecker, 1991) to understand the dynamics of the program overall. We had planned the initial evaluation to not be summative, or "outcome" oriented, since there would be few if any analyzable outcomes so early in the process. Rather, we planned a "process evaluation" (Patton, 1997) looking for any early conflicts, glitches, and successes, and trying to explain them. I conducted intensive interviews with all of the executive directors, community organizers, and mentors involved in the project. In addition, I attended a variety of other meetings, including training meetings. Melissa Jeter, the research assistant, did intensive participant observation of the training meetings. We also used surveyed attendees at the first training meetings, and Melissa analyzed those questionnaires.
At the same time I was engaged in the formal evaluation, Madeline Talbott was doing outcomes evaluation with the three CDCs. At the initial planning meeting each group set goals for how many doors they would knock, how many hours of doorknocking they would do, and how many new members they would sign up. Madeline pushed all the groups to continuously self-evaluate how well they were doing in relation to those goals. Some of the outcomes of that process then became part of the evaluation report.
The title to this subsection is really a double entendre. On the one hand, it is about changing the evaluation process as you go. On the other hand, it is about making change in the program itself. A good evaluation process is designed to effect the system, and hopefully in an ongoing way (Millett, 1996). Patton (1997) distinguished between evaluations that render judgements, facilitate improvements, and generate knowledge. The goal was first to facilitate improvements and, second, to generate knowledge. The object of the evaluation wasn't to have a flashy evaluation report at the end but to have a better functioning program. That meant making the evaluation a continuous feedback process. And so, as the evaluation progressed, change happened because people began reflecting more on their practice than they might have otherwise.
One way the evaluation proces itself changed was in how we used the surveys from the trainings. We hoped the surveys would show how people reacted to the initial trainings, and what they wanted from future trainings. Because they were uniformly positive, however, it was difficult to pick out anything that should be changed. That is encouraging in one sense, but always makes me worry that if the survey didn't generate more interesting findings then maybe it asked the wrong questions. Also, when people wrote what they wanted from future trainings, there were no clear themes. So we discovered that, rather than try to plan the next trainings from the survey responses, we would need to do it almost entirely from an upcoming June planning meeting.
The evaluation process also began changing the program in important ways as Madeline, the lead mentor, and I developed a very interesting relationship trading reflections of Toledo culture. We developed together an analysis of what was happening in the project based partly on her in-depth experience with community organizing in a wide variety of settings, and my particular understanding of Toledo. For perhaps the first time in my academic life, I found the "expertise" tables turned. I am used to the academic-activist partnership where the activist has the experiential understanding and I have a more abstract understanding based on books and travels to other places. Here, I had the experience and Madeline had the more general understanding.
There were other important changes along the way that began affecting the flow of the Toledo project, which are better understood in the next section.
Step 4: Uncovering Creative Tensions
The conflict orientation of participatory research forces the evaluator to become immersed in the "messy fray" of people and politics (Patton 1997:57). It is thus tricky to adopt a conflict orientation when facilitating an internal evaluation, as the researcher runs the risk of causing conflicts rather than resolving them. I have found it useful to deal with this risk by using the concept of "creative tensions."
What are creative tensions? Wadsworth (1991:5) says "evaluative research commences with observing a discrepancy between an 'is' and an 'expectation.'" Kenneth Benson's (1977; 1983) "dialectical methodology" uncovers the contradictions between the goals of an organization and the practice of those goals which may create internal tensions. For any project trying to do something creative, such tensions will exist. In fact, tensions give rise to many creative projects, as people struggle to define their dissatisfaction with the status quo. So the first source of tension is between what people are used to and what they want to do differently. The second source of tension arises from the often-varied definitions people bring to any creative project. Because the project is new to everybody, everybody tries to define it partly from their own unique experience, often leading to misunderstandings in action. The third source of tensions are those caused by the outside world--of race, class, and sex/gender, and other structural inequalities that are almost never completely avoided in any organization. And a fourth creative tension comes from the organizational and institutional settings of the participants, which may conflict with the development of new definitions or skills.
Dealt with openly and constructively, creative tensions are conflicts that are good. They are not weaknesses in a program, but simply realities--usually rooted in social structural conditions. Understood this way, participants can see personality conflicts as rooted in external conditions of inequality and make the outside world, rather than each other, the target (Wadsworth, 1991; Grills et al., 1996; Andrews, 1996). Getting people to "depersonalize" conflicts between individuals is extremely difficult, however. Most people are much more comfortable thinking like psychologists--assigning blame to individuals--than they are thinking like sociologists--assigning blame to the system. Under those circumstances, some "creative" tensions become just plain-old tensions, and may be best dealt with quietly.
People's collective identification of creative tensions, and their collective attempts to manage them, lead to even more program innovations. And tensions, when effectively organized, are often the source of creativity and innovation. When the process is truly participatory, once the group together agrees on what the issues or creative tensions are, they are more likely to do something about them (Uphoff, 1991). Here is where community organizing, participatory planning, and popular education come in, as we will see.
As the Toledo Project research progressed, three creative tensions emerged. First, the three CDCs were very different in their abilities to do community organizing, and some needed much more intensive intervention than others, so the initial mentoring model being used by ACORN needed to diversify to better fit the different organizational conditions. Second, the speakerphone organizer debriefings seem disrupted by attempting to be both an accountability mechanism and a training mechanism, as well as by geographic distance. Third, the initial trainings showed confusion over what community organizing is, and showed that there is a severe shortage of community organizing skills across the city.
Other tensions were difficult to define as "creative" and make public. In those cases I found myself offering personal feedback or simply being a sounding board. This is difficult to do without betraying the confidences of others, but participants seem to accept when I explain that I am being vague in order to protect the confidences of others (and hopefully develop trust that I will then also protect their confidences).
There was one tension that arose where I found myself in a very interesting role. The Toledo Project is typical in being funded at a bare bones level, and since the grant paid both ACORN and myself, concern was developing among some of the groups that not enough grant money was being used to support their community organizers. I talked with various participants about what my role should be in this issue. Ultimately, I ended up being an advocate for more organizing funds. For the evaluation facilitator to play such a role is, as I have discussed, partly a violation of the principles of participatory evaluation. But in this case I was advocating with the participants, not for them. In addition, people from every structural interest group in the Toledo Project encouraged me to take this role--partly playing on my status as a partial outsider who could more easily be seen as "objective." I redrafted the evaluation report to emphasize the groups' concerns about funding and suggest some steps to address the issue. This redrafting is part of a participatory validation process described next.
Step 5: Participatory Validity Checking
One of the most personally rewarding parts of any participatory research project is rough draft time. Again, because I was working with already organized groups who didn't have time to write, I composed the draft. And in participatory evaluation, where the evaluation facilitator's role is to hear and further the expression of different voices, there also may be no one else in a position to do the writing. When an outsider does the writing, it is extremely important for the writing process to be strictly accountable. I do this through a participatory validity process. Depending on the complexity of the situation, I go through a variety of validity steps, including returning transcripts of interviews back to participants for their editing (before anyone else sees them); returning write-ups of organizational case studies to the organization for editing (before anyone else sees them); and returning full drafts to participants for editing, with individuals and organizations kept anonymous (which is the process being used to write this paper). These steps are especially important to make sure the creative tensions are understood and constructively dealt with.
I have experienced amazing successes in developing a technically valid analysis using this approach. When individuals see the draft they often say " Oh, so that's what you wanted to know," and they give me more and better information. The new understandings that lead people to give me more information also shows the popular education dimension of this process. They also edit their own quotes, usually for grammar (I type up quotes as verbatim as possible) but also sometimes to elaborate the content. And while individuals sometimes do political editing of their quotes, it has never been to change a story as much as to better communicate it--where they may have viciously slammed an antagonist in the original interview, they edited their language to take out the verbal slurs.
The time pressures in the Toledo project, working on an initial four-month evaluation cycle complicated this validity checking. Too much was happening too fast to begin interviews, send them back for individual comments, write a draft, and send it out for comments. Consequently, I skipped the step of doing a validity check on individual quotes, and sent out a draft that did not identify individuals, but only their structural position (organizer, director, mentor). With only a few days between their receipt of the draft and the meeting where we were to discuss it, I received only a couple of comments before the meeting. In one of those cases, however, the draft did elicit much more information, and I spent an hour on the phone with an ACORN mentor redrafting one section of the report. Again, this is partly due to the lack of buy-in to the evaluation process at this point, but more buy-in would develop during the planning action phase.
This step is the fruit of the process. It is possible to just turn in the report and go away. It would even be easy to give a report to the group, turn in the evaluation and go away. But the real fun is in following through on the evaluation process all the way to the group's next action phase. This is much like facilitating the initial planning meeting, only instead of planning a round of evaluation, participants are planning their next organizational steps.
How does the evaluation facilitator do this? My method is in contrast to Patton (1997:324-9) who seems to say the evaluator should make recommendations, though with participation from users. I think the evaluator should not make recommendations. The evaluation facilitator's job is to identify creative tensions and/or outcomes, and then facilitate a planning process where participants come to better understand those creative tensions and make their own recommendations. Here the process of popular education becomes central, as participants engage in a learning-teaching process about their organization or program.
The June planning meeting of the Toledo Project was organized partly around the evaluation process, divided into two parts. During the afternoon meeting, the ACORN mentors and CDC directors and organizers met to discuss the creative tensions identified in the evaluation report, and develop plans to deal with them. In doing so, we were using a combination of popular education and empowerment planning principles. Their discussions were again slow in starting. One member expressed the lack of groupness that had yet been generated by saying they did not yet feel comfortable expressing some of their thoughts in the group. To deal with the creative tension of different organizations being at different stages and needing different kinds of mentoring, the group recommended sending new organizers out for training with ACORN; doing focused trainings for individual organizations; developing executive director-organizer relationships more; and doing more assessment and diagnosis with each organization. To address the creative tensions in the organizer debriefing process, the group recommended conducting one on one debriefings, with the executive director when possible; and encouraging organizers to get together at each others' actions, and debrief together with each other afterward, including members in the process.
As I described earlier, and issue emerged during this meeting that would become the most important source of involvement in the planning process. One CDC director, worried about the level of funding for their community organizer, voiced concern about the funding of the entire program. The conversation was picked up by another CDC director, and we started to think about how to write this concern into the evaluation. The redrafting process allowed me to send the redrafted section back to three participants who were particularly interested in it, and I got quick feedback on the redraft. As another CDC director experienced difficulty hiring a new community organizer because they couldn't offer him enough money, I offered to put her story in the evaluation report as well, to which she readily agreed. At this point the evaluation began to be a much more collective project.
An evening meeting that also included organization leaders and members dealt with the third creative tension--how to do the trainings. Melissa Jeter (the research assistant) presented the survey results so people could learn the diversity of needs and interests for future trainings. Madeline then organized them into small groups to outline what the next three training meetings should look like. Amazingly, they all outlined a progression of topics that would start with recruiting members, then building a campaign, and ending at the last training with an action. The implementation of these plans, discussed later, will show how crucial a good planning process is to make the most of any evaluation process, especially when the evaluation research results are unclear.
I followed up this meeting with a late night coffeehouse conversation with the lead mentor, Madeline, since we had built in no formal way to "evaluate the evaluator" this early in the program. The feedback was very interesting. In contrast to working with academics and activists, as I had in other evaluations, working with building constructive conflict among organizers is still something I am getting used to. Finding ways to get organizers to engage in constructive discussion, especially when they had not yet built a sense of trust with each other, I have found especially challenging, showing that the same "tricks" will not work the same from group to group. We also discussed ways to turn more control of the process over to participants--at a meeting with a majority of participants being African American, most of the facilitation was done by two white folks--Madeline and me. So another positive impact of this evaluation process is consciously looking at ways for participants to take more and more control of it.
Outcomes for the Program
During the second phase the program participants made a number of important changes. The TCCN/NIA coalition, was dropped from the program. The NIA side of the coalition was having great difficulty developing new leadership, making it very weak in turning people out for actions. The relationships between TCCN and NIA had also not stabilized, making it clear that this coalition needed basic organizational development assistance before they could benefit from organizing training. ONYX lost its executive director and its community organizer, who left for other jobs. They hired my research assistant, Melissa Jeter, as their new organizer, and ACORN took her to Brooklyn and then Chicago for two months of intensive training. They also took Ramon Perez, the Lagrange organizer, to Chicago for intensive training. As a result of the ongoing discussion of how to train organizers, Madeline also created a reporting system to keep the organizers organized and promote accountability. In Toledo, the series of three trainings for community leaders, decided upon through the June planning process, commenced. This training series ended with a real action on a real slumlord that started a coalition between LDC and ONYX.
Outcomes for the Evaluation Process
In the second phase of the program I learned through experience what Yoland Wadsworth (1991) means by "everyday evaluation on the run." There was less planning of this second evaluation cycle, since the pieces were the same and participants wanted the same programmatic areas evaluated. So planning the evaluation was more of a "checking in" with participants to see whether they wanted to continue the evaluation process along the same lines.
I also found my role becoming much less formal and separate. On two occasions parts of the program hit snags, and I was the first person to be told. In both cases I worked quietly on interventions with participants. The eventual dismissal of TCCN/NIA from the program came after a great deal of ongoing discussion and was part of ACORN's ongoing evaluation of each of the organizations involved. Evaluation and intervention, in general, were becoming much less separate and much less linear as the evaluation, participatory research, empowerment planning, and popular education facets became integrated.
In this second phase, the evaluation became much more of a research project. If the evaluation is limited to only testing whether a set of in-concrete objectives were achieved, the side effects and unexpected outcomes will be ignored. In this project, something quite unexpected happened that deserved careful research. One of the funders' goals was for this program to result in strong city-wide organizing, but after the first three months everyone was pretty convinced that would be a long-range goal. At the third training session for leaders, however, the ACORN trainer got the LDC and ONYX staff to identify a joint target, which happened to be a city-wide slumlord, and organized them to march in front of his posh suburban house. That "training exercise" started a joint campaign against the slumlord that is ongoing. The campaign itself is fascinating because it promotes a popular education strategy to identity the challenges in coalition work and decide how to grow the project with more groups and more targets.
The organizing components of participatory evaluation also became crucial in the second cycle, focused around the issue of organizer training. Remember, this had been the most difficult creative tension in the first evaluation phase, and would be even more so in the second round. Both organizers, and Madeline, had very mixed reactions to the out of town organizer training experiences. Writing the evaluation report in such a way that would contribute to people viewing the tensions as creative rather than personal, was daunting. Here the method of participatory validity checking became crucial. I showed my draft of the organizer's critique of the out-of-town training first only to the organizers, since their words could easily be misinterpreted. I also checked other people's comments on this section before making the draft more generally available.
As the groups bought into both the program and the evaluation process, they also began to engage more in the popular education aspects of the evaluation. So much of a good evaluation process is about helping people understand not just what they are doing, but how they think about what they are doing. As the program progressed, a new creative tension emerged between the ACORN model and the CDCs. That tension had been in the background all along, but as some of the CDC's initial resistance to the ACORN model wore off, people began wondering how much like ACORN they wanted, and didn't want, to become. Madeline invited me to lead a discussion of that tension at one staff meeting. We organized what was supposed to be an hour of discussion and became two hours. We compared the ACORN model, the LDC/Village Council model, the ONYX model, and the TCCN/NIA model, filling a 12 foot chalkboard with notes that were later transferred to disk and made part of the second evaluation report. Everyone became much more clear about their own model, and the other models, as a result.
Most interesting, however, has been the integration of popular education and participatory planning. The second evaluation cycle culminated with another draft evaluation report and two planning meetings in January of 1999. At the staff planning meeting I facilitated another long chalkboard discussion, this time focused on a visioning process looking down the road two years. It led to a fascinating discussion where people compared the organizing options of building a separate city-wide community organizing body, a multi-CDC umbrella organization, or continued to do organizing through the separate CDCs. People explored the implications of the various models for the organizers, for their own organizations, and for the city. As a consequence of the process, people began thinking seriously about how to expand the program during the second year to build toward city-wide organizing. The group is not yet wedded to any model, but the discussion is begun.
CONCLUSIONS--LESSONS OF THE INTEGRATED MODEL
What does this case say about participatory evaluation? First, it says that understanding and employing the participatory research, community organizing, popular education, and empowerment planning approaches are crucial to the success of participatory evaluation. For real participatory democratic change to occur, learning, research, organizing, and planning all must be carefully intertwined.
This example also says that an evaluation process is best when it's relatively invisible. Evaluation has typically been separated out from basic organizational processes because it has been done to, at, for, on, or against people rather than by them (Wadsworth, 1991). As evaluation becomes part of a community-controlled process it is no longer as threatening and no longer as separated from the ongoing activities of the group. At the same time, we should not make evaluation too invisible since, as Wadsworth (1991; xi) argues, "the purpose of evaluation is to purposefully spend some time thinking about what we are doing." An evaluation process provides opportunities for explicit reflection.
Third, this case shows how important it is for the evaluation to be focused on program improvement rather than judgement-making. As I've noted, there are three evaluation uses: rendering judgements, facilitating improvements, and/or generating knowledge (Patton 1997). The negative consequences of evaluation--the destruction of individuals, the creation of resentments, etc., can be avoided if the evaluation results are used for neither prizes nor punishments, and if the process is never hurried (Uphoff, 1991).
Fourth, this form of evaluation requires special skills of the evaluator. Since successful participatory evaluation is so dependent on the personal involvement from people (Patton, 1997: 43-52), the evaluator needs special skills in building relationships, facilitating groups, managing conflict, walking political tightropes, and effective interpersonal communications (Patton, 1997: 50-52). An evaluation facilitator who does not have these skills needs to partner with others who do. One way to look at it is that participatory evaluation is really a participatory research project, which is really just a part of a broader social change project. Making social change requires that crucial roles be filled--and the research role is only one of those (Stoecker, 1998).
Finally, it may be that such a process is becoming much more widespread and valued than even its proponents realize. This evaluation project, and three others I have been involved in, are supported with real money by funders. Millett (1996) also argues for the increasingly enlightened way that funders are approaching evaluation. That does not mean there will no longer be tensions between an organization's need for process evaluation and a funder's need for summative evaluation, but there certainly is not the all out antagonism there used to be.
ACORN--Association of Community Organizations for Reform Now. Available at http://www.acorn.org/.
Alinsky, Saul. 1971. Rules for Radicals. New York: Vintage Books.
Alinsky, Saul. 1969 [1946]. Reveille for Radicals. New York: Vintage Books.
Andrews, Arlene Bowers. "Realizing Participant Empowerment in the Evaluation of Nonprofit Women's Services Organizations: Notes from the Front Line." Pp. 141-158 in David M. Fetterman, Shakeh J. Kaftarian, and Abraham Wandersman (eds.) Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks: Sage.
Beckwith, Dave, with Cristina Lopez. 1997. "Community Organizing: People Power from the Grassroots." COMM-ORG: The On-Line Conference on Community Organizing and Development. http://comm-org.wisc.edu/papers97/beckwith.htm
Benson, Kenneth. 1983. "A Dialectical Method for the Study of Organizations." In Gareth Morgan (ed.) Beyond Method: Strategies for Social Research. Beverly Hills: Sage.
Benson, Kenneth. 1977. "Organizations, a Dialectical View." Administrative Science Quarterly 22:1-21.
Brown, L. David, and Rajesh Tandon. 1983. "Ideology and Political Economy in Inquiry: Action Research and Participatory Research." Journal of Applied Behavioral Science 19:277-294.
Butterfoss, Frances Dunn, Robert M. Goodman, Abraham Wandersman, Robert F. Valois, and Chinman, Matthew J. 1996. "The Plan Quality index: An Empowerment Evaluation Tool for Measuring and Improving the Quality of Plans." Pp. 304-331 in David M. Fetterman, Shakeh J. Kaftarian, and Abraham Wandersman (eds.) Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks: Sage.
Cassidy, Robert. 1980. Livable Cities: A Grass-Roots Guide to Rebuilding Urban America. New York: Hold, Rinehart, and Winston.
Dean, G. J., and W. D. Dowling. 1987. "Community Development: An Adult Education Model." Adult Education Quarterly 37:78-89.
Dugan, Margret A. 1996. "Participatory and Empowerment Evaluation: Lessons Learned in Training and Technical Assistance." Pp. 277-303 in David M. Fetterman, Shakeh J. Kaftarian, and Abraham Wandersman (eds.) Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks: Sage.
ESLARP--East St. Louis Action Research Project. 1996. Available at: http://imlab9.landarch.uiuc.edu/~eslarp.
Fals-Borda, Orlando. 1991. "Some Basic Ingredients." Pp. 3-12 in Orlando Fals-Borda and Mohammad Anisur Rahman (eds) Action and Knowledge: Breaking the Monopology with Participatory Action-Research. New York: Apex Press.
Fetterman, David M. 1997. Collaborative, Participatory, and Empowerment Evaluation. http://www-leland.stanford.edu/~davidf/empowermentevaluation.html
Fetterman, David M. 1996. "Empowerment Evaluation: An Introduction to Theory and Practice." Pp. 3-46 in David M. Fetterman, Shakeh J. Kaftarian, and Abraham Wandersman (eds.) Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks: Sage.
Fetterman, David M. 1996b. "Conclusion: Reflections on Emergent Themes and Next Steps." Pp. 379-384 in David M. Fetterman, Shakeh J. Kaftarian, and Abraham Wandersman (eds.) Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks: Sage.
Fetterman, David M. 1994. "Empowerment Evaluation." Evaluation Practice 15:1-15.
Fetterman, David M. 1994b. "Steps of Empowerment Evaluation: From California to Capetown." Evaluation and Program Planning 17:305-313.
Fetterman, David M., Shakeh J. Kaftarian, and Abraham Wandersman (eds.). 1996. Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks: Sage.
Freire, Paulo. 1982. "Creating Alternative Research Methods: Learning to Do it By Doing It." Pp. 29-37 in Budd Hall, A. Gillette and R. Tandon (eds.) Creating Knowledge: A Monopoly? Participatory Research in Development. New Delhi: Participatory Research Network Series 1.
Friedman, Elizabeth. 1978. "Crest Street: A Family/Community Impact Statement." Policy Paper #2 of Center for the Study of the Family and the State. Institute of Policy Sciences and Public Affairs, Duke University.
Gaventa, John. 1993. "The Powerful, the Powerless, and the Experts: Knowledge Struggles in an Information Age." Pp. 21-40 in in Peter Park, Mary Brydon-Miller, Budd Hall, and Ted Jackson (eds.) Voices of Change: Participatory Research in the United States and Canada. Westport, Connecticut: Bergin and Garvey.
Gaventa, John. 1991. "Toward a Knowledge Democracy: Viewpoints on Participatory Research in North America." Pp. 121-133 in Orlando Fals-Borda and Mohammad Anisur Rahman (eds) Action and Knowledge: Breaking the Monopology with Participatory Action-Research. New York: Apex Press.
Grills, Cheryl N., Karen Bass, Didra l. Brown, and Aletha Akers. 1996. "Empowerment Evaluation: Building Upon a Tradition of Activism In the African American Community." Pp. 123-140 in David M. Fetterman, Shakeh J. Kaftarian, and Abraham Wandersman (eds.) Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks: Sage.
Hall, Budd. 1993. "Introduction." Pp. Xiii-xxii In Peter Park, Mary Brydon-Miller, Budd Hall, and Ted Jackson (eds.) Voices of Change: Participatory Research in the United States and Canada. Westport, Connecticut: Bergin and Garvey.
Hamilton, Edwin. 1992. Adult Education for Community Development. New York: Greenwood Press.
Hasell, Mary Joyce. 1987. "Community Design and Gaming/Simulation: Comparison of Communication Techniques in Participatory Design Sessions." Simulation & Games 18:82-115.
Heron, John. 1996. Co-operative Inquiry: Research into the Human Condition. Thousand Oaks, CA: Sage.
Horton, Myles, and Paulo Freire. 1990. We Make the Road by Walking: Conversations on Education and Social Change. Ed. By B. Bell, J. Gaventa, and J. Peters. Philadelphia: Temple University Press.
Jones, Bernie. 1990. Neighborhood Planning: A Guide for Citizens and Planners. Bigtown: Planners Press, American Planning Association.
Kennedy, Marie. 1996. "Transformative Community Planning: Empowerment Through Community Development." Prepared for the 1996 Planners Network Conference." Available at http://www.plannersnetwork.org/combased.htm.
Lynd, Mark. 1992. "Creating Knowledge Through Theater: A Cast Study With Developmentally Disabled Adults." The American Sociologist 23:100-115.
Marsden, David, and Peter Oakley. 1991. "Future Issues and Perspectives in the Evaluation of Social Development." Community Development Journal 26:287-328.
Millett, Ricardo A. 1996. "Empowerment Evaluation and the W. K. Kellogg Foundation." Pp. 65-76 in David M. Fetterman, Shakeh J. Kaftarian, and Abraham Wandersman (eds.) Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks: Sage.
Morris, Aldon. l984. The origins of the civil rights movement: Black communities organizing for change. New York: Free Press.
Nyden, Philip, Anne Figert, Mark Shibley, and Darryl Burrows (eds). 1997. Building Community: Social Science in Action. Thousand Oaks, CA: Pine Forge Press.
Nyden, Philip, and Wim Wiewal. 1992. "Collaborative Research: Harnessing the Tensions Between Researcher and Practitioner." The American Sociologist 23:43-55.
Park, Peter. 1993. "What is Participatory Research? A Theoretical and Methodological Perspective." Pp. 1-20 In Peter Park, Mary Brydon-Miller, Budd Hall, and Ted Jackson (eds.) Voices of Change: Participatory Research in the United States and Canada. Westport, Connecticut: Bergin and Garvey.
Park, Peter and Lee Williams. [in previous volume]
Patton, Michael Quinn. 1997. Utilization-Focused Evaluation: The New Century Text, 3rd Edition. Thousand Oaks: Sage.
Patton, Michael Quinn. 1997b. "Toward Distinguishing Empowerment Evaluation and Placing It In a Larger Context." Evaluation Practice, 18(2):147-163.
Pyatok, Michael, and Hanno Weber. 1978. "Participation in Residential Design: a Method for Generating Choice and Its Ideological Consequences." Pp. 173-204 in Henry Sanoff (ed.) Designing With Community Participation. Stroudsburg, Penn.: Dowden, Hutchinson & Ross, Inc.
Reardon, Kenneth M. n.d. Community Development in Low-Income Minority Neighborhoods: A Case for Empowerment Planning. Unpublished manuscript.
Stevenson, John F., Roger E. Mitchell, and Paul Florin. 1996. "Evaluation and Self-Direction in Community Prevention Coalitions." Pp. 208-233 in David M. Fetterman, Shakeh J. Kaftarian, and Abraham Wandersman (eds.) Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks: Sage.
Stoecker, Randy. 1998. "Are Academics Irrelevant: Roles for Scholars in Participatory Research." COMM-ORG, The On-Line Conference on Community Organizing and Development. http://comm-org.wisc.edu/papers98/pr.htm.
Stoecker, Randy. 1991. "Evaluating and Rethinking the Case Study." The Sociological Review 39:88-112.
Stoecker, Randy, and Edna Bonacich. 1992. "Why Participatory Research?" The American Sociologist 23:5-14.
Stoecker, Randy, and Susan Stall. 1996. "Community Organizing or Organizing Community? Gender and the Crafts of Empowerment." COMM-ORG: The On-Line Conference on Community Organizing and Development. http://comm-org.wisc.edu/papers.htm
Stringer, Ernest T. 1996. Action Research: A Handbook for Practitioners. Thousand Oaks, CA: Sage.
Uphoff, Norman. 1991. "A Field Methodology for Participatory Self-Evaluation." Community Development Journal 26:271-286.
Vargas, Laura, Vargas. 1991. "Reflections on Methodology of Evaluation." Community Development Journal 26:266-270.
Wadsworth, Yoland. 1991. Everyday Evaluation on the Run. Melbourne: Action Research issues Association.
Wadsworth, Yoland. 1984. Do it Yourself Social Research. Melbourne: Victorian Council of Social Service and the Melbourne Family Care Organisation.
Williams, Lee. 1996. "First Enliven, Then Enlighten: Popular Education and the Pursuit of Social Justice." Sociological Imagination 33:94-116. Also at http://comm-org.wisc.edu/si/sihome.htm.