9th Annual Nonprofit Leadership Conference

Mission and Leadership: Staying the Course in Difficult Times

April 3, 2003, 8:30 am to 5:15 pm; Northwest Rooms, Seattle Center

Program Evaluation and Public Interest Nonprofits: Measuring Progress towards Mission

Luncheon Sessions Abstract

Presenter -- Stacy L. Holmes (MPA '01 Evans School of Public Affairs)

Overview

The purpose of this MPA degree project was to investigate the ways in which public interest nonprofits try to capture and evaluate their progress towards meeting their stated missions. For this project, program evaluation emphasized what Rossi et al. (1999) define as program monitoring -- a form of evaluation describing how well a program operates, and includes comparisons to program theory, program outputs (products delivered), and program outcomes (results of the products being delivered). In addition, the project emphasized internal rather than external program evaluation. Although external, or independent, evaluations were not excluded from the analysis, this project focused on evaluation as an internal management tool.

The final report included an analysis of existing research and qualitative interview findings on the program evaluations conducted by public interest nonprofits. The project also included an application of the findings and concepts to a client organization.  The interviewees included eleven public interest nonprofits whose missions focused on promoting environmental and social responsibility (ESR).

Interviews with Public Interest Nonprofits

The areas that were emphasized consistently by all interviewees when evaluating progress were media presence, network expansion, and the production of educational messages. Most of the interviewees had a deliberate process through which they reflected on their progress, though some were more formal than others. Only three interviewees indicated that pressure from funders had motivated them in the past to plan and evaluate certain parts of their programs in specific ways. All others indicated that the motivation for engaging in their evaluation activities came from within the organization. The progress indicators that were most often collected and counted across all interviewees were:

§ Changes in membership numbers and demographics

§ Number and type of assistance and consultation provided by ESR

§ Number and type of distributions of information with special emphasis on mainstream media and including newsletters, research studies, conferences, speaking events, books, website hits, and public information campaigns

§ Number and type of external corporate or governmental policy changes that were associated with the program's activities.

The production and use of case studies and success stories to reflect on progress was an additional theme during the interviews. The majority of the ESRs also expressed frustration at trying to find out what motivates individual and corporate behavior change, and at measuring the impact of program activities on any behavior change. These interview findings provided examples of how some ESR nonprofits meet the challenges of monitoring program progress.

Recommended Research and Tools

This project included an analysis of related research and evaluation tools as an attempt to fill the gaps highlighted during the interviews. Through using models of social movements, theory maps, and multi-discipline academic research, ESR organizations can:

§ clarify assumptions;

§ emphasize the learning atmosphere of the organization;

§ educate potential funders about the role that ESR’s play in history and the larger society; and,

§ enrich the evaluation components without reinventing the wheel each time.

The academic disciplines that provide useful information for ESR planning and design include consumer theory, marketing research, sociology, anthropology, cognitive science, human evolution theory, and environmental/social psychology.

Client Recommendations

This project was inspired by a request from a growing nonprofit organization, The Conscious Consumer (TCC). As part of implementing an ever-changing strategic plan, the executive director of TCC asked for a consultation on useful performance indicators for his organization. In order to meet this request, the author applied the interview results and research to further customize a recommended plan for program monitoring. Recommendations included specific indicators for measuring change in: database/clearinghouse capacity (main TCC service), impact on database users, message dissemination, and network building. Assumptions which underlie each activity and indicator were stated and illustrated through a theory-map example.

For questions or permissions, contact Stacy L. Holmes at 206.543.7954 SLHOLMES@u.washington.edu or stacylholmes@yahoo.com.