Our next guest in the Open Forum on Outcomes, Hildy Gottlieb, is well known through the nonprofit community. President of the Community-Driven Institute, and the author of The Pollyanna Principles: reinventing “nonprofit organizations” to create the future of our world, she is a frequent commentator on blogs, in articles, and other forums where our sector’s practitioners regularly go for guidance, insights and ideas. We asked Hildy "What’s Next in Outcome Measurement?" and she took a characteristically thought provoking view
HG: From my perspective, “What’s Next in Outcome Measurement?” is a critical decision point.
Will we continue to consider the effectiveness of individual organizations in a vacuum? Or will we instead find ways to measure the real end purpose of outcomes measurement - the extent to which the quality of life in our communities is improving?
To date, outcomes measurement has focused on the former, based on the assumption that it is important to weigh the relative effectiveness of individual organizations. Among other purposes, this provides data for funders / donors to use in determining which organizations to support.
But in truth, the reason it is important that funders invest wisely is because impact in our communities is all that matters. When viewed against that end goal of community / global change, measuring only the effectiveness of individual organizations falls short of the mark.
Within the context of the ultimate end goal of improved communities, therefore, What’s Next would be a system for first measuring the larger context of community-wide improvement, and only then measuring the performance of individual organizations within that larger context.
Ken Berger & Robert M. Penna: The distinction Hildy makes here, between measuring the effectiveness of individual organizations, as opposed to measuring collective improvement within communities, is one that has often arisen in our sector. In fact, Mark Friedman addresses this exact issue in his Results-based AccountabilityTM model when he separates “program or agency accountability,” from “population accountability.” We agree that communal progress ought to be ascertained. But that cannot be done without having a valid, fact-based sense of the outcomes -positive, null, or negative- of the organizational efforts underway in our communities. As such, this measurement hardly takes place “in a vacuum” as Hildy suggests.
We also differ with Hildy in her implication that, beyond serving as an investment guide, there is little value in measuring the effectiveness of individual organizations. The majority of outcomes experts will tell you that the true importance of outcome frameworks and measures is organizational learning, the opportunity to systematically track progress to ascertain precisely how well a given approach, program, or intervention is working. If you do not know the answer to the question of "What’s working?" no matter what you learn about community or population progress, you will never know which efforts to continue and expand and which to end or change. At a time when all nonprofits are keenly feeling the competition for donor dollars, and when the total pot of potential resources from governmental, foundation and corporate sources has shrunk, we believe that it is critically important to be able to ascertain which nonprofit efforts are having the greatest positive results. Efforts to develop community-wide impact measurement are fine, but let’s not sit on our hands and ignore the individual organizational efforts in the meanwhile. We should be using the tools we have on hand now.
At present, the accountability and effectiveness of individual organizations is the only basis upon which we can make informed decisions regarding how to invest scarce social change resources. Charity Navigator sees its role as assisting in this decision, by helping to disseminate information on which efforts, which organizations and programs, are having the most beneficial impact in their communities.
HG; cont’d: Community-wide measurement would help whole communities determine what is working and not working. Broadly disseminated results could serve as a learning tool, so communities and organizations everywhere could apply that learning to create even more community improvement.
Measuring community-wide change will require shared commitment, shared discussion of what is important - what indicators whole communities want to measure. It will require improving the skills of all organizations to ensure all community benefit efforts are indeed improving communities.
Such an effort will require shared data (as well as someone to gather, sort and disseminate the data; someone to do record keeping, analysis, investigation and reporting). It will require funding.
And it will require contributions from all organizations working on a particular issue. That last step will ensure what has, to date, been the first (and dare I say only) step in outcomes measurement - that we have a tool for measuring the performance of individual organizations.
But we will no longer be measuring individual performance for the primary purpose of measuring those individual organizations. We will instead be measuring the effectiveness of those organizations for the sake of whole communities holding themselves accountable for creating a better future for our world. We will have placed individual monitoring within the context of the only thing that matters - community-wide measurement of community-wide change.
What’s next for outcomes measurement is therefore a critical decision.
Will we continue to measure the individual parts of the whole, with virtually no regard to the ultimate outcome - community improvement? Or will we devote the considerable brain-power of this sector towards developing indicators and systems that measure what we really want to see - improvement in our communities?
The choice - and the results of that choice - are what I see as What’s Next. It is clearly one of the most important decisions this sector will make.
KB&RP: We think it is a mistake to view our sector as a monolith. Most of us speak of the “nonprofit sector,” as shorthand by which we refer to a variety of actors and institutions. But in fact, the “sector” is composed of a spectrum of interests, organizations, efforts and funders. It includes roughly 1.9 million nonprofit organizations and is the engine for as much as 10% of our country’s GDP. Within this trillion dollar+ sector, organizations focus upon targeted needs and activities precisely because they believe that the focus of their particular mission has not gotten the attention it requires in the face of matters the larger society of activists, residents and funders believes are more pressing.
So too are our communities themselves far from monolithic. Prioritization and the allocation of resources are always contentious issues. Therefore, a preliminary and exclusive focus upon “community” improvement is a misleading and perhaps dangerous one. Furthermore, there are numerous issues, from the environment to challenges facing the disabled, which cross the borders of communities. For example, the issue of drug abuse will never be solved in one community if the supply of those drugs goes on unabated half a county, half a state, or half a continent away.
In conclusion, communities are important; we agree. But people make up those communities and a macro focus on community measures can very often mask the very real problems these people have. Much of the nonprofit sector is focused upon the issues facing individuals and families. Knowing which organizations’ efforts most effectively help them is invaluable to practitioners, donors and our society as a whole.
So we wish Hildy well in her advocacy for community wide assessment of outcomes. But we do not believe it is the logical first step. In the end, , we hope that we can work in tandem and find areas of collaboration as we move forward together to focus nonprofit resources on those efforts that provide the greatest benefits to our communities.
 Dr. Robert Penna is an independent outcomes consultant and is assisting Charity Navigator in managing this Open Forum. He is the author of Outcome Frameworks and the forthcoming Outcomes Toolbox.