Monday, January 4, 2010

Hildy Gottlieb on Outcomes - Open Forum Participant

Our next guest in the Open Forum on Outcomes, Hildy Gottlieb, is well known through the nonprofit community. President of the Community-Driven Institute, and the author of The Pollyanna Principles: reinventing “nonprofit organizations” to create the future of our world, she is a frequent commentator on blogs, in articles, and other forums where our sector’s practitioners regularly go for guidance, insights and ideas. We asked Hildy "What’s Next in Outcome Measurement?" and she took a characteristically thought provoking view

HG: From my perspective, “What’s Next in Outcome Measurement?” is a critical decision point.

Will we continue to consider the effectiveness of individual organizations in a vacuum? Or will we instead find ways to measure the real end purpose of outcomes measurement - the extent to which the quality of life in our communities is improving?

To date, outcomes measurement has focused on the former, based on the assumption that it is important to weigh the relative effectiveness of individual organizations. Among other purposes, this provides data for funders / donors to use in determining which organizations to support.

But in truth, the reason it is important that funders invest wisely is because impact in our communities is all that matters. When viewed against that end goal of community / global change, measuring only the effectiveness of individual organizations falls short of the mark.

Within the context of the ultimate end goal of improved communities, therefore, What’s Next would be a system for first measuring the larger context of community-wide improvement, and only then measuring the performance of individual organizations within that larger context.


Ken Berger & Robert M. Penna[1]: The distinction Hildy makes here, between measuring the effectiveness of individual organizations, as opposed to measuring collective improvement within communities, is one that has often arisen in our sector. In fact, Mark Friedman addresses this exact issue in his Results-based AccountabilityTM model when he separates “program or agency accountability,” from “population accountability.” We agree that communal progress ought to be ascertained. But that cannot be done without having a valid, fact-based sense of the outcomes -positive, null, or negative- of the organizational efforts underway in our communities. As such, this measurement hardly takes place “in a vacuum” as Hildy suggests.

We also differ with Hildy in her implication that, beyond serving as an investment guide, there is little value in measuring the effectiveness of individual organizations. The majority of outcomes experts will tell you that the true importance of outcome frameworks and measures is organizational learning, the opportunity to systematically track progress to ascertain precisely how well a given approach, program, or intervention is working. If you do not know the answer to the question of "What’s working?" no matter what you learn about community or population progress, you will never know which efforts to continue and expand and which to end or change. At a time when all nonprofits are keenly feeling the competition for donor dollars, and when the total pot of potential resources from governmental, foundation and corporate sources has shrunk, we believe that it is critically important to be able to ascertain which nonprofit efforts are having the greatest positive results. Efforts to develop community-wide impact measurement are fine, but let’s not sit on our hands and ignore the individual organizational efforts in the meanwhile. We should be using the tools we have on hand now.

At present, the accountability and effectiveness of individual organizations is the only basis upon which we can make informed decisions regarding how to invest scarce social change resources. Charity Navigator sees its role as assisting in this decision, by helping to disseminate information on which efforts, which organizations and programs, are having the most beneficial impact in their communities.

HG; cont’d: Community-wide measurement would help whole communities determine what is working and not working. Broadly disseminated results could serve as a learning tool, so communities and organizations everywhere could apply that learning to create even more community improvement.

Measuring community-wide change will require shared commitment, shared discussion of what is important - what indicators whole communities want to measure. It will require improving the skills of all organizations to ensure all community benefit efforts are indeed improving communities.

Such an effort will require shared data (as well as someone to gather, sort and disseminate the data; someone to do record keeping, analysis, investigation and reporting). It will require funding.

And it will require contributions from all organizations working on a particular issue. That last step will ensure what has, to date, been the first (and dare I say only) step in outcomes measurement - that we have a tool for measuring the performance of individual organizations.

But we will no longer be measuring individual performance for the primary purpose of measuring those individual organizations. We will instead be measuring the effectiveness of those organizations for the sake of whole communities holding themselves accountable for creating a better future for our world. We will have placed individual monitoring within the context of the only thing that matters - community-wide measurement of community-wide change.

What’s next for outcomes measurement is therefore a critical decision.

Will we continue to measure the individual parts of the whole, with virtually no regard to the ultimate outcome - community improvement? Or will we devote the considerable brain-power of this sector towards developing indicators and systems that measure what we really want to see - improvement in our communities?

The choice - and the results of that choice - are what I see as What’s Next. It is clearly one of the most important decisions this sector will make.

KB&RP: We think it is a mistake to view our sector as a monolith. Most of us speak of the “nonprofit sector,” as shorthand by which we refer to a variety of actors and institutions. But in fact, the “sector” is composed of a spectrum of interests, organizations, efforts and funders. It includes roughly 1.9 million nonprofit organizations and is the engine for as much as 10% of our country’s GDP. Within this trillion dollar+ sector, organizations focus upon targeted needs and activities precisely because they believe that the focus of their particular mission has not gotten the attention it requires in the face of matters the larger society of activists, residents and funders believes are more pressing.

So too are our communities themselves far from monolithic. Prioritization and the allocation of resources are always contentious issues. Therefore, a preliminary and exclusive focus upon “community” improvement is a misleading and perhaps dangerous one. Furthermore, there are numerous issues, from the environment to challenges facing the disabled, which cross the borders of communities. For example, the issue of drug abuse will never be solved in one community if the supply of those drugs goes on unabated half a county, half a state, or half a continent away.

In conclusion, communities are important; we agree. But people make up those communities and a macro focus on community measures can very often mask the very real problems these people have. Much of the nonprofit sector is focused upon the issues facing individuals and families. Knowing which organizations’ efforts most effectively help them is invaluable to practitioners, donors and our society as a whole.

So we wish Hildy well in her advocacy for community wide assessment of outcomes. But we do not believe it is the logical first step. In the end, , we hope that we can work in tandem and find areas of collaboration as we move forward together to focus nonprofit resources on those efforts that provide the greatest benefits to our communities.


[1] Dr. Robert Penna is an independent outcomes consultant and is assisting Charity Navigator in managing this Open Forum. He is the author of Outcome Frameworks and the forthcoming Outcomes Toolbox.

16 comments:

Bonnie Koenig said...

I don't believe that Hildy is saying that outcomes measurements for individual orgs and communities are an either/or - she is wisely saying that they must go together, and that the latter has not been given the focus that the former has. We have a tendency in our society (as we have become more specialized in our education and professions) to segment what are in reality integrated/holistic challenges that call for integrated/holistic approaches and solutions.

The evaluation of individual organizations is certainly important for many of the reasons that you have outlined but individual organizations do not (and should not) exist, operate, nor develop programs, policies and services in a vacuum. To what end they are doing their work (improvements of their communities) should be a crucial part of outcomes measurements.

JaneG said...

I agree with Bonnie - Hildy is in fact a very strong supporter of organizational effectiveness measures and is certainly not suggesting dropping that activity in favour of only community-wide measurement. However, as she has noted, traditional measures have often focussed mostly on internal efficiencies when external results are far more important.

Another important reason for an emphasis on community-wide measures is that often change is initiated and maintained by groups of people who are not even associated with any formal organization. Their efforts are critical, but they are completely missed when only organizations are considered.

I understand the resistance. We have struggled to have good tools and resources for organizational effectiveness, and want to use them. And we should. Our tools for community-wide effectiveness are not only newer and less tested, they are also more community-driven. Experts have to give up some of their power, and realize the wisdom is in the community. Is the community is changing it desires to, or not?

Gayle L. Gifford said...

I'm with Hildy and Bonnie on this.

While I'm an avid champion of data, measurement and organizational learning, I also know how incredibly complex this measurement of societal outcomes is. I have repeatedly seen first hand that no NGO is an island onto itself. And I'm repeatedly reminded that a critical part of being effective is having effective implementers at the helm.

Just coincidentally, this past week I've been reading Michael Pollan's "In Defense of Food". Pollan's discussions of the inadequacy of the science of nutritionism to make us healthier reads as a cautionary tale for those of us who are are embarking on the same scientific reductionism around NGO outcomes.

Pollan points out that despite all the "scientific" analysis and altering of food and eating, our Western diet (including the foods, the farming, and the processing) has only proliferated the diseases that result from insulin resistance, obesity and other factors with unknown impact on our health (e.g. antibiotics and hormone additives).

Pollan notes that new studies appear almost daily that contradict past definitive studies on what we should or shouldn't be eating.

Pollan writes:
"Scientists study variables they can isolate... yet even the simplest food is a hopelessly complicated thing to analyze, a virtual wilderness of chemical compounds, many of which exist in intricate and dynamic relation to one another, and all of which together are in the process of changing from one state to another. So if you're a nutrition scientist you do the only thing you can do, given the tools at your disposal: Break the thing down into its component parts and study those one by one, even if that means ignoring those subtle interactions and contexts and the fact that the whole may well be more than, or maybe just different from, the sum of its parts." (p62)

Pollan is not necessarily suggesting that we shouldn't continue with science, but he raises alarm bells at science that does not appreciate or study the interactions of complicated systems.

One could easily apply these same words to NGOs.

Like Hildy & Bonnie, I am very worried that the complex interactions between nonprofits, their people, and their communities will be ignored and lost in a reductionist rating system that rewards isolated actions.

I am also concerned about harmful unintended consequences that can't be measured in the short term, or a failure to measure or value positive unintended consequences that are not necessarily the primary mission of a particular NGO but are valuable secondary outcomes nonetheless (e.g. development of community leaders), about the capacity of the richest NGOs to game the rating systems while the least resourced continue to be starved of funding, or the ignoring or devaluing of important societal outcomes that may only occur at the most micro levels (e.g. human dignity, leading a fulfilled life) (see my story "Hope, Dignity and Quality of Life are also valuable outcomes -- even when measured in hours." http://bit.ly/8iYuEk)

Yes, what Hildy is suggesting is hard, but it is very much worth doing. If the goal is making beneficial societal change, then funders need to reward community-wide learning systems by funding them.

While I know that Ken is approaching this much more thoughtfully than most, I have deep concerns nonetheless. I can see a rating system that notes whether NGOs are attempting to measure outcomes, but I am very, very concerned at rating systems that definitively ascribe societal outcomes to any particular NGOs.

Maureen said...

I agree that both community assessment and the effectiveness of individual organizations are important to building a better future. It is imperative that we have the tools to sort out the effective nonprofits from the ineffective nonprofits, supporting those who are doing the most good.

I think that funders may have a role to play in ensuring that the community aspect is addressed. All funders have a focus for their grantmaking. Community Foundations have a regional focus, many large private foundations are focused on a number of program areas. If funders can take a more active role in 'taking the temperature' of their funding communities and sharing that information with the world. If we can find a way to match that information with our list of effective nonprofits. If we all take responsibility for our communities, finding the gaps and seeking to fill them - oh what a world it could be!

Ken Berger said...

Bonnie, Jane, Gayle and Maureen,

We are pleased to hear that Hildy is a strong supporter of organizational effectiveness measures. When we read her book and her blogs we do not find it highlighted anywhere to the degree that community-wide measurement is. Given the sorry state of affairs in the NP sector, where most do not measure themselves, we believe it is going to be a long time coming before individual measurement happens for most, let alone community-wide measurement! We believe that this individual effort MUST come first before we can talk in a substantive way about community measures, not the other way around.

We agree that the most effective NPs are fully engaged in their communities with other NPs, government, business and their supporters. We also understand the value in setting community-wide measures to coordinate efforts. However, given where most NPs are at and the resources available to them, individual performance management and measurement should be the first priority.

Let us agree to disagree on this. As we said in our original response to Hildy, we only wish you well in these efforts. We are happy to work in parallel and collaborate with you whenever we can.

Ingvild Bjornvold said...

Ken is right that we need to start with the individual nonprofits. How can we possibly expect to measure progress at the community level if we can't even do so at the organizational level? And how can community level measurement be meaningful without understanding whether the efforts of individual organizations are helping or hurting?

Hildy Gottlieb said...

Ken et al:
Thank you for this lively debate. I am sorry I have been in full-day sessions and unable to participate since this posted.

I just want to address Ken's suggestion that my work does not address individual organizational effectiveness within community effectiveness. I guess to say I am surprised is beyond understatement.

My entire book, The Pollyanna Principles, focuses on building organizations that are effective in all ways - externally and internally. Our website is filled with dozens of articles on organizational effectiveness, as is my blog. We have a YouTube channel that addresses how organizations can hold themselves accountable for both end results and then the means to accomplish those results.

As I am sadly in a rush and heading into another full day of strategy, I will provide those links and then unfortunately have to run.

http://pollyannaprinciples.org/
http://www.youtube.com/communitydriveninst
http://www.communitydriven.org

The bottom line is this: if we want our communities to improve, we must aim at that improvement, and only then use organizations as one among many tools towards that improvement. If we aim only at the tools - the organizations - we are missing the opportunity to create the kinds of visionary community improvement that is possible.

Hildy

Ken Berger said...

Hildy,

I point to a sample of notions suggested in the Pollyanna Principles book:

"1. What indicators will show a community that feels spiritually whole and integrated?
2. Imagine that the norm in measurement is celebration and reflection!
3. Funding everyone. Collective capacity building rather than individual capacity building."

All of this to me seems pointing toward a lack of MEANINGFUL!, individual accountability for performance.

Teri Behrens said...

This discussion is a great example of why NGO's, funders and evaluators could benefit from systems-thinking based approaches to program design, implementation and evaluation. In any community and with any social issue (hunger, homelessness, etc.) there are many organizations, polices, relationships, and resource allocations that are both part of the problem and part of the solution. One important outcome that should be assessed by/for any NGO is how well it understands and works within the community's systems. The American Evaluation Association published a book on systems thinking last year (Systems Concepts in Evaluation); the current issue of The Foundation Review has an article by David Stroh on systems thinking tools.

Steve said...

Ken and Hildy:

I was impressed by your statement on measurements based on improving the communities quality of life.

One of the programs the Silver Star Families of America operates is a Hospice Program.

Our Chaplain distributes Prayer Blankets to our dying veterans.

Can we measure this? Yes I suppose we can count blankets.

Does it get to the "root of the problem?"

Mercy I don't know.

But I DO know that it improves, not the community, but individuals quality of life and we are proud to be involved.

Maybe some things are not meant to be measured.

Ken Berger said...

Steve,

I commend your wonderful work. Providing blankets is a measurable activity (although not an outcome, some programs are output focused and that is more than fine). I suspect that there is much more that could be measured such as the amazing psychological boost you may give to people who have served our country.

Best,
Ken

Renata Rafferty said...

I've always been in firm support of "outcomes" assessment at the organizational level. The standards I suggest may be somewhat atypical, and ultimately I believe that a common sense "360" can often tell us as much "truth" about an organization's effectiveness as any set(s) of numbers. That's why I am a major cheerleader for CN's trip down the path less traveled by other ratings entities.

But simply substituting a different measuring stick -- which we're all still struggling to define -- will be as limiting (or misleading) as the financial ratio mirage if we continue to assess organizations on a silo basis.

Frankly, I think Hildy Gottlieb has jumped the shark on the rest of us. When -- and if -- we ever settle on some commonly understood and respected organizational outcomes assessment methodology (what a mouthful!), we will then inevitably need to move on to where Gottlieb has already landed ... that is, how important is simply measuring the independent outcomes of disparate organizations?

Ultimately, isn't our goal as the avant-garde of changemakers, as "social" venturers and entrepreneurs, and (gasp) as philanthropists, to improve communities, and ultimately the community of the human race and the global environment in which we live?

I think Hildy's point is "let's get to the point." And she has attempted to -- if not define a standard of measurement of that scope -- at least develop and put into practice an approach to community change that leapfrogs to that endpoint where the rest of us will eventually follow.

In my own work with philanthropists and foundations, I stress the importance of tying one's philanthropic investment to one's most basic beliefs about the inalienable rights granted by our very humanity.

And though we may give in order to effect some small or specific change, ultimately, it's about moving the ball toward some large, ultimate and presumably shared goal and vision for humanity.

The "world" is too big for us to take on, but we can take on the lofty goal of changing the fabric of our own community to more closely approximate our vision for humanity.

Frankly, that's way too big of a concept for which I would ever try to develop an evaluation methodology.

Whether Gottlieb can, will or even wants to be successful at developing systems for outcomes measurement at that level remains to be seen.

But, hell, I applaud her for pushing -- and exciting -- us to think about and measure our work against the ultimate goal line and not just at the "first down" marker (okay, so it's Superbowl month, I couldn't resist).

This has been a fascinating discussion -- thanks for bringing this controversial topic to the blogosphere!

Ken Berger said...

Renata,

Thank you for your thoughtful (as always) comments. I too wish Hildy well on her endeavor. I think we all have a full plate!

I just want to clarify how we frame the issue. No one I know who focuses on individual NP measurement assumes that they are not intending to have it result in "community improvement" as it has been incorrectly characterized by some on Twitter. Rather, it is a question of where to begin to make community progress.

A good example is the Harlem Children's Zone that, as an individual organization, is very much concerned about the entire Harlem community and how it can make it a better place through its efforts, block by block or zone by zone.

Also, it is understood that a high impact organization is fully engaged with the community and providing catalytic energy to advocate, educate and fully engage others in the effort (government, business, other NPs and individuals in the community, etc.).

Renata Rafferty said...

I've always been in firm support of "outcomes" assessment at the organizational level. The standards I suggest may be somewhat atypical, and ultimately I believe that a common sense "360" can often tell us as much "truth" about an organization's effectiveness as any set(s) of numbers. That's why I am a major cheerleader for CN's trip down the path less traveled by other ratings entities.

But simply substituting a different measuring stick -- which we're all still struggling to define -- will be as limiting (or misleading) as the financial ratio mirage if we continue to assess organizations on a silo basis.

Frankly, I think Hildy Gottlieb has jumped the shark on the rest of us. When -- and if -- we ever settle on some commonly understood and respected organizational outcomes assessment methodology (what a mouthful!), we will then inevitably need to move on to where Gottlieb has already landed ... that is, how important is simply measuring the independent outcomes of disparate organizations?

Ultimately, isn't our goal as the avant-garde of changemakers, as "social" venturers and entrepreneurs, and (gasp) as philanthropists, to improve communities, and ultimately the community of the human race and the global environment in which we live?

I think Hildy's point is "let's get to the point." And she has attempted to -- if not define a standard of measurement of that scope -- at least develop and put into practice an approach to community change that leapfrogs to that endpoint where the rest of us will eventually follow.

In my own work with philanthropists and foundations, I stress the importance of tying one's philanthropic investment to one's most basic beliefs about the inalienable rights granted by our very humanity.

And though we may give in order to effect some small or specific change, ultimately, it's about moving the ball toward some large, ultimate and presumably shared goal and vision for humanity.

The "world" is too big for us to take on, but we can take on the lofty goal of changing the fabric of our own community to more closely approximate our vision for humanity.

Frankly, that's way too big of a concept for which I would ever try to develop an evaluation methodology.

Whether Gottlieb can, will or even wants to be successful at developing systems for outcomes measurement at that level remains to be seen.

But, hell, I applaud her for pushing -- and exciting -- us to think about and measure our work against the ultimate goal line and not just at the "first down" marker (okay, so it's Superbowl month, I couldn't resist).

This has been a fascinating discussion -- thanks for bringing this controversial topic to the blogosphere!

Alan Arthur said...

Folks -
Every donor makes their own individual determination about whether the purpose/goals of a particular nonprofit are relevant to their lives and giving goals, or not. They also make their own determination about whether an organization's purported progress towards their organizational goals is appropriate, based upon either emotive evidence or outcome data.

Most donors probably require some reasonable portion of both, but I think research definitively shows that most current donors lean towards emotional or relational decision-making.

Organizations like ours must always do their work in the context of a broader set of community needs and goals, but I doubt seriously that most donors are going to be emphasizing that perspective in their giving, even if they should.

The impact of Charity Navigator and other rating systems helps donors make better choices, but its greatest impact is in helping nonprofit organizations themselves spend more time thinking about the important issues of effectiveness and efficiency. Those who consistently address effectiveness and efficiency will perform better, and have a higher chance of doing their work longer. And should.
Best Always,
Alan

Ken Berger said...

Alan,

I agree with your characterization of the state of things as it stands at the moment.

However, one of our goals is to educate donors as to the importance of results and effectiveness of charities. Over time, we hope that the awareness that, "not all charities are equal" and that identifying the most high performing charities is vitally important to maximize your charitable support, will become much more common place.

Best,
Ken