Comments on Ken's Commentary: Hildy Gottlieb on Outcomes - Open Forum Participan...

Alan, I agree with your characterization of the s...

2010-04-07T12:23:13.685-04:00

Alan,

I agree with your characterization of the state of things as it stands at the moment.

However, one of our goals is to educate donors as to the importance of results and effectiveness of charities. Over time, we hope that the awareness that, "not all charities are equal" and that identifying the most high performing charities is vitally important to maximize your charitable support, will become much more common place.

Best,
Ken

Folks - Every donor makes their own individual det...

2010-04-07T12:16:58.189-04:00

Folks -
Every donor makes their own individual determination about whether the purpose/goals of a particular nonprofit are relevant to their lives and giving goals, or not. They also make their own determination about whether an organization's purported progress towards their organizational goals is appropriate, based upon either emotive evidence or outcome data.

Most donors probably require some reasonable portion of both, but I think research definitively shows that most current donors lean towards emotional or relational decision-making.

Organizations like ours must always do their work in the context of a broader set of community needs and goals, but I doubt seriously that most donors are going to be emphasizing that perspective in their giving, even if they should.

The impact of Charity Navigator and other rating systems helps donors make better choices, but its greatest impact is in helping nonprofit organizations themselves spend more time thinking about the important issues of effectiveness and efficiency. Those who consistently address effectiveness and efficiency will perform better, and have a higher chance of doing their work longer. And should.
Best Always,
Alan

I've always been in firm support of "outc...

2010-01-12T14:50:32.929-05:00

I've always been in firm support of "outcomes" assessment at the organizational level. The standards I suggest may be somewhat atypical, and ultimately I believe that a common sense "360" can often tell us as much "truth" about an organization's effectiveness as any set(s) of numbers. That's why I am a major cheerleader for CN's trip down the path less traveled by other ratings entities.

But simply substituting a different measuring stick -- which we're all still struggling to define -- will be as limiting (or misleading) as the financial ratio mirage if we continue to assess organizations on a silo basis.

Frankly, I think Hildy Gottlieb has jumped the shark on the rest of us. When -- and if -- we ever settle on some commonly understood and respected organizational outcomes assessment methodology (what a mouthful!), we will then inevitably need to move on to where Gottlieb has already landed ... that is, how important is simply measuring the independent outcomes of disparate organizations?

Ultimately, isn't our goal as the avant-garde of changemakers, as "social" venturers and entrepreneurs, and (gasp) as philanthropists, to improve communities, and ultimately the community of the human race and the global environment in which we live?

I think Hildy's point is "let's get to the point." And she has attempted to -- if not define a standard of measurement of that scope -- at least develop and put into practice an approach to community change that leapfrogs to that endpoint where the rest of us will eventually follow.

In my own work with philanthropists and foundations, I stress the importance of tying one's philanthropic investment to one's most basic beliefs about the inalienable rights granted by our very humanity.

And though we may give in order to effect some small or specific change, ultimately, it's about moving the ball toward some large, ultimate and presumably shared goal and vision for humanity.

The "world" is too big for us to take on, but we can take on the lofty goal of changing the fabric of our own community to more closely approximate our vision for humanity.

Frankly, that's way too big of a concept for which I would ever try to develop an evaluation methodology.

Whether Gottlieb can, will or even wants to be successful at developing systems for outcomes measurement at that level remains to be seen.

But, hell, I applaud her for pushing -- and exciting -- us to think about and measure our work against the ultimate goal line and not just at the "first down" marker (okay, so it's Superbowl month, I couldn't resist).

This has been a fascinating discussion -- thanks for bringing this controversial topic to the blogosphere!

Renata, Thank you for your thoughtful (as always)...

2010-01-12T09:43:42.391-05:00

Renata,

Thank you for your thoughtful (as always) comments. I too wish Hildy well on her endeavor. I think we all have a full plate!

I just want to clarify how we frame the issue. No one I know who focuses on individual NP measurement assumes that they are not intending to have it result in "community improvement" as it has been incorrectly characterized by some on Twitter. Rather, it is a question of where to begin to make community progress.

A good example is the Harlem Children's Zone that, as an individual organization, is very much concerned about the entire Harlem community and how it can make it a better place through its efforts, block by block or zone by zone.

Also, it is understood that a high impact organization is fully engaged with the community and providing catalytic energy to advocate, educate and fully engage others in the effort (government, business, other NPs and individuals in the community, etc.).

I've always been in firm support of "outc...

2010-01-11T22:23:10.398-05:00

Steve, I commend your wonderful work. Providing b...

2010-01-10T22:05:17.503-05:00

Steve,

I commend your wonderful work. Providing blankets is a measurable activity (although not an outcome, some programs are output focused and that is more than fine). I suspect that there is much more that could be measured such as the amazing psychological boost you may give to people who have served our country.

Best,
Ken

Ken and Hildy: I was impressed by your statement ...

2010-01-09T20:34:30.962-05:00

Ken and Hildy:

I was impressed by your statement on measurements based on improving the communities quality of life.

One of the programs the Silver Star Families of America operates is a Hospice Program.

Our Chaplain distributes Prayer Blankets to our dying veterans.

Can we measure this? Yes I suppose we can count blankets.

Does it get to the "root of the problem?"

Mercy I don't know.

But I DO know that it improves, not the community, but individuals quality of life and we are proud to be involved.

Maybe some things are not meant to be measured.

This discussion is a great example of why NGO'...

2010-01-07T17:15:36.217-05:00

This discussion is a great example of why NGO's, funders and evaluators could benefit from systems-thinking based approaches to program design, implementation and evaluation. In any community and with any social issue (hunger, homelessness, etc.) there are many organizations, polices, relationships, and resource allocations that are both part of the problem and part of the solution. One important outcome that should be assessed by/for any NGO is how well it understands and works within the community's systems. The American Evaluation Association published a book on systems thinking last year (Systems Concepts in Evaluation); the current issue of The Foundation Review has an article by David Stroh on systems thinking tools.

Hildy, I point to a sample of notions suggested i...

2010-01-07T11:28:55.678-05:00

Hildy,

I point to a sample of notions suggested in the Pollyanna Principles book:

"1. What indicators will show a community that feels spiritually whole and integrated?
2. Imagine that the norm in measurement is celebration and reflection!
3. Funding everyone. Collective capacity building rather than individual capacity building."

All of this to me seems pointing toward a lack of MEANINGFUL!, individual accountability for performance.

Ken et al: Thank you for this lively debate. I am ...

2010-01-07T10:08:45.049-05:00

Ken et al:
Thank you for this lively debate. I am sorry I have been in full-day sessions and unable to participate since this posted.

I just want to address Ken's suggestion that my work does not address individual organizational effectiveness within community effectiveness. I guess to say I am surprised is beyond understatement.

My entire book, The Pollyanna Principles, focuses on building organizations that are effective in all ways - externally and internally. Our website is filled with dozens of articles on organizational effectiveness, as is my blog. We have a YouTube channel that addresses how organizations can hold themselves accountable for both end results and then the means to accomplish those results.

As I am sadly in a rush and heading into another full day of strategy, I will provide those links and then unfortunately have to run.

http://pollyannaprinciples.org/
http://www.youtube.com/communitydriveninst
http://www.communitydriven.org

The bottom line is this: if we want our communities to improve, we must aim at that improvement, and only then use organizations as one among many tools towards that improvement. If we aim only at the tools - the organizations - we are missing the opportunity to create the kinds of visionary community improvement that is possible.

Hildy

Ken is right that we need to start with the indivi...

2010-01-05T17:31:48.554-05:00

Ken is right that we need to start with the individual nonprofits. How can we possibly expect to measure progress at the community level if we can't even do so at the organizational level? And how can community level measurement be meaningful without understanding whether the efforts of individual organizations are helping or hurting?

Bonnie, Jane, Gayle and Maureen, We are pleased t...

2010-01-05T12:13:30.588-05:00

Bonnie, Jane, Gayle and Maureen,

We are pleased to hear that Hildy is a strong supporter of organizational effectiveness measures. When we read her book and her blogs we do not find it highlighted anywhere to the degree that community-wide measurement is. Given the sorry state of affairs in the NP sector, where most do not measure themselves, we believe it is going to be a long time coming before individual measurement happens for most, let alone community-wide measurement! We believe that this individual effort MUST come first before we can talk in a substantive way about community measures, not the other way around.

We agree that the most effective NPs are fully engaged in their communities with other NPs, government, business and their supporters. We also understand the value in setting community-wide measures to coordinate efforts. However, given where most NPs are at and the resources available to them, individual performance management and measurement should be the first priority.

Let us agree to disagree on this. As we said in our original response to Hildy, we only wish you well in these efforts. We are happy to work in parallel and collaborate with you whenever we can.

I agree that both community assessment and the eff...

2010-01-04T15:38:25.851-05:00

I agree that both community assessment and the effectiveness of individual organizations are important to building a better future. It is imperative that we have the tools to sort out the effective nonprofits from the ineffective nonprofits, supporting those who are doing the most good.

I think that funders may have a role to play in ensuring that the community aspect is addressed. All funders have a focus for their grantmaking. Community Foundations have a regional focus, many large private foundations are focused on a number of program areas. If funders can take a more active role in 'taking the temperature' of their funding communities and sharing that information with the world. If we can find a way to match that information with our list of effective nonprofits. If we all take responsibility for our communities, finding the gaps and seeking to fill them - oh what a world it could be!

I'm with Hildy and Bonnie on this. While I...

2010-01-04T14:32:06.142-05:00

I'm with Hildy and Bonnie on this.

While I'm an avid champion of data, measurement and organizational learning, I also know how incredibly complex this measurement of societal outcomes is. I have repeatedly seen first hand that no NGO is an island onto itself. And I'm repeatedly reminded that a critical part of being effective is having effective implementers at the helm.

Just coincidentally, this past week I've been reading Michael Pollan's "In Defense of Food". Pollan's discussions of the inadequacy of the science of nutritionism to make us healthier reads as a cautionary tale for those of us who are are embarking on the same scientific reductionism around NGO outcomes.

Pollan points out that despite all the "scientific" analysis and altering of food and eating, our Western diet (including the foods, the farming, and the processing) has only proliferated the diseases that result from insulin resistance, obesity and other factors with unknown impact on our health (e.g. antibiotics and hormone additives).

Pollan notes that new studies appear almost daily that contradict past definitive studies on what we should or shouldn't be eating.

Pollan writes:
"Scientists study variables they can isolate... yet even the simplest food is a hopelessly complicated thing to analyze, a virtual wilderness of chemical compounds, many of which exist in intricate and dynamic relation to one another, and all of which together are in the process of changing from one state to another. So if you're a nutrition scientist you do the only thing you can do, given the tools at your disposal: Break the thing down into its component parts and study those one by one, even if that means ignoring those subtle interactions and contexts and the fact that the whole may well be more than, or maybe just different from, the sum of its parts." (p62)

Pollan is not necessarily suggesting that we shouldn't continue with science, but he raises alarm bells at science that does not appreciate or study the interactions of complicated systems.

One could easily apply these same words to NGOs.

Like Hildy & Bonnie, I am very worried that the complex interactions between nonprofits, their people, and their communities will be ignored and lost in a reductionist rating system that rewards isolated actions.

I am also concerned about harmful unintended consequences that can't be measured in the short term, or a failure to measure or value positive unintended consequences that are not necessarily the primary mission of a particular NGO but are valuable secondary outcomes nonetheless (e.g. development of community leaders), about the capacity of the richest NGOs to game the rating systems while the least resourced continue to be starved of funding, or the ignoring or devaluing of important societal outcomes that may only occur at the most micro levels (e.g. human dignity, leading a fulfilled life) (see my story "Hope, Dignity and Quality of Life are also valuable outcomes -- even when measured in hours." http://bit.ly/8iYuEk)

Yes, what Hildy is suggesting is hard, but it is very much worth doing. If the goal is making beneficial societal change, then funders need to reward community-wide learning systems by funding them.

While I know that Ken is approaching this much more thoughtfully than most, I have deep concerns nonetheless. I can see a rating system that notes whether NGOs are attempting to measure outcomes, but I am very, very concerned at rating systems that definitively ascribe societal outcomes to any particular NGOs.

I agree with Bonnie - Hildy is in fact a very stro...

2010-01-04T13:08:52.429-05:00

I agree with Bonnie - Hildy is in fact a very strong supporter of organizational effectiveness measures and is certainly not suggesting dropping that activity in favour of only community-wide measurement. However, as she has noted, traditional measures have often focussed mostly on internal efficiencies when external results are far more important.

Another important reason for an emphasis on community-wide measures is that often change is initiated and maintained by groups of people who are not even associated with any formal organization. Their efforts are critical, but they are completely missed when only organizations are considered.

I understand the resistance. We have struggled to have good tools and resources for organizational effectiveness, and want to use them. And we should. Our tools for community-wide effectiveness are not only newer and less tested, they are also more community-driven. Experts have to give up some of their power, and realize the wisdom is in the community. Is the community is changing it desires to, or not?

I don't believe that Hildy is saying that out...

2010-01-04T11:49:32.833-05:00

I don't believe that Hildy is saying that outcomes measurements for individual orgs and communities are an either/or - she is wisely saying that they must go together, and that the latter has not been given the focus that the former has. We have a tendency in our society (as we have become more specialized in our education and professions) to segment what are in reality integrated/holistic challenges that call for integrated/holistic approaches and solutions.

The evaluation of individual organizations is certainly important for many of the reasons that you have outlined but individual organizations do not (and should not) exist, operate, nor develop programs, policies and services in a vacuum. To what end they are doing their work (improvements of their communities) should be a crucial part of outcomes measurements.