Comments on Ken's Commentary: Announcing an Open Forum on Outcomes

Rob, Thank you for your very thoughtful comments....

2009-08-07T09:24:20.617-04:00

Rob,

Thank you for your very thoughtful comments. I entirely agree with your core points and share the concern that efforts to create social value may become nothing more than an adjunct issue or trendy superficial funding strategy rather than the real deal.

Let us fight for the real deal.

Ken, The biggest challenge facing effective use o...

2009-08-07T02:25:46.114-04:00

Ken,

The biggest challenge facing effective use of outcome systems to date is that they are feared, too complexly perceived or cavalierly introduced as an adjunct to evaluation or as a supplemental administrative tool simply to track program performance.

This is at best an immature view that almost entirely misses their powerful potential.

Outcomes must become an operational necessity regarding the design and implementation of any program or service, as well as the defining category of any intended success, period.

Furthermore, our field is getting way off track in the implementing of outcomes and outcome systems because we face two structural forces creating substantial drift in their application:

1. Consultants and professionals desire to promulgate outcome methods as an adjunct evaluation product or service that not-for-profits can procure after the fact or as a way to satisfy funder compliance.

2. Philanthropists and funders also continue to avoid their responsibility for establishing outcomes at a level of measurable and verifiable end client gains.

Because of this we instead default to outcomes as a set of intrinsic metrics based on implementing agencies a la "capacity building" or efficiency gains in services delivered, as if such activities were a legitimate proxy for achieving greater human gains to a level of significance outside organizations in targeted populations of need.

We too well know our industry is a third-party payor system, where funders are the de facto customer as regards not-for-profit development activities; as such resource and talent is drained from investing in product development that should begin by defining the significant end client gains intended and then proceed to vet such assumptions by compressing the innovation cycle of delivering such benefits to clients more effectively. instead, our not for profits organizations are trying to produce cars while spending time complying with mandates for better buggy whips. There is a huge mismatch in evolutionary scope and timing.

There are effective corrective actions and upgraded frameworks for correcting each of these forces in turn, but the bottom line is this:

Outcomes will never systemically deliver traction in the philanthropic industry until they outgrow the immature perspective of captive professionalism and administrative supplements. Outcomes must be rightly perceived as the initiating DNA, operating focus and subsequent day to day management skill within high-performing, not for profit organizations and be DEFINED AS END CLIENT GAINS--i.e. you don't seek EXTRA funding to do outcomes, it must be the initiating premise of your mission and simply the way you do business every day with your clients.

Only when we require such information to do our daily work will the quest for harmonizing outcomes make sense within and outside the charitable organization. And this operational necessity to know how outcomes are delivered must be explicit, transparent, shared and clearly desired at all levels: from clients to line staff to program directors to executive to Board members to partners and funders. At that point outcomes become natural and easily used, just like paper--ubiquitously valuable yet hardly remarked upon.

Kind Regards,
Rob Hanna
Social Wealth Partners

Joe, Welcome aboard Joe. Let me know if you need ...

2009-08-05T16:54:02.713-04:00

Joe,

Welcome aboard Joe. Let me know if you need me to come out to Dublin to help you!

Just come across your discussions and am very inte...

2009-08-05T16:49:09.691-04:00

Just come across your discussions and am very interested to take note of all the comments and participate in the Open Forum. I am in the early stages of researching a PhD examining "How does board governance contribute to the success of Not For Profit organisations ?" so I hope to learn and also contribute to the discussions... Joe Houghton, Smurfit Graduate School of Business, Dublin, Ireland (joe.houghton@learningtalk.ie)

Reb, I agree that Whatworks.org is a great resour...

2009-07-31T19:14:46.162-04:00

Reb,

I agree that Whatworks.org is a great resource for nonprofits to learn about developing their ability to better manage their performance.

I do not know the other group but hope to learn more about them.

Carolyn - Allison in an earlier comment mentioned ...

2009-07-31T19:03:09.849-04:00

Carolyn - Allison in an earlier comment mentioned innonet.org. This is an excellent resource for nonprofit professionals to use so they don't have to reinvent the wheel. You may find some work already done in regard to HIV and early child development and how it relates to measuring meaningful impact. Whatworks.org is another great resource that has been funded by the Urban Institute which shares best practices on taxonomies and again, meaningful impact. Both these organizations may have helpful information that you can use as a springboard to your benefit.

Collaboration is vital in this effort to deliver excellence in all that we do to make the world a better place.

Carolyn, Thanks so much for your insights. I am i...

2009-07-17T12:46:32.076-04:00

Carolyn,

Thanks so much for your insights. I am increasingly convinced that, whereas the for profit sector has been able to boil down their engine to measure success in one word - profit, the NP sector can boil it all down to one word too - impact. At CN we believe that there are three components required to get an NP to be high performing (financial health, accountability and outcomes) but ultimately all these components should lead to impact of significant social value.

I think this is a fascinating debate and there are...

2009-07-17T11:42:33.683-04:00

I think this is a fascinating debate and there are many important threads here - the "far factor", the cost and challenge of measuring, the debate on what is truly measurable. It is a debate that we are keenly interested in and have been for quite a few years at Save the Children. Having come from the corporate world before joining the non-profit sector over 11 years ago, I have to admit to having been mystified on how a very large, far flung non-profit like ours could actually achieve results without clear measurement of indicators of performance. Of course we had the easily measured financial measures, on which we did very well, and the project by project indicators that donors have demanded for many years. But how were we measuring and importantly talking about, the change we were making globally in key programmatic areas like HIV Aids progams for OVC's or Early Childhood development work, or Child Survival around the world?

The answer was were were not and most organizations were not. Tragically, this lack of focus on visible and straight forward results that people can understand reinforces a fact of life that we all deal with in our work - that the main indicator that the public, corporate partners, volunteers, donors and other interested stakeholders use to decide whether to support a specific non-profit in ANY way, is a set of financial measures that I openly admit have little to do with outcomes for beneficiaries. And that is because we have not given people anything else by which to judge us.

This lack of performance indicators also contributes to a "beneath the surface" but important factor that manifests itself in several interesting ways - the deeply held feeling of many that maybe business, with it's clear "bottomline" orientation, is actually better at achieving socal change results than a bunch of "do-gooders" who can't tell us what they have achieved. All one has to do is look at the Bush adinistration pattern of spending US aid funds over the past 6-7 years to see this manifested - the amount of funding going to for-profit "contractors" has grown tremendously, while the share to NGO's has shrunk dramtically. This ties directly to a belief (misguided it is now becoming evident in many case) that for-profit organizations who sign up for "concrete" measures are more accountable for results to vulnerable populatons and can be more successful in driving change.

I think the announcement by CN to take this on is extremely exciting and we look forward to working on it with many others, but it will also be a real challenge. I have found in our work on global indicators of outcomes for children that our biggest stuggle has been with defining which indicators to use. I think CN is on the right track of letting organizations define those performance indicators for themselves. Right now, a set of common ones seems unattainable, though I hope I see the day when this is possible so you can actually get us all pulling in the same direction to do things like save children's lives and make sure every child has access to education.

But the "system" then for comparing indicators across organizations will be tough. We have experienced this in our own organization in comparing results in the same sector across countries and it has been a source of large debate.

In addition, I strongly support the comments of many that it is very important to make sure we are incorporating the voices of our beneficiaries more strongly in our indicators of success. Yes we may have delivered water, food and shelter materials to millions of people in an emergency, but what if what would have really changed their lives even more was an ability to develop new livelihood options or get kids back to school quicker? We need to most importantly know what those we serve value and how they think we are doing on providing them. This is an area that really just now is getting the attention it needs.

So thanks Ken for launching this and we look forward to helpng make it happen!

Carolyn Miles
COO
Save the Children

James and LeAnn, Agreed on all counts!

2009-07-02T09:23:44.398-04:00

James and LeAnn,

Agreed on all counts!

I think these two additions are necessary. I worke...

2009-07-01T18:36:53.390-04:00

I think these two additions are necessary. I worked for a charity who just stopped granting wishes, however continued to collect from donors - 10's of thousands of dollars each week!! Donors are called upon to support a charity - they assume the money is going where the charity says it's going... not always the case. Make them prove where the money is going and the outcome - we (donors) need to know the money we donate is doing good and not just lining Executive Directors and Presidents pockets! Trust me, this is necessary. Not all charities are on the up and up....

It seems that some organizations will not measure ...

2009-07-01T17:53:44.939-04:00

It seems that some organizations will not measure outcomes because they fear they will not do it perfectly. We must all beware of "letting perfection be the enemy of good." An organization can start with managing those things it can identify and monitor. Just stopping waste, or ineffective solicitation communications, is an outcome. And we all learn as we go along, so keeping useful records of what we have learned while trying to achieve our stated goals is itself a measurable outcome.
I am fortunate to be involved in an organization that can actually count the people we help and the island habitat we assist them in preserving, but even we seek to measure our efficiency in what we do.
Jake Walker, Seacology
Saving the World one Island at a time.

Saundra to Allison, I agree with Saundra that pre...

2009-06-12T09:26:31.625-04:00

Saundra to Allison,

I agree with Saundra that pressure will need to be applied from social investors (donors with their eyes wide open) for more nonprofits to embrace performance management. However, there are also some encouraging signs within the sector too. Read the comments above from Andrew Niklaus and Bridget Laird above as examples!

Allison, I agree that the fear factor needs to be...

2009-06-11T18:56:27.523-04:00

Allison,

I agree that the fear factor needs to be reduced before aid agencies use the new gizmos for evaluations. But I don't think that will happen from inside the aid world. There have already been numerous attempts from within to improve practices, but initiatives such as HAP-I (not evaluation but accountability) do not have any clout unless donors back them.

That's why I've decided to focus on changing donor habits. If donors begin to reward only those aid agencies doing evaluations, and do not punish negative findings (unless management does nothing to improve the organization based on those findings). With the requirement of evaluations becoming standard, evaluation findings will become more common, thus less stigmatizing. And the fear may change to what will happen if they don't do evaluations.

Unless donors force it, I don't think the industry will change its ways. There's too much money and momentum keeping it the way it is.

Saundra, You raise a good point about another pro...

2009-06-11T15:54:07.759-04:00

Saundra,

You raise a good point about another problem with many nonprofits not sharing information. I believe that the term - Balkanization (to break up into hostile units) is part of the problem. I have seen far too many examples of nonprofits that behave in a very adversarial, hostile, secretive and defensive fashion. They will go over a cliff before they will open up.

There is a myopic, closed mindedness among some agencies that boggles the mind and goes beyond fear of loss of funding when they have bad results. I hope that as we begin to measure transparency (as well as outcomes) we can highlight those who unclench their fists!

Allison, I agree that fear is an important factor...

2009-06-11T15:37:46.278-04:00

Allison,

I agree that fear is an important factor as to why most nonprofits do not embrace a performance management system that measures the social value of their work. However, I have worked in enough nonprofits to tell you without hesitancy that there is another big challenge - selfishness that trumps mission. Sadly, there are far too many examples of nonprofits where the leadership cares little about outcomes beyond lining their pockets. I believe that the heightened measurement and accountability that is evolving will help to erode this horrific problem.

Thanks Ken for catalyzing this very interesting di...

2009-06-11T15:23:57.923-04:00

Thanks Ken for catalyzing this very interesting discussion. And my heartfelt greetings to David Hunter whom I have known for many years and highly esteem!

I founded Innovation Network (www.innonet.org) many years ago to help nonprofits and foundations use evaluative tools to learn how to improve their efforts. I have more recently been engaged in research and writing about ways that social media can improve the effectiveness and learning for causes.

The ambitions of this effort are admirable, at the same time, the challenges are quite daunting. And it begins at the beginning: who is trying to learn and for what reasons. For years funders like United Ways have tried to improve an evaluative framework to no great effect, in fact, I often hear groups not only disdain the imposed process required by United Ways but admissions from groups that they were telling them what they wanted to hear.

The barriers haven't been cost or even technical skills (although I have argued that the social science construct is often too difficult for groups to comfortably use and that lighter learning models are needed, but more on that another time!)the barrier is fear. Fear that your efforts are not as successful as you had hoped or have been telling people that they are, fear that your funding will be pulled (which isn't just an idle fear), fear that your board will be disappointed. A pervading, paralyzing, overwhelming fear.

Unless and until the fear factor can be reduced (and it can only be reduced by developing new norms and cultures within nonprofits themselves not be outside entities whether they are funders, or, perhpas, if you don't mind my saying, Charity Navigator) then the development of new tools, gizmos, processes, frameworks will fall flat.

Thanks again, Ken, for facilitating this conversation.

I am pleased to see Charity Navigator opening a fo...

2009-06-11T13:29:53.180-04:00

I am pleased to see Charity Navigator opening a forum on evaluating outcomes of non-profit work. Evaluating the work of non-profits is critical to both improving the quality and professionalism of aid work, and to empowering donors to hold aid agencies accountable.

As it stands, far too few aid agencies do any real evaluation of their work and when evaluations have negative findings they are often not shared with donors. Of the 200 aid agencies that responded to the tsunami in Thailand I know of less than a handful that actually evaluated their work of those, none shared their findings with donors. I've gone on a little rant about this very topic on my own blog - http://informationincontext.typepad.com/good_intentions_are_not_e/2009/06/its-time-to-stop-telling-pretty-stories-and-start-really-evaluating-the-impact-of-aid.html

As others have said, just because non-profits have the best intentions does not mean that their program is helpful. It is time that the aid world improved their professionalism by consistently evaluating their work and making their findings public. Only then will aid really improve.

Best of luck to Charity Navigator and Ken in their efforts.

Helen I have not seen any research on the subject...

2009-06-08T14:54:39.635-04:00

Helen

I have not seen any research on the subject. However, I believe that common sense dictates that the outcome of the nonprofit's work is important to people!

We believe that donors are relying on us to be the experts on "intelligent giving" (so that they can become effective social investors). They are trusting us to determine what constitutes a "good" nonprofit. It is clear to us that outcomes are critically important in that determination.

As to the use of anecdotes and case studies, it gets back to my earlier comment - if you can't measure it, you can not be held accountable. That is not acceptable to us.

As a past consultant at a couple of development ba...

2009-06-08T11:29:03.021-04:00

As a past consultant at a couple of development banks, it is not just non-profits that have problems clearly defining their outcomes/outputs/results or measuring their performance. There is limited accountability for the billions of dollars of loans/investments the development banks make, despite large evaluation departments to take on the cost.

Although instinctively it should be the case that donors are more interested in non-profits with demonstrable results and outcomes, is there anyone out there who has done the research to find out if this is actually the case? Given the history of donations to 'good causes' only, rather than for 'good results', it would be interesting to know.

And, to add to Anonymous' comment, what can be measured in relation to performance, impact and results, should be, and what can't can still be collated in case studies as additional anecdotal data and examples.

Bridget, I salute you and congratulate you on bei...

2009-06-05T10:02:56.313-04:00

Bridget,

I salute you and congratulate you on being one of the leading organizations in this battle for the heart and soul of the social sector.

I work for a results-driven nonprofit that’s been ...

2009-06-05T09:58:55.342-04:00

I work for a results-driven nonprofit that’s been focused on outcomes and accountability from its beginnings. We did not want to be another well-meaning program trying to help kids living in poverty. We wanted true impact.

We quickly found ourselves in the very small minority of nonprofits that consider outcome-based practice not only valuable but mandatory. Constantly we’ve had to defend to supporters why we spend precious time and money on this effort. And the outcomes we seek at WINGS for kids – instilling a strong social and emotional education – are by no means easy to measure.

But it is without question the only way to know whether we’re effective and why.

After we began working with David Hunter to improve upon the way we tracked our efforts, it was amazing to see how this data allowed us to align our efforts to outcomes and respond quickly with course corrections. We became more effective – and we constantly receive information that tells us in precise detail how and why.

It takes time to implement these new technologies. But the payoff is big. I can now sleep better at night because we ensure that our hard work is actually making a difference for the kids we serve.

Bridget Laird
COO, WINGS for kids
Charleston, S.C.

Ingvild, Thanks so much for your contribution. I...

2009-06-04T11:17:55.425-04:00

Ingvild,

Thanks so much for your contribution.

I recommend that people check out http://www.socialsolutions.com/ to learn about their tools to help nonprofits develop an "efforts to outcomes" system.

If Charity Navigator can successfully integrate ou...

2009-06-04T10:11:42.283-04:00

If Charity Navigator can successfully integrate outcomes into its rating system, I think it will be an important contribution to changing the nonprofit sector to the better. It puzzles me that so many people, like Anonymous, argue that we should “Hire the right people, give them resources, tell them what you want done, and let them try things until they hone in on the right strategy.” Clearly, the assumption is that whatever we do, it must do some good.

As Ken, David Hunter, Andrew Niklaus and others have pointed out, however, good intentions and hard work do not necessarily result in any good at all – in fact, countless evaluations have shown that it can result in harm! It’s time to turn the conversation from our responsibility to nonprofits (they have no inherent claim on our support) to our responsibility to all the people who rely on those organizations to help them improve their lives.

Innovation, trial and error – it can be good, but only so long as it is based on research about what is already known, so an organization can feel confident that the new approach is likely to benefit rather than harm people, and so long as the organization has a performance management system in place to capture what works and what needs to be improved – immediately (not years down the road). A lot is known about what works, and I agree with George Overholser that it is worth the effort to expand a really effective program like the Nurse-Family Partnership.

It is true that it is expensive to evaluate programs, and very few programs are in a position to go through rigorous evaluation by external researchers. We shouldn’t require that of all nonprofits, and it wouldn’t make sense for Charity Navigator. What we should require is that they manage performance internally. Performance management is different from evaluation, and all nonprofits can do it at a reasonable cost. It is about gathering and reviewing real-time information – internally – to answer some key questions:

• Are we reaching the people we intend to reach?
• Are we implementing our programs at the frequency, duration and level of quality we believe is necessary in order to achieve the outcomes?
• Are our clients making progress toward the expected outcomes?

Few organizations are managing performance because they think it is too difficult. First Place for Youth, which Andrew Niklaus represents, the Latin American Youth Center, Roca, Our Piece of the Pie, Wings for Kids and other organizations, however, show us that it is indeed possible. It is not as difficult as most people think, as Ken the Accountant also pointed out.

And when donors have the choice between organizations who manage performance and those who do not… Well, I think the choice is clear. More and more organizations are recognizing the need for performance management, and it would be great if more funders would recognize the need to fund those efforts. They are critical to producing the social value we all want, and which we owe to the people who need nonprofit organizations’ services.

Ingvild Bjornvold
Social Solutions

Ken the Accountant, Amen brother!

2009-06-03T16:58:43.224-04:00

Ken the Accountant,

Amen brother!

william, Indeed I hope we can hear from all persp...

2009-06-03T16:54:57.647-04:00

william,

Indeed I hope we can hear from all perspectives on the issue. That is why we are calling for an Open Forum here.

At the same time I intend to be transparent about my perspective - that outcomes (which can be defined as the measurable social value of the work) are critically important and the very reason for nonprofits to exist.

Thanks for the reminder though and may those who agree as well as disagree express their views!