Monday, February 16, 2009

A Scary Finding on Outcome Measurement

Back in December 2008, I made it clear that Charity Navigator intends, over time, to report on charity's outcomes (see blog entry - A Measure of Outcome). As a first step down that road, we have been on a "listening tour" that includes researching existing tools that may help us in developing our system. To that end, we have identified over 50 organizations that have at least something to say on the matter. We are culling through them with the hopes that we can find components that are relevant. Our goal is to develop a measure of outcomes that provides meaningful information for charity donors (our users).

We have been testing out what information charities are currently compiling in the area of outcome measurement. We also intend to use the information to help us in developing our system. If there are some universally agreed upon outcome measures in a particular category of charities, it could help inform us on good standards. We assumed that most charities have SOME system of evaluating their outcomes. However, you know what they say when you assume. Suffice to say, you can end up with a bad ending! So far, less than 10% of the charities we have polled have provided us with information in this area.

I recently met with a colleague who funds organizations who can provide him with evidence of a good system of measuring their outcomes. He is getting similar results. The scary reality, he suspects, is that most charities (the overwhelming majority) have not even taken their first step down the outcome road. A couple of other experts I have bounced this around with have corroborated our findings.

I know from my many years of running charities that day-to-day survival mode is often the overriding focus and concern. In the current economic climate that reality has only intensified for most charities. So the lack of focus on outcome measurement is not likely to change any time soon, unless there are outside forces that demand it and resources that facilitate the process. We continue to assume that the larger agencies may be compiling this information, but may be reluctant to make it public. Even if this is true, only 4% of all charities have annual revenues in excess of $10 million. So our suspicion remains that the vast majority of charities are doing slim to nil in the area of outcome measurement.

I think that the experts, foundations and charity advocacy groups are going to need to educate government policy makers and the general public about the significant importance of publicly available, outcome measurement information before this situation will change. All grants, whether from government, foundations or corporations, should include a percentage to fund outcome measurement.

Why is this so important? As I noted last week (When Can Donors Trust a Charity?), I believe that an outcome driven culture is vitally important for a charity to be at its best and to be trusted. With all of the scandals and lack of confidence in charities, objective data will become more and more important for the public's perception of a charity's ongoing legitimacy. In such a climate, it's scary news that most charities probably are not measuring and documenting their outcomes.

Nonetheless, we are going to continue down this road and implement an outcome measurement system once we are confident it contains the right elements. We will also be a voice for the importance of outcome measurement to whoever will listen! However, I now anticipate that whenever we begin to evaluate charities on outcomes (probably no time soon), most will not do well, if for no other reason than that they are not documenting what they are doing. I hope I am wrong and will not let assumptions get in the way of the outcomes!

P.S. My colleague Vincent Bogucki here at Charity Navigator observed the following about all of this:
"Socrates made the statement at his trial, 'The unexamined life is not worth living'. Socrates freely chose death over life without measuring outcomes. Charities may have that choice forced upon them."

37 comments:

Gabrielle said...

Ken

I always enjoy reading your commentary and I also like the newly energized Charity Navigator. Maybe it's an hallucination but The CN feels alive. I give you the credit for that.

I remember you from the Floating Hospital days. So happy you found a place to shine!
Gabrielle
Children's Health Fund

Ken Berger said...

Gabrielle,

Thank you so much for you kind words. They mean a great deal to me. I wrote about something scary and I now get something so nice in return!

Kevin Jones said...

if 90 % of non profits are not measuring, maybe the value of measurement needs to be measured, count the cost and benefits. do the metrics create new value? or are they a transaction cost without additive value? does measuring half a dozen ways for each funder divert and distract from the mission? is more good created because it is measured, or less because of the cost of measuring? if only 10 percent do it, maybe the measurers need to recalibrate their metrics to make their measuring worth more.

Ken Berger said...

Kevin,

In my opinion, there is no question that measuring the outcomes of a charities work is worth its while. I believe that donors will demand it as we go forward. Furthermore, if you do not know whether or not you are truly having an impact, how do you know that what you are doing is truly fulfilling your charity's mission?

I also think you are on the right track in indicating that the metrics must be as cost effective and simple as possible to use. Otherwise, they will not be viable for most charities that are quite lean in infrastructure. Some of the tools that I am learning about out there may fit the bill (no pun intended!).

Ken Berger said...

Kevin,

In my opinion, there is no question that measuring the outcomes of a charities work is worth its while. I believe that donors will demand it as we go forward. Furthermore, if you do not know whether or not you are truly having an impact, how do you know that what you are doing is truly fulfilling your charity's mission?

I also think you are on the right track in indicating that the metrics must be as cost effective and simple as possible to use. Otherwise, they will not be viable for most charities that are quite lean in infrastructure. Some of the tools that I am learning about out there may fit the bill (no pun intended!).

Soytri said...

Kevin makes a great point. The value of measuring and metrics needs to accrue to the provider of the programs/services first and foremost with more informed decision-making about program delivery and resource allocation NOT because it makes donors happy. And the cost associated with gathering and analyzing the data needs to be less than the benefit accrued from having the knowledge.

When the priority is to make donors happy, the best expenditure of effort and resources is often in marketing, not in assessing true social returns.

Ken Berger said...

Soytri,

I must disagree with the idea that we do not need to prioritize getting donors information on charity outcomes. Making donor's "happy" should not just be about spending marketing dollars.

I believe there is such a thing as a bad donation and a good one. A bad donation is based on a sales pitch to pull the heart strings that may have no substance behind it. A good donation is where the heart meets the mind. Outcome information will not just make donors happy in the feel good way. I see more and more demand for objective data. It is indeed an important management tool for performance, but it is EQUALLY important for donors.

Soytri said...

Ken,
I agree that in an ideal world donation decisions would be made based on where the best outcomes are. (Disclosure: I'm a development director for a management support org whose clients are other nonprofits, and also consult with our clients to help them develop their logic models and eval systems... so I'm a believer in the importance of defining one's desired impact and clearly articulating one's outcomes).

But I question whether many donors, especially in the "long tail" of philanthropy, really care about outcomes... or if they care more about an emotional story, or who asks them for the gift, or who else is already giving to the organization (i.e. marketing).

These are all legitimate reasons for giving. The question for me is how to you make people prioritize impact over these other rational reasons for giving.

My guess is that there is a marketing component to this, too....

Soytri said...

Following up on comment from yesterday, Sean's latest Tactical Philanthropy blog post (where he references Godin's theory of rational marketers and irrational customers) is relevant to this discussion - http://tacticalphilanthropy.com/2009/02/philanthropy-ideas-that-spread.

bob said...

I agree that outcome measurement for any endeavor is a must. Can you please provide charity website with outcome measurements?

Bob

Ken Berger said...

Bob,

As I mentioned in the blog entry, we are looking at over 50 organizations that are doing this. Once we get further along we can share our findings but at this stage I am unsure of exactly who is doing what.

Btw, do we know each other?

Ken Berger said...

Soytri,

It is true that different kinds of donors have different levels of interest in charity information. However, I believe that we should provide what donors SHOULD be considering (like the over 3 million people who visited our site last year). The information can't hurt the others! Also, research I have read shows that donors going to the internet for information has doubled in just 3 years. All evidence indicates that the trend will only continue. Further, I think we have a responsibility to lift the bar on giving decisions to include issues of charity performance.

Paul Horne said...

Just read Dan Pallotta's new book "Uncharitable," which is all about the importance of outcome measurement, and how focusing on a charity's overhead percentage is a critically flawed way of measuring a charity's value. I hope you've read the book and I'd love to see you interview Pallotta on this issue.

Ken Berger said...

Paul,

I did read it and commented on it in a blog entry a few weeks back called - Is it a Sacrifice to Work for a Charity? Pallotta makes some good points, but I think he misses the mark quite a lot. Financial measures are important. Pallotta throws out the baby with the bath water quite a lot. Fundamentally the issue is this - A for profit model is not the same as a non profit model. To think otherwise is folly.

Andreas said...

Dear Ken,

Why do you not show which these 50 charities are and ask your users to rank the usefulness of their individual methods?

Kind regards,
Andreas Nilsson, Stockholm
(a new fan of Charity Navigator)

Ken Berger said...

Andreas,

Most of these entities are not charities. They are largely consulting firms and think tanks. Also, we are vetting them now to see who is REALLY doing relevant work that could inform our system development. We got the majority of the names from lists that Tactical Philanthropy and Lucy Bernholz compiled.

Ken

William said...

Well said, Ken. Very well done. And I couldn't agree more with Kevin that the value of measurement itself needs to be reconsidered. He's right to say that the reasons why outcome measurement—as usually practiced—is not more common stem from its failure to add value, that it diverts attention from the mission, and that it costs more than it is worth. But this doesn't have to be the case. There is a large technical literature and research tradition in the area of what makes measurement meaningful, both qualitatively and quantitatively. My passion in life is translating that work into practical tools that add value. It often happens that the dictum “you manage what you measure” is a warning as to how distracting accountability and transparency can be, relative to what you originally set out to do. But if your measures do not actually add up and if you don't actually know what you're measuring, which is quite often the case, then how do you know what you're managing? Properly calibrated tools, on the other hand, embody the mission. When you manage well-made measures, you're managing your core purpose. Good measurement lowers transaction costs and reduces friction, making for a more efficient market in the way it transforms the opaque local and concrete particulars of one collection of indicators and ratings into transparent global and abstract measures. Produce markets, for instance, need generalized metrics like weight in order to overcome uncontrollable variation in the size of oranges. Methods for arriving at the same kind of meaningful, practical, and scientifically rigorous measurement for social capital are readily available. I just wish more people knew about it. See www.rasch.org or my web site, www.LivingCapitalMetrics.com, for more information.

Vicknj said...

Right on target and it doesn't surprise me. I think there are a number of reasons why this is the case. One, it cost money to compile data, non-profits don't like spending money on anything other than programming so foundations, donors have to accept that some of their money goes to overhead. Another reason generational. I think Gen X and Gen Y (I am Gen X) tend to value outcomes as a necessity. I learned in my MPA courses the importance of measuring, older generations it wasn't so important. YOu sat down with a donor and they wrote a check, now you need concrete results. Last infrastructure. How many non-profits have databases where there isn't buy in from all departments? Without the buy in gather outcome information can be hard.

The question I would ask is how does one get organizations to recognize the importance of performance measurement?

Ken Berger said...

Vicknj,

I think the way a charity comes to know the importance of performance measurement can be both via the carrot or the stick. The carrot is when they see the concrete results it brings by looking at other organizations that have implemented it. The stick is when it is required by funders. Both have merit and are needed!

Ken

Anonymous said...

The Obama administration has proposed eliminating tax deductions for charitable contributions, which nobody wants to see. However, it might serve all parties if tax deductions were only allowed for contributions to charities that meet certain criteria. The criteria might include those that you already report, and at some point, outcome data that meets a certain standard should be a required criterion. As the financial advisor Harry Dent says, we're now in a Shake-Out season. It is actually healthy, for both the business and charity worlds, if only the fittest survive.

Ken Berger said...

Anonymous,

I totally agree with you that the fittest organizations should survive. However, requiring charities to conduct outcome measurement to get a tax deduction is worrisome to me. Who decides what outcomes are acceptable? The government? That is a scary thought!

Rebecca said...

Although our industry has a long way to go in regards to serving our communities efficiently and effectively, there are some organizations that are doing good work in the area of outcome measurement. United Way of America, Girls Incorporated, the Boys and Girls Club, Youth Villages, Nurse Family Partnership are just a few of these. As a MPA candidate working on my thesis I chose outcome measurement as a topic because I see the need to raise the bar of excellence in the independent sector. Ken – I am happy to offer what little help I can be in your organization’s quest to ensure that social value is attained through the thousands of nonprofit organizations in our country.

Ken Berger said...

Rebecca,

I would love to compare notes and see how we might collaborate. Email me at kberger@charitynavigator.org.

Anonymous said...

As the person in charge of measuring outcomes for a $14 million nonprofit org, I have seen first hand the benefit of measuring program effectivness.

Putting aside the obvious benefits (such as improving the quality of our programming, eliminating/chaning the programs that we've found actually cause harm), we've been able to have our investment in our evaluation work more than pay for itself.

To establish our own in-house three person evaluation team, we have spent approximately $500K over the past 3 years. In that time, we have raised over $1 million in additional/new funds for programming as a DIRECT result of our outcomes work.

Those organizations that fail to devote time and resources to outcomes measurement are living in the past. I often use this analogy:

Most nonprofits/charities have staff devoted to fundraising (getting money in the door).

Most nonprofits/charities have staff devoted to managing money once it is raised (finance and accounting people).

But very few nonprofits/charities have staff devoted to measuring program/service effectivenss.

Why?

In my mind, all three of these things should be equally treated. If a nonprofit/charity fails to measure the effectiveness of their programming, they have no idea what their dollars are doing.

Would we be satisfied with nonprofits/charities that failed to have accounting/finance staff? Or that failed to have audits? Probably not.

Then why are we satisifed with nonprofits/charities that fail to document the effectiveness of their work?

Isaac D. Castillo
Director of Learning and Evaluation
Latin American Youth Center
isaac@layc-dc.org

Ken Berger said...

Isaac,
Well said!

Barbara said...

In this economy, the organizations that add value will be the ones who remain standing. Our Habitat for Humanity organization is evaluating everything we do through that lens. "Activity does not mean progress" is my motto.

We have a Dashboard (that I have shared with other Habitat affiliates) that tracks our outcomes against benchmarks. This highlights how different departments' performance (or lack thereof) affects other departments. It also streamlines the board meetings; we no longer spend lots of time weeding through monthly department reports or finances....everything of importance is on the Dashboard and one can check the reports if more information is needed. Some of the items are directly off of Charity Navigator's current rating methodology (working capital, cost to raise $1).

This has been a huge benefit to the organization in terms of identifying problems and needed policies.

I applaud Ken's desire to come up with a performance-based measurement for Charity Navigator.

Ken Berger said...

Barbara,

I in turn applaud the wonderful work you describe. Great stuff!

Ken

Anonymous said...

I'm new to the this blog, but have written some academic papers on the topic. So I'll toss my own humble opinion into the ring.

I think the discussion so far has ignored an important point. Some nonprofits produce things which are more easily measured than others. For example Habitat has very countable output. Alternatively, an advocacy organization or a think-tank might have output that is very difficult to quantify. There is a famous economics paper: B Holmstrom, P Milgrom - Journal of Law, Economics, & Organization, 1991 that poses an analogous problem. In brief, the paper warns to be careful what you measure, because those things which can be measured will tend to be emphasized relative to things which cannot, independent of which is actually more important. For example, it is relatively easy to measure how many patients ad NP nursing home facility treats, it is difficult to measure how kind they are to patients. Place incentives (i.e. rankings)on that which is easier to measure and it is likely that particular dimension of service will receive disproportionate attention.

One can easily imagine where, either across NP firms, or within a particular organizations, those tasks which can be easily measured by CN will get undo emphasis.

Accountability has its place, but donors need to be reminded that if it was easy to measure output, then for-profits could do it.

Ken Berger said...

Anonymous,

You make an excellent point and have put your finger on a major challenge for us in exploring whether or not there is ANY measure that can cut across a wide variety of types of charities. I also agree that we must be careful not to go down a dead end path that measures the wrong thing or is too vague to be meaningful. Stay tuned.

Vivek said...

We are very young non-profit who believes in measuring the outcome of our work. We are working to help needy children though education. Any suggestion on how we can measure our results/impact.

www.EmpowerKidsFoundation.org

Ken Berger said...

Vivek,

You are certainly off to the right start! Here are two good resources I am aware of -

1) The Center for What Works http://www.whatworks.org/

2) Social Solutions
http://www.socialsolutions.com/

Best of luck and please let us know how things progress for you!

Ken

cactusmitch said...

"Wilderness" seems to be a cash register word for charities. The marketing of "eco-tourism," and the slick ads, (say in the Sierra Club publications,) have me wondering about the net effects of the "good efforts" of well intentioned folk. Outcomes must be estimated if not measured.

Reb said...

Ken - has there been any update on how many nonprofits are actually implementing outcome measurement? At the beginning of this article you state 10% are doing this. If no update, do you know of anyone who has facts on actual number of organizations?

Ken Berger said...

Reb,

What we are finding is that, if you are looking for meaningful evidence of outcomes (not just having a performance management system but one that really shows results), it actually DROPS to 1 or 2% according to our expert colleagues. I have not seen any hard research on this question.

As to where you might find a group of organizations that are doing this work, you can contact the list of members of the Alliance for Effective Social Investing (www.alleffective.org). Many of the members are working to fund or assist these organizations. Jeff Mason is the Chair of the group so you can start with him. I also particularly like the work of David Hunter and David Bonbright.

Best,
Ken

Tom King said...

A vast number of small nonprofits face a grim future. These mom and pop responses to local needs and challenges operate on a financial shoestring, fueled by passion, love for their communities and a string of miracles that keep them open. They don't understand outcomes measures, they only know the people they help or the good things they do. They've little idea how to quantify that. They can't assign three staff members to "quantify the efficacy of their outcomes measures". They don't have 3 employees to spare and the volunteers have no idea what you're talking about.

What they do know is that they can no longer compete effectively for grant funds or significant donations because these donors are asking for things they don't knnow how to give them. They need technical expertise they can't afford. There's a whole industry of "consultants" out there providing that expertise. I'm one of them. The problem is the ones who really need us can't afford us. There's a natural selection process going on that's weeding out the little guys. It's a shame really. They do some of our best work.

Tom

Ken Berger said...

Tom,

You make an excellent point. However the opposite point is also true - that there are many small and inefficient nonprofits that should either get acquired by a larger agency or close their doors. Over 90% of charities are less than 1 million in annual revenue and 45% are less than $25,000. We have had an explosion of teeny weeny charities that speaks to - on the one hand the heart of people to help and on the other hand the lack of thinking/researching existing agencies to work with. Sometimes it is necessary and unique circumstances that require a small charity. However, it is also very common to find ego, ignorance and terminal uniqueness as motivators.

Scott Bechtler-Levin said...

Ken,

Thank you for another thoughtful post and the work you and others are doing to expand the criteria available to judge nonprofit effectiveness.

As you know, many outcomes of nonprofit activity benefit those 'outside' the explicit mission of the organization. Those reflected outcomes are quite beneficial to society and the nonprofit sector, as whole.

As an organizations focused on building the capacity of nonprofits through knowledge sharing and collaboration, we find that some of the most interesting outcomes occur when dissimilar organizations discover that the resource or tool they shared was able to help another organization in a very different sub-sector.

The data from thousands of these online exchanges could become an interesting 'proxy' for an organization's contribution to cross-sector collaboration and a validation of how other organizations value that contribution. With no additional effort by the charities’, organizations like Charity Navigator could aggregate this new “collaboration index” as a criteria to help donors evaluate potential philanthropic investments.

Scott Bechtler-Levin
IdeaEncore Network
www.IdeaEncore.com