Monday, December 8, 2008

A Measure of Outcome

Charity Navigator evaluates far more charities every year than anyone else. Every day during the holiday season we are quoted by a number of media outlets somewhere in the country. Furthermore based on surveys we have conducted we can see that the outcome of our effort is to have tremendous influence on the giving behavior of donors to US based charities. That is a major part of our core mission.

We believe that our rating system gives a decent snap shot of the financial efficiency and capacity of charities in every sector from health care to animal welfare, from human services to the arts and humanities. However, this is not a time for us to rest on our laurels or to assume that we have got the whole picture in hand. I noted a while back that one of my core values is the belief in a continuous improvement process. That means we need to continually strive to make our services better. I believe that we are now at a point of opportunity to do just that by expanding our ratings into a whole new dimension.

We have advised donors that to make their giving decisions, our ratings are a "part of the puzzle". We specifically recommend in our 10 Best Practices of Savvy Donors that, in addition to reviewing our charity ratings, donors (among other things) should:

"1. Be Proactive In Your Giving - Smart givers ... are specific about the change they want to affect. For example, they don't just support generic cancer charities, but instead have targeted outcome goals for their giving, such as providing mammograms to at-risk women in their community....

8. Start A Dialogue To Investigate Its Programmatic Results - Although it takes some effort on their part to assess a charity's programmatic impact, donors who are committed to advancing real change believe that it is worth their time. Before they make a contribution, they talk with the charity to learn about its accomplishments, goals and challenges. These donors are prepared to walk away from any charity that is unable or unwilling to participate in this type of conversation."

"Programmatic impact", "accomplishments, goals and challenges" all relate to one degree or another to the outcomes of the work of the charity. What is an outcome? I believe a good definition is given in the book, Quicker, Better, Cheaper? (Edited by Dall W. Forsythe) "Outcomes are the events, occurrences or changes in conditions, behavior, or attitudes that indicate progress toward achievement of the mission and objectives of the program. Thus, outcomes are linked to the program's (and its agency's) overall mission - its reason for existing."

Over our six plus years, we have come to the conclusion that some donors are not taking the additional steps we recommend to assess outcomes and are relying almost entirely on our ratings. In addition, while we assume that financially strong organizations are far more likely to be effective in their outcomes, it may not always be the case. Alternatively, charities with mediocre financial strength may not necessarily have mediocre outcomes. Therefore, to help donors know with greater certainty which charities are achieving program results, we are exploring integrating outcome measurement into our rating system.

As a first step, we have begun conversations with a variety of experts in the field to see the current state of such measurement tools. In large measure, from what I have seen so far, it appears that the field is in its infancy and we will have to do some heavy lifting to get a meaningful tool that can be broad enough to evaluate all categories of charities, while not becoming overwhelmingly cumbersome and complicated. In other words, this is going to take us some time to develop. It could be years rather than months. In the meanwhile, we have a number of other efforts we will be making along the way to get the ball rolling. I will give you the details of these efforts in future blog entries.

Another challenge that leads us to the conclusion that developing this tool will take time is the fact that there is no standardized data source from a third party to get the information (like the IRS 990 that we use for financial analysis). Therefore, we will probably need to gather the information directly from the charities we evaluate. To say that many of them will not be thrilled by such a prospect is an understatement! However, at the end of the day, I think that most will welcome this expansion of our rating system with the hopes that it will more comprehensively capture what they do and how they do it. It will also further our core mission of providing guidance to donors on making intelligent giving decisions.

The leadership of Charity Navigator believes it is well worth the challenges it entails and as a result we are setting a goal over time of offering an expanded rating system to more comprehensively evaluate nonprofits and separate great organizations from the rest. We look forward to collaborating with our colleagues who are working on this issue to improve the state of charitable giving. I am certain that we are up to the challenge and will get to the right outcome as soon as we can!

19 comments:

Sean Stannard-Stockton said...

Bravo Ken! Good luck, you've got your work cut out for you. But I am very, very glad to hear where you are headed.

solopar said...

I loved your article and the direction that you are taking. I would love to see something like you can find at consumerreports.org about various products, but related to charities. I spent a little time looking for the best heart association to donate too, but didn't find the type of content that I was looking for. Even on your site, I didn't find the kind of information that helped me make a decision to donate. I did find information on a charity that I already know about and did make a donation to them. The point however is to provide better content to help drive donors to make pledges without having prior knowledge of the charity that they are pledging to. Hmmm... just a couple thoughts - hopefully to spark a little conversation. Thanks for your blog!

Ken Berger said...

Sean,

Thank you for your words of encouragement!

solopar,

Thanks for the feedback. We DO consider the information we provide an important component in making a decision to donate, however outcome is another very important component that should be included in the best giving decisions. As a first step, in the early part of 2009 we intend to have charities self report on their outcomes. Hopefully that will make it easier for you to get you some of the ADDITIONAL information you need going forward.

Nick Deychakiwsky said...

Ken,
I also applaud the direction you are taking, and urge you to bear in mind that there are others that have already started traveling down the path you are embarking on. Especially given the economically tight times, I hope you will be all traveling together.
Nick Deychakiwsky

Ken Berger said...

Nick,

I am hopeful we can collaborate with as many others as possible. To begin with, we have joined the Working Group for Effective Social Investing and the Independent Sector. I am also contacting as many experts (including Sean Stannard-Stockton of Tactical Philanthropy) as I can to talk one-on-one and see who the possible partners are out there.

Ed B said...

Ken,

I welcome your move to sort through the "outcome" question - which will not be easy. For our charity it will be easy - but imagine that will not always be the case. Thanks.

Anonymous said...

Ken

You are indeed an innovator! Outcome measures of reaching the mission rather than just measuring activity are long overdue. The tidal wave of "baby boomers" will be with you! I have been reading Social Innovation Review on this subject for a year now. They are in the forefront with you. Their Spring 2008 edition "More Bang for the Buck" article reports on the groundbreaking efforts of Messrs Neuhoff and Searle. Some charities like Kickstart and Year Up are moving in this direction. Just mentioning outcome measures in your reports will help us find more of them! I'll be one donor who will be watching your progress.

Jim from Wisconsin

Peter Mason said...

Excellent Ken. As both a donor and someone who works for a non-profit, I'm excited for this work. I work for a nonprofit that measures its results because we are a learning organization that has decided to constantly improve on how we fulfill our mission. Water For People (http://www.waterforpeople.org) can attest to how constant monitoring and evaluation of our programs is both a challenge and a blessing as it takes time energy and diligence, but allows us to truly know the long-term, sustainable affect of our work. I hope that more organizations with leaders such as you (and us!) in the vanguard can bring us all into the next generation of success at solving problems in the world. Cheers -

Peter Mason
Director of Marketing and Communications
Water For People

Anonymous said...

I was so pleased to read this post. As a nonprofit professional, I have come across your website many times and respect what you are trying to do, but feel strongly that your rating system does not tell the whole story. Thank you for your effort to conduct a more thorough analysis of nonprofits and improve your rating system.

Gonzalo Ibarra said...

I hope so some day we can to have a Charity Navigator for Latin America. Well Done Ken !!

bwallace said...

I am excited to hear of your interest in moving beyond the basic analysis of program vs management and fundraising expense as a measure of the quality of nonprofit organizations.

InterAction is the largest coalition of US based non-profits engaging in international relief and development. For many years now our members have been required to meet comprehensive standards of accountability which were developed at their request, and members engage in biennial self certification of their compliance. In addition, InterAction holds regular meetings of a Monitoring and Evaluation Working Group which provides an avenue for our members to share best practices and develop standards of effective measurement, monitoring and evaluation.

Given the depth of our work within the international development community I have often wondered at the usefulness of the baseline that you have been providing to those who are trying to determine in which organizations to invest.

We would be happy to share the work that we have done and are doing and to support your admirable efforts to provide more information to donors.

Barbara Wallace
VP Membership & Standards
InterAction

Nik C said...

Ken,

Great idea, and you do well to highlight the very real challenges that arise in attempting to evaluate non-profit work. If we don't count dollars, what should we count?

I'm perplexed by this: non-profits have been around for decades. Why is it that we're still inventing the wheel when it comes to outcome measurement? After a couple generations, we still don't have a standardized method of evaluating social progress. It's maddening... and troubling.

Every NP I've worked for struggles with the same metrics problem, and I always spend a good portion of my work year wrestling with the Evaluation sections of our grant proposals.

While I imagine that some missions are easier to measure fulfillment against (# of children's palates surgically corrected), I'm not at all sure that *the vast majority* of missions are not very difficult to quantify.

Despite many years and MANY billions, NGO aid work in Africa continues. And would an after-school tutoring program be considered a success if 30% of participant increased their test scores by 5%, or if one student eventually discovered the cure for cancer? If one judges "effectiveness" by the greatest good done for the greatest number, wouldn't the one student curing cancer be a better outcome? How to reconcile that with the millions more we all serve in some way?

I look forward to reading about the progress you make in addressing outcome measurement... but I won't hold my breath.

Human Development Foundation said...

Ken,

Look forward to your next iteration of charity evaluation process.

HDF (human development foundation) adopted UN Millennium Development Goals as its target goals (http://www.undp.org/mdg/basics.shtml).

It is always challenging to show donors month-by-month or even quarter-by-quarter progress towards these goals....But we are trying very hard.

Another benefit of your expanded charity evaluation is going help organizations like HDF.com, to approach donors, and charity funds to consider giving grants where the ratio of expenses vs. program spending is other than the 25%/75%. We can show great work to these donors, but they always cite the above ratio as the dominant criteria.

I look forward to your revised/expanded evaluation of charities.

Shahid.Khan@hdf.com

Domus said...

Our nonprofit (www.domuskids.org) has been measuring outcomes since 2002 and tweaking them every step of the way. The lack of standardization—every funder and consultant and expert has a different opinion on how they think it should be done—is frustrating to our staff and makes us wonder, when we’re feeling low, if we’ll ever master measuring the true impact of our work with vulnerable kids and families. If, in the course of your efforts, you can bring together major players and create a measurement rubric, millions of us toiling away at this will be forever grateful to you.

Ken Berger said...

Domus,

Thank you for your thoughts. I will be interested to get your opinion as we move forward in the development of our rating tool.

Ken Giunta said...

Dear Ken:

Having worked in the field of NGO accountability and standards for over ten years now, I very much appreciate your comments and advocacy for the expansion of measures used to assess the effectiveness of charities. While I have respect for the underlying premise of indicators offered by groups like Charity Navigator, they only tell a small part of a much larger story that needs to be told about any organization.

Mission and management systems are critical components of organizational relevance and effectiveness. In the NGO world, spending and overhead rates are quite different for an international relief organization, that is both quick to respond and quick to complete their work following a crisis, than they might be for a development agency that must, by necessity, stay in a country for a protracted period and hire permanent staff to help rebuild a health or education system that has been devastated by a conflict or a natural disaster. Does this mean that the latter type of organization is less efficient or effective than the former?...Not at all. It simply means that they have different missions and different management systems.

Assessing an organization positively based solely on its low overhead rates is also short-sighted. How exactly does a charity deliver any program without overhead, and why would any donor want to support a charity that does not allocate resources to the infrastructure needed to deliver the services promised? All organizations need staff, sound management, financial controls, human resources and governance systems, and basic physical infrastructure in order to do their work. How well an organization has developed these systems and infrastructure says more about their worth than their overhead. Personally, I find it odd that so many donors will only support overhead at levels well below where their own organizations operate. It would seem to be a much wiser and sounder investment to support the health and strength of the “institution” that is providing the services that the donor values.

I agree with you that donors should place more emphasis on measuring the effectiveness, efficiency and program outcomes of the organizations they choose to support. At the same time, charities also need to understand and accept that measuring program effectiveness and outcomes should be integral to every program design. With all this said, donors and charities must appreciate and agree that measuring program effectiveness is not an overhead cost, but rather, part of the cost of doing business. If donors demand it, but are not willing to support monitoring and evaluation as a valid direct program expense, we will just be “whistling Dixie” once again.

With much appreciation for these important next steps you are taking.

Ken Giunta
kjg212@aol.com

Ken Berger said...

Mr. Giunta,

I agree with much of what you have said in your comment. I just want to reiterate that we use seven variables in our current rating system - not just overhead. Regardless, we are intending to add a number of new variables in the future (hopefully not too distant) that encompasses outcomes.

Anyway, love the first name too!

Robert M. Penna, Ph.D. said...

In a January 4, 2009 comment here, “Jim from Wisconsin” mentioned a Neuhoff and Searle article in the Spring 2008 issue of the Social Innovation Review, “More Bang for the Buck.” In that piece, the authors describe a number of tools or ideas from the corporate sector that nonprofits ought to emulate in pursuit of greater productivity. Among these are standardizing best practices, investing in staff and critical activities, and managing costs aggressively. Good ideas all; but why stop there?

Six Sigma contains a number of analytical tools, “Critical to X,” the “Hidden Factory,” and the “VOC/VOP/VOB” balance among them, that would benefit any nonprofit that used them to analyze its procedures and offerings. SERVQUAL analysis is an excellent tool for uncovering the gaps that may exist between what a nonprofit thinks it is offering in and through its programs, and what its intended beneficiary audience comprehends is being offered. Finally, such notions as Value Engineering and Failure Mode Analysis can both greatly help nonprofits avoid costly mistakes in program design and implementation.

The point here is that if nonprofits are truly interested in increasing effectiveness, perhaps they should think of their initiatives more as “products,” and explore more of the tools common in the corporate world for assuring product quality.

Ken Berger said...

Robert,

Thanks so much for joining in the conversation.

For our readers, Dr. Penna is the lead author of OUTCOME FRAMEWORKS. His latest book is coming out any day now and is called THE OUTCOME TOOLBOX.