Wednesday, February 6, 2013

The First Step Toward a Measure of Outcome

On December 8, 2008 I posted a blog here titled, “A Measure of Outcome”. In that blog entry I discussed the critical importance of knowing the outcomes of the work of charities and made the following promise, “we are setting a goal over time of offering an expanded rating system to more comprehensively evaluate nonprofits and separate great organizations from the rest”. Over the intervening years we have been working hard to do just that. The launch of CN 2.0 in 2011 was a step in that direction. However, on January 23, 2013 we took our greatest leap forward in that effort. We launched the new Results Reporting dimension to our evaluation of charities (the overall system is now called CN 3.0 and includes three dimensions – (1) financial health, (2) accountability & transparency, and (3) results reporting).

The reason this is such a big deal has been explained in a number of places on our web site as well as in that original blog post back in 2008. Today, I would like to share a specific story that I think sums it up. A couple of months ago, before we shared the draft concept note regarding Results Reporting with a group of 140 experts, we wanted to run it by a hand full of people we considered the most respected experts in the field. One of that small group was the former head of the Hewlett Foundation (our largest foundation funder) and now a professor at Stanford Law School, Paul Brest . He had told us in the past that he was maintaining a “wait and see” attitude toward the work Charity Navigator was doing in the area of outcomes and nonprofit performance. A few days after sending him the materials, our Board Chair (Pat Dugan) and I had lunch with Paul to get his reaction. I will never forget what he said when we asked him what he thought of what were working on. He replied, “this is the most important work going on in the nonprofit sector.” After almost falling off my chair, I turned to Pat and said, “Did you hear what Paul just said?” Pat smiled and simply replied, “Yes I did.” We mentioned Paul’s comment (with his approval) in our discussion of Where We Are Headed and at the end of that document, gave our own interpretation of why we think Paul made that statement. We explained that,
We believe it [i.e. the work we are doing on results reporting and what will follow] is that important because donors will have access to much more robust information than ever before about each charity’s ability to bring about long lasting and meaningful change in the world. It is also important because many more charities will become focused on measuring and managing their performance. In other words, we believe this work and the new rating system that is evolving out of it is critically important because our users will be able to direct even more money to high performing charities.  Ultimately we believe this will lead to a significant and measurable improvement in human welfare and acceleration in solutions to our world’s most persistent problems.
I think that sums it all up in a nutshell. I give my most heartfelt thanks to Paul Brest for putting some serious “wind in our sails” as we move forward on this endeavor to help change the nonprofit sector and, by extension, the world for the better!

In the coming months I will be using this blog site to explain a bit about each of the five elements of the Results Reporting dimension of our evaluation system. Stay tuned!

6 comments:

Tosca Bruno-van Vijfeijken said...

I have a similar reaction to what Paul told you: this will influence the sector in fundamental ways. Even if we can quibble about details of the approach, it will drive the sector to focus more on outcomes, outcome assessment and public reporting, and that is valuable. My hat off to you as a change leader, Ken!

Toni Goldfarb said...

Ken, congratulations on this important new effort.

A question: Will you be using each charity's published reports to analyze the effectiveness of their programs? I receive many impressive reports from charities I support, often containing photo features about individuals who have been helped or showing ongoing projects with obvious results. However, I'm always a bit skeptical.

Here's why: As a medical writer, I know that the majority of research studies which get published in medical journals show positive results; negative results have a much lower chance of journal acceptance. I'm sure it's the same for philanthropic groups, whose publications aren't likely to feature programs that failed.

With that outlook, I'm eager to know what methods you've devised that could have equal success in uncovering data on funds used for unsuccessful programs, as well as those used for projects that successfully met the organization's stated goals.

Ken Berger said...

Tosca,

My hat is off to you as well for your wise counsel and all the help that you and your students have provided in getting us this far. I look forward to continuing to work with you on this as we go forward.

Thanks!
Ken

Ken Berger said...

Toni,

The more reflective the published reports are of quality analysis, the better. I anticipate that many or even most charities will "cherry pick" what reports they make public initially, but over time we anticipate more and more reporting (of both "good" and "bad" results) will occur.

Charities that publish evidence that they had "bad outcomes", as well as those that show evidence they are learning from their "mistakes" or poor performance, are a rarity today. This will probably take a number of years to change. At this stage, ANY reports, even if slanted in one direction, are a beginning. As it stands today, we have next to nothing for most charities, so the "cherry picking" is in the direction of picking not even the "juiciest plums" to share!

Once we have multiple years of data, if there are goals/outcome indicators set and no published reports on how the organization performed, we can anticipate either "cherry picking" or simply a lack of evaluation occurring. That will be where we can start to discern the gaps. Beyond that, we will learn as we go what other signals there are for missing reports. Perhaps we can rely upon concerned stakeholders to share these gaps with us? I believe these evolutionary steps will then be built into our analysis of the quality of reporting at that stage of our evaluation systems development.

Best,
Ken

Todd J. Sukol said...

Moving in the right direction, Ken. Those of us that have watched (and sometimes criticized) CN over the years have an obligation to acknowledge the wisdom, dedication and progress toward creating barometers of effectiveness. Looking forward to seeing how 3.0 plays out and moving forward yet again. Kudos. Todd Sukol, www.domoremission.com

Ken Berger said...

Thanks Todd!