Tuesday, June 28, 2011

Managing to Outcomes Forum

On June 13, 2011 we co-hosted an event (along with the NYU Wagner School of Public Service) titled, "Managing to Outcomes". We have lots of information to share about the event and begin with a panel presentation that reviewed CN 2.0 as a case study that may be used by colleges around the country in their course work. See below.

Executive Summary - Charity Navigator 2.0 Case Study
Charity Navigator (hereafter CN) is America’s largest and most influential charity rater. CN was founded in 2001 with the simple purpose of providing those looking to make charitable contributions with an easy way to help them determine where to direct their funds. In 2010, over 4.7 million visits were made to our web site, which impacted billions of dollars of charitable donations.
CN until now has rated charities on a single dimension - financial health. Critics have pointed out that relying on this measure alone may send more donations to organizations with ineffective programs or unjustly penalize organizations that may incur higher overhead but are ultimately more efficient and effective. In response to this critical feedback and our own research, at the end of 2008 CN announced plans to change the rating system to include other important factors in making a wise social investment/charitable giving decision. We have been working toward that goal ever since.
The new system (which is called CN 2.0) considers the three dimensions we believe are the critical components to such donor decision making. We continue to believe that financial health is a vitally important dimension to assure that the organization is sustainable and fiscally efficient in its operations. The second dimension - accountability & transparency will be factored into the rating system this summer (2011). This dimension includes the governance and ethical practices of the organization as well as how openly available information is to all key stakeholders. The third dimension, and ultimately the most important one – results, is currently in the testing phase. This dimension considers the key question as to whether the organization is reporting on how effectively it operates its program(s) and meeting its mission of providing meaningful change to the people and communities it serves.
Once preliminary testing on the results dimension is completed, data will start to be gathered and tabs of information will begin to go up on our charity rating pages in 2012. However we anticipate it will not be reflected in charity ratings for a few years. This is because we must wait until all the charities have been evaluated on this dimension before we integrate the data into the overall rating (otherwise our rating system is inconsistent from one charity to the next). In addition, this last dimension is the most complex and time consuming to gather and testing will continue throughout the data gathering period with further improvements as we go. We believe it is critical to undertake this effort because our research shows that nearly 80% of our users rely primarily or solely on our current one dimensional ratings. Therefore, we believe to that it is our obligation to supply donors with this more comprehensive 3-D view of a charity.
At the same time, over the next 5 years CN plans to almost double the number of charities (to 10,000—those that command ~70% of the revenue flowing into the sector each year) that we rate. This will be accomplished by creating a web platform through which we can train, certify and guide a corps of engaged and committed volunteer raters (625 projected within five years).
Last year, CN received a $75,000 grant from the William and Flora Hewlett Foundation to begin the important work of developing and testing the 3-dimensional CN 2.0 rating tool. Our first pilot occurred in the Fall of 2010, using graduate students as volunteer raters. In the post-rating survey, many students and their professors pointed out the need to simplify and reduce the degree of subjective judgment in the results rating criteria to ensure that those ratings can be undertaken reliably by a large number of different and diverse raters.
For the Spring 2011 Pilot, Charity Navigator strove to address this student and professor feedback in three structural changes, including the development of a web-based rating application with simple yes/no questions and the requirement to provide evidence (via the charity’s web site page links) for all yes answers. We also developed clearer and more elaborate instructions in the rating manual. The second change was to implement a two-step rating process in which students rate a charity individually and then they come together in groups of 3 to create a consensus group rating. Thirdly, the Spring Pilot did not allow students a free choice as to the charities that they rated. Instead, Charity Navigator assigned students specific charities that we already understood to be either high performers (based on outside expert reviews of results) or very low performers on the current CN ratings. The evaluation of the Spring pilot is currently in process and will lead to further improvements for the next round of testing.
We have just been awarded a $100,000 Hewlett renewal grant to continue work on further development and testing of the rating tool. Paul Brest, Hewlett’s President, has noted that
CN 2.0 “is no small task, but one we believe to be important for the future health and dynamism of the nonprofit sector”.
Looking further down the road—once CN 2.0 has been fully realized—there are a number of additional initiatives we intend to pursue. Firstly, we hope to eventually rate 20,000 charities which garner ~85% of the revenue that comes into the nonprofit sector each year. Secondly, we hope that the “cognitive surplus”[1] will help us to go beyond the largest charities and that many thousands more will be rated by a growing core of engaged and committed volunteers who are interested in smaller charities in their local area. Finally, we also foresee a not too distant future in which we will assess and evaluate the comments and approaches of various partner rating organizations, and consider how to incorporate their findings into a weighted overall rating that would represent CN 3.0 – the next level of ratings for the future.
In sum, we believe that our revised rating system has the potential to begin a process that will completely transform the landscape of charitable giving. We foresee that, through this effort, there will be an increase in the number of social investors that use our services, which will lead to a sizable increase in giving to higher performing nonprofits. Ultimately we believe this will lead to a measurable improvement in human welfare and acceleration in solutions to our world’s most persistent problems.

[1] Shirky, Clay. 2010. Cognitive Surplus: Creativity and Generosity in a Connected Age. New York, NY: Penguin Press.

1 comment:

Geri Stengel said...

This is good news, indeed. I hope nonprofits recognize the value of participating in the evaluation, even if it takes a little time, because it will help the sector establish much needed standards. Thank you, Charity Navigator.