Our second participant in the Open Forum on Outcomes is David Bonbright who is the co-founder of Keystone Accountability. David has more than 20 years of experience in the field of social and economic development. He helped to create Keystone to address the weak accountability among funders and civil society organisations to their primary constituents – those meant to benefit from their work – and to innovate measurement tools that support meaningful and lasting developmental impacts. Over the last five years, Keystone has worked with a wide range of organizations to design new practical ways of planning, measuring, learning and communicating social change that foster accountability and learning among all constituents, and enhance positive outcomes.
Ken Berger and Robert Penna*: Should charities be assessed on their use of outcomes?
David Bonbright: Yes, if by that one means how they plan for, measure and learn from outcomes. They must ask and make their best answer to the question: What difference does our organization make? Because of the complexity and relative cost of answering this question for most charities, we need to find 'good enough' answers that are comparable across charities. This takes us to the realm of things that we can measure relatively easily in the short term that have a strong proven correlation to outcomes. The leading such predictive measure in the world of commerce is customer loyalty. Keystone is adapting comparative customer loyalty measures for charities with considerable success. For one great example applied to US high schools see the Gates Foundation - Center for Effective Philanthropy project
KB & RP: Do individual donors understand the value of outcomes? If not, what’s the best way to educate them to its value?
DB: I think folks get this intuitively. They have just never had the data, so they don't know they can expect it. For years I have been asking people a simple question, "If you were convinced that your giving was making a real difference, would you give more?" Virtually everyone says yes.
Comments from KB and RP: It is not surprising that there is an almost universal desire to be more generous. Even in these difficult economic times, surveys show that people intend to give as much or more than in the past. It reflects great optimism and generosity but not necessarily what will end up happening or the value of outcomes! So do donors actually put the charities to which they contribute to the test: DO THEY ask for outcomes? DO THEY consider whether the “outcomes” cited by charities make sense? Do THEY ask for any information at all? Unfortunately, more often than not the answer is no. Most donors, particularly individual donors, have not been informed about the importance of measurable outcomes as an indicator of nonprofit effectiveness. We hope over time to help remedy that situation!
KB & RP: What does outcome information tell a potential donor?
DB: It tells them what difference the organization makes. In the case of measures that are predictive of outcomes -- which are most often what we have to rely on -- it tells them the qualities and interim results of the organization that are likely to lead to outcomes. We need to benchmark outcome data across charities.
Comments from KB & RP: The key here, we think, is the quality of the “outcome information” offered. Are the charities in question actually supplying outcome information, or are they erroneously listing activities and outputs and calling them outcomes? Experts in the field tell us that there are characteristics of “good” outcomes. A logical question therefore is whether the outcomes cited by a charity meet these criteria. It all comes down to how faithfully a charity is actually applying outcomes.
KB & RP: Can “outcomes” really be applied across the board throughout the charitable sector? Are there areas where “outcomes” simply don’t apply? If so, how is the value of investments in those areas to be determined?
DB: Yes, I believe they can be applied across the board. Or maybe it is better to say 'outcomes thinking' as in many cases the best we can do is track the interim results that are likely to lead to outcomes. For example, the efforts of a counselor with at risk youth may only become visible years after the counseling. But at the time of the counseling, there are signs that can be picked up by asking youths across an entire counseling program that are highly indicative of later outcomes. Another way to say this is that there are short-term outcomes that can be measured and reported, and an associated logic that explains the likelihood of longer term outcomes. In some cases for certain kinds of work -- say writing books on moral ethics -- it is very difficult to make confident predictions about the relationship between short term results (how many people read the book) and outcomes. But even there, meaningful data can be readily gathered, such as what proportion of people who read the book register on the associated website. A survey of a sample of those who never heard of the book, who read the book and did not register on the website, and those who read the book and did register will allow a useful analysis of the outcomes from the book being written and published.
Comments from KB & RP: We agree that outcomes can and should be applied across the charitable sector: no portion of the sector should be “immune” from outcomes or measures of effectiveness. But certain subsets of the sector are considerably ahead of others in the identification and measurement of outcomes. In such areas as direct services, housing, substance abuse, community development and environmental protection, measures or indicators have existed for some time. But in other areas, nonprofit organizations are still struggling with how to apply the outcomes concept(s) to their work. In the areas of culture, the arts, and in advocacy, agreement has not solidified as to what the outcomes or indicators really should be. There are other examples that could be given, but the point is that more work still needs to be done to truly make outcomes applicable as a matter of practice across the charitable sector.
What do you think?
*Dr. Robert Penna is an independent outcomes consultant and is assisting Charity Navigator in managing this Open Forum. He is the author of Outcome Frameworks and the forthcoming Outcomes Toolbox.