Monday, August 10, 2009

David Bonbright on Outcomes - Open Forum Participant

Our second participant in the Open Forum on Outcomes is David Bonbright who is the co-founder of Keystone Accountability. David has more than 20 years of experience in the field of social and economic development. He helped to create Keystone to address the weak accountability among funders and civil society organisations to their primary constituents – those meant to benefit from their work – and to innovate measurement tools that support meaningful and lasting developmental impacts. Over the last five years, Keystone has worked with a wide range of organizations to design new practical ways of planning, measuring, learning and communicating social change that foster accountability and learning among all constituents, and enhance positive outcomes.


Ken Berger and Robert Penna*: Should charities be assessed on their use of outcomes?
David Bonbright: Yes, if by that one means how they plan for, measure and learn from outcomes. They must ask and make their best answer to the question: What difference does our organization make? Because of the complexity and relative cost of answering this question for most charities, we need to find 'good enough' answers that are comparable across charities. This takes us to the realm of things that we can measure relatively easily in the short term that have a strong proven correlation to outcomes. The leading such predictive measure in the world of commerce is customer loyalty. Keystone is adapting comparative customer loyalty measures for charities with considerable success. For one great example applied to US high schools see the Gates Foundation - Center for Effective Philanthropy project

KB & RP: Do individual donors understand the value of outcomes? If not, what’s the best way to educate them to its value?
DB: I think folks get this intuitively. They have just never had the data, so they don't know they can expect it. For years I have been asking people a simple question, "If you were convinced that your giving was making a real difference, would you give more?" Virtually everyone says yes.

Comments from KB and RP: It is not surprising that there is an almost universal desire to be more generous. Even in these difficult economic times, surveys show that people intend to give as much or more than in the past. It reflects great optimism and generosity but not necessarily what will end up happening or the value of outcomes! So do donors actually put the charities to which they contribute to the test: DO THEY ask for outcomes? DO THEY consider whether the “outcomes” cited by charities make sense? Do THEY ask for any information at all? Unfortunately, more often than not the answer is no. Most donors, particularly individual donors, have not been informed about the importance of measurable outcomes as an indicator of nonprofit effectiveness. We hope over time to help remedy that situation!

KB & RP: What does outcome information tell a potential donor?
DB: It tells them what difference the organization makes. In the case of measures that are predictive of outcomes -- which are most often what we have to rely on -- it tells them the qualities and interim results of the organization that are likely to lead to outcomes. We need to benchmark outcome data across charities.

Comments from KB & RP: The key here, we think, is the quality of the “outcome information” offered. Are the charities in question actually supplying outcome information, or are they erroneously listing activities and outputs and calling them outcomes? Experts in the field tell us that there are characteristics of “good” outcomes. A logical question therefore is whether the outcomes cited by a charity meet these criteria. It all comes down to how faithfully a charity is actually applying outcomes.

KB & RP: Can “outcomes” really be applied across the board throughout the charitable sector? Are there areas where “outcomes” simply don’t apply? If so, how is the value of investments in those areas to be determined?
DB: Yes, I believe they can be applied across the board. Or maybe it is better to say 'outcomes thinking' as in many cases the best we can do is track the interim results that are likely to lead to outcomes. For example, the efforts of a counselor with at risk youth may only become visible years after the counseling. But at the time of the counseling, there are signs that can be picked up by asking youths across an entire counseling program that are highly indicative of later outcomes. Another way to say this is that there are short-term outcomes that can be measured and reported, and an associated logic that explains the likelihood of longer term outcomes. In some cases for certain kinds of work -- say writing books on moral ethics -- it is very difficult to make confident predictions about the relationship between short term results (how many people read the book) and outcomes. But even there, meaningful data can be readily gathered, such as what proportion of people who read the book register on the associated website. A survey of a sample of those who never heard of the book, who read the book and did not register on the website, and those who read the book and did register will allow a useful analysis of the outcomes from the book being written and published.

Comments from KB & RP: We agree that outcomes can and should be applied across the charitable sector: no portion of the sector should be “immune” from outcomes or measures of effectiveness. But certain subsets of the sector are considerably ahead of others in the identification and measurement of outcomes. In such areas as direct services, housing, substance abuse, community development and environmental protection, measures or indicators have existed for some time. But in other areas, nonprofit organizations are still struggling with how to apply the outcomes concept(s) to their work. In the areas of culture, the arts, and in advocacy, agreement has not solidified as to what the outcomes or indicators really should be. There are other examples that could be given, but the point is that more work still needs to be done to truly make outcomes applicable as a matter of practice across the charitable sector.

What do you think?

*Dr. Robert Penna is an independent outcomes consultant and is assisting Charity Navigator in managing this Open Forum. He is the author of Outcome Frameworks and the forthcoming Outcomes Toolbox.

4 comments:

Ingvild Bjornvold said...

It is great to see such discussion around the question of outcomes and performance, and as Ken knows, I fully support Charity Navigator’s quest to find ways of rating nonprofits in this area.

As the conversation continues, however, it strikes me how important it is for us to clarify what an outcome actually is. People work with different definitions, but in my work, we think of outcomes as relatively enduring changes in people attitudes, health, behaviors, social condition and so on, which can reasonably be expected to result from the work of the program.

The implication of that definition is that not all programs can produce outcomes. Take a soup kitchen, for example. While it provides meals to people who drop in, it does not change anything about the reasons why they need food from the soup kitchen in the first place. It does not improve their outcomes.

This doesn’t mean it’s not a valuable service; it most certainly is, but we need to look at these “output programs” differently from outcome producing programs. They can also manage performance (meals must comply with food safety standards, etc.), but it won’t make sense to ask them to promise and report on outcomes which are unrealistic.

Ken Berger said...

Ingvild,

Thanks for the reminder that not all nonprofits have outcomes to be measured. However, in your example, wouldn't we hope to at least see that soup kitchen making referrals to other organizations that do undertake efforts that would lead to more enduring changes in peoples lives?

Ingvild Bjornvold said...

Yes, a soup kitchen making referrals to other programs that will work toward outcomes would be excellent. All the things the soup kitchen measures, though, including the referrals, are about program quality and outputs, not outcomes. The soup kitchen can and should manage performance, but the performance is all about outputs rather than outcomes.

Any system that compares organizations on a broad scale will need to take that into account. I believe it is very important for us to be clear about the difference, so we don't add to the already reigning confusion and nonprofits' feelings of pressure that they should be able to achieve often unrealistic outcomes. It is OK for soup kitchens and similar programs to produce outputs - their services are needed, too. Nonprofits should know that.

Ken Berger said...

Ingvild,

We shall take this reality into account. Thanks.