Monday, August 24, 2009

David Hunter on Outcomes - Open Forum Participant

Our next blogger for the Open Forum on Outcomes is Dr. David Hunter. Dr. Hunter has some 30 years of experience running and consulting to human services organizations. He pioneered the use of performance management methods and developed a unique approach to helping organizations implemement Strategic Performance Management. He is also a member of the Alliance for Effective Social Investing and has developed an assessment tool (along with Steve Butz) to measure the potential for a nonprofit to create social value.

Ken Berger and Robert Penna*: Should charities be assessed on their use of outcomes?

David Hunter: I'm not sure what the question means. Should nonprofits track their outcomes? else can they know they are doing a good job?! Should investors use outcomes to assess how good a return they are likely to get by investing in a nonprofit? Sure...but impacts are even better!!! And to be confident that outcomes/impacts will continue to be produced reliably, investors should also look at the nonprofit's internal performance data.

(Of course for some nonprofits one can expect only outputs...e.g., for a soup kitchen. Here the issue is tracking indicators of output quality.)

KB & RP Comments: The issue the question was meant to frame is whether nonprofits should be held accountable by both institutional and individual investors (donors with their eyes wide open) for the “return” they deliver on the monetary investments made in their work. We wholeheartedly agree with David that impacts are better than outcomes. To achieve impact however, an organization will need to be able to rule out any other variable for causation of the benefit provided. This is a very costly endeavor that most organizations will not be able to undertake. Therefore, for most, measuring outcome indicators is as good as it is likely to get. We also agree with David that it is essential for nonprofits to have an internal performance management system that provides data that tracks their outcomes. In other words, investors should ask whether a nonprofit is doing what it claims to be doing and effecting the changes it claims.

KB & RP: What is the real state of the outcomes movement? Is it spreading and growing, has it hit a plateau, or is it declining due to the economic downturn or other factors??

DH: I think it is undeniable that "outcomes" have become a hot topic. I don't know if one can say there is an "outcomes movement" or not...I think the economy has had no effect on the nature of outcomes discussions...they are ubiquitous...

Of course, there is many a slip between speechifying and action...and lots of outcomes talk remains just that. I think it is fair to assert that many funders are, in general, still amazingly unconcerned about the matter in practice .

KB & RP Comments: Even before the contraction in the economy took place, many nonprofits were already making the claim that they could not “afford” to identify, manage toward, track or verify outcomes. Many said they could not afford the training necessary to understand outcome concepts or to master the tools available for putting outcomes into practice, that they could not afford to devote resources toward tracking outcomes and certainly could not afford any verification. A tightened economy has only increased these visceral reactions. Counterintuitive though it may be, these tough financial times are precisely when both nonprofits and funders ought to be making the necessary investments in implementing outcomes:
1. It is especially in times of tightened fiscal resources that nonprofits should want to demonstrate their true effectiveness;
2. These times more than ever should dictate that funders start providing the wherewithal nonprofits need to master and implement outcomes.
Unfortunately, what we have seen instead is a retrenchment on the part of many organizations, away from any investments in outcome training or quality control. Across the economy the current hard times are forcing forward-looking for-profit organizations to push ahead with improvements in products and efficiency that had been too long delayed. In the nonprofit sector, by contrast, the urge instead seems to be to move backwards.

KB & RP: What are the “costs” associated with outcomes, and who should bear them? Are funders doing enough to enable nonprofits to fully implement outcome measurement?

DH: [The first question you ask is] the wrong question. The costs associated with outcomes are simply the costs of high performance. Yes it often (but not always!!!) costs more to work at high levels of quality and effectiveness -reliably and sustainably - than to work without any concern for such matters. But so what? The issue is not cost per unit of service or any other such exclusively cost-based metric. The issue is the social value that is produced...and whether it makes sense to pay what it costs to produce it - both sides of the equation: costs and value must always be considered in relation to each other. Running an inexpensive program that helps nobody is, arguably, much more expensive (all things considered...including opportunity costs and costs to the wider society) than running an expensive program that profoundly improves peoples lives and prospects! Are you serious? Who should pay for outcomes? Anybody who wants them!! [As for the second question, the answer is] “No.” The nonprofit sector is the only social domain where people don't think they should pay fair market value for what they are buying...hence grants with little or no overhead (and the self-satisfied crowing that accompanies them).

KB & RP: Comments: We agree wholeheartedly with the closing sentiments in David’s reply. The issue of high performance is inexorably tied to capacity: in order to do its job correctly, a nonprofit need more than simply the threadbare ability to “run” a program. Unfortunately, for many institutional investors, (barely) providing the resources for running a program is as far as they are willing to go; and this is especially true of governmental investors. We agree with David when he says that that the nonprofit sector is the only social domain where people often don't think they should pay fair market value for what they are buying. We also concur that many (most?) grants are approved with little or no allowance for overhead. The type of investments that would allow a nonprofit to truly implement outcomes are often simply not made. Funding for staff training, for guidance in rethinking their approach to the issues they tackle, tools for tracking and managing towards good, sustainable outcomes, and the ability to verify what they have accomplished, are very often the types of investments funders should make, but don’t. In fact, as soon as distressing ripples become apparent on the economic horizon, these are exactly the kinds of investments and expenditures that are jettisoned first. Institutional funders, philanthropic and governmental, are being both unfair and shortsighted when they ask for outcomes but refuse to provide the means whereby the nonprofits they support might actually accomplish and produce what the funders are asking for. The result is a game of misdirection and obfuscation that too closely resembles the old Soviet joke describing the Workers’ Paradise: “The government pretends to pay us, and so we pretend to work.” For many nonprofits today the situation might most aptly be described as “Funders pretend to want outcomes, and so we pretend to produce them.”

“Who should pay for outcomes?” David asks. The answer, he says, is “Anybody who wants them!!” We wonder whether the sector as a whole really wants them.

KB & RP: What are investor/funder organizations doing, beyond those asking for outcomes, to truly foster the movement and its spread throughout the sector?

DH: Damned if I know.

KB & RP: Comments: We share David’s sense of frustration, but would hazard the guess that the answer to the question is actually “very, very little.”

KB & RP: Has the outcomes field, its jargon, concepts and suggestions, gotten “too complicated?

DH: Yes; [because of] too many narcissists insisting on the primacy of their own locutions (cynically I'd say: efforts to control the jargon are part and parcel of strategies to control the consultation market!)

KB & RP Comments: We believe that the issue has to do with the fact that so few practitioners in the nonprofit field have any idea what those consultants are talking about.

Since the early nineties there has been a significant amount of growth in the number of outcome frameworks available to the nonprofit sector. This number is still growing and we suspect will only become larger as corporate-based approaches like the Balanced Scorecard and Six Sigma make their way across the great divide and begin to gain acceptance in the charitable sector. Yet while all this growth and refinement in the field is taking place, many practitioners still do not understand the essential difference between an output and an outcome.

As we stated in our response to David’s last reply, some of the responsibility for this situation rests with the funders who are failing to provide the means whereby practitioners in the nonprofit field could get up to speed –and stay there- on the practices and thinking currently being debated and developed in the outcomes field. We also see some nonprofits who are trying to avoid the matter entirely.

KB & RP: Alternatively, is there a danger the field will become co-opted for PR value but lose its critical value, meaning and depth?

DH: The degeneration of anything good into a morass of mediocrity or worse is an ever-present possibility...if not a likelihood. Where I weigh the probability in this case depends on what I ate for breakfast on the day you ask me the question.

KB & RP: Answering this question is one of the primary rationales for Charity Navigator’s intention to report upon the use of outcomes in the nonprofit sector. As a hedge against the danger David sees, of the current focus on outcomes sliding into a “morass of mediocrity,” we intend to report on nonprofits’ use of outcomes in as much depth as possible. We believe that investors, particularly individuals and those institutional investors with significantly limited resources, deserve to know whether the nonprofits asking for their money are actually accomplishing anything. Over the course of a long time, many nonprofits have gotten quite good at seeking donations based upon need (“The problem is SO big…”), or upon activity (“We’re working SO hard…”). Now many are becoming equally good at using the right jargon and seeking support based upon claims of effectiveness. We believe that investors have a right -and a responsibility- to know the truth, and so we intend to do all we can to provide guidance on this issue. Watch this space for more information on this soon.

*Dr. Robert Penna is an independent outcomes consultant and is assisting Charity Navigator in managing this Open Forum. He is the author of Outcome Frameworks and the forthcoming Outcomes Toolbox.

Monday, August 10, 2009

David Bonbright on Outcomes - Open Forum Participant

Our second participant in the Open Forum on Outcomes is David Bonbright who is the co-founder of Keystone Accountability. David has more than 20 years of experience in the field of social and economic development. He helped to create Keystone to address the weak accountability among funders and civil society organisations to their primary constituents – those meant to benefit from their work – and to innovate measurement tools that support meaningful and lasting developmental impacts. Over the last five years, Keystone has worked with a wide range of organizations to design new practical ways of planning, measuring, learning and communicating social change that foster accountability and learning among all constituents, and enhance positive outcomes.

Ken Berger and Robert Penna*: Should charities be assessed on their use of outcomes?
David Bonbright: Yes, if by that one means how they plan for, measure and learn from outcomes. They must ask and make their best answer to the question: What difference does our organization make? Because of the complexity and relative cost of answering this question for most charities, we need to find 'good enough' answers that are comparable across charities. This takes us to the realm of things that we can measure relatively easily in the short term that have a strong proven correlation to outcomes. The leading such predictive measure in the world of commerce is customer loyalty. Keystone is adapting comparative customer loyalty measures for charities with considerable success. For one great example applied to US high schools see the Gates Foundation - Center for Effective Philanthropy project

KB & RP: Do individual donors understand the value of outcomes? If not, what’s the best way to educate them to its value?
DB: I think folks get this intuitively. They have just never had the data, so they don't know they can expect it. For years I have been asking people a simple question, "If you were convinced that your giving was making a real difference, would you give more?" Virtually everyone says yes.

Comments from KB and RP: It is not surprising that there is an almost universal desire to be more generous. Even in these difficult economic times, surveys show that people intend to give as much or more than in the past. It reflects great optimism and generosity but not necessarily what will end up happening or the value of outcomes! So do donors actually put the charities to which they contribute to the test: DO THEY ask for outcomes? DO THEY consider whether the “outcomes” cited by charities make sense? Do THEY ask for any information at all? Unfortunately, more often than not the answer is no. Most donors, particularly individual donors, have not been informed about the importance of measurable outcomes as an indicator of nonprofit effectiveness. We hope over time to help remedy that situation!

KB & RP: What does outcome information tell a potential donor?
DB: It tells them what difference the organization makes. In the case of measures that are predictive of outcomes -- which are most often what we have to rely on -- it tells them the qualities and interim results of the organization that are likely to lead to outcomes. We need to benchmark outcome data across charities.

Comments from KB & RP: The key here, we think, is the quality of the “outcome information” offered. Are the charities in question actually supplying outcome information, or are they erroneously listing activities and outputs and calling them outcomes? Experts in the field tell us that there are characteristics of “good” outcomes. A logical question therefore is whether the outcomes cited by a charity meet these criteria. It all comes down to how faithfully a charity is actually applying outcomes.

KB & RP: Can “outcomes” really be applied across the board throughout the charitable sector? Are there areas where “outcomes” simply don’t apply? If so, how is the value of investments in those areas to be determined?
DB: Yes, I believe they can be applied across the board. Or maybe it is better to say 'outcomes thinking' as in many cases the best we can do is track the interim results that are likely to lead to outcomes. For example, the efforts of a counselor with at risk youth may only become visible years after the counseling. But at the time of the counseling, there are signs that can be picked up by asking youths across an entire counseling program that are highly indicative of later outcomes. Another way to say this is that there are short-term outcomes that can be measured and reported, and an associated logic that explains the likelihood of longer term outcomes. In some cases for certain kinds of work -- say writing books on moral ethics -- it is very difficult to make confident predictions about the relationship between short term results (how many people read the book) and outcomes. But even there, meaningful data can be readily gathered, such as what proportion of people who read the book register on the associated website. A survey of a sample of those who never heard of the book, who read the book and did not register on the website, and those who read the book and did register will allow a useful analysis of the outcomes from the book being written and published.

Comments from KB & RP: We agree that outcomes can and should be applied across the charitable sector: no portion of the sector should be “immune” from outcomes or measures of effectiveness. But certain subsets of the sector are considerably ahead of others in the identification and measurement of outcomes. In such areas as direct services, housing, substance abuse, community development and environmental protection, measures or indicators have existed for some time. But in other areas, nonprofit organizations are still struggling with how to apply the outcomes concept(s) to their work. In the areas of culture, the arts, and in advocacy, agreement has not solidified as to what the outcomes or indicators really should be. There are other examples that could be given, but the point is that more work still needs to be done to truly make outcomes applicable as a matter of practice across the charitable sector.

What do you think?

*Dr. Robert Penna is an independent outcomes consultant and is assisting Charity Navigator in managing this Open Forum. He is the author of Outcome Frameworks and the forthcoming Outcomes Toolbox.