Monday, September 14, 2009

Tris Lumley on Outcomes - Open Forum Participant

Tris Lumley is Head of Strategy at New Philanthropy Capital (NPC) in London, UK. NPC describes itself as "an organisation that enables charities and funders to do good, better. Our mission is to help make effectiveness central to how charities work and how funders give."

Tris has led the development of NPC’s framework for charity analysis, published in the 2005 report Funding success, and the development of NPC’s framework for analysing campaigns, published in the report Critical masses. Tris has also co-authored a number of reports within the community sector, on child abuse, adult and child refugees, and older people. He is a member of the Charity Commission’s SORP Committee and the ImpACT Coalition.

Before joining NPC, Tris worked in consulting and research, first at Deloitte Consulting and finally at FreshMinds. He gained MA and MPhil degrees at St John’s College, Cambridge, in Natural Sciences and the History and Philosophy of Science.

Tris can be reached at

Q: Should charities be assessed on their use of outcomes?

A: They should, and they must. We believe that a charity should measure its outcomes, because it will equip the organisation to manage its outcomes better. It’s possible always to deliver the best intervention purely by luck (without checking on its results) but it’s unlikely. When a charity is measuring its results, it has data on which to base reviews of its work, and therefore to adjust its approach accordingly. We also believe that a charity must measure its outcomes because of the privileged status it holds. A charity exists for the benefit of the people it’s trying to help. It has privileged status because of this, which sets it apart from for-profit companies or government agencies that do similar work. In order to demonstrate that it does deliver benefit to the people it’s supposed to, and crucially to demonstrate that as much benefit as possible is being delivered, the charity must measure its outcomes for those beneficiaries.

So charities should measure outcomes because the practice will make them better charities. And charities must measure outcomes because their privileged status relies on a claim about the benefit they’re creating.

Ken Berger & Robert Penna Comments: We agree with Tris’ analysis, but would add two additional and related reasons in support of charities using outcomes. The first is that these charities do indeed hold a privileged position: they not only act as trustees for the funds their investors give them, but, because of their tax exempt status, they are effectively subsidized by the rest of the taxpaying public whether or not they receive government grant money. In recognition of this dual support, charities have a responsibility to show something beyond rhetoric as evidence of the beneficial use to which these funds have been put. What these investors - philanthropies, individual donors, and government - don’t want is to see their money wasted, particularly on programs that don’t work. This is particularly true in tough economic times when everyone is carefully watching what they spend. Reliable outcomes are virtually the only guarantee of the effectiveness donors deserve. Our second point, a corollary of the first, is that in seeking new or additional funding, charities again owe something more than rhetoric and assurances of commitment and hard work. Prospective donors should have information at hand upon which they can make an informed social investment, a giving decision, in other words, made with their eyes wide open. Of the yardsticks that might be used, verifiable, meaningful and sustainable outcomes are clearly the best indicator of effectiveness, while fiscal health and accountability can help to inform investors of the capacity, sustainability, and ethics of these organizations.

Q: Do individual donors understand the value of outcomes? If not, what’s the best way to educate them to their value?

A: Intrinsically, of course they do. Donors give to charities because they believe in the work they’re doing, and in the impact they (may) create. But that often rests on intuitive assessments of a charity’s vision, the severity of the need it’s addressing, and a sense of the competence of the charity’s leadership. What we don’t see that often is donors looking rigorously to measure potential impact before making giving decisions.

Why is that? Well, those donors that do want to look under the hood of a charity and examine its outcomes face further challenges, because the right information generally just isn’t available. So donors tend to end up making decisions based on much less useful (and sometimes harmful) information instead. Like administrative cost ratios, for example. We believe donors ask about admin costs because they care about as much as possible of their donation going to the beneficiary (i.e. creating impact). But telling them about admin costs reinforces the problem. Far better to explain to the donor what that admin cost achieves, in other words to lead donors to thinking about outcomes, not inputs.

In order to educate donors, we need some real coordination across the field. Charity fundraisers need to start trying to answer difficult questions about outcomes. Chief executives need to start agreeing among themselves how best to report their outcomes so that charities can be compared. Foundations need to invest in charities’ capacity to measure outcomes, and start acting on the information these efforts produce. Analysts need to agree not to publish admin cost ratios any more, and to publish more useful and comparable outcome measures instead.

KB & RP: Tris makes several excellent points, not the least of which is that an unvarnished reliance upon things like a charity’s administrative costs can lead to the wrong decisions. As we have noted elsewhere, measures of financial health are one of three dimensions that must be considered by a wise social investor. Furthermore, the most critical dimension to be addressed is the outcome of the organization’s work. It is important, however, to remember for a moment why proxies such as admin costs and similar measures were examined in the first place. The historical record clearly shows that for as far back as anyone might care to look, charities, whether philanthropically or governmentally funded, did a terrible job of measuring, assessing or otherwise reporting on their performance. Sadly, for their part many funders were more interested in head counts and compliance than they were in measures of real effectiveness.

But Tris suggests another point that is worth exploring. We initially asked if individual donors understand the value of outcomes. Tris believes that "intrinsically, of course they do." We are not so sure….

While institutional donors have their own frequently byzantine rationales for giving, many individual donors, some well off and many not of any particular wealth, often make their decisions to give to a nonprofit based upon one or more of the following six reasons:

1. A personal connection to a specific nonprofit
2. A personal connection to a specific cause or issue
3. A personal connection to the person doing the asking
4. A resonance of personal beliefs or values with an issue or the field in which a certain nonprofit may be working
5. A sense, impression, or knowledge of a great need to be addressed
6. A "brand loyalty" to or generalized confidence in a given well known nonprofit

Unfortunately, not one of these reasons either inquires of, nor addresses the effectiveness of the nonprofits to which one might be donating. In these cases, the concept of outcomes is immaterial: no matter their innate importance or value, they are being trumped by these other, often emotional considerations. It is our hope that, as public awareness develops regarding the importance of meaningful outcomes, this sorry state of affairs will change for the better and more money will end up in the hands of higher performing nonprofits.

Q: Why are so few nonprofits actually using outcome measurement?

A: My moderate answer to this question is that adopting and implementing an outcome measurement approach is hard, and takes significant time and money to pull off. But given the fact that I believe charities both should, and must, measure outcomes, I don’t think this answer is enough. My more cynical answer is that nonprofits are organisations made up of individuals, who respond to incentives. The incentives simply aren’t there yet for outcomes measurement to really gain momentum. The moral incentives (being a better nonprofit; helping more people by doing better) may exist, but next to financial incentives they can often be too weak or fragmented to drive behaviour. Financial incentives, on the other hand, are clear and strong. If nonprofits can access the funding streams they need to develop and grow without using outcome measurement, they will. If funders use their collective power to ensure that nonprofits need to report their outcomes in order to secure funding, the world will change, and outcome measurement will become ubiquitous.

KB & RP: As we have stated previously in this space, there are numerous reasons why more nonprofits are not using outcome measurement in any meaningful way. Yes, it is harder than simply doing nothing. A great part of the problem, however, is that the use of outcomes requires resources to understand and implement. Sadly, as times get tight, these are exactly the types of expenditures that nonprofits believe they cannot afford. As the vice-president of one very large nonprofit recently and incongruously told us, "With things as tight as they are, it will be a long time before we’ll be spending anything on quality assurance or performance improvement." Now think about that for a moment. What this woman was saying was that at the very time when she probably has to make the best, most compelling, fact-based case to her individual and institutional donors, she is planning instead to fall back on need, passion, and probably a few head counts; that, amazingly, with funding extremely tight and uncertain, this is exactly the time to not focus on quality or performance. On one level, her statement is laughable; on another, however, it is sad because it points to the stark fact that because donors usually do not make the resources available for assuring quality or enhancing performance, practitioners believe that they must make a budgetary choice between service and effectiveness. Many donors, unfortunately, seem to think that merely asking for, or even demanding high performance is enough. How they think such performance is going to occur, however, is anyone’s guess.

Q: Are funders doing enough to enable nonprofits to fully implement outcome measurement? What are investor/funder organizations doing, beyond those actually asking for outcomes, to truly foster the movement and its spread throughout the sector?

A: No. One could argue that charities on their own should be strong enough to implement outcome measurement, but we need funders to provide the financial incentives. Funders are not doing enough to help individual organisations on at least three fronts: 1. funding charities’ efforts to improve their own measurement, 2. acting on the findings of charities’ evaluations, 3. holding their own organisations up to the yardstick of outcomes measurement. And beyond that, funders are failing to help develop the marketplace as a whole. Without funders that are prepared to invest in infrastructure, building the marketplace, sharing experiences, and developing standards where appropriate, a lot of the effort that’s going on at an organisational level can’t flourish and be replicated.

There’s a real danger in the current climate that funding retrenches towards core services, and away from any infrastructural or developmental work. We need funders to become more committed to outcomes measurement, not less!

KB & RP: We agree wholeheartedly with Tris with one exception. As with any other skill or knowledge base, implementing outcome measurement requires a certain amount of learning. What to measure, how to measure, and how to make sense of those measurements, are among the skills any nonprofit needs to master to fully benefit from the use of outcome measures and management. The overwhelming failure of donors to make this sort of investment in the market and in their own investment partners is illogical, sad, and dispiriting. We somehow expect the nonprofits we have tasked with addressing some of society’s greatest challenges, to attain - perhaps by osmosis - the knowledge and skills that they need to perform to current and growing expectations. We have armed them with (at best!) the tools of the 20th Century, but are asking them to succeed in the 21st.

Perhaps worse than this, however, is that at the same time that the opinion among practitioners is effectively unanimous that more needs to be done to improve nonprofits’ effectiveness, from the institutional donors comes a resounding and ponderous silence in reply. What is the sound of one hand clapping? The same as the sound of the nonprofit world trying to have a dialogue with donors about the need to invest in nonprofit capacity, specifically in the capacity to understand, manage and measure outcomes. Unfortunately, until this sad soliloquy becomes a determined dialogue, not much is likely to change.


Ingvild Bjornvold, Director of Advocacy, Social Solutions said...

Nice interview. I agree with most of Tris Lumley's thoughtful perspectives, and yours, but I think there is one point to add to individual donors' motivations.

I believe Tris is right when he says that intrinsically, individual donors care about outcomes. And while I think you are also right to list some other motivating factors, I believe it is important to articulate one very important underlying reason why individual donors care, yet do not demand good information about outcomes in force.

People simply can't imagine that what nonprofits do isn't doing good. Most people believe that, as long as most of the money goes to programming (as opposed to overhead), that money is making a difference. People don't understand that it is incredibly difficult to change the life courses of the people who receive human services. Success is no certainty in HIV prevention, violence prevention, parenting education and so on. Somehow, though, the public thinks the work is easy and doesn't require serious performance management.

That is the underlying public assumption that has to change.

Ken Berger said...


I agree with you that most people assume that the services being provided are helping and effective. I am not convinced that they think it is "easy" but certainly that it is getting done. One of our jobs is to educate the public to the sad fact that the evidence is not yet available on effectiveness for most nonprofits right now and to motivate the nonprofits to change.