Monday, June 1, 2009

Announcing an Open Forum on Outcomes

In the blog entry of April 13, 2009, I wrote of a new direction for Charity Navigator, stating that “we are ... seeking to transform our evaluation system of charities to include two additional dimensions (beyond financial health) – accountability (including transparency) and outcomes.” The blog entry concluded with the promise that “You will continue to hear about these components of making wise giving decisions in the months ahead.”

So now, a few “months ahead”, one of the ways we are planning to move forward is by conducting an open forum on what we believe is one of the most important topics facing the nonprofit and philanthropic worlds today. The issue of outcomes has alternately inspired, dispirited, empowered, and angered nonprofit and philanthropic directors, managers, and practitioners since the early 90s when the concept first took hold in these sectors. Proponents of outcome-based practice see it as the surest way to achieve true effectiveness. Critics see efforts at identifying, managing towards, and tracking outcomes as a burdensome drag on nonprofits, diverting resources and attention away from the real work upon which these organizations should be focusing.

In the February 16, 2009 blog entry, “A Scary Finding on Outcome Measurement,” we noted that we had been on a “listening tour” regarding outcomes, and came away with the impression that fewer than 10% of charities seem to be measuring outcomes at all. A colleague with whom I checked these findings suggested that the overwhelming majority of charities have not even taken their first step down the outcome road.

The debate that followed the posting of that column was spirited. Kevin Jones commented that “If 90 % of non profits are not measuring, maybe the value of measurement needs to be measured.” Texas-based reader Soytri noted that if the effort associated with outcomes is to be worth it, the cost associated with gathering and analyzing the data needs to be less than the benefit accrued from having the knowledge; and blogger Vicknj pointed to a generational difference of perspective as a possible reason why the leadership of more charities are not embracing outcomes.

As the discussion spread beyond our pages, Sean Stannard-Stockton’s Tactical Philanthropy blog received a comment from outcomes consultant Dr. Robert Penna. He suggested that charities’ confusion over the availability of outcome-based tools is the root cause of so few of these organizations using them. “In the end,” he wrote, “it is up to the investor community to make the necessary expertise available to their client, partner and grantee organizations.”

All of this raises at least four questions:
A. What is the value of outcome measurement and the practices associated with it for nonprofits and social investors (donors who seek results from their investment in a nonprofit that lead to social value)?
B. Why are so few nonprofits actually using outcome measurement and so few donors asking for it?
C. Are funders doing enough to enable nonprofits to fully implement outcome measurement?
D. Regardless of the issue of funding, should nonprofits be held accountable for documenting meaningful outcomes?

As we have noted before, Charity Navigator is committed to the concept of reporting upon the outcomes success of the nonprofits we survey, as well as applying them to our own work. With that decision made, we believe that we have the opportunity to educate ourselves and advance knowledge in the field by hosting this open forum. We hope to have comment and opinion offered by some of the leading thinkers in the nonprofit sector. We trust that our readers will both participate in and profit from the exchange of information and ideas that we anticipate will result. We look forward to all of your thoughtful comments and suggestions as we travel this road together.

39 comments:

Anonymous said...

Some outcomes just cannot be measured. How do you know if you have decreased homelessness in your town? It is expensive, if not impossible to count the number of homeless people in one small town, let alone a city. How do you know if you've changed racial prejudice unless you set up expensive psychological testing? How do you know if you've educated, provoked, supported, inspired? It costs a lot of money to measure things, especially psychological things.

If you can't trust that your people are being effective and efficient with your money, you have the wrong people. Hire the right people, give them resources, tell them what you want done, and let them try things until they hone in on the right strategy. THAT is how things really get done.

If you insist on measuring outcomes, you will get the equivalent of the "no child left behind" debacle. Real education decreased as teachers focused only on "teaching for the test." Obsession with measuring outcomes leads to bland programs and no social change.

Ken Berger said...

Anonymous,

As you may know, I worked in the charitable sector for over 25 years before I came to CN. In fact, the vast majority of my work was with the homeless and the chronically mentally ill.

It is with that background in mind that I must strongly disagree with your perspective. Further, I think you have an incorrect understanding of what an outcome is. For example, counting the homeless is an output, not an outcome.

More importantly, the failures at measuring you mention are when they are imposed from the outside and do not measure what is meaningful. The outcome tools we are looking at are meant to empower the nonprofit to determine ITSELF what a meaningful outcome measure is (referring to commonly agreed upon measures within its field). In addition, when we think about speaking to the clients of a charity, we empower THEM to decide if the outcome was of value to them.

Without meaningful measurement of outcomes, a charity runs a very real risk of doing more harm than good even with the most caring and dedicated staff. I will give you an example. An organization serving battered women has found (through outcome measurement) that support groups for those they serve have markedly increased the quality of their lives. So they decide to have a group for the batterers to help them gain insight into the issues that lead to their abusive behavior. However, because they are objectively measuring the results, they discover that the group is having the very opposite effect. The batterers are egging each other on and becoming even more violent. If the charity was not measuring its outcomes, the charity would have done more harm than good.

Without MEANINGFUL outcome measurement, we will remain ignorant. I conclude with the thoughts of my colleague David Hunter:

The bottom line is -
If you can't measure it, you can't manage it.
If you can't manage it, you can't deliver it.
If you can't deliver it, you have no social value.

And that means that if a social sector agency doesn't measure, it can't be ethical because it has no way of knowing whether it's helping or hurting people. It can not hold itself accountable for anything and neither can society. Therefore, a social sector agency that doesn't measure should not be in business!

Jeff Mason said...

Right on Ken!

I like the term "performance management" as opposed to "outcomes management." Performance management is about having a clear strategy for how you will do what you want to do, diligently collecting and analyzing data to understand what works in progressing towards your goals and what doesn't work, and using this knowledge to continuously improve. This is what happened in the domestic violence example Ken provided.

If every nonprofit actively managed their performance we would have a very different world.

david@dekhconsulting.com said...

I agree fully with Ken Berger's response to Anonymoous - and wonder why that person won't sign his or her name with conviction.

You may be interested to know that NYC routinely counts homeless people in a very sophisticated way: to gain confidence that it has found virtually all homeless people to count, it puts "proxy" people in all areas where homeless folks are known to congregate, and even in less obvous areas. And if all "proxies" are not found and counted by surveyors, the count is not accepted as complete. How much easier it would be to do this in Anonymous's "small town."

The point is: where there is a will there is a way. Sadly, anybody who has been accountable for managing the work of an organization knowns that hiring good folks with good intentions doesn't necesarily ensure getting good results. And an organization that wants to go on working in ignorant bliss while ignoring this simple fact and potentially doing little good or even harm...well, in my view it doesn't deserve to get tax deductable support (those who want to support it out of sentimentality can do so from their own pockets).

I've been working with nonprofits for a long time and can assure you that I have yet to find a case where an organization that truly wanted to understand whether it was getting good outcomes - meaningful in terms of its own vision and mission (not funder imposed) - could not develop straightforward and relatively inexpensive means for doing so.

The issue is not cost: it costs money to get good results...and the cost of measuring and tracking them is simply part of the cost of doing buisness. Nonprofits have no inherent claim on our support, and most certainly not if they don't have this as a simple guiding value for their work...not because funders impose it on them, but because (e.g., in the social sector) the folks who depend on them to help improve their lives and prospects absolutely need them to operate in this way...otherwise how can the organization continue to improve its services and make adjustments as indicated?!

Ken's point that this is an ethical issues is right on the mark!

I wonder whether Anonymous would send his or her children to a physician who never bothered to keep track of whether operations resulted in health improvement or health loss (or death)! And adjusted his or her practices accoringly. Why should nonprofits expect to be protected from similar accountability?

David Hunter
Hunter Consulting LLC

Ken Berger said...

Jeff,

Point well taken.

Ken

Ken Berger said...

David,

I greatly respect your perspectives and insights. In the spirit of transparency, I note here that Charity Navigator has signed an agreement with Dr. Hunter to take us through a theory of change process with the goal of developing a performance management system that will lead to great outcomes as an organization. As noted in an earlier blog, our organization does not just intend to "talk the talk" but also "walk the walk".

Andrew Niklaus said...

What a stimulating conversation to enter into. I have heard the argument many times about performance management detracting from the work of providing direct services to participants. But I believe strongly that any organization that ignores the impact of their work (or lack thereof) is reckless. Performance management is done solely to benefit the participant. It is THE difference maker between a good program and a great program. In fact, I would argue that it is the programs that do not seek information about their work with a passionate pursuit that are the ones destined to have the most bland, uninspiring services. Data tells a story! It tells you what you have done, and if you have done it well. Data can tell an organization where it has been, and most importantly where it is going. Data can tell you how to form a strategy to meet your goals, objectives and fulfill your mission. The use of data as the cornerstone to a robust performance management strategy will provide an infinite loop of information that can be fed back into programs to continually prove and improve the efficacy and impact of an organization. Who would not want this? This is the reason we work in the social service field - to create meaningful change. Agencies that believe they can do it on goodwill, personality and instinct might as well be throwing a dart at a list of ideas, only to chose the one that the dart hits. The stakes are too high to not measure and improve. Be fearless and challenge yourself to know if you are doing good, bad, or nothing. You owe it to your participants.

I would also respectfully add that the word charity does not fully and accurately describe the work of high impact organizations. I believe my agency is deeply dedicated to tracking outcomes, improving programs and managing our performance. I also believe that no one, not a government entity, municipality, foundation or individual gives us anything. They are investing in our work, and they want a return on that investment. Stakeholders want to see us house transition age foster youth and help them enroll in school and earn their diploma. Our investors want our participants to get jobs and make a successful transition into independence. The only way, and by the only way I mean the only ethical way, that we can show stakeholders the impact of their investment is to rigorously track what we do, provide usable, clear data, and then use that data to secure more investments from the community. It is the most logical, simple cycle - and it works.

Thanks.

Ken Berger said...

Andrew,

Thanks for your great feedback. One small point on the word charity, this is the term used by the IRS to describe 501 (c) 3s which are the type of organization we evaluate. However, I agree that the common understanding of the term does not fully capture the universe we are talking about - nonprofit organizations who should be providing social value.

George Overholser said...

I would add that things do get different when we get into the industrial strength zone of "scaling what works".

Take Nurse Family Partnership, for example. Here's a case where it took years of expensive program tinkering to get what looked to be an effective visiting nurse program for first-time pregnant low income moms.

Then it took years of scientific testing in Denver to build a strong enough case to attract initial government funding. Then, after they failed at first to replicate the results in another city, with a different population, it took years more (plus much money) to make the program design robust enough for faithful scaling.

Finally, after a third randomized trial, that showed once more a $7-to-$1 payback to taxpayers, the program has finally been slated by Obama for a $7 billion per year government roll out.

The philanthropic "burn" en route to this government-sustained roll out: $80 million.

Perhaps it could have cost less, and certainly this is an approach reserved for only the most high-reaching non-profiteers, but I am convinced that a major scale-up like this, engineered to maintain high fidelity to a scientifically proven model, and enabled by compelling-enough evidence to prompt major government investment, requires VERY SIGNIFICANT up front philanthropic investment.

On the other hand, if this continues to hold up, "merely" $80 million of philanthropic investment will create a stream of $49 BILLION per year in taxpayer benefits. (Not to mention a whole BUNCH of happier moms and kids!!!)

Ken Berger said...

George,

Thanks you for you insights!

The example you gave shows the INCREDIBLE results that can occur over time. However, I believe that the work of Nurse Family Partnership had dramatic outcomes year after year throughout this evolutionary process. As I understand it, the major cost was in the scaling up of a great model that was being tested in a variety of localities over time.

So people need not assume that the core performance management system for the agency cost $80 million!

We are most concerned about the vast majority of charities who have not yet even put their toe in the outcome waters! The views expressed by our first commentator (Anonymous) reflect that thinking. On the other hand, I hope the take away from what you have described encourages them to proceed!

william said...

Ken, I agree with Andrew that this is a very stimulating conversation. I read the comment of Anonymous and my first thought was “he raised some genuine concerns.” But your response to those concerns was as Jeff said “right on”.
I also was concerned with outcome /performance measurements being imposed from the outside that really do not measure what really meaningful and could be more harmful than helpful because of the constraints and limitations place on them. However, you made it very clear who should determine the measurement of outcome.
Although Jeff, Andrew and David all had excellent points which I basically agree , however we need to hear the issues and concerns of those that disagree with ken’s position in this matter. The issues herein are too important to not have a clear and open discussion. Without meaningful discussions we will not be able to convince those in the nonprofit sector that meaningful outcome measurement is necessary or we will remain ignorant of the outcome

Ken the Accountant said...

Outcome measurement isn't as difficult as some believe and while it isn't coming fast enought for me, I do see positive signs that momentum buliding.

For example, about a year ago United Way of America, went public with it's 10 year goals for having an impact on Education, Income, and Health in this country. They pledged to measure the change in key national statistics. The challenge for United Way now is to tie thier annual investments of donated funds to programs that can show that they are producing outcomes in those key areas. Why is it such a challenge? Because so many organizations that United Ways fund believe they are entitled to funding.

Unfortunately the truth of Anonymous' comment that "If you can't trust that your people are being effective and efficient with your money, you have the wrong people" is coming home to roost. Unfortunately for Anonymous though, it is precisely that point that will drive the non profit sector further down the road to outcome measurement, because trust must be earned.

As an accountant with more than 25 years of for profit and non profit experience, I have seen far to many organizations in both the for profit and non profit sector take for granted the trust granted to them by their supporters. Personally I always tell people that I don't want their trust, I want them to hold me accountable because if I'm only accountable to myself, you'll never really know if I am trustworthy.

Who among us would continue to put money into a mutual fund that never tells investors what their return on investment is? Don't we all look at the fund manager's track record on rate of return (outcome) before we trust him/her with our money?

It is the outcome produced by an organization that tells donors and the community at large whether or not the organization is worthy of additional investment. Like it or not, donors are hungry for this information so as I see it, this outcome train is leaving the station and if an organization doesn't get on board, it will be left behind.

Ken Berger said...

william,

Indeed I hope we can hear from all perspectives on the issue. That is why we are calling for an Open Forum here.

At the same time I intend to be transparent about my perspective - that outcomes (which can be defined as the measurable social value of the work) are critically important and the very reason for nonprofits to exist.

Thanks for the reminder though and may those who agree as well as disagree express their views!

Ken Berger said...

Ken the Accountant,

Amen brother!

Ingvild Bjornvold said...

If Charity Navigator can successfully integrate outcomes into its rating system, I think it will be an important contribution to changing the nonprofit sector to the better. It puzzles me that so many people, like Anonymous, argue that we should “Hire the right people, give them resources, tell them what you want done, and let them try things until they hone in on the right strategy.” Clearly, the assumption is that whatever we do, it must do some good.

As Ken, David Hunter, Andrew Niklaus and others have pointed out, however, good intentions and hard work do not necessarily result in any good at all – in fact, countless evaluations have shown that it can result in harm! It’s time to turn the conversation from our responsibility to nonprofits (they have no inherent claim on our support) to our responsibility to all the people who rely on those organizations to help them improve their lives.

Innovation, trial and error – it can be good, but only so long as it is based on research about what is already known, so an organization can feel confident that the new approach is likely to benefit rather than harm people, and so long as the organization has a performance management system in place to capture what works and what needs to be improved – immediately (not years down the road). A lot is known about what works, and I agree with George Overholser that it is worth the effort to expand a really effective program like the Nurse-Family Partnership.

It is true that it is expensive to evaluate programs, and very few programs are in a position to go through rigorous evaluation by external researchers. We shouldn’t require that of all nonprofits, and it wouldn’t make sense for Charity Navigator. What we should require is that they manage performance internally. Performance management is different from evaluation, and all nonprofits can do it at a reasonable cost. It is about gathering and reviewing real-time information – internally – to answer some key questions:

• Are we reaching the people we intend to reach?
• Are we implementing our programs at the frequency, duration and level of quality we believe is necessary in order to achieve the outcomes?
• Are our clients making progress toward the expected outcomes?

Few organizations are managing performance because they think it is too difficult. First Place for Youth, which Andrew Niklaus represents, the Latin American Youth Center, Roca, Our Piece of the Pie, Wings for Kids and other organizations, however, show us that it is indeed possible. It is not as difficult as most people think, as Ken the Accountant also pointed out.

And when donors have the choice between organizations who manage performance and those who do not… Well, I think the choice is clear. More and more organizations are recognizing the need for performance management, and it would be great if more funders would recognize the need to fund those efforts. They are critical to producing the social value we all want, and which we owe to the people who need nonprofit organizations’ services.

Ingvild Bjornvold
Social Solutions

Ken Berger said...

Ingvild,

Thanks so much for your contribution.

I recommend that people check out http://www.socialsolutions.com/ to learn about their tools to help nonprofits develop an "efforts to outcomes" system.

Bridget Laird said...

I work for a results-driven nonprofit that’s been focused on outcomes and accountability from its beginnings. We did not want to be another well-meaning program trying to help kids living in poverty. We wanted true impact.

We quickly found ourselves in the very small minority of nonprofits that consider outcome-based practice not only valuable but mandatory. Constantly we’ve had to defend to supporters why we spend precious time and money on this effort. And the outcomes we seek at WINGS for kids – instilling a strong social and emotional education – are by no means easy to measure.

But it is without question the only way to know whether we’re effective and why.

After we began working with David Hunter to improve upon the way we tracked our efforts, it was amazing to see how this data allowed us to align our efforts to outcomes and respond quickly with course corrections. We became more effective – and we constantly receive information that tells us in precise detail how and why.

It takes time to implement these new technologies. But the payoff is big. I can now sleep better at night because we ensure that our hard work is actually making a difference for the kids we serve.



Bridget Laird
COO, WINGS for kids
Charleston, S.C.

Ken Berger said...

Bridget,

I salute you and congratulate you on being one of the leading organizations in this battle for the heart and soul of the social sector.

Helen Knight said...

As a past consultant at a couple of development banks, it is not just non-profits that have problems clearly defining their outcomes/outputs/results or measuring their performance. There is limited accountability for the billions of dollars of loans/investments the development banks make, despite large evaluation departments to take on the cost.

Although instinctively it should be the case that donors are more interested in non-profits with demonstrable results and outcomes, is there anyone out there who has done the research to find out if this is actually the case? Given the history of donations to 'good causes' only, rather than for 'good results', it would be interesting to know.

And, to add to Anonymous' comment, what can be measured in relation to performance, impact and results, should be, and what can't can still be collated in case studies as additional anecdotal data and examples.

Ken Berger said...

Helen

I have not seen any research on the subject. However, I believe that common sense dictates that the outcome of the nonprofit's work is important to people!

We believe that donors are relying on us to be the experts on "intelligent giving" (so that they can become effective social investors). They are trusting us to determine what constitutes a "good" nonprofit. It is clear to us that outcomes are critically important in that determination.

As to the use of anecdotes and case studies, it gets back to my earlier comment - if you can't measure it, you can not be held accountable. That is not acceptable to us.

Saundra said...

I am pleased to see Charity Navigator opening a forum on evaluating outcomes of non-profit work. Evaluating the work of non-profits is critical to both improving the quality and professionalism of aid work, and to empowering donors to hold aid agencies accountable.

As it stands, far too few aid agencies do any real evaluation of their work and when evaluations have negative findings they are often not shared with donors. Of the 200 aid agencies that responded to the tsunami in Thailand I know of less than a handful that actually evaluated their work of those, none shared their findings with donors. I've gone on a little rant about this very topic on my own blog - http://informationincontext.typepad.com/good_intentions_are_not_e/2009/06/its-time-to-stop-telling-pretty-stories-and-start-really-evaluating-the-impact-of-aid.html

As others have said, just because non-profits have the best intentions does not mean that their program is helpful. It is time that the aid world improved their professionalism by consistently evaluating their work and making their findings public. Only then will aid really improve.

Best of luck to Charity Navigator and Ken in their efforts.

Allison Fine said...

Thanks Ken for catalyzing this very interesting discussion. And my heartfelt greetings to David Hunter whom I have known for many years and highly esteem!

I founded Innovation Network (www.innonet.org) many years ago to help nonprofits and foundations use evaluative tools to learn how to improve their efforts. I have more recently been engaged in research and writing about ways that social media can improve the effectiveness and learning for causes.

The ambitions of this effort are admirable, at the same time, the challenges are quite daunting. And it begins at the beginning: who is trying to learn and for what reasons. For years funders like United Ways have tried to improve an evaluative framework to no great effect, in fact, I often hear groups not only disdain the imposed process required by United Ways but admissions from groups that they were telling them what they wanted to hear.

The barriers haven't been cost or even technical skills (although I have argued that the social science construct is often too difficult for groups to comfortably use and that lighter learning models are needed, but more on that another time!)the barrier is fear. Fear that your efforts are not as successful as you had hoped or have been telling people that they are, fear that your funding will be pulled (which isn't just an idle fear), fear that your board will be disappointed. A pervading, paralyzing, overwhelming fear.

Unless and until the fear factor can be reduced (and it can only be reduced by developing new norms and cultures within nonprofits themselves not be outside entities whether they are funders, or, perhpas, if you don't mind my saying, Charity Navigator) then the development of new tools, gizmos, processes, frameworks will fall flat.

Thanks again, Ken, for facilitating this conversation.

Ken Berger said...

Allison,

I agree that fear is an important factor as to why most nonprofits do not embrace a performance management system that measures the social value of their work. However, I have worked in enough nonprofits to tell you without hesitancy that there is another big challenge - selfishness that trumps mission. Sadly, there are far too many examples of nonprofits where the leadership cares little about outcomes beyond lining their pockets. I believe that the heightened measurement and accountability that is evolving will help to erode this horrific problem.

Ken Berger said...

Saundra,

You raise a good point about another problem with many nonprofits not sharing information. I believe that the term - Balkanization (to break up into hostile units) is part of the problem. I have seen far too many examples of nonprofits that behave in a very adversarial, hostile, secretive and defensive fashion. They will go over a cliff before they will open up.

There is a myopic, closed mindedness among some agencies that boggles the mind and goes beyond fear of loss of funding when they have bad results. I hope that as we begin to measure transparency (as well as outcomes) we can highlight those who unclench their fists!

Saundra said...

Allison,

I agree that the fear factor needs to be reduced before aid agencies use the new gizmos for evaluations. But I don't think that will happen from inside the aid world. There have already been numerous attempts from within to improve practices, but initiatives such as HAP-I (not evaluation but accountability) do not have any clout unless donors back them.

That's why I've decided to focus on changing donor habits. If donors begin to reward only those aid agencies doing evaluations, and do not punish negative findings (unless management does nothing to improve the organization based on those findings). With the requirement of evaluations becoming standard, evaluation findings will become more common, thus less stigmatizing. And the fear may change to what will happen if they don't do evaluations.

Unless donors force it, I don't think the industry will change its ways. There's too much money and momentum keeping it the way it is.

Ken Berger said...

Saundra to Allison,

I agree with Saundra that pressure will need to be applied from social investors (donors with their eyes wide open) for more nonprofits to embrace performance management. However, there are also some encouraging signs within the sector too. Read the comments above from Andrew Niklaus and Bridget Laird above as examples!

James said...

It seems that some organizations will not measure outcomes because they fear they will not do it perfectly. We must all beware of "letting perfection be the enemy of good." An organization can start with managing those things it can identify and monitor. Just stopping waste, or ineffective solicitation communications, is an outcome. And we all learn as we go along, so keeping useful records of what we have learned while trying to achieve our stated goals is itself a measurable outcome.
I am fortunate to be involved in an organization that can actually count the people we help and the island habitat we assist them in preserving, but even we seek to measure our efficiency in what we do.
Jake Walker, Seacology
Saving the World one Island at a time.

LeAnn said...

I think these two additions are necessary. I worked for a charity who just stopped granting wishes, however continued to collect from donors - 10's of thousands of dollars each week!! Donors are called upon to support a charity - they assume the money is going where the charity says it's going... not always the case. Make them prove where the money is going and the outcome - we (donors) need to know the money we donate is doing good and not just lining Executive Directors and Presidents pockets! Trust me, this is necessary. Not all charities are on the up and up....

Ken Berger said...

James and LeAnn,

Agreed on all counts!

Carolyn Miles said...

I think this is a fascinating debate and there are many important threads here - the "far factor", the cost and challenge of measuring, the debate on what is truly measurable. It is a debate that we are keenly interested in and have been for quite a few years at Save the Children. Having come from the corporate world before joining the non-profit sector over 11 years ago, I have to admit to having been mystified on how a very large, far flung non-profit like ours could actually achieve results without clear measurement of indicators of performance. Of course we had the easily measured financial measures, on which we did very well, and the project by project indicators that donors have demanded for many years. But how were we measuring and importantly talking about, the change we were making globally in key programmatic areas like HIV Aids progams for OVC's or Early Childhood development work, or Child Survival around the world?

The answer was were were not and most organizations were not. Tragically, this lack of focus on visible and straight forward results that people can understand reinforces a fact of life that we all deal with in our work - that the main indicator that the public, corporate partners, volunteers, donors and other interested stakeholders use to decide whether to support a specific non-profit in ANY way, is a set of financial measures that I openly admit have little to do with outcomes for beneficiaries. And that is because we have not given people anything else by which to judge us.

This lack of performance indicators also contributes to a "beneath the surface" but important factor that manifests itself in several interesting ways - the deeply held feeling of many that maybe business, with it's clear "bottomline" orientation, is actually better at achieving socal change results than a bunch of "do-gooders" who can't tell us what they have achieved. All one has to do is look at the Bush adinistration pattern of spending US aid funds over the past 6-7 years to see this manifested - the amount of funding going to for-profit "contractors" has grown tremendously, while the share to NGO's has shrunk dramtically. This ties directly to a belief (misguided it is now becoming evident in many case) that for-profit organizations who sign up for "concrete" measures are more accountable for results to vulnerable populatons and can be more successful in driving change.

I think the announcement by CN to take this on is extremely exciting and we look forward to working on it with many others, but it will also be a real challenge. I have found in our work on global indicators of outcomes for children that our biggest stuggle has been with defining which indicators to use. I think CN is on the right track of letting organizations define those performance indicators for themselves. Right now, a set of common ones seems unattainable, though I hope I see the day when this is possible so you can actually get us all pulling in the same direction to do things like save children's lives and make sure every child has access to education.

But the "system" then for comparing indicators across organizations will be tough. We have experienced this in our own organization in comparing results in the same sector across countries and it has been a source of large debate.

In addition, I strongly support the comments of many that it is very important to make sure we are incorporating the voices of our beneficiaries more strongly in our indicators of success. Yes we may have delivered water, food and shelter materials to millions of people in an emergency, but what if what would have really changed their lives even more was an ability to develop new livelihood options or get kids back to school quicker? We need to most importantly know what those we serve value and how they think we are doing on providing them. This is an area that really just now is getting the attention it needs.

So thanks Ken for launching this and we look forward to helpng make it happen!

Carolyn Miles
COO
Save the Children

Ken Berger said...

Carolyn,

Thanks so much for your insights. I am increasingly convinced that, whereas the for profit sector has been able to boil down their engine to measure success in one word - profit, the NP sector can boil it all down to one word too - impact. At CN we believe that there are three components required to get an NP to be high performing (financial health, accountability and outcomes) but ultimately all these components should lead to impact of significant social value.

Reb said...

Carolyn - Allison in an earlier comment mentioned innonet.org. This is an excellent resource for nonprofit professionals to use so they don't have to reinvent the wheel. You may find some work already done in regard to HIV and early child development and how it relates to measuring meaningful impact. Whatworks.org is another great resource that has been funded by the Urban Institute which shares best practices on taxonomies and again, meaningful impact. Both these organizations may have helpful information that you can use as a springboard to your benefit.

Collaboration is vital in this effort to deliver excellence in all that we do to make the world a better place.

Ken Berger said...

Reb,

I agree that Whatworks.org is a great resource for nonprofits to learn about developing their ability to better manage their performance.

I do not know the other group but hope to learn more about them.

Joe Houghton said...

Just come across your discussions and am very interested to take note of all the comments and participate in the Open Forum. I am in the early stages of researching a PhD examining "How does board governance contribute to the success of Not For Profit organisations ?" so I hope to learn and also contribute to the discussions... Joe Houghton, Smurfit Graduate School of Business, Dublin, Ireland (joe.houghton@learningtalk.ie)

Ken Berger said...

Joe,

Welcome aboard Joe. Let me know if you need me to come out to Dublin to help you!

Rob Hanna said...

Ken,

The biggest challenge facing effective use of outcome systems to date is that they are feared, too complexly perceived or cavalierly introduced as an adjunct to evaluation or as a supplemental administrative tool simply to track program performance.

This is at best an immature view that almost entirely misses their powerful potential.

Outcomes must become an operational necessity regarding the design and implementation of any program or service, as well as the defining category of any intended success, period.

Furthermore, our field is getting way off track in the implementing of outcomes and outcome systems because we face two structural forces creating substantial drift in their application:

1. Consultants and professionals desire to promulgate outcome methods as an adjunct evaluation product or service that not-for-profits can procure after the fact or as a way to satisfy funder compliance.

2. Philanthropists and funders also continue to avoid their responsibility for establishing outcomes at a level of measurable and verifiable end client gains.

Because of this we instead default to outcomes as a set of intrinsic metrics based on implementing agencies a la "capacity building" or efficiency gains in services delivered, as if such activities were a legitimate proxy for achieving greater human gains to a level of significance outside organizations in targeted populations of need.

We too well know our industry is a third-party payor system, where funders are the de facto customer as regards not-for-profit development activities; as such resource and talent is drained from investing in product development that should begin by defining the significant end client gains intended and then proceed to vet such assumptions by compressing the innovation cycle of delivering such benefits to clients more effectively. instead, our not for profits organizations are trying to produce cars while spending time complying with mandates for better buggy whips. There is a huge mismatch in evolutionary scope and timing.

There are effective corrective actions and upgraded frameworks for correcting each of these forces in turn, but the bottom line is this:

Outcomes will never systemically deliver traction in the philanthropic industry until they outgrow the immature perspective of captive professionalism and administrative supplements. Outcomes must be rightly perceived as the initiating DNA, operating focus and subsequent day to day management skill within high-performing, not for profit organizations and be DEFINED AS END CLIENT GAINS--i.e. you don't seek EXTRA funding to do outcomes, it must be the initiating premise of your mission and simply the way you do business every day with your clients.

Only when we require such information to do our daily work will the quest for harmonizing outcomes make sense within and outside the charitable organization. And this operational necessity to know how outcomes are delivered must be explicit, transparent, shared and clearly desired at all levels: from clients to line staff to program directors to executive to Board members to partners and funders. At that point outcomes become natural and easily used, just like paper--ubiquitously valuable yet hardly remarked upon.

Kind Regards,
Rob Hanna
Social Wealth Partners

Ken Berger said...

Rob,

Thank you for your very thoughtful comments. I entirely agree with your core points and share the concern that efforts to create social value may become nothing more than an adjunct issue or trendy superficial funding strategy rather than the real deal.

Let us fight for the real deal.

Anonymous said...

I was curious if you ever considered changing the page layout
of your website? Its very well written; I love what youve got to say.

But maybe you could a little more in the way of content
so people could connect with it better. Youve got an awful lot of text
for only having 1 or two images. Maybe you could space it out better?


Review my blog :: jeux enfants

Ken Berger said...

Anonymous,

We have begun to work on improving the layout. Thanks!

Ken