Thoughtful Blueprints, Well-Built Houses 

A grantmaker’s guide to evaluating a nonprofit’s theory of change

“If we just start nailing that lumber together, we’ll end up with your dream house.”
—No architect, ever

Nonprofit mission statements evoke powerful visions of social change. To translate these lofty rallying cries into action, many thoughtful organizations have developed a detailed blueprint illustrating the links between their inputs, activities, and desired impacts.

Some refer to this blueprint as a logic model; we at Arbor Brothers employ another common term: theory of change. Whether distilled into a stand-alone document, narrated in a grant application, or floating around in the executive director’s head, our experience suggests the clarity and logic of this blueprint are key indicators of an organization’s ability to generate meaningful social outcomes.

As grantmakers, we see it as our responsibility to evaluate the strength of every potential grantee’s theory of change and invest accordingly. We don’t require this in a formalized document before we make a grant, but we do try to understand an ED’s thinking regarding the elements that compose it. This brief post outlines our approach to doing so. (This is written with direct service organizations in mind, but the methodology can be adapted for many different nonprofit types.)

To ground ourselves in a nonprofit’s theory of change, we ask EDs the following three questions:

• What is the risk profile of the population served?
• What is the quality and dosage of service provided?
• What are the data supporting the claimed outcomes?

The strongest blueprints will depict tightly logical and intuitive connections between the organization’s inputs, activities, and demonstrated outcomes, conveying a sense of “hanging together.”

What is the risk profile of the population served?

Consider two nonprofits operating college access programs: Nonprofit A serves low-income students and has a college matriculation rate of 99%. Nonprofit B also serves low-incomes students and has college matriculation rate of 65%. Which should you invest in? What if you were to learn that nonprofit A has a rigorous application process and only accepts top-ranking middle school students, whereas nonprofit B works with academically unscreened participants who have recently been arrested?

Nonprofit A’s targeted recruitment efforts are neither intrinsically good nor bad, but they do prompt the question “What is the counterfactual?” Said another way, “Given where these young people are starting, where would they likely end up without the program?” If statistics suggest 85% of nonprofit A’s participants—but only 5% of nonprofit B’s participants—would be college-bound without intervention, would that change your relative preference?

We like to understand an organization’s intentional (and unintentional) screening criteria and resulting cohort profile first, as that orients our expectations around the next two questions.

What is the quality and dosage of service provided?

For fundraising purposes, many direct service nonprofits feel pressure both to make strong claims of impact and to serve a larger population over time. This often creates a tension between depth and breadth, pitting sometimes subjective outcome measures against an easily countable figure: the number of participants served. This can result in an unfortunate trade-off: providing lower quality support to a broader group of people by downwardly defining “served” from “made a meaningful impact on” toward “interacted with at least once.”

For example, mobile apps, drop-in centers, and other “scalable” light-touch models can play important roles in improving people’s lives by interfacing with lots of folks and connecting them to other resources. But, since most participants only engage with these resources periodically, we should expect that these models generate only modest impact on each participant served. In our experience, helping individuals make meaningful, durable change in their lives requires them to develop new skills, mindsets, and/or behaviors. That change comes neither quickly nor cheaply, and programs claiming otherwise often prove too good to be true.

There are no bright-line tests for evaluating this aspect of an organization’s theory of change, but, after meeting an ED, the question we ask ourselves is “Given the quality and depth of the intervention, how likely is the program to meaningfully change the life trajectory of a participant?”

What are the data supporting the outcomes you claim?

We all respond to emotionally compelling stories, and good nonprofits tell them. Great nonprofits tell them well and—even if they don’t lead with it—have a track record backing up their claims of impact with statistics. Anecdotes are not data sets, and emotional appeals should rest on, not replace, robust bodies of evidence.

Even if a program is too early-stage to have statistics, or operates activities that don’t lend themselves easily to data gathering, we ask EDs “How, specifically, do you define success? What program tweaks have you made recently to generate better outcomes?”

At the end of the day, of course, all of this analysis really boils down to sound judgment. There is no universal tool for determining which “blueprints” are likely to yield the sturdiest houses, nor which of the many worthy nonprofits would most benefit from your support. Our commitment to ourselves, and our hope for our fellow grantmakers, is that we continue to grapple with these thorny but essential theory of change questions.

After graduating from the University of Michigan with an engineering degree, Scott began his career teaching middle school mathematics with Teach for America in the Washington Heights neighborhood of New York City. He taught for three years before moving to the south end of the island for an MBA from NYU, where he helped launch the nation’s first student-managed venture philanthropy fund. Scott joined Lehman Brothers and later Neuberger Berman, spending six years managing investments in infrastructure and alternative energy companies. Scott serves on the New York and National Boards of Directors for Summer Search, a youth development organization.