Getting good data that can determine success has always been difficult from programs that work to change behavior, as it is not always easy to measure behavior change — though programs working with criminals have arrest records — especially within the two or three year window most nonprofits interact with clients.
This article from the Foundation Center explores the issue.
An excerpt.
“There's no question that the need for better data is on the minds of just about everyone who is working to address seemingly intractable social problems. If you run a nonprofit, you've undoubtedly felt the push from funders to demonstrate the impact of your programs. If you're a foundation program officer or an individual donor, you are probably looking for data that enable you to compare programs and choose the most effective ones. And in today's tough economic climate, government leaders at the local, state, and federal levels are urgently seeking ways to use data to make better use of increasingly limited resources.
“Fortunately, we're on the brink of a sea change in how we generate and use data to address social problems — and change is exactly what we need. Although significant data on social issues exist, much of it is not publicly available or is not action oriented. Indeed, quality information about nonprofit performance is scarce and not typically standardized to make it possible to compare organizations working on the same issue. As a result, we don't know whether the billions of dollars invested annually in nonprofit organizations by the public and private sectors is achieving the desired results — or any results at all.
“The enormous potential to improve the quality of and access to information is analogous to the information revolution that took place in the private sector during the twentieth century. The first stage of that revolution was inaugurated by Section 13 of the Securities Exchange Act of 1934, which required publicly traded companies to file annual reports (known as 10Ks) with the U.S. Securities and Exchange Commission. The information revolution continued with the rise of the tech sector in the United States in the late 1970s. Back then, tech start-ups were growing rapidly. Many investors, however, lacked data about industry trends, which companies were "hot," and how those companies were performing on a comparative level. One of the innovations that helped provide more transparency at the time was the development of an independent financial research industry. Reports, conferences, and advice began to be offered by the likes of the Yankee Group, Forrester, and Gartner Research. That information, in turn, provided investors with the insights they needed to make informed investment decisions and greatly increased the amount of growth capital available to tech companies, both young and established.
“In the nonprofit sector today, by contrast, the only standardized source of information is the 990 tax form. And while the 990 provides financial information, it offers no indication of whether an organization is fulfilling the charitable purpose for which it was awarded tax-exempt status in the first place. Imagine, then, what an information revolution similar to the one that transformed the private sector in the the last quarter of the twentieth century might mean for twenty-first-century efforts to invest in social change. Rigorous and readily available assessments would ensure that "social investors" are able to identify the most successful approaches to our most pressing social issues. Nonprofits with access to better information could use that information to assess their programs and make informed decisions about ways to improve those programs. Funders would be able to more effectively compare programs and select the most promising grantees at every step of the innovation cycle — from early-stage testing of new models to replicating already proven approaches. And collaborations involving the nonprofit, government, and business sectors would be able to use data to help ensure that their efforts resulted in sustained social impact.”