Measuring our results: Making sure that we’re changing lives for good
For a long time, Rotary thought about measurement in terms of the money and time that we invested in a project, or the number of people that we trained through our project and programs. While these types of information are useful to gather, our approach to measuring our results is changing.
It remains important to celebrate the money, time, and other resources we bring to help address needs in a community, yet we cannot stop there. We have bigger goals than that. Our members want to improve the quality of life in communities near and far. And to ensure that is happening—and that we are individually and collectively learning and growing—we need to double down on measuring the positive change that happens as a result of our actions.
And that is what impact is at Rotary. It is the positive, long-term change resulting from our actions. Our members join Rotary to grow in their experiences and in their service. And part of that growth requires reflection and learning.
Now we want to know for certain if we’re creating positive change — to determine while our projects are in progress if they’re working or if we need to make adjustments, and to understand exactly what kind of results they’re having. And we want to make sure the change is sustainable.
This is why our Action Plan asks all of us not only to do good work but also to measure the full results of our efforts — which means incorporating monitoring and evaluation into every project that we carry out.
Dr. Axel von Bierbrauer, medical project manager for a project in the Rarieda district of Kenya, acknowledges that this requires a new mindset. “Most people believe that acceptable data is ‘training 100 people through our project.’ Measurement doesn’t stop at counting how many people you trained. It’s the next step — checking the quality.” In short, measurement includes not only counting the number of community health workers trained within a project but also determining if these workers are sufficiently applying the training in their work with patients.
Isis Mejias, an ambassador for the Water, Sanitation, and Hygiene Rotary Action Group and the founder and director of the organization Global WaSH, says measurement used to focus on the number of available clean water sources. “But that’s not capturing what happens when children live without disease,” Mejias says. “When we measure results, we can see how projects positively change people’s lives. This means job opportunities, gender equity, time savings, improved health — all of which is not seen if you just pay attention to the number of new water sources.”
Beyond showing our impact, she says, gathering quality information about project results can also be a source of inspiration. One project partner in Uganda told Mejias that she hadn’t realized the connection between water quality and school attendance, or that girls miss one week of school each month because they don’t have a safe toilet and the hygiene support they need when they’re menstruating. “Measuring impact empowered this partner to continue her work and make it better,” Mejias says. “It’s not just about the number of pumps installed, it’s about touching someone’s life directly. It’s about envisioning the ripple effect of having clean water”
Marcelo Haick, a trustee of the Rotary Foundation and chair of the Cadre, believes that the ability to clearly demonstrate our results is also a powerful incentive for attracting and retaining members. “Young people, especially, have expectations when they join Rotary — and they leave when they don’t see those expectations fulfilled,” he says. “They’ve grown up and work in a data-driven culture —measuring and sharing our results is one way we can show them how we’re making a genuine impact.”
So, how do we know what measurements we need? One way to start is to find out what information others have collected—. Randall Blair, principal researcher at the evaluation consulting firm Mathematica, helps foundations and government agencies determine the best types of programs to invest in. Blair, who is also a new Rotarian, says a good starting point is to simply search the internet early in your planning and use the large amount of data that’s available.
You can search for a big concept, like ‘reducing maternal mortality or proven methods to increase literacy.’ “To make the search more effective, add the names of trusted international organizations like the United Nations and the World Bank’ Blair says. “This kind of early phase research helps identify solutions that are already known to work—as well as vetted measures of success.”
In terms of how we collect the information we need, no one method fits every situation. For some projects, large-scale surveys are the best way to gather information about the results. For others, the outcomes might be assessed using economic indicators, interviews with community members, or focus group discussions with parents and officials.
Determining how to collect data can also involve using Rotary members’ renowned networking power. Rotary clubs can easily find and work with local government entities, nongovernmental organizations, universities, or other research institutions that von Bierbrauer says already have “the skills, knowledge, and networks to strategize and conduct quality measurements, collect and manage data, and analyze effectiveness.”
Mejias used that approach for a project in Uganda. She connected with a local university that was instrumental in developing a baseline survey of indicators in the community. University students went house by house to conduct the survey. Mejias says that Rotary members could have similar roles gathering data, while delegating the analytical work to a partner.
We should also establish how often we’ll collect information, says Viveca Serder, a technical coordinator for the Cadre in basic education and literacy: “If we don’t know the starting literacy rate, we can’t measure our success — so we have to do this from the very beginning.”
Next, she says, think about: “What will happen? How should we use the money? Then, write down your thoughts about what you want to happen in one year, two years, and after Rotary leaves a project. All of this is important to do early on, so the intended long-term effects are part of the project design and we know where we are starting from and what changed because Rotary was there. Many times, that will need to be assessed years after a project ends.”
This method of designing a project — identifying your desired results and the steps needed to get there — is a process known as a theory of change. It can apply to initiatives of any size.
“It’s a road map of what’s supposed to happen,” Blair says. “For example, in a teacher training program, you first document the money you will spend on training, how you you plan to target schools and teachers, and what the training will include. Then, you can plan out the behavior change and benefits that you expect from the training, provided your assumptions hold. Will teacher engagement with students go differently, and will students better master the material through that engagement? Anything that results from the training is something we can measure along the way.”
However you define your theory of change, having what Blair calls “your vision for success” will help you make a bigger difference.
“As long as everyone uses that same plan detailing how you’ll achieve your desired results, your project will benefit,” Blair says. “Through this, years later, you will have changed the system of behavior so that it never goes back to how it was before. That’s impact.”
Connect with your local district resource network to learn more.