Code for All Summit: Defining and Measuring Impact

At the Code for All Summit this spring, Stacy Donohue from the Omidyar Network hosted a panel about defining and measuring impact. Joining on the panel was Jennifer Pahlka (Code for America), Adi Eyal (Code for South Africa), Matthew McNaughton (Code for the Caribbean), and Laura Walker MacDonald (SIMLab). Understanding what change we want to make and how we know if we are achieving it is a challenge for all civic technology organizations. During the session the panel discussed how different Code for All partners define goals and measure impact.

You can watch the first part of the panel by clicking here:

Jennifer Pahlka / Code for America

Jennifer Pahlka kicked off the panel by talking about Code for America’s vision statement and how Code for America is going about achieving those impacts.

Code for America believes government can work for the people, by the people in the 21st century.

Code for America builds open source technology and organize a network of people dedicated to making government services simple, effective, and easy to use. Initially, Code for America would take on a variety of challenges – but has learned that they can be more effective by focusing their efforts to affect deeper change.

Right now, Code for America is focusing it’s work in four focus areas: Health and Human Services, Safety and Justice, Economic Development and Communications and Engagement.

Code for America is working by influencing the civic technology ecosystem, changing government practices, delivering focus area outcomes, and working to make this effort sustainable.

As part of that shift, Code for America is moving aways from measuring things like “how many apps’ and moving to measuring the more permanent sustainable changes that can be seen. These include the number of pilot-scaled projects, products made available to government, government adopting better technology, tech people hired by government, number of official brigades and number of governments that have adopted new skills and preferences after having worked with Code for America. One of the challenges of adopting these metrics is that these metrics can’t be generated by a computer – the information has to be gathered by people.

Adi Eyal / Code for South Africa

Next up was Adi Eyal who serves as the Director of Code for South Africa. Code for South Africa is a civic technology lab using data and technology to change lives. Code for South Africa is still in it’s early stages.

One of their first projects regarded the government’s procurement of pharmaceuticals. The challenge with this project was that there is no real benchmark of how to determine if price points are fair are not. When doing their research, they found vast price differences for the same drug across countries.  For example, South Africa pays 60 times what Malawi pays for the drug Albendazole even though South Africa has the biggest market.

The team conducted the study for almost two years and used the results to push drug companies like Bayer to lower their price – saving the government 10 Million Rand ($2 million US).

While the project had impact, Eyal states that the effort wasn’t sustainable and didn’t call the project a success partially because of low stakeholder engagement. They were working with country governments and it was difficult to gain traction because efforts like these require staff whose job it is to care – efforts like these require people on the ground.

A different project involved showing residents what their medicine should cost. In South Africa, the price of medicine is regulated so that there’s fixed limits on what people can be charged – but that price doesn’t include things like taxes and the dispensary fee. Eyal had the data and put together a quick app that aimed to help residents to avoid being ripped off. When the app was launched, it turned out that people were using the tool to help them find the generic brand since the tool categorized medicines with all the same ingredients together.

The project ended up being accidentally taken offline when changing servers and that led to the discovery of a doctor who was using the site to help prescribe medicine that his patients could afford. When measuring the success of the project, the question of impact was a maybe. While they know they have one real user who is finding a lot of use out of the app, they don’t know who else is using the app. (Code for South Africa hadn’t really marketed before.) While the app is sustainable given the low-cost to maintain, it’s hard to measure it’s success without knowing who is using it.

Eyal also mentioned a project that was done in cooperation with Black Sash that helps monitor government service delivery. Code for South Africa developed posters (they’re cheap, they don’t break, easily distributed) to help spread the results of the project. Eyal points out that while ‘solve the problem’ is a popular term in civic tech, a lot of times we’re not solving problems – but supporting the work of government and community organizations.

This project was impactful and sustainable, it’s important to note that it’s success was in supporting rather than solves.

Another project that stressed the importance of not doing things alone was the Living Wage project. LivingWage is a project that helps people check to make sure that they are paying their domestic workers a living wage. In many cases, residents were not aware of what the living wage was. While the app was very successful during their first week, Eyal stated that the project needs a partner that can help sustain the app’s effort. Code for South Africa went on a roadshow to try to find a partner and is still looking for one at the present time. In terms of metrics, it’s hard to measure the success of the app without knowing if people are paying better salary. While the tech part of this is sustainable, ideally the site would be on the Department of Labor’s website so that when people search for “How much should I pay a domestic worker” it would go to the government’s site.

Matthew McNaughton / Code for the Caribbean

Code for the Caribbean is an organization that’s helping to bring civic innovation to the core of Caribbean governance. Code for the Caribbean is a project of the Slashroots Foundation. Their director, Matthew McNaughton, also spoke on the panel on how they are defining and measuring impact.

Code for the Caribbean focused immediately on a single issue. Jamaica is one of the most indebted countries in the world with almost half their budget being allocated towards debt management services.  For every dollar that the Jamaican government has, 55 cents of that goes towards the debt. After that, about 25 cents goes towards paying wages for public sector employees leaving 20 cents to run the rest of the government including services, infrastructure, and investments.

Given this context, McNaughton points out that the app was never ‘sexy.’ Given these challenges, there wasn’t any traction for just building an app. One of the bigger challenges that Jamaica faces currently is praedial larceny (cattle theft) – which causes a lost of $5 billion dollars to the economy each year. Given that problem set, it’s not the easiest thing to measure success or impacts.

At the high level, Code for the Caribbean measures success by seeing how many government officials are starting to adopt open data and civic innovation. At this year’s International OpenData Conference, the Caribbean had a large contingent which McNaughton was very proud of.

Code for the Caribbean is also working with the new CIO of Jamaica to develop a national open data portal. On the institutional level, they’ve seen data sharing agreements across agencies and the information quality is improving.

Code for the Caribbean is also looking at service delivery indicators including the quality of care being delivered by government agencies. As part of that effort, they’ve been working with government agencies to set up ways to measure that delivery.

McNaughton says that the the process by which any agency becomes data driven is a non-trivial one – as data usage increases, so does the appetite for more data.

Laura Walker MacDonald / Social Impact Lab Foundation (SIMlab)

The last speaker on the panel was Laura Walker MacDonald who serves as the CEO at the Social Impact Lab.(SIMLab) SIMLab helps people and organizations use inclusive technologies to build systems and services that are more accessible, responsive, and resilient.

SIMlab mostly concentrates on apps that use SMS messaging. One of the group’s first projects was FrontlineSMS – a suite of software that helps organizations build services with text messages. While it started as a non-profit group, they spun off FrontlineSMS as it’s own business and started the SIMlab. 

During that time, the team had tried to measure the impact of the platform – something that was harder to do since the platform didn’t have any speciality. You could use the platform in any social change project where SMS might be useful. The team’s metrics instead centered around it’s performance as an app. Today, the app has been downloaded in 200 countries and over 200,000 times.

MacDonald stresses that it’s important for organizations to understand “how you suck, the ways in which you suck, and to try not to do that again.” It’s not just knowing about what you’re impact was, but monitoring things in real time, checking in, and understanding how things are going so you can tweak your plan.

Part of her teams process for defining and measuring success was going turning their blog posts about what they did into numbers that could be tracked. MacDonald also states that how you measure comes down to who is giving you money and why – how they define value is how you should define value. Because the team was in the international development space, the app itself was being evaluated on social impact – even though the project was really about making good software. As a result of that, when they spun off SIMlab part of their role is to help organizations who are using SMS software evaluate their efforts.

One of the main reasons for SIMlab spinning off Frontline into it’s own company is that as a for-profit software company, they could evaluate FrontlineSMS in terms of being good, useful software. As a social good organization, SIMlab is still evaluated on social impact.

MacDonald also recommends that you know why what you’re doing, what your definition of value is, what your theory of change is and that’s what you should evaluate and manage.

You can find out more about Code for All on the Code for All website at www.codeforall.org.