There is craft in creating a structure that fits your organisation, but anyone can put the essentials in place and here are five steps to get you going.

We need to get serious about impact

Social services are going to be hit by a wave of need as we progress into 2021.  Many of the temporary COVID fiscal measures provided by governments will drop away removing the safety net for those with reduced incomes in this global recession.  We need to start building a social economy that really works for people, and that delivers value for money.

In a tight funding environment, donors, including the more discerning public investors, will compare similar services based on their quality of impact evidence.  The maturity of impact management systems will be put to the test this year!

Setting up a system

So where do you start?  Most purpose organisations have a scattered collection of case-studies or quantitative measures that have been assembled for certain funding streams.  Knowing how to set-up a system that is integrated into your organisation is the next level of maturity.

Naturally, there will not be a one size fits all solution for every organisation.  The scale, services type, stakeholders and quality of the existing and needed evidence base will all influence your approach.

The essence, however, remains the same for all organisations.  You need to build an overarching impact narrative that clearly identifies the intended outcomes; and then set up a quantitative reporting mechanism to track your causal contribution to any changes.

Five essential steps

1.       Map a Theory of Change

The first step for any organisation is to map your impact story through a Theory of Change.  This will articulate your most material outcomes and capture how and why your services achieve these changes.

I sometimes encourage organisations to draw up a simple, central change logic to get started.  This ensures a basic architecture can be mapped out for your system.  If you take this approach, be prepared to revisit your Theory of Change and build more detail onto it as you progress (perhaps on a programmatic basis) because it will ultimately be the blueprint specifying what outcomes are measured.

Your Theory of Change is your impact narrative and your strategy.  Remember that best practice is to involve internal and external stakeholders in identifying meaningful outcomes from your work.

I’ve laid out a method for building a Theory of Change in this blog.

2.       Review existing reporting

Your Theory of Change will provide you with a list of impact areas to measure that are created by your work.  These can then be cross compared to existing evaluation and outcome data to review your existing coverage.

This exercise is about identifying any gaps.  Are there outcomes you do not evidence, but which are critical to your value creation story?  For example, funders might identify “innovation” or “relationship building” amongst funded services as critical, but they might have overlooked collecting relevant data as they are often focussed on end outcomes for beneficiaries.

The cross comparison should reveal variations in methods of reporting on impact, and this will provide an opportunity to harmonise approaches across services and funding streams.

A bonus from this exercise is that organisations often find they can save work by stopping some data collection.  This is because they discover they are collecting data that is either irrelevant (perhaps a relic from old processes) or too poor in quality to be useful.

3.       Develop a framework

The review will lead into planning ongoing impact management systems.  This is where you will need to make some trickier decisions about the quality of evidence you require (robustness), while balancing this out with the available resources for managing that impact data.

Your material outcomes and implementation theory should guide an examination of the existing evidence base and what kind of ongoing tracking (or primary data collection) is needed to prove your impact. If you are implementing a service that has strong evidence underpinning it, for example, a hotline for domestic violence victims, you may not need to collect as much evidence on the impact as a new or novel service.

Your Theory of Change will become the blueprint or structure for documenting a framework so here you should capture:

-          What evidence (indicator types) is appropriate for tracking each of your outcomes

-          How you will collect the data

-          Your sample size and frequency of data collection

-          How you will (if at all) evidence causality.

-          Benchmarks if desired (if prior years or by sector)

Setting yourself up with an indicator bank can also make ongoing management more seamless as users will have some flexibility to ensure the data they collect is relevant to the context in which they operate, yet consistent with the needs of your framework.

Designing data collection tools, such as surveys, should also be drafted at this point.  Logs should be included for recording secondary data streams or using activity monitoring data like the number of service users.

4.       Build staff capacity

Strong organisational systems depend upon staff engagement and capacity. If your team believe in the value of impact management, they will be a wellspring of observations and ideas that will make your data more meaningful.  Teams who do not understand the purpose of the data they collect are likely to collect less data and cut corners.

Work with your teams to build their skills in managing good impact data and analysis including qualitative and quantitative sources.  More about creating impact thinkers in your organisation in this blog.

5.       Set-up data systems

Today we are lucky to have a range of digital solutions that can streamline impact management.  Your aim in designing a digital system should be to have a central dashboard for collating evidence and reviewing performance across services and time.

Purpose-built impact management platforms normally include tools for collecting the data (automated online surveys), pre-programmed analysis and a dashboard.  If a standalone impact management system works for your organisation you should certainly check out the available products.   Just remember to use your own discernment on whether the type and nature of the data you are collecting is appropriate as these systems will reduce flexibility and control.

Another option is to integrate your impact data into existing organisation software.  This might be a CRM or a finance system.  Ultimately this will put impact data within easy access of senior decision makers and reduce duplications.

Integration can be time intensive and is likely to require some engineering.  Before you take that leap, you might want a simpler solution which is to design a more manual database and data visualisation dashboard for example with Power BI or Tableau.  This will give you more control and tailoring.

Regardless of your chosen approach, make sure you have set up an appropriate database for managing data collection.  Accompany this with written operating procedures.  Ultimately you want to pick the solution that enables you to make joined-up, smart decisions.

Play and repeat

When it comes to road testing your new framework and system, my advice would be start small.  The Theory of Change should outline the impact strategy for your whole organisation, but you can test the impact measurement system by piloting it in one area, adapting, and then rolling it out more broadly across other departments.

An impact management system should be an ongoing learning tool.  The collection and review of data should take place routinely to enable you to properly allocate resources, stop work, pivot, innovate and scale.  Make sure you system is genuinely flowing and interacting!

Final thoughts: five essentials for an impact management system

We are in the midst of enormous social turmoil.  It will not be enough for purpose organisations to continue doing what they have always done and hoping for the best.  

As we respond to the social challenges of 2021 we need to see more organisations getting serious about impact by making it a core part of their operations.  The five essential steps above will help services, organisations and funds explore how they can deepen their efforts.

So why do so many people feel uneasy about the difference their work creates?  And how can your organisation enable the skills and culture for impact management?

COVID19 has made us all more aware of our everyday invisible impacts. We wear masks to stop tiny droplets on our breath from affecting other people, we touch less things, go less places and make tough choices not to see loved ones to avoid the small but lethal risk of spreading illness.

Managing intangible impact is hard, but with the right information and guidance it is something that everyone can take greater responsibility for.  I find people often miss the words and ideas that would help them organise thoughts on their job's impact.

This matters because it can be the difference between harnessing significant organisational value like greater innovation or positive customer sentiment; or it could mean exposing your organisation to major risks in your operating environment.  

Many organisations box-off the idea of impact with high-level metrics like "number of lives touched".  This approach to impact is meaningless to the majority of the workforce and contributes to a sense of cynicism.  Much like an orchestra, each part of an organisation needs to have the capability to play its own tune, making the music of impact meaningful to each instruments, while also contributing to the overall symphony.

So how can we unlock an impact mindset within an organisation?  There are two basic things to think about: (i) getting strategic buy-in from leaders; and (ii) delivering a top-notch learning and development program.

Secure buy-in from the top

The rise of the Chief Impact Officer reflects the importance organisations now place on planning, delivering and evaluating impact.  However not everyone is lucky enough to work in a organisation that leads with impact, sometimes the initiative may come from the middle.

Championing impact management when you are not the final decision maker can be like playing snakes and ladders – one minute you step up a few rungs of the ladder, but then a new strategy or structure imposed from above will see you slide back down.  

If you are a changemaker outside of the leadership ranks, do not go at it alone.  Consider starting a working group that represents and communicates with the whole organisation.  If you don't already have leadership backing, then a willing pressure group will probably make it hard for them to resist your efforts.

Build impact thinking capacity

The second step is to build competence and confidence in managing impact. To grow an impact mindset you need to first aquire some basic knowledge and then have the opportunity to put it into practice.  So, how can you use learning and development time within the organisation to embed an impact mindset?  Capacity building is central to our work, so here are a few tips :

1.      Locate your learning and development experts

·         You may have a specialist team, or the skills may be within your HR function. Meet with that team, or person, and discuss the learning need for impact measurement and management.

·         If learning and development is devolved to teams and line managers, or dispersed geographically, your working group will need to map those individuals and plan how you will engage with them.

2.      Internal or external training?

·         The advantages of using an external trainer include access to specialist knowledge and experience in impact management.   An external trainer will save you time, be able to listen and advise teams neutrally on how to put impact management into practice, and can bring broader ideas and energy into the mix.

·         The alternative is to design a learning initiative that can be led internally by people who know your organisation intimately, its strengths and its sticky points. If cash is a more important resource than time this could be the option for you.

·         The third option is to use the best of both worlds. Run a ‘train the trainer’ session and co-produce the internal learning programme with external oversight.

3.      From training to learning

·         When designing your learning initiative don’t stop at the training programme. More than ever, today’s students are learning on their feet and taking advantage of the many learning contexts. Ask yourself, how do people in my team learn best? Do they like to read books or interact? Do they enjoy developing theories or trying out new concepts in practice?

·         Consider options such as online courses, shadowing more experienced impact practitioners, mentoring, action learning groups, secondments, simulations or reading seminars. The solution should fit your own context but try mixing up the methods so there is something for every learning style.

4.      Make it personal: the ‘what’s in it for me?’ factor

·         To make your investment in learning deliver, participants need to see how impact thinking is going to improve their job. As a trainer, my heart sinks every time I hear someone say: “I’m here because my boss sent me”. To take something meaningful away, participants must recognise their learning needs and the value the training offers. For most people, its realising that impact will help them explore the questions of "how can I be more effective?" and "did I make a difference today?".

Final thoughts: develop impact thinkers throughout your organisation

There is a hunger, especially amongst younger employees, to know that their work is making a meaningful contribution.  We live in a world of data but most people struggle to draw the connection between their short-term activities and the ultimate contribution their work is creating.  

Learning time is seen by most people as a reward in their job.  So why not invest it into growing impact mindsets.  A leadership mandate to roll out both formal training and continual learning practices will pay dividends both in employee fulfillment and organisational value.

Enabling the skills and culture for impact management means bringing greater precision into the way your organisation pursues intangible value.  It will mean building trust and confidence across your organisation that what people do matters.

This article is an updated version of an original post on NEF Consulting in 2016.

In Social Value practice we regularly use three impact concepts to help calculate your share of any outcomes.  This post summarises what they are and how to apply them.

What is impact?

Impact in Social Value practice refers to the amount of additional change you have generated.  It is the change that would not have occurred in the absence of your work.  Or put another way, it is the portion of value that is caused by an intervention.

This is a more precise use of the word impact than you are likely to see in other evaluation practices where it often just refers to long-term change.  The conceptual clarity is useful because cause-and-effect considerations are normally missing in general outcomes measurement.

Approaches to estimating impact

A clean approach to estimating impact is to use a good control group selected as part of a randomised control trial. The difference between the outcomes experienced by your stakeholders (receiving a treatment or intervention) and your control group (stakeholders not receiving a treatment) will provide you with your impact.  For example, the wellbeing and academic outcomes experienced by a group of children entering an innovative foster care program could be compared to a group of children who did not participate in the foster service.

Unfortunately, randomised control trials are never as squeaky clean as evaluators might have you believe.  Returning to our example above of the foster care program, what characteristics would need to be consistent between the treatment group and the control group to make them a reliable comparison?  Family history, prior experience of the care system, type of school they attend, age, where they live and even genetics will all play a role in influencing these children’s outcomes.

Randomised control trial sizes are intended to mitigate the variation you might see between your treatment and control groups.  But even then, there are ethical questions around knowingly depriving a group from a beneficial intervention or treatment.  Constructing a control study is therefore beyond the means of most small services and community projects.  Luckily there are other helpful ways for us to estimate the counterfactual.

Social Value practitioners use three elements to assess impact. Once you have measured the size of the outcomes that have taken place, you can then take slices out of that pie to account for these three aspects of the counterfactual.  The concepts we use are:

Let us look at each in turn and describe how to apply them in your impact calculations.

Deadweight

Deadweight is the level of outcome that would have occurred regardless of your intervention; or the amount of change that would have happened anyway.

How do you determine what might have happened to your stakeholders if your work did not exist?  A simple way is just to ask your stakeholders.  Qualitative data can be used to create a visual narrative, perhaps with multiple possible pathways that your stakeholders could otherwise have taken.

For example, if women did not receive help at a refuge, they may otherwise have remained with an abusive partner, or they may have moved in with other family, or they may have ended up homeless.  You can ask stakeholders to rank the likelihood of each pathway and use this to describe the difference of your service.

If you would like to put a number on the deadweight, you will need a comparison indicator or benchmark.  This benchmark should relate to the characteristics of your participants as closely as possible.

The comparison indicator may come from government data that can provide you with outcome trends either locally or nationally for a similar segment of the population to your stakeholder group.  For example, you may look at ABS data on employment rates, crime trends or neighbourhood indicators of deprivation over time. Alternatively, you may find useful benchmarks from other research organisations or from social programs themselves.

Remember you are looking for trend data, so either a change from one point in time to another or multiple data points from successive years.  Comparing your outcome data, which will represent a change to a static point-in-time percentage is a very common pitfall when using deadweight benchmarks.

Once you have your deadweight, it should be deducted from your measured outcomes (removing a slice of the outcome pie).  The exception is when your initiative prevented a harm (a prevention program), in which case it should be added to your measured outcome level.  The harm you prevented is an invisible outcome so you would not have been able to gather direct data on it.

Deadweight is most easily handled as a percentage.  For example, you run a transitional housing program that aims to get people into secure long-term housing.  You do this by supporting people to deal with the needs that left them in housing crisis.

After six months of support, 80 per cent, or eight out of every ten participants, have secured long-term housing.  However, the national rate of people in housing crisis receiving long-term housing is 50 per cent or one in two during the same period.  The national benchmark must therefore be subtracted from the outcome you measured.  This leaves you with an outcome additionality of 30 per cent (see chart below).

Attribution

Attribution is the portion of outcomes that your intervention can take credit for.  It is applied to the outcomes after you have accounted for deadweight.

Further influences on outcomes could be other services, individuals like family members or even something about individual’s circumstances like their health or financial resources.  These influences are usually specific to your stakeholder group (perhaps they are refugees in housing crisis) and therefore are unlikely to be captured in the more general deadweight you used (for example the national rate of long-term housing placements).

Estimating the amount of attribution to each factor is not an exact science.  In the absence of a complete dataset for performing causation analyses (like a regression analysis), we simply ask stakeholders to estimate the attribution to each factor.

I think this is most clearly illustrated with an actual survey question.  Using the example of our transitional housing program, the stakeholders who experienced the outcome of moving into long-term housing might be presented with a survey question such as the following:

In the example above, the survey prompts the respondent to think of other influences on their outcomes.  This is good questionnaire design because it enables the respondent to consider wider influences.  You may have identified other influences as part of the “external factors” in your Theory of Change research.

Other approaches could include stakeholder workshops where various service providers describe their contributions and debate their influence.

Displacement

Displacement is where one outcome displaces another.  A benefit is therefore not created from nothing but instead it is moved around a system.  Displacement does not apply to all outcomes, but it commonly affects outcomes in the employability, criminal justice, land use and environmental sectors.

To give an example, one of my clients, a Local Authority, had an issue with youth anti-social behaviour on the streets in the evenings.  They decided to provide free cinema tickets to divert young people into a different social activity.  After a month of the scheme, anti-social behaviour complaints had dropped, and the scheme appeared to be a success.

However, one small detail had been overlooked.  There was no cinema in the local ward where the young people live, so the film tickets required the young people to travel to a neighbouring area.

After going to the film, the young people were sitting down on the streets outside the cinema and drinking and engaging in their normal anti-social activities.  In this case, the positive outcome of reduced anti-social behaviour had not occurred at all, the anti-social behaviour had simply been moved to another location.

A similar phenomenon exists with employability schemes.  In a market where there are a limited number of jobs, one person’s success in securing a job will indirectly prevent another jobseeker from experiencing the same outcome.  Associated outcomes relating to taxes paid, productivity gains, reduced welfare payments and wages received need to be offset to account for these benefits being denied to other stakeholders.

Carbon emissions from driving an electric car are similarly displaced from your personal exhaust pipe to the fossil fuel energy production sites if you are using grid energy.  The emissions are less, but if the grid is electrified by coal and gas the emissions still exist, just elsewhere.

To apply displacement, a percentage estimate must be made on the strongest evidence available.  In the case of employment outcomes, you may be able to access local output area databases with timely displacement rates.  In our environmental example above, emission intensity factors for grid energy could be accurately factored into your calculation.

For other outcomes the data available to support your displacement estimates may be less certain.  You may not know exactly where crime is being displaced to, or how the wellbeing of people on a housing waiting list is affected.

You must make a judgement about the reasonable amount of time you will spend researching credible data on displacement depending on its importance to your value story.  If in doubt, be conservative and over-estimate the effects so that you can claim you know that “at least” this much impact is taking place.

Calculation

The calculation above is a useful guide for how to combine your outcome and impact data.  The exact mathematics will depend on the format and logic of your data.

This calculation works best when deadweight data is in the same format as the outcome level data (for example, both are a percentage of the same population).  But if the deadweight data comes from a completely different indicator scale (for example the outcome might be academic outcome data, and the deadweight could be sentiment about how much would have happened anyway) then it is better to multiply the outcome by the deadweight proportion.

Try to take account of any overlap in your impact data.  You may adjust your attribution percentages to account for forces which might already be assessed by the deadweight estimate.  For example, if the outcome was academic outcomes for girls, and the deadweight was the national average improvements for girls, then some attribution factors like potential help from families might be adequately accounted for in the deadweight.

Ultimately the coverage of counterfactual variables between the three impact elements needs to be assessed on a case-by-case basis, and the calculation adjusted with a clear rationale.

Final thoughts: how to calculate your additional value

Traditional reporting and evaluation approaches fall short of telling you about cause-and-effect.  To understand the additionality created by your intervention you need an approach for assessing the counterfactual.

Social Value practice helps you in considering and estimating three elements of your impact.  The three concepts of deadweight, attribution and displacement clarify the role of your intervention.  They can be measured and applied as part of a calculation, or they may simply form part of the narrative and framing of your outcome data.

Just thinking about these factors will greatly sharpen your impact strategy by revealing external influences and collaborators.

This post outlines what benchmarks are (what), challenges motives for their use (why), signposts to current sources (how), and considers when sector-based benchmarks might be ready for wider use (when).

Impact investment and the pursuit of benchmarks

An impact benchmark is a measure that helps stakeholders understand the performance of a project over time, or compared to similar interventions. Essentially they are comparison points.  The idea of comparing impact is major focal point in the impact investment field right now.  Respondents to the GIIN’s global impact measurement and management survey ranked benchmarks the most important resource for improving impact performance.

Impact investors are not alone in feeling this imperative to compare.  I am systematically asked by clients to help them benchmark their performance.  And it is one of the greatest disappointments when we break the news to clients that it is inappropriate to compare to other impact studies, even using structured methods like SROI ratios (I know, I know, it is frustrating for us practitioners too!).

The push towards standardisation in social impact reporting is a centralising call within our field.  I have no doubt that it is the future.  However, in the short term, efforts to standardise are fraught with complications.

To contextualise this, it took hundreds of years to achieve standards in the accountancy field (and they are still slightly different all over the world).  Impact measurement metrics must mature in the same manner.  However, the challenge is much larger, because the standards we agree on must encompass a much broader range of subjects and metrics.

Do you know why you want a benchmark?

I have a gripe with benchmarks.  The use of the word is often loose and lazy.  People ask for comparators without really knowing why they want them or what research question they will answer .  There is a blind innate social pressure that we feel to compare, and this is dangerous in a field where context is everything, and metrics are still very diverse.

The temptation is to reach for the first framework which sounds relevant and credible.  The problem with this kind of (lazy) impact reporting structure is that you often measure the wrong things.  Most available frameworks, even the IRIS+ metrics are full of outputs (as oppose to outcomes) which are poor impact measures.  Additionally, unless the intervention is very standard you are likely to be measuring the wrong things, which will mislead your understanding of real impact.  This is why we always advise starting with a Theory of Change.

To give you a simple example, you might have two school attendance programs.  Perhaps one is working with a general cohort of children and the other is working with a group of children with special needs.  The standardised impact metric around school attendance might be number of days of unexplained absence.

The first, general cohort are much more likely to make improvements on this indicator relative to the amount of effort put into an intervention.  For example, a 75% decrease in absences might come from fining parents.  However, small changes for the children with special needs, for example, a 20% decrease, might be indicative of life-changing outcomes.  

Making it to school might be the end result of very complex self-esteem and independence building activities.  But as an investor, your quick sector-level comparison would lead you to place your funds with the first cohort because according to the benchmark, this is where the most impact is taking place.

The advantages of benchmarks

Benchmarks might not be an exact science, but there are good reasons to pursue them if they are handled with care.  For impact investors they will facilitate two important breakthroughs.

Setting targets and predicting impact

Most impact management is still done retrospectively, after the intervention is completed.  And even then, the findings are not always channeled back into design improvements.  For example, the GIIN found only 65 per cent of impact investors are using impact data to set and revise goals.  This is a major missed opportunity for predicting and enhancing performance.  Where relevant benchmarks exist, discussions can be had on success factors and the appropriate level of impact to pursue in your own context.

Aggregating data

The bulk of the impact investment market is made of large institutional investors and asset managers like banks, development finance institutions, pension funds and insurance companies.  To efficiently manage impact at that scale there is a need for consistent metrics which can be aggregated across investments, even funds.   To aggregate we need to be sure that we are combining apples with apples, not pears.

How to benchmark: three places you can find impact investment benchmarks now

A social benchmark is composed of an outcome level, a defined stakeholder group, and a set time period.  Critically, the benchmark should be calculated in a consistent manner, for example, survey questions on life satisfaction are measured on several different response scales.  Make sure yours is consistent with your benchmark and that you are calculating change against that scale (see my post on how to select indicators).

To achieve this level of control you have three options:

1.       Past performance

Using the baseline performance of your own intervention is a great way of benchmarking.  It ensures that you are picking a benchmark that accurately reflects your context and that the metrics are calculated in the same way.  It will allow you to understand and explore impact performance for individual investments within each reporting period.

2.       Control or comparison groups

Good data on the broader trends for the population that you are helping may be available in some cases.  For example, there is good data on outcomes for young people in out-of-home (residential) care and for ex-offenders.  This will often be available in the form of national, government datasets or sector publications.  This kind of benchmark will give you the additionality of your program, rather than the performance of your program compared to similar interventions.

3.       Established intervention areas

Some interventions are more standardised than others and have impact reporting frameworks which are widely used across sectors.  If you have an investment in one of these areas, you can benchmark with greater confidence.

Interventions with frameworks include housing (see the GIIN’s 2020 report on benchmarking housing investments and HACT’s impact valuation workin the UK), healthcare, education and early childhood services (often based on country frameworks) and environmental initiatives (see the Green Bonds Standards).  There are more and more of these frameworks springing up, for example I can think of at least four digital inclusion indices including the Australian one and the World Benchmarking Group’s work on Digital Inclusion.  So check to see if there is sufficient data available covering the outcomes you are interested in by size and relevance of the datasets that you are benchmarking against.

When will sector benchmarks be widely available?

These days it seems like everyone is developing a standard or a framework that will one day encompass all social (environmental and economic – but especially social) outcomes.  Impact geeks like me love following these initiatives, but for the ordinary impact investor they must be exhausting.

The GIIN’s 2019 survey of impact investors found that on average, impact investors are each using three different frameworks to measure their impact (such as the Sustainable Development Goals (SDGs), IRIS+, UNPRI, GIIRs e.t.c.).  Curiously, they also found that no framework is actually seeing a decline in use since their last survey in 2017.  Additionally, investors who are focussed primarily on social outcomes have the least coherence in their choice of tools, metrics and frameworks.

It is not surprising then that investors would like to see simpler sector-based benchmarks.  So, with all these frameworks and tools to navigate, when might that be a reality?

A few hurdles remain before impact investors will have reliable and easy access to sector benchmarks.  For benchmarks to exist, there will need to be:

a)       A shared understanding of the principles of impact – the Impact Management Project (IMP)’s dimensions of impact has accelerated a common frame of reference;

b)      A mapping of existing outcome metrics – the IRIS+ catalogue has made huge progress on this front but the linkage to outcomes and financial proxies needs work;

c)       Controls on how metrics are calculated – this is a major missing piece;

d)      A conceptually discrete taxonomy of outcomes – many efforts have been made on this front including by Big Society Capital and IRIS but these are still too orientated around interventions rather than capital or outcomes;

e)      Fluency between financial reporting and impact reporting formats.

The controls around how metrics are calculated (point c) is the most important next step for investors to be able to compare and aggregate with confidence.  Where benchmarks currently exist, for example in the IRIS+ catalogue, there is still fuzziness around how to calculate them and what levels of assumptions are acceptable.

A straightforward metric like number of jobs created is still interpreted very differently by impact reporters.  Does it count if the job is a short-term contract, or a zero-hours contract, or what if the person quits after a month?  We need a workable rule here such as the person needs to remain in the job for six months.  Similarly, there are metrics which sound comparable but may be misleading like “number of children with access to education”, “number of children enrolled in school” and “children attending school”.  These might seem comparable but in practice they can mean very different things.

The good news is there is a lot of (tech enabled) energy going into standardisation right now.  I think we will need to wait at least five more years before you begin to see impact reporters confidently benchmarking their impact across sectors.  In the meantime, you should check the leading frameworks to ensure you are aligned with the best practice in impact investment reporting.

Final thoughts: impact investment and the pursuit of benchmarks

Every system of measurement has defined units, methodological rules and reference standards. The impact measurement field is going through a process of rapid maturation towards this, a process that took financial accounting several centuries to achieve.  The difference is that impact measurement metrics are far more diverse than classical accounting.

It is brilliant to see the impact investment field aspiring for the best in impact performance and reporting.  As you look to apply benchmarks to your impact portfolio remember to start by answering the fundamental question of why you want a benchmark.  This might lead you to discover there are effective ways of comparing your performance which do not depend on whole sectors aligning and comparing their value.

The death of George Floyd has been seismic in the conversations around our justice systems and how best to achieve social equity. It has bought into sharp focus the systemic privilege that some in our societies have enjoyed, while others unfairly struggle and sometimes find themselves at the mercy of brutal police and “corrections” services.

As a white woman, I am very fortunate, and would like to acknowledge that I have not experienced discrimination based on the colour of my skin.  I was saddened and incensed to learn about the death of George Floyd and the nature of his killing, and I can only imagine how scared, angry and exhausted those in the Black community in America must feel.  The price he paid is also symbolic of the discrimination experienced by people in the justice systems in two countries that mean a lot to me: The United Kingdom and Australia.

In both these countries we need to do and demand better outcomes from criminal prevention and justice programs. There is a large body of literature showing that socio-economic status is a determinant of both the likelihood of personal offending (putting aside institutional crimes for the moment) and the likelihood that you will have contact with judicial systems including the police[1].

Unfortunately, ethnic minorities are also over represented in both UK and Australian low socio-economic status indicators and in prison populations. Aboriginal and Torres Strait Islanders for example, make up 3 per cent of Australia’s population but 29 per cent of the prison population[2].

Defund the police

There has been a flurry of calls to “defund the police” in the last few days.  This is to root out and refresh a culture of racism in police forces.  But these ideas are not far-fetched or extreme.  They build on a healthy movement of practice and research into crime prevention.

The idea does not mean replacing the police, but simply reducing their role and giving them proper support.  Put simply, this is about re-allocating resources towards safety and away from punitive responses to crime.  It is about raising up underprivileged groups and addressing the determinants of crime.

In emergencies this would look like a mental health worker or mediator attending a public disorder situation triggered by someone having a mental health crisis.  A shelter provider might guide someone homeless to a safe place when they are obstructing the streets. Or it would be providing youth clubs for young people who may be vulnerable to delinquent behaviour.

Commission outcomes not services

Too often social programs like these are designed based on guesswork and assumption rather than evidence.  For example, it is assumed that the police are best placed to respond to all emergencies even if they do not require force or detention.

Government commissioning documents are normally riddled with prescribed activities and outputs based on how things have been done in the past. The thread of evidence on what works is often lost.  This further undermines trust in vital social structures like the police, and frankly, it is a waste of money.

There are two easy things which commissioners and anyone designing or delivering a social service can do to improve their effectiveness at the point of commissioning:

1.     Identify delivery success factors (activities or approaches which have been proven to create impact elsewhere) for example:

a.     ex-offender reintegration programs including five community buddies;

b.     or diversion programs to be family-based rather than only engaging with the individual[3].

The evidence base for these factors should be available from a systematic review of similar programs at the point of design.  There is a strong evidence base for many crime prevention programs, but you may need to undertake the review yourself.

2.     Identify intended outcomes: commissioners should request outcomes (not outputs) and evaluate service success based on these.  For example: “a reduction in re-offending” rather than “100 hours of rehabilitation activities per ex-offender”.  The service design to achieve the outcomes can then be left open to the provider.  A Theory of Change showing how and why the service will achieve this outcome (and other outcomes) over several years should then be provided in the tender response.

Co-produce services

“If you have come here to help me you are wasting your time, but if you have come because your liberation is bound up with mine, then let us work together.”

– Lilla Watson (as part of an Aboriginal activist group, 1970s)

Co-production is an acknowledgement that service professionals (like the police) need the communities they serve as much as the communities need the services.  It is about citizens and professionals sharing power.

I love the quote above from Lilla Watson  because it reminds me that a good and free society is a social contract between all of us.  The idea of co-production puts this into practice by getting service users to work in partnership with those who have helpful skills (professionals) to design, deliver and evaluate better social services.

In the 1970s, the Chicago police approached Nobel prize winning economist, Elinor Ostrom, to help them understand why crime increased on the streets when the police spent more time in their patrol cars and less time walking the streets.

She coined the term “co-production” to explain why police who were constantly in contact with the community were more effective.  They were more available to and trusted by the community. They were also able to use the knowledge and skills of the community to better achieve safety, for example, by sharing and distributing information.

Co-production was more recently popularised in the UK by the New Economics Foundation[4].  This call for deep meaningful engagement matches wisdom in the Social Value space as involving stakeholders is the first of the Social Value principles.

If we want our safety and justice services to be fairer and more effective, then we should combine commissioning for outcomes with the principles of co-production:

·       See people as assets: service users are partners not burdens on a system

·       Build on people’s capabilities: focus on what people can offer not their problems

·       Develop mutuality: design services which are distributed and embedded in communities not centralised and detached

·       Blur the roles between producers and consumers: we need service users to contribute to their safety, it is not “delivered” to them

·       Grow networks: strong peer networks transfer knowledge and build trust.

The Don Dale Youth Detention Centre example

Shocking abuse of the young people in custody at Don Dale Youth Detention centre sparked a national Australian outcry against police brutality in 2016 and triggered a Royal Commission inquiry. Two years ago, I was able to meet some of the young people still in the centre as part of some work I was doing in Darwin.

When I spoke with the young people, the overwhelming message I heard was their desire for “respect”.  It is an important concept in local Larrakia culture, and in this context, it is about being seen and valued by professionals in the criminal justice system.  When respect is given, it is also reciprocated.

The young people often came from chaotic family backgrounds, but regular encounters with white police officers reinforced their feelings of being misunderstood and marginalised: they were “another white person telling them what to do”.

Instead, they talked of elders and community role models who had “good lives” and would “be nice” to them.  These were the individuals who were able to listen to their problems, the individuals who would encourage them to make “wise decisions” and set goals.

We also spoke to the police who are the entry point into custody.  They said they would like less involvement in the young peoples’ lives and acknowledged that they made more progress through school visits than interventions on the street.

A major turning point in the antipathy between law enforcement and the local Aboriginal population was a partnership with the Larrakia Nation Aboriginal Corporation.  The Corporation were funded to better protect local people and they set up an outreach patrol.  The patrol can be called instead of the police in instances of intoxication, domestic violence, anti-social behaviour and homelessness.

The patrol response team have no official power, but the community trust them.  Young people related how they had helped them find safer living environments such as relatives’ homes, or they had transported them to the YMCA and other youth centres at night instead of hanging out on the streets.

The new prison management were also exploring different services.  After an attempted break-out, they invited community and guards to engage in a joint mediation with the young people involved.  They had approached community services who could connect the guards with the young people through traditional methods like Yarning Circles and music.

Both the police and the prison were beginning to integrate co-production and focusing more on outcomes.  They stopped assuming they know the right way to deliver services and invited the community to help them build up responsible young people.  This helped both the police and guards have more positive interactions with the young people, which was founded on “respect” for the young people and their community.

Final thoughts: why “defund the police” is not a radical idea

It makes sense to defund bloated police departments that have lost the trust of their communities and instead fund upstream preventative social services.  It is a systemic approach that will go some way to addressing centuries of unjust privilege.

The Black Lives Matter campaign is teaching us the importance of putting the people who are affected by social harms at the centre of  solutions.  I mentioned my whiteness at the beginning of this post because I cannot claim to know what it feels like to walk in the shoes of someone who is marginalised as a result of their skin colour or ethnicity.  That does not mean professional like me should step back, instead, we should lean in as facilitators and hold space for those most affected to co-produce more effective social outcomes with us.

[1]Newburn, T., 2016, Social disadvantage, crime and punishment, LSE Research Online, available at: http://eprints.lse.ac.uk/68133/1/Newburn_Social%20Disadvantage%20and%20Crime.pdf

[2]Australian Bureau of Statistics, 2020, 4512.0 - Corrective Services, Australia, March Quarter 2020

[3] Farrell, J., Betsinger, A. and Hammond, P., 2018, Best Practice in Youth Diversion, The University of Maryland School of Social Services, available at: https://theinstitute.umaryland.edu/media/ssw/institute/md-center-documents/Youth-Diversion-Literature-Review.pdf

[4]New Economics Foundation, 2008, Co-Production: A manifesto for growing the core economy, available at: https://neweconomics.org/uploads/files/5abec531b2a775dc8d_qjm6bqzpt.pdf

The posts should give readers the confidence to select and defend a set of indicators that measure their outcomes.  The first section walks you through the decision-making process of selecting an indicator; and this second half can be used as a checklist for ensuring you have a good indicator.  In summary:

1.       How to select an outcome indicator - last post

2.       Checklist for good outcome indicators - this post

Indicators should be selected to quantify the most important outcomes you identified when developing a Theory of Change.  Go back to that step if you haven’t first mapped your intended outcomes.

Checklist for good outcome indicators

The following checklist will help you test and stretch your indicators to determine which should be prioritised for each outcome.  It will also help you to avoid common pitfalls when selecting indicators.

Take your draft indicators and ask yourself if they reflect the following characteristics:

Each of these characteristics are discussed below.

Good indicators are valid

Indicators are a way of knowing whether (or not) an outcome has taken place.

Good indicators originate from asking how we can most accurately know an outcome is being achieved.  They are relevant to, and fully capture your outcome - also known as construct validity.  Bad indicators come from looking around and asking what can most easily be counted.

Think about those reports which describe the activities delivered, how much money was spent, and how many hours were volunteered.  This kind of improvised impact reporting is still a common pitfall in outcomes measurement.  It can easily be spotted because it asks us to stretch our imagination to link things like “volunteer hours” with outcomes like “increase in skills”.

Good indicators are stakeholder verified

Indicators should be in clear, concise language that is understandable and meaningful to your respondents.  A simple way of checking indicator clarity is to ask your stakeholders if they think the indicator is a reliable sign of the outcome.

For example, I worked with a youth service to understand healthy friendships.  We asked the young people if they thought the indicator “are there people in your life whom you could ask for support in a time of crisis?” was a good way of measuring friendships.

They told us that this was confusing, because they would probably go to a carer first (like a parent) in a time of crisis before a friend.  Instead, they suggested it was about “mates who are always interested in what I have to say”.  Using the language (like “mates”) which is common to your stakeholder group will also unlock better responses.

Good indicators balance subjective and objective metrics

A rule of thumb is to have three or more indicators per outcome.  Three indicators will give you a chance to triangulate, validate and spot any outliers in the data you gather.  Within those three indicators it is helpful (although not always necessary) to have a balance of subjective and objective indicators.

Yes, indicators can be subjective data; and they often are subjective in social impact research.  Subjective data represents thoughts, feelings and opinions.  It might be an indicator for social wellbeing like “how would you score the statement: I’ve been feeling close to other people”.

Many people are nervous about subjective data because they think it is less defensible than objective data.  However, most social impact outcomes are psycho-social, so the most reliable way of knowing whether they have taken place is to ask people about what is happening for them on the inside.

Filling social impact reports with objective indicators is a common pitfall.  Returning to our example above, an equivalent objective indicator for social wellbeing might be: “how many hours have you spent with other people this week”.  However, this data will be riddled with unknowns, for example a dysfunctional or abusive relationship could look very positive in the light of this indicator.

Good indicators are influenced by your intervention

I often see organisations shooting too wide with their indicators.  They may include one single indicator for example for mental wellbeing, like a rating on life satisfaction. The trouble with a big indicator like this, is that it is influenced by many variables. And let’s be honest, it is probably only influenced a small amount by your intervention.

It is better to focus on the component area of wellbeing that your intervention influences.  For example, if you are running a befriending service, you might ask about feelings of inclusion or reduction in loneliness.

Attribution indicators (impact blog to follow) will go some way to correcting any overclaiming on compound or higher order indicators.  However, the most common result of picking high-level indicators is that you measure no change regardless of your good work.

Good indicators are measurements of one outcome only

Your indicator should ask about change in one thing only.  This sounds obvious, but it is easily to slip into composing questions which sweeps up multiple indicators.  This is called “double-barreled” questioning.

For example, you might want to find out about the quality of someone’s relationships and ask, “Have you been feeling secure and understood in your primary relationships?”.   This can be hard for someone to answer.  What if they feel secure, knowing that their significant others are there for them, but not fully understood?  Or what if they feel understood by someone but can’t depend on them?

Watch out for that sneaky word “and” in your indicators and see if you need to separate topics into two indicators, or to prioritise the more relevant area.

Good indicators are low in bias

Bias is everywhere and while you will not be able to eliminate it entirely from your indicators, you can select indicators which keep it to a minimum.  Addressing all the characteristics in this checklist will reduce your bias substantially, but there are a couple of bias types which frequently pop up when designing social indicators:

·       Social desirability: this is where the indicator attempts to gauge people’s opinions, feelings or behaviour in relation to something that is either desirable or stigmatised.

Behaviour change measures often attract this bias.  For example, if you asked someone to rate their adherence to coronavirus lockdown rules, they are more likely to over-estimate their compliance because they have both a moral and legal obligation.

This bias can be minimised by rewording the indicator, breaking down the (socially desirable) concept into component parts, or by prefacing it with a normalising statement that appeals to their honesty.

·       Recall bias: this is where the indicator may overly rely on the completeness of someone’s memory.  For example, you ask them to rate how they were feeling a month ago, or even what they ate for dinner!  This can be reduced by putting a near-term parameter around the indicator like “Yesterday, overall how happy did you feel…” or “Over the last two weeks, how would you rate…”.  Sometimes we also try to anchor the indicator to a memorable event in the period we are interested in, for example, “Think back to last Christmas, to what extend did you….”

·       Comprehension bias: this can creep in when the respondents to a survey are asked to make judgements on things that they do not fully comprehend.  For example, asking children to rate their emotional intelligence.

Good indicators are not suggestions for activities

An important pitfall to avoid when creating indicators is prescribing delivery activities and/or creating perverse incentives.

I worked on an evaluation looking at the effectiveness of a community organising programme.  The underpinning philosophy was to take time to listen to local residents’ concerns and visions.  Effective engagement was largely judged by the number of doors that the organisers managed to knock on.

This created two problems.  Firstly, the indicators prescribed an activity, meaning the organisers who listened to people in places other than on peoples’ doorsteps, looked less effective.

Secondly, good quality engagement and listening was dependent upon having a meaningful conversation.  However, the organisers could count short, dismissive interactions toward their total number of door knockings.  This created a perverse incentive to keep the listenings short and shallow.

Good indicators are actionable

Actionable indicators give you information that you can respond to in the design of your intervention.  You should have options for responding immediately if you have met the checklist criteria above on “Good indicators are influenced by your intervention”.

Actionable indicators are also practical in terms of data collection and analysis.  Biting off more than you can chew is a common pitfall in impact analysis.  If your team rolls their eyes at the thought of additional data collection, then you might be measuring too much.

There is no point in over-cooking an impact evaluation in the hope of achieving more rigour, when you end up compromising the response rate.  As a rule of thumb, surveys should take no longer than 15 minutes to complete and should be appropriate to the abilities of your respondents.  Wherever possible bake data collection into the service delivery, and do not be afraid to remove indicators that you are not using.

In terms of analysis, you need to have enough credible indicator data to action an analysis.  Another pitfall I often see is the selection of indicators for which there is insufficient (or unreliable) historical data to analyse a change. This is most common for objective indicators, for example, “number of professional partnerships” during a period, when many of these partnerships may have been fluid without proper documentation.

Final thoughts: how to select an outcome indicator

We all know what bad data looks like when we see it.  We can smell something fishy even if we cannot articulate why we find it unconvincing.

To design strong indicators that you feel confident to defend, follow Part 1 of this guide.  It will walk you through how to select an indicator and compose a survey question where appropriate.

This checklist will further help you in testing and refining your choice of outcome indicators.  The most fundamental element to get right is construct validity: picking an indicator which really represents the change you are trying to measure.  Take your time with stakeholders on this step before then ensuring they are balanced, sensitive to your intervention and actionable.  Good luck!

The posts should give readers the confidence to select and defend a set of indicators that measure their outcomes.  This first section will walk you through the decision-making process of selecting an indicator; then the second half can be used as a checklist for ensuring it is a strong and reliable.  In summary:

1.       How to select an outcome indicator – this blog

2.       Checklist for good outcome indicators – next blog

Indicators should be selected to quantify the most important outcomes you identified when developing a Theory of Change.  Go back to that step if you haven’t first mapped your intended outcomes.

After measuring your outcome levels an advance step will be to adjust your outcome levels for impact, or the portion of change you created (sometimes referred to as additionality).  I will give guidance on impact considerations in a separate post.

In this post I’m focussing on social capital as an example topic, because in a time of isolation, our social connections matter more than ever.  If you are looking for examples of indicators (as I know many people are), watch this site for a future blog with some of the indicator banks I often use for a variety of social impact topics.

How to select an outcome indicator

Indicators are a way of knowing whether (or not) an outcome has taken place.

Typically, indicators are metrics that we track change against, to understand the size or magnitude of an outcome.  There are two parts to an indicator we need to get right: the identification of the outcome state (the question); and the measurement of its level (the scale).  Guidance on each of these are provided below.

Where will your indicator data come from?

Questionnaires are the best means to gather social indicator data.  Social impact practice is founded on the belief that you should engage directly with stakeholders and that they can be trusted to self-report on many social and personal outcomes they experience.

I realise this can be a stumbling block if you have limited resource.  Many of my clients groan at the idea of having to contact current and past service users.  In that case, it might be better not to quantify your outcomes.  One solution could be to embed simple, easy questionnaires into the delivery of your service rather than doing a stand-alone large evaluative data gathering exercise.

Additionally, where appropriate, you can use indirect indicators to infer your outcome.  This ladies and gentlemen are where outputs can cautiously be put to use.  You can scan your operational data for information you may already be collecting and that could signal the outcome.

For example, new calls to a helpline may indicate recognition of a problem, attendance rates at social coffee mornings might indicate rates of social interaction, or perhaps usage data from an online learning platform may signal awareness.  In each case, you should ask how relevant and accurate this output data is to the outcome you are seeking to measure.

Secondary data may also be helpful.  To test the effectiveness of a service tackling neighbourhood loneliness you might look at usage patterns of nearby mental health services or rates of single occupancy in homes depending on the nature of your intervention.

It is worth noting before I go on, that economic and environmental outcomes will primarily rely on objective data, which in turn, will be outputs or information gathered from secondary sources.  For example, an energy reduction programme will use meter readings to calculate electricity and carbon savings.  A programme looking to stimulate income and jobs will use salary and employment rates as indicators.  There are more and less reliable ways of gathering such objective data, however the focus of this article (and the next sections) is on social indicators so will not be explored here.

Indicator questions

You have three options when selecting social outcome indicator questions:

There is a race towards standardisation currently taking place in the impact measurement field.  This is accompanied by a lack of confidence to design indicators (the bespoke option) and a disproportionate faith in the quality of pre-existing indicators.

The social impact practice is still evolving.   Until unique outcomes have been mapped and defined for all possible social experiences (imagine the ultimate social outcomes framework to rule all frameworks) in my opinion, will still need some bespoke indicators.

Let us explore an example of these three approaches focussing on the quality of peoples’ personal relationships.   Imagine we are running a mothers’ group for women experiencing post-natal depression.  We are looking for an indicator that will tell us if attending the group has improved women’s feelings of being supported and understood.

Here are six options for standardised indicators, from the ABS General Social Survey, Australian Unity Personal Wellbeing Index (PWI) and the European Social Survey:

Looking at the six indicator questions, which do you think best fits the outcome of feeling supported for women experiencing post-natal depression?  Perhaps the loneliness indicator from the European Social Survey and the having friends and family to confide in from the ABS General Social Survey would touch on the specific experiences of isolation that new mothers can have.

Overall, all six are good options, but most of them suffer from being too general to capture the specific change relating to our intervention.  For example, a woman may have many people whom she feels she can meet or confide in, but not about her motherhood anxieties.

We could take our two favoured indicators and adapt them to better represent the experiences our intervention.  For example:

Finally, we could select our own bespoke indicator by listening to the experiences described by women attending the group.  Perhaps they mentioned the weight of their worries being lifted by meeting other people who could fully empathise with their experiences leading us to create an indicator like:

Indicator scales

“There are three kinds of lies: lies, damned lies, and statistics” – Sir Charles Dilke

In social impact research we are measuring outcomes.  Outcomes mean a change.  To determine whether a change has taken place we need to know what the baseline level of the outcome was, and where it ended up.

It is still common to see binary indicators in social impact studies.  Binary indictors are where you ask whether an outcome has taken place: yes or no?  The problem with binary indicators is that you cannot quantify the level or amount of outcome that has taken place.  It was either 0% change or 100%: they are a blunt instrument.

This can result in over-claiming.  For example, if you ran an afternoon event and then asked if it had improved attendees’ relationships, they are likely to respond yes.  If you placed a homeless young person in a special care home for a year and gave them the same survey, you would probably also measure the same improvement in relationships (100%).

Binary indicators do not tell you how much or for how long the outcome lasted.  This matters if you want to improve what you do, and it will really matter if you plan on attaching a monetary value to an outcome (later in your analysis).  It is best practice therefore to use a scale so that you measure the amount of movement (hopefully positive) that an individual traveled along a scale.

As you can see in the indicator examples in Figure 3 above, there are various factors to play with when designing your scale:

·       Short vs. long. Short scales, for example with three points, are less sensitive to small changes and/or can lead to overclaiming.  Long scales, for example with 11 points (as shown in the Australian Unity PWI) can be difficult for some people to hold in their heads and are not recommended for telephone surveys and people with learning difficulties.

·       Even vs. odd numbers on a scale.  Even numbers force respondents to express an opinion one way or the other, odd numbers allow respondents to sit in the middle.

·       Defined scale vs. undefined.  In a defined scale, each point along the scale is specified (for example in the ABS indicator) and this can be extended into sentences if you want to control how people interpret each point.  For example, returning to our European Social Survey question on people who care about me: “agree” could instead say “there are some people who care about me, but I cannot talk to them about things that really matter”.  

Once you have a scale in place there are two common ways of analysing the indicators:

1-     Average percentage change: this approach converts movements along scales into percentage changes.  It calibrates the maximal possible outcome state as 100% and the minimum point as 0%.  The calculation approach is illustrated below in Figure 6.  

The benefits of this approach include that you can monitor subtle improvements (or declines), you can harmonise different scales and you can accurately convert outcome levels in valuations (later in your analysis).  The disadvantage of percentages is they can be difficult to communicate in isolation, as they involve a little more explaining.

2-      The number of people experiencing the outcome.  Alternatively, you can set a threshold level of movement along a scale to count the outcome as having taken place. For example, an individual must positively jump two points on a scale or move from a disagree to an agreement point.  You can then analyse the percentage of individuals who achieved the outcome.

Some general advise with scales:

·       use at least four or five points for written questionnaires

·       aim for consistency across indicators: for example, if you can use a five point strongly agree to strongly disagree likert scale for most of your indicators then do that

·       keep it simple: only design new defined points on the scale if you really need to.  There are plenty of tested likert response scales you can use, here is a useful selection.

Final thoughts: how to select an outcome indicator

Indicators should be chosen or designed once you have a Theory of Change or logic model.  Questionnaires are the most common ways of gathering information on social outcomes.   You can either: (i) use indicators that are standardised in these questionnaires, or (ii) you can adapt existing indicators to better suit your context, or (iii) design bespoke indicators from scratch.

Take time to craft the right indicator question and multi-point scale to capture the change.   Test your indicators with some stakeholders, especially if you have adapted or prepared some bespoke indicators.

I find coming up with the right indicators hard work and you probably will too.  The sophistication in this exercise comes from producing something so simple that it looks easy.

It is no longer just an opportunity for educators, but an imperative.  So here are five levels of impact to build into the evaluation of any learning and development program, and ideas on how to put them into practice for an edtech course.

The breakdown of learning evaluation into levels comes from well-known theorists in the education space.  A general trend across their models is that they move from the micro (immediate, individual changes) towards outcomes at the macro level (longer-term and for more stakeholders).  In this sense, they are similar to a Theory of Change diagram which moves from short-term to long-term outcomes.

Some of the key thinkers guiding practice include Donald Kirkpatrick (1959), Kaufman, Keller and Watkins (1995) and Robert Brinkerhoff (2006).  Below, I summarise their thinking into five levels (building on Kirkpatrick).

If you cover all these areas, you should be able to tell whether your new online training is expensive digital clutter, making your learners cry “not another platform to download?!”, or a game-changing edtech solution that shakes up whole sectors.

Remember, your evaluation does not have to cover all five levels.  Pitch your evaluation level based on the audience for the report and the size of the course.  That said, Levels 1 and 2 should be achievable for almost all trainers.

Level 1: Reactions

This level is about whether the content met the expectations of students in terms of design, delivery and experience of the course.

A national training provider once approached us with data from a classic feedback form.  It included questions like “how would you rate …the trainer’s presentation? …the length of the course? ...the exercises?” culminating with “I am happy I attended this course – Yes/ No?”.

Our contact inherited the program, and when a funder asked for an impact report, she believed her feedback data would hold all the answers.  She is not alone in this belief.  Most people are familiar with seeing a form like this at the end of training (if anything at all) and unthinkingly assume that learning effectiveness is being captured.

That is not to say these forms of useless.  Feedback forms like my clients’ do work for a very simple evaluation.  They provide Level 1 information by assessing both sentiment (whether your learners enjoyed the experience) and feedback on the delivery method (whether the content is engaging).  This will be important as you roll out a new digital format as the experience may be new to both the provider and the learner.

To gather data at Level 1 a simple end of course feedback form should suffice.  Plug it into the last page of your course before users are invited to close a platform, or send it immediately by email to maximise return rate.

If you are working in a smaller setting, perhaps with a small class on, why not gather quotes from oral feedback.  You could simply by asking for direct reactions at the end of class (of course this could be biased if it is led by the trainer).

If you are running a larger course, consider setting up a reference group or cohort.  This could be representative sample of your learners, or a group of experienced trainers that can provide qualitative feedback through independent sessions or interviews.

Level 2 – Learning

Level 2 is about the achievement of learning outcomes, such as an increase in skills and knowledge in a topic.

It is tempting to believe that just because you delivered a killer course, with an engaging, interactive online platform, that you must have increased knowledge.

At my workplace we have an annual web-based training in professional independence.  It is an essential course for ensuring that professionals in the firm do not (intentionally or unintentionally) take advantage of sensitive client information.  And hats off to the company, they update the design of the course each year with videos, puzzles, case-studies and bite-size delivery of information.

The thing is, many people have worked in the firm so long, that they click through the content as fast as they can.  You need to mark the whole course as complete and then pass a test.  The old hands can pass the test without engaging in the course because its their business to know this stuff inside-out.

So, has there been any learning?    For a segment of learners this is unlikely because they were able to pass the test just as well before embarking on the training, as after.  This could be the same for your course, or perhaps someone had technical challenges or learning difficulties.  You will never know until you check.

To gather data on Level 2, you need to assess the level of knowledge, skill or awareness on a topic prior to the student undertaking the course (a baseline).  You then need to assess their level again at the end of the course.

Ideally this would be a test taken at the start and then compared to results on a very similar test at the end of the course to compare scores.  This is a water tight approach (assuming your test accurately assesses the learning outcomes).  However, this can be hard to implement practically as time is often precious on courses, and continual testing can cause fatigue.

The other option is to include some self-report.  This could include self-report on the baseline level of knowledge.  For example, in some circumstances you can simply ask a learner if they are a complete novice.  It gets a little tricker if you want to define an intermediate level, for example one person’s idea of conversational Spanish might be very different to another’s.

Finally, you can ask people to self-report their pre and post knowledge on the topics you covered retrospectively using a simple scale.  This would look a little like the following:

Level 3 – Behaviour

This level means testing whether your content led to any application of the learning.

Application of the learning means using the skills and knowledge gained from the course to do something different.  It might be changing a workplace habit after attending a time-management course, or building a new software plug-in if someone attended a coding course.

Behaviour change is notoriously difficult to measure in social impact assessments.  This is because it is often delayed, and traditionally social impact research has relied centrally on self-report measures which can easily be biased.

To gather data on Level 3, it is desirable to introduce a follow-up survey or measure.  This means monitoring the behavioural (and learning) impact till or at an appropriate period after the course.  This could be 2 weeks if your course had learnings which were simple to apply, or for a larger course, maybe six months.

Ideally you will be able to combine objective measures (observable changes) with subjective reports (opinion-based data).  For example, if your course covered pro-environmental habits or lifestyle changes, you might ask people only for objective data by monitoring their energy or water bills, or weighing their landfill rubbish.

If the anticipated behaviour changes are complex and variable, you may prefer to compose some simple self-report scales, or even ask people to share case-studies through open questions.  Remember, qualitative data can be more reliable than quantitative data if the numbers are meaningless.

Level 4 – Business Return on Investment

This level examines the business benefits to the course provider.  Benefits may be direct income or the achievement of strategic objectives.  The returns can be considered based on whether your course intended for internal audiences or for external attendees.

If your online course is open to external attenders, strategic objectives may include building the organisation’s reputation, or to strengthen partnerships.

The success criteria for internal courses may include outcomes identified at Levels 2 and 3 because the expectation is that you are building organizational capability.  However, they may also include medium-term outcomes like employee engagement and/or efficiency.

To gather data on Level 4 you will probably be looking at a variety of indirect sources.  For external courses, marketing data can be used by cross comparing attendees’ names with subsequent purchases, referrals or sign-ups.  Phone interviews and the follow-up survey could also be used to gather changed sentiment on the organisation.

For internal courses, you may look at year-on-year employee engagement surveys, or specific productivity measures based on the organization and function.  Remember you need baseline and follow-up data and some consideration of causality (more to follow on this in a later post).

Level 5 – Wider contribution

The final level examines the impact the course has created on the capacity of the sector you are serving, and wider society.

This level is similar to the long-term outcomes in a Theory of Change. It expresses the broader contribution your training is making.  For example, an online course in playing the guitar may lead to greater appreciation and health of the music sector.  An internal training in data protection, might contribute towards greater safety of personal data and therefore more trust in institutions.

Gathering data on your specific contribution to these high-level outcomes may be challenging.  Larger, sector level studies looking at the role of online courses could help.  For example, we recently worked with a national sustainability initiative.  We were able to explore the collective contribution of both online and offline training run by over 500 organisations.  The study explored whether understanding, values and behaviours had shifted from a variety of online and offline courses and the success factors which made individual courses most effective.

In the absence of useful larger studies, you may like to support the evidence base by hosting a round table discussion on the role of online courses, or by interviewing experts.  You could also add an open-ended question into your data gathering tools for Levels 1 and 2.

Final thought: the fundamentals to measure online training effectiveness

The glowing novelty of technology and the dizzying array of platforms to try, should not be a distraction from the ultimate test of course effectiveness: the achievement of outcomes.

Edtech is giving traditional classroom settings a good run for their money.  However, initial reactions like engagement and entertainment are still given a primal importance by most course providers.  So in proving the value of your training make sure you are reaching beyond Level 1, at least to Level 2, to demonstrate the impact of your education offer.

The key to progressing a Theory of Change is to get a minimal viable product down on the page.  You should then return to the details and finesse with your stakeholders through interviews and workshops.  You do not need to have all the answers first time around.  We work in complex, changing social systems, so iteration will be your friend.

The framework

At NEF Consulting, a talented colleague of mine, Jenny Rouse, drew up a simple template for building your Theory of Change.  It helps organisations capture all the component parts of a Theory of Change in one place.  Start with this template.  Ultimately you will want to find your own way of presenting the diagram, so that it has your organisation’s personality.

Steps

1.     Set scope

A Theory of Change is contained within what I call the bookends of the diagram: the aim and the need.

The aim is the picture-perfect description of what you would like to achieve.  It is a vision realised for your stakeholders.  I often like to call it the guiding star for your work.  In that sense it doesn’t have to be something achievable, but something that shows your direction of travel.  So big esoteric statements like  “happy, healthy people in a resilient neighbourhood” or “a thriving science and technology sector” can work.  Try to make it relevant to your stakeholders and avoid recycling mission statement as this often includes activities or the way an organisation works.

The need on the other hand, defines the problem you are addressing, or the cause for your work.  You can choose the level of complexity you provide here.  You could focus on something like “supported living residents are experiencing loneliness” or you could chart key systemic causes for social isolation in society.   A clear and insightful understanding of the drivers and context for your work will enable better intervention design.  As Einstein once said:

Given one hour to save the planet, I would spend 59 minutes understanding the problem and one minute resolving it.

Finally, you will notice the diagram is split into phases: short, medium and long-term.  This linear layout works really well for project and program level Theories of Change.  You might find organisational and systemic ones look more cyclic.  What is critical is that you layout a flow, or causal chain of outcomes leading from your need towards your aim.

So as part of step one, set out:

A.         Your overall aim* - identify a goal statement: the end point of the Theory of Change.

B.         The need for your work – what problem is the programme aiming to address?

C.   Periods – what will short, medium and long-term refer to in the journey towards your aim?  Could it be pre-intervention, during intervention and post-intervention?

*I am often asked whether you should start with the aim or the need when plotting a Theory of Change.  The answer is either!  Most social change activists have a better understanding of what they are trying to achieve (what they are for rather than against).  However, it is possible to use Theory of Change as an innovation and planning tool.  In that case, you might start with a really good understanding of the problem you are addressing, and then design an intervention (activities) and associated outcomes by plotting forwards on the diagram.

2.     Identify material (most important) stakeholders

For your primary stakeholder(s) you are going to develop a journey of change across the short, medium and long-term periods.

D.  Select a primary stakeholder for your intervention and maybe critical secondary stakeholders.

Identifying your stakeholders sounds simple, but the broader the initiative (think organisation scale) the more difficult this exercise will be.  In fact, I often find this to be the most challenging step in developing a Theory of Change or impact management system.

Stakeholders are material, if you create significant change for them through your work.  Your stakeholders are not supporting actors or partners.  The change should also be relevant to your overall aim.  As a useful rule of thumb, I suggest picking somewhere between one and five stakeholders to focus on.  The more you add, the more unwieldly your Theory of Change will become.

3.     Map outcomes

The next step is to populate a causal flow of outcomes from the need to the aim.  I suggest doing this initially on post-its (sticky notes) with one outcome per post-it.  You will inevitably want to move the outcome sequence around as you begin to chart a timeline.

Critically, make sure you are describing outcomes not outputs.  So, we are looking for “increase in skills” rather than “achieve a certificate”.  The certificate is an output of the learning, but your stakeholder may have already had sufficient skills to obtain a certificate regardless of your activity.  So it does not necessarily mean the outcome has taken place.

There is no single, hard and fast way of mapping the outcome chain, but why not:

E. Plot outcomes  by working backwards from the goal to the intermediate outcomes, and finally to the short-term outcomes.  You should be asking “what change has to happen in order for this to be achieved?”. This ensures that the focus is on what has to be done to achieve the goal and intended outcomes, rather than on what the current activities are.

If you have multiple stakeholders, I recommend initially plotting causal chains in rows for each stakeholder separately and with a different colour post-it.  You can later intermix the connections to better tell the story.

Once you have all the outcomes mapped, ask yourself whether you have also been honest about any negative outcomes.  Mapping, and potentially measuring, negative side effects will help to minimise their effects.

4.     Add activities and explore assumptions

Activities are what you do as part of your intervention to create the outcomes.  They should be included in your diagram to show how you are driving the outcomes chain.

How you present your outcomes will depend on the nature of your intervention and your personal preference.  Sometimes activities directly drive outcomes, for example, providing a study grant or scholarship may increase the accessibility of education.  In this case, you may wish to present the activities as a stepping stone within the outcomes chain (shown in box A in the diagram below).

Other times, your activities may have a more complex indirect relationship with outcomes.  An example might be campaigning leading to changes in public opinion. In these cases, you may prefer to group them – perhaps under the respective short- or medium-term outcomes that they are driving.  Either way:

F. Add activities – one per post-it, ideally on a new colour.  Then arrange them as closely to the activities they drive as possible.  Add arrows where there is a clear direct relationship between activities and outcomes and also between sequential outcomes.

Once you have the activities noted, it is critical to explore the logical links between activities and the outcome chains you have created.  Document how and why you deliver your activities in the way you do.  So use these as prompt to:

G. Document assumptions -

# How are you going to deliver the activities?  What characteristics in the way you work are important?

# Why are these activities important and causally related to the outcomes?

# Why do the outcomes captured sequentially build upon each other?

5.     Advanced step (i): adding context

No intervention exists in a vacuum.  There will always be factors which either hinder or help you move stakeholders across your Theory of Change and towards your ultimate aim.  Adding a list of enablers and preventers will build another layer of sophistication into your thinking.  It will highlight preventers which need to be managed, for example poor transport links or unexpected pandemics, and enablers that need to be maximised, for example other actors working for your cause.

Enablers and preventers can be at any scale from individuals, like family members, to systemic considerations, like a depressed job market. Add what you feel is most relevant to your context.  It is worth noting that single factors are often both enablers and preventers.

H. Add enablers - what are the conditions or actors, that will help the positive changes you have identified be achieved?

I. Add preventers – what are the barriers, which might prevent your intended outcomes from being realised?

6.     Advanced step (ii): material outcomes and line of accountability

If you plan to use your Theory of Change as a tool for evaluating your impact, or collecting impact data, you will need to decide which outcomes your success should be judged against.

There are two final features of the Theory of Change diagram that will help you set the boundaries for your responsibility.  Firstly, there is the line of accountability; the point up to which it is reasonable, and plausible for you to have had significant control over the outcomes being realised.

This line often lies between the medium and long-term outcomes on diagrams.  However, it can sometimes be wiggly, or come earlier or later depending on how much is appropriate. Hearing about a line of accountability is normally a big relief for those putting a Theory of Change together.  The whole Theory of Change diagram paints the big picture.  The line of accountability shows your role within that story.

J. Draw a line of accountability around the outcomes which you believe are reasonably within your sphere of influence.

Once you have a line of accountability you will want to select the material outcomes on your diagram for measurement.  Material outcomes are those which are:

·       Relevant to the aims of your project

·       Significant in size due to your actions

You will be selecting outcomes prior to your line of accountability if it is linear.  If you have a chain of causally related outcomes, it makes sense to pick the final outcome in a chain to minimise double counting.  For example, a workplace campaign might have an outcome chain including: greater pride in the organisation > greater interest in work > greater employee engagement.   In this case, you may select only “greater employee engagement” as material.

K. Mark material outcomes – select the most relevant and significant outcomes before the line of accountability for taking forward for measurement.

7.     Finally review and iterate

By this point you will have a first draft.  Does this capture a full picture of the change you are supporting?  Are the outcomes articulated in the right way? Are the assumptions underpinning the model plausible?

Once you are happy with your first draft it is time to take it out to your stakeholders.  Interview or workshop with your beneficiaries, partners and anyone else who is affected by your work to test whether you have put together a plausible and accurate model.  Listen to the language your stakeholders are using, and try to reflect this in your model.

Once you are happy with the detail, find a diagram shape or infographic layout that elegantly captures your Theory of Change.  You do not need to include everything in one diagram.  We are aiming for accessibility: a diagram that needs the least possible explanation.  That said, do combine your diagram with a narrative(s).  The nature of this narrative will depend on whether it is for a report or pithy copy for a website "about us" page.

Most of all, have fun!  Welcome the complexity and meaningful conversations.  I love Theory of Change because it sharpens organisation's thinking and helps them to think systemically.  

“Never let a good crisis go to waste” – Winston Churchill

The silver linings from the COVID19 recession are delicately emerging around the thundering clouds of this crisis.

Social innovation is blossoming

All over the world, community mutual aid groups are forming in the absence of institutional actors.  Supply chains are adapting and familiar brands like Burberry, Dyson and Teslar are reacting swiftly to produce much needed medical equipment.

Some countries like Ethiopia are opening the financial sector to enable more mobile money.  Arts, culture, education, sports and leisure services are fast adapting to provide online services.

Our charities, social enterprises and social services are also feeling the imperative for innovation.  Reduction in income streams from funders delaying or withdrawing, from household cutbacks and from an immediate halt from subsidiary income streams like room rental or paid services mean many will move into survival mode.

This comes at a time when demand for their services will be greater than ever.  The challenge facing the social sector is to create more positive impact with less investment.  It is a time to embrace flexibility and to allow for new learning.

Institutional donors are being asked to permit changes in contract deliverables, timelines and reporting.  Emergency funding should hinge on trust and regular communication.

Impact management and impact measurement are part of innovating

It is tempting to assume that impact reporting can also be moved to the backburner.  However, impact management is just (if not more) important during periods of innovation than established delivery.

Quick and timely feedback will be critical to ensure emerging social needs are being met.  Similarly, funders will be making difficult decisions during this time about effectiveness.  They will be looking across their portfolios and asking which services provide essential impact.

The good news is that impact reporting can be nimble and combined with innovation to make sure we do not end up adrift during this period.  Two methodologies are worth borrowing from during this time: Developmental Evaluation  and Most Significant Change (MSC).   I have pulled five key tips from the two approaches to bake into any COVID19 adaptations to social services:

1.       Gather timely data

During periods of innovation you want to be collecting less impact data, more frequently.  This means your reporting period could be a week – it could even be a day.  You should collect short-term feedback and quickly analyse and consolidate the learnings.

The diagram below illustrates how the action learning cycle, which might take a year or more for an established programme, should be sped up for rapid feedback during the innovation phase.  As your project matures with more consistent features, the length of time for a typical review period and the sophistication of evaluation methods should both increase.

Remember to keep good records in a database and note when changes in program design took place.

2.       Short qualitative data points

The data you will be collecting needs to be easy to manage.  I would advise focussing on a small number of open-ended questions. These should gather feedback on change (outcomes) and why or how those changes are happening.  Try to sketch up a rapid Theory of Change to guide your questions and return to this as you gather data.

To give an example of what the questions look like, for an education service, the question could be as simple as “How much did you know about X topic before the intervention, and how has that changed now?”.

Prepare outcome questions on a few focal areas, then cycle between these with different stakeholders.  This way you reduce the burden of questions on each stakeholder or interaction.  It is worth noting that lighter data collection does not mean you can go light on rigour.  Consider bias in terms of your sample and question structure.

MSC invites the collection of stories to understand what is changing and why it matters.  This matters to avoid feedback becoming soundbites.  You can create mini case-studies to preserve context.  You might cover the background of the interviewee, what happened for them and the difference it has made.

3.       Hit the telephones

Physical distancing during the coronavirus lockdowns means speaking to stakeholders face-to-face should be minimised.  This is not a reason to avoid collecting data.  Service users may even be more receptive than ever to forming a contact and more inclined to provide feedback.

Collect phone numbers or provide videoconference invites to connect remotely with users to conduct short interviews or surveys.  Remember to take time to establish a connection and build trust before diving into feedback as this can sometimes be a little harder over the phone.

4.       Focus on what is working

Evaluation calls for the consideration of positive and negative impact.  When you are innovating it is important to acknowledge what is not serving your stakeholders, but this is not the time to measure it.

Be guided by what is creating positive change for the most stakeholders.  Dig into why it made a difference and how you can amplify that positive outcome.

5.       Involve everyone

Innovation gurus tell us the best ideas come from everywhere.  The same goes for evaluation insights.

Both MSC and Developmental Evaluation emphasise the importance of involving your stakeholders in not just the data collection, but also the data analysis.  They both encourage you to chew over the findings with people from every level within your organisation.  If you have a close funder, talk it over with them too.  This is the way you will generate new solutions, scale what is working and build new collective solutions with partners.

Final thoughts: five tips for evaluating social innovation in response to coronavirus

The social sector will be under immense pressure over the next few months.  It will take a collective effort from all directions to safeguard the wellbeing of those most vulnerable.  Guesswork is not a smart approach to creating impact during normal times and it should not be relied upon now.  Simple, light impact measurement techniques can be built into even the earliest stages of social innovation.

The impacts of coronavirus have crashed into our lives at breakneck speed.  The tightening of State borders, the shutdown of non-essential services and the terrifying increase in diagnoses will have already left no one untouched.

Some of the most cherished icons of Australian culture have been early victims.  The sports news plays eerily with no fixtures and beach goers are scolded for their “selfish” refusal to take social distancing seriously.

The health and social fall out of coronavirus will not however be felt evenly across Australia.  We know already that older people and those with existing health conditions are more in danger from the virus itself.

The queues outside Centrelink reflects the next wave of vulnerability.  Some of these people will struggle to self-isolate because they cannot pay the rent.  Some will be worrying more about filling the kitchen shelves with the next family meal rather than hoarding pasta.  Some of their children won’t be homeschooled as the prospect of an internet connection slips even further from reality.

Australia is expecting the number of people on benefit payments to more than double in the next few weeks.  This hints at the size of the precarious workforce, but others will still be heading out each day as they have no choice but to continue stacking supermarket shelves, providing care services and driving rideshares.

We are being urged to “be kind” as we head into the most profound socio-economic crisis, possibly since the last world war.  This is accompanied by articles advising us to get our minds over the matter.  We should stay positive, show goodwill and look after our neighbours.

This is a nice sentiment, but with a structural breakdown on this scale there needs to be a much more profound investment in our social fabric.  Wellbeing is driven by both material and psycho-social factors.

If you live in one of Australia’s poorest communities, you can already expect to live 10 years less than those in richer neighbourhoods [1]. Living with deprivation means you are more likely to have a poor diet, breath worse air and do less exercise.  Asthma, heart disease and diabeties rates suddenly stop feeling like a problem for tomorrow when coronavirus is sweeping across the country.

Inequality is also bad for social cohesion.  Again, overcrowded homes and toxic household relationships are much more probable if you are at the bottom of the social ladder.  But there is also a cost for those at the top, as those living in big houses, behind tall gates are less likely to know and trust their neighbours.

Australia is becoming a less equal society [2] and this pandemic will undoubtedly bring that into focus in the cruelest way possible.

We need to start talking seriously about a wellbeing rescue package to accompany health and economic plans.  This should renew focus on the social determinants of health and the critical social, civic and care sectors which are working to make Australian society more inclusive.

There will be significant (and much needed) investments flowing into social causes in the coming months.  I would urge donors and investors to focus on models with proven, lasting social impact and rather than stories of kindness and positive thinking.

[1] Seccombe, M, 2019,  The Fatal Cost of Australia’s Rising Inequality, The Saturday Paper, available at https://www.thesaturdaypaper.com.au/news/economy/2019/11/02/the-fatal-cost-australias-rising-inequality/15726132009011, [Accessed 24 March 2020]

[2] The Productivity Commission, 2018, Is Australia becoming more unequal [online], Australian Government, available at https://www.pc.gov.au/news-media/pc-news/unequal, [Accessed 24 March 2020]

This article, written with colleagues 18 months ago, explores methods for putting a monetised value on our health.  Coronavirus is throwing up questions around risk and value in health, so this could be a timely moment for a deep dive into this topic.

Monetisation – the placing of monetary values on outcomes – is a fundamental element of the Social Return on Investment (SROI) methodology. However, surprisingly, it is an area where there are a diversity of approaches.

In some ways this diversity of approaches is a good thing; the difference in scope and purpose of SROIs means that sometimes different approaches are called for. However, these different approaches to monetisation can lead to otherwise similar SROIs showing very different SROI ratios. For some people, this lack of consistency in approach is one of the main drawbacks of SROI.

The original SROI guide says, on page 46, that:

“as SROI becomes more widespread, monetisation will improve and there will be scope for pooling good financial proxies”.

This article aims to contribute to this improvement by outlining how Quality Adjusted Life Years (QALYs) can be a useful tool in monetising outcomes.

QALYs are a measure of a person’s heath. They are defined by NICE (the National Institute for Health and Care Excellence) as:

“a measure of the state of health of a person or group in which the benefits, in terms of length of life, are adjusted to reflect the quality of life. One QALY is equal to 1 year of life in perfect health”.

The relative success of medical interventions can be measured in QALYs. For example, if a medical intervention leads to an individual ‘gaining’ a QALY, this could mean a number of things. It might mean that the individual is expected to live one year longer, and that one year of life will be lived in perfect health. More realistically, they may live two years longer, and those two years will be lived at 50% health. Or their life expectancy may not change at all, but for the final five years of their life they gain 20% health per year.

We believe that QALYs are useful because they allow a whole range of different outcomes to be compared on a like-for-like basis, using the same unit of health. For example, QALY measures have been established for areas such as:

·       Physical health: A 2013 study used a QALY approach to estimate the cost of ill health attributable to environmental noise, specifically looking at the costs of hypertension-related acute myocardial infarction, stroke and dementia due to environmental noise levels.[1]

·       Mental health: The Centre for Mental Health has examined the impact of mental health problems in terms of QALYs. They used data from the Health Survey for England 1996, Department of Health (1998), and the Measurement and Valuation of Health (MVH) survey[2]. They report that the average loss of health status is estimated at 0.098 of a QALY for each individual with a moderate mental health problem, and 0.352 of a QALY for those with a severe mental health problem.

·       Environmental impact: A toolkit called CAPTOR (Schmitt et al., 2015) produced for the West Yorkshire context in the UK, a low emission zone, gives users the ability to calculate the QALY gain by reducing Particulate Matter 2.5 (PM2.5) and Nitrous Oxide (NO2/NOx) concentrations. The QALY value per person per tonne of PM2.5 reduced is 0.000044 and reflects general mortality risk, pregnancy complications and lung diseases. At a population level the health value can quickly add up.

·       Fear of crime: A paper examining estimating the economic and social cost of fear of crime used QALYS to estimate the health loss from episodes of fearfulness of crime. The mean annual health loss per person was 0.00065 QALYs.

What is a QALY worth?

QALYs do not in themselves give a monetary value to outcomes. However, if outcomes such as the ones above are expressed in QALYs, they can then be converted into a monetary figure fairly easily, and on a consistent basis. All that is required is a monetary value per QALY. There are several approaches to placing a monetary value on a QALY, and we discuss some of these below.

Approach 1: NICE QALY threshold 1: £20,000 – £30,000

NICE use a cost-effectiveness threshold of £20,000 – £30,000 for one QALY. Crudely, a medical intervention is judged to be good value for money if it costs less than £20,000 – £30,000 for every QALY it creates for the patient. It means that, in theory, the NHS would be prepared to pay up to £30,000 for each QALY created or saved for a patient by a medical intervention.

Approach 2: NICE QALY threshold 2: £18,317

To provide an empirical basis for the NICE QALY value estimates, academics looked at health spending and its effects on health outcomes across England to consider the opportunity cost of health spending. The allocation of funding to one intervention/treatment will improve the health of some people but will create an opportunity cost – funding will be displaced from elsewhere. If money is displaced, for example, from spending on improving mental health to cancer screening, cancer survival rates are likely to increase but outcomes for those with clinical depression are likely to get worse. Analysts examined programme health budgets across England to consider the effects of marginal changes in expenditure related to over 20 different diagnoses and conditions, and the subsequent impact or mortality and QALYs. The study found that the cost per QALY threshold £18,317 in 2008 prices – or £23,234 in current (2018) prices.

Approach 3: Stated preference – ‘willingness to pay’ for a QALY: various

As an alternative empirical approach to QALY valuation, the European Commission undertook a 3-year research project with organisations from across 10 European countries. The aim of this project was to develop robust methods for determining the monetary value of a QALY across various European member states. Two primary approaches were developed to estimate the monetary value of a QALY.

The ‘chained approach’ involved administering a questionnaire in which respondents were asked to value avoidance of small, but certain, negative health states through answering questions which would ascertain their utility value for each health state. They were then asked how much they were willing to pay to avoid a certain duration/risk of that health state. Responses to these two components were combined to estimate willingness to pay (WTP) per QALY gained.

The ‘direct approach’ sought to elicit values of a QALY in a more direct way by administering questionnaires which used graphical and textual descriptions to present different scenarios representing one-QALY gains and asking related WTP questions to respondents. Due to the broad and complex nature of the research, the study does not arrive at a specific value or narrow range of values for a QALY. Instead a number of different QALY values are provided depending on the country and the approach used. These are outlined in the report: European Value of a Quality Adjusted Life Year.

Approach 4: A meta-analysis of QALY values in Australia: $151,000

In Australia the Government recommends using a value which combines a number of techniques. In 2008 professor Peter Abelson published a meta-review (OBPR, 2014) of both Australian and non-Australian studies on the Value of a Statistical Life (VSL). Pulling together the most relevant empirical research for the Australian context he provided a VSL of AUD3.5million based on the mean age and a discount rate of 3%. This equated to a QALY value of AUD151,000 in 2007 and when adjusted for CPI is roughly $190,500 in 2018 money. This high value represents cost of illness to an individual (income and medical costs), willingness to pay through insurance rates and contingent valuation. The relatively high wages in Australia combined with WTP considerations mean this high value can often be one of the most sensitive numbers in an Australian SROI analysis.

Recommendations
We believe that QALYs can be a useful tool for people who are undertaking SROI. They are particularly useful when evaluators are able to measure changes in health with some degree of rigour, and when it is valuable to communicate in language that makes sense to commissioners as well as wider audiences.

There are several mechanisms for placing a monetary value on a QALY. Like other approaches to monetisation, there are different views as to the most appropriate value to use, and analysts should consider the purpose of their research and its audience when choosing the best approach, as well as considering other options in the sensitivity analysis. Sensitivity analysis is the process by which data and assumptions are tested, to see which have the greatest effect on the model or analysis.

Our belief is that, overall, QALYs strengthen SROI analyses, and can contribute to the increase in consistency and rigour of monetisation.

References

[1] Harding, A-H, Frost, G. A., Tan, E. & Tsuchiya, A. (2013). The cost of hypertension-related ill-health attributable to environmental noise. Noise & Health. 67 (15): 437- 445. DOI: 10.4103/1463- 1741.121253.

[2] MVH Group (1995), The Measurement and Valuation of Health: Final Report on the Modelling of Valuation Tariffs, Centre for Health Economics, University of York

This article was originally prepared for nefconsulting.com and was published there in September 2018.

Authors
Alison Freeman, EY

Oliver Kempton, Envoy Partnership

Sarah Arnold, New Economics Foundation

Samrawit Mariam, formerly at NEF Consulting

When I was working in London I met a CEO of a small but old youth homelessness charity.  She had been in the role for a year and was brimming with intelligence, charisma and innovative new ideas for services.  She was all set to take a respected, minor charity to the next level and she wanted help to prove the impact of her plan.

The funny thing is, her team kept resigning and she was slowly taking on more and more of the work herself.  I was surprised to enter her office and see her working up graphics for the website when she also had to stay up all night to finesse Board papers.

Of course, when I asked about the Theory of Change for her new strategic ideas, she looked like she wanted to poke me in the eye with a pen.  A “theory” is the last thing you imagine will help increase your impact when you are stuck in a downward cycle of overwork and your team keeps getting smaller and smaller.

I pointed out that most serious charities, public sector services and increasingly private sector programs have a Theory of Change.  It is both a strategy and evaluation tool.

Like many busy folk in the charity sector, she was looking for a quick fix.  But I discussed with her five compelling reasons why if you don’t already have one, you might just find Theory of Change transformative for achieving your impact ambitions.

1.       It illustrates why and how your strategy will work

As part of your Theory of Change you will identify an ultimate aim or impact destination for your work.  However, big aims like “resilient young people in appropriate accommodation” might be achieved through many different pathways.

I’ve seen some pretty flat Theories of Change with big motherhood statements and no grit.  A strong Theory of Change will help you grasp the specific context you are working in.  It will specific highlight barriers you need to tackle, for example a “lack of public support for homeless teenagers” or “difficulty in forming secure relationships”.

Many interventions sound similar on paper, but a Theory of Change will give you a platform to explain how you deliver your activities and why that matters.  For example, they might be “human-centred” or “peer-led”.  Perhaps you want to make the service culturally appropriate and therefore the delivery team all speak the language of service users.

2.       It engages stakeholders (including your team)

The consequence of presenting your strategy’s logic in a Theory of Change is that it will be better understood by those it affects: first and foremost, the team delivering it.

Those working day-in and day-out on your change initiative are experts on the topic area.  It would be silly not to engage them in the process of creating the Theory of Change.  They might throw up some practical challenges or better still, some gems on what works.  Ultimately, they will own the Theory of Change in action, so it is essential that their perspectives are reflected.

The other expert stakeholder you are going to want to engage are the beneficiaries and stakeholders most affected by your work.

I’ve noticed the social sector can often be afraid of conducting Theory of Change research or workshops with their primary stakeholders.  Sometimes this is a genuine desire not to burden time-poor stakeholders, but more often, I’m afraid it’s a symptom of the continued lip service given to meaningful stakeholder input.

I’ve seen some of the most honest and frankly ingenious Theories of Change when organisations are able to trust that listening (really listening) to their stakeholders will allow them to deliver better services and greater impact, .

3.       You will measure the right thing

Part of developing your Theory of Change will involve selecting the most material (most important) outcomes from your work.  These are the outcomes you will measure and be held accountable against.

This can be the most difficult stage in a Theory of Change, especially if you lack internal clarity.  But once you are measuring your material outcomes, it will leave you with a relevant picture of your effectiveness.

Two of the most common pitfalls I see with bad impact measurement are (1) measuring what’s easy and (2) measuring too much.

Measuring what’s easy might mean you are collecting output data, probably management data, like how many people turned up to your event.  Then you will be guessing how this correlate to an outcome.  Obviously not a clever approach, but its terrifying just how many “for good” initiatives operate this way.

Additionally, I see organisations scrambling with pre-cooked lists of indicators and just selecting measures that “sound right” or “defensible”.  This is a seductive approach to impact measurement because you can sling a bunch of data into your annual report or dashboard, but it will give you a poor picture of your own impact.

Secondly, I see many initiatives drowning in a patchwork of measures that have been inherited from past managers or perhaps frameworks requested by funders.  This makes collecting outcome data a drag and something that your team is going dread.  Theory of Change in this case can paradoxically make proving your impact leaner and more elegant.

4.       It will guide operational decisions

A comprehensive Theory of Change will distill your understanding of the drivers for your work, how you seek to address social challenges and the other players working on the same issue.  It will help you marry together a gap in the system with the unique competences or approaches that your organisation is bringing to the table.

I’ve seen this exercise help countless programs re-prioritise.  For example, a charity working broadly on food issues had a small policy team and a big community-education team.  They were offered a significant grant to expand their policy work. Their Theory of Change revealed that the policy team had a role, but the core value they were delivering with the community needed to be their focus.  They turned down the grant.

I also worked with a national care service.  They developed a Theory of Change emphasising the importance of tailoring support around the service users.  This surfaced tension around the top-down management style.  Using the Theory of Change as a backbone reference document they shifted to a decentralised, flat management model, holding case-workers accountable for outcomes not activities.

5.       It will attract funders and/or investors

A Theory of Change is the first step you should take on your impact measurement journey.  Mapping the logic of your work is good hygiene and any serious funding body or social impact investor will expect to see one.

The quality and effort put into Theories of Change vary widely, so this can be your chance to stand out.  Or perhaps its your chance to get going! Remember you should regularly be revisiting your Theory of Change to ensure it still reflects your actions and thinking.

Coming back to our homeless charity CEO, once she started coming through the Theory of Change process, she became a serious convert.  The challenge she had been facing was clarity: while she had initially won over her stakeholders with ambitious ideas, it wasn’t until she had used the Theory of Change tool that she bought this together in a clean impact story.

The Theory of Change produced with her team and stakeholders was more realistic that her initial strategy and therefore more appealing to grant makers.  As for the team, one of program counsellors confided in a workshop that they had previously felt criticised, but now they were “on the same page” and excited about updating their work.

Theory of Change is a powerful framework in the impact toolbox.  If you would like to get going with your own, keep an eye out for a blog coming shortly on how to build your own Theory of Change.

Forget the car and smartphone markets, it turns out the every-man's cost of coronavirus panic can be measured in toilet paper.

This week there has been a run on loo roll across Australia.  The nation clearly decided that if there is a chance of catching the bug, they are certainly not going to face it without a clean bum.

Since the emergence of novel-coronavirus in December, we have been in a media storm of “just how bad is this outbreak?” conversations.  Some people feel we are being whipped into a frenzy, while others argue authorities such as Iran have been too slow to diagnose and control its spread.  In the meantime, we are hanging onto our chairs for when the W.H.O. announce that it is officially a “pandemic”.

Social risks and costs of illness

Why does it matter what we call it or how bad it is?  It matters because these are ways of sizing up the total social risk, and relatedly how much value there is in trying to prevent it.  So what might be included in the social cost of coronavirus?

The social risk of this virus includes all the ways catching the illness might affect our lives and those around us.  If a carer catches coronavirus for example, those dependent on them may also suffer.  Then there are all the fun things like parties you can’t got to, harming our psycho-social wellbeing, access to utilities will be reduced, not to mention potential lost income.  So implicitly, when we think about the risk, we’re considering the value of these factors along with good physical health.

The great toilet paper shortage of 2020 suggests that we think this social risk is pretty big (objective data aside).  The panic also tells us that there is an upstream social cost before even getting sick.  That cost is fear.  In social impact research we often talk about “safety” and “perceived safety”.  The funny thing is, regardless of the objective risk, perceived safety matters.

As we kick into fight or flight mode, our willingness to co-operate reduces and other anxieties are stoked. We are less courteous, less courageous, less creative and generally less constructive in survival mode.  This suggests the social harm of coronavirus has already begun.

How much should governments invest?

Governments and institutions are also grappling with the potential social harm from the disease. Right now, there is no vaccine for novel-coronavirus, and the interventions that governments are making are costly.

Shutting down the entire city of Wuhan was not a decision the Chinese Government would have taken lightly.  A cost-benefit decision would have been made factoring in potential future risk with certain short-term socio-economic costs.  Other governments are also already picking up pay cheques for chartering flights, quarantining citizens and checking peoples’ temperatures in public places.

If the W.H.O. declares the disease a pandemic there will be an expectation that governments elevate the management from health agencies and mobilise significant resources into stopping its spread domestically. It is funny how in a crisis we are willing to loosen the purse strings.

We need to get better at valuing prevention

The value at stake is very clear when you’re staring down the barrel of a gun.  But public health campaigns continue to be undervalued: like keeping older people active so they are less susceptible to illness or building community bonds, so we look after our neighbours when they are sick.

These early action programs are often perversely described as “expensive”.   This is of course a bad framing of the preventative value they deliver.  Climate change is threatening a world with many new diseases.  Health economics and social value research have guidance on how to reliably model avoided future harm.   We need to get better at routinely calculating long-term value if we don’t want to be fighting alone over the last packet of toilet roll.

How we feed an extra two billion people on our little planet by 2050 is one of the seismic challenges of this century.

The problem isn’t just about the growing number of mouths. It is about the potential loss of 20% of our arable farm land due in part to soil degradation, a changing climate and water scarcity. So what role do women have in driving sustainability?

In 2016 I saw first-hand how hunger has already been a daily reality for many living in rural Northern Ghana. As part of my work conducting a Social Cost Benefit Analysis for CARE’s Pathways programme I was able to meet over 200 farmers from communities in the region.

Women and children last

They told me how prolonged periods of food shortages were directly related to tired soil, persistent pests, a long dry season and the same crops being farmed season after season. Sadly, the hunger was often worst for women and children. Men, who control family income, are always provided for first.

Mediators, nurturers and huge economic potential

Some say that having more women in politics and leadership roles would solve many of the world’s most pernicious problems.  Women are often great mediators, tend to hold more nurturing values and represent a huge untapped economic potential. When working at NEF Consulting I saw this evidenced  time and time again. So what happens when you give women a spade in Northern Ghana?

Since 2012 the Pathways programme has trained women in sustainable farming techniques: inter-cropping, composting, nitrogen fixing crops, low tillage, reduced run-off and eco-harvest techniques. This has been combined with gender dialogues, market access training and support for village savings groups.

Richer soils, richer families

The result has been transforming and a real pleasure to witness as an evaluator.  It is rare to see beneficiaries so wholeheartedly pleased with an intervention. Environmentally, soils have become richer, more fertile and now support diverse flora. Economically, women and their families’ incomes have leap-frogged and their savings can now see them through difficult periods.

Socially, women and men have started to make decisions together. More children (especially girls) are completing school, domestic violence has all but disappeared and everyone feels healthier thanks to better diets.

“In the past men would take everything to the market and use the money for pitu [liquor] and meat and bring nothing home.  Now if there’s money we decide how to spend it together, we drink and eat together.  There is peace and harmony.”

No need to compromise

Programmes like Pathways are archetypes for achieving the Sustainable Development Goals. They prove that in solving one crisis, such as hunger, we don’t need to compromise on other societal targets, such as biodiversity, to achieve sufficient yields.

During causation exercises participants routinely rated women’s capacity – their unleashed skills, energy and passions – as the most important driver for this programme’s success.  This is a truly uplifting signal of the role women will play in building our future sustainable economy.

Work with CARE

NEF Consulting was asked by CARE to calculate the overall social value of Pathways across all stakeholders. This includes women and their families, in Ghana, Mali and Malawi.  To find out more, see the full report here.

This post was originally published on nefconsulting.com in September 2016.