Evaluating the Soft Outcomes of Learning and Development Programmes
Evaluation is critical to assessing the effectiveness of learning and development programmes for young people, yet little is often done to provide hard evidence of soft outcomes. Soft outcomes are those which may not necessarily lead to a qualification or employment, but which are nonetheless important stepping-stones towards this or moving a person forward.
As an example, a programme that is aimed at disadvantaged young adults with behavioural problems may not necessarily produce a high level of certification and employment, yet the participants may have grown in terms of self-confidence, respect for elders and self-esteem. All of these can make the difference between them continuing a negative path or steering them towards a more positive one in the future.
Evaluation in South London
One organisation that is well aware of the importance of soft outcomes is Connexions South London, an organisation committed to the growth and development of young people in South London. Connexions South London, works with many partners and helps fund many initiatives designed to help young people further their education and get into employment. Monitoring these initiatives is important to ensure funding is being directed to the most successful areas, yet some initiatives are dealing with more disadvantaged groups than others and so measurement of success is not always a level playing field.
Jeremy Thorn, Development Manager (PA Training and Quality Improvement) at Connexions South London comments: “We have carried out many evaluation studies of projects that aim to guide young people into training and education or employment. One overall measure of success is in whether participants achieve qualifications and enter employment. Qualitative research often highlights many other benefits from our schemes, which are not solely about the qualifications achieved or getting into employment. Many of our schemes have touched participants in a way that has steered them to a positive path of self-discovery and given them life-skills that they previously never had, such as self-esteem, anger management and enhanced communication skills. Many people take these things for granted, but for some young people these are the building-blocks for accomplishment and taking control of their lives.”
Framework for measurement
Connexions South London has commissioned Marketry, a specialist in research and evaluation consultancy within the learning and skills sector to develop a framework from which to measure soft outcomes, as well as hard outcomes.
Judy Wilson, managing director of Marketry comments: “Despite the widespread interest in the subject, there is still no agreed definition of soft outcomes. A link has already been made between hard outcomes and specifically education, training and qualifications and economic prosperity. Our aim is to provide a more scientific approach into measuring soft outcomes to provide similar, but perhaps less direct, conclusions.”
In terms of defining soft outcomes, there are four definitions that appear to be used generally either in research, by organisations looking at soft outcomes or by government. Soft outcomes are:
· ‘achievements’ by people (e.g. interpersonal skills or organisational abilities)
· the intangible, subjective, un-quantifiable benefits of interventions
· intermediate and logical steps towards ‘hard outcomes’
· results or immediate effects related to labour market initiatives
Based on initial feasibility studies, it is possible to measure soft outcomes, however there are some very important things to consider. Soft outcomes need a reference position or ‘baseline’ and subsequent reviews to define and measure them. This can be time-consuming and requires commitment and extra resources to make it work.
Collecting the data
The need for very robust data may mean that some projects with limited information may not have enough information to provide real evidence of any soft outcomes. Soft outcomes might get confused with other real-time maturing effects and changes in people’s lives. This is particularly the case with young people who are traveling fast through mainstream education and mature quickly – what may appear as an outcome might have happened anyway. Results of soft outcomes are also not always fixed. Whereas something like a qualification may always be something a young person has achieved, in terms of soft outcomes; young people may go through peaks and troughs in life, during or after a project, such as family break up, which may reverse positive outcomes.
What is clear is that in measuring soft outcomes, measurement must remain consistent throughout baseline and follow-up studies. This ensures data that can be pulled together and assessed using scientific research methods. Soft outcomes can be measured in a variety of ways through balanced score cards, psychometric testing, games that engage, use of existing administration systems and data and project evaluation. The best approach may well use a combination of these.
Soft outcomes contribute to society and economy
Judy Wilson at Marketry would like to see a greater emphasis on soft outcomes within government. She comments: “If this government really wants to make a difference to the socially excluded; soft outcomes are the key stepping stones to examine their contribution to society and the economy. To truly find out whether support schemes lead to greater personal achievement and ultimately greater economic prosperity, there needs to be investment and commitment for long-term studies that attempt to examine and measure soft outcomes. Long-term studies should compare disaffected people and the support they have received against those that have not. Only then might we find out the true worth of soft outcomes and their overall benefits to the national economy.”
Marketry is a market research consultancy that specialises in providing research services to publicly funded organisations. Established in 1992, it offers an extensive portfolio of services including benchmarking studies, evaluation, business audits, mystery shopping, feasibility studies and market analysis. Further information about the company can be found at www.marketry.co.uk