Applying the Lessons from Evidence-based Medicine to Public Policy

By Annette Boaz

Reproduced by permission of the Public Management and Policy Association

In his 1996 presidential address to the Royal Statistical Society, Professor Adrian Smith challenged the field of public policy to adopt a more ‘evidence based’ approach.

Most of us have aspiration to live in a society which is more, rather than less evidence based … there has been the growth of a movement in recent years calling itself ‘evidence based medicine’, which perhaps has valuable lessons to offer.

The policy and social science research communities have responded with a variety of research and development activities. Initiatives include the establishment of the Department for Education and Skills’ EPPI center, the Centre for Management and Policy Studies in the Cabinet Office and the Social Care Institute for Excellence (SCIE) and our own ESRC sponsored UK Centre for Evidence Based Policy and practice which works with a network of seven specialist ‘nodes’. At an international level the launch of the Campbell collaboration, designed to carry out systematic reviews of interventions in the fields of education, criminal justice and social work, signals an international interest in developing evidence based approaches to policy and practice. However, much of the groundwork of developing and testing methodologies is happening here in the UK.

The systematic review

A key feature of the evidence-based approach in health care is the emphasis on systematic reviews of research and we have begun our work by looking at the potential of systematic reviews as a tool for public policy research. The systematic review approach acknowledges the large body of existing research and seeks to synthesize the findings from relevant, good quality, studies. Of course, there is already a considerable amount of research review in social policy research. It is likely that anyone who has carried out a piece of research, will usually at an early stage, have been involved in conducting some sort of review of existing literature. However, systematic reviews present a distinctive approach. The key principles include:

• focusing on answering a specific question(s)
• using a protocol to guide the process
• seeking to identify as much of the relevant research as possible
• appraising the quality of the research included in the review
• synthesizing the research findings in the studies included
• aiming to be as objective as possible about the research in the review
• preparing a review that can be regularly updated in order to remain relevant.

Using protocols to make the review process clear and transparent, clearly articulating the research question the review is designed to explore, developing and carrying out comprehensive searches and appraising the quality of research identified for inclusion, all seem like useful standards for developing a good quality evidence base. It has also been argued that the widespread use of systematic review methods can help to:

• drive up standards in primary research (and the reporting of research)
• identify gaps in the evidence base
• inform the development of future research questions.

Broader approach needed

There are aspects, however, of the approach that are proving difficult and inappropriate to transfer. For example the focus of systematic reviews in healthcare has traditionally been on evidence of effectiveness. In public policy research review, questions aim to explore not only ‘what interventions work’ but how, why, when and where they work – and under what conditions. To answer these questions a variety of research methods need to be employed. Where systematic reviews in medicine have traditionally identified, appraised and synthesized randomized and controlled trials as the ‘gold standard’ of evidential quality, reviews in public policy are likely to pull together a much wider variety of evidence. This distinction has implications for both the appraisal and synthesis of studies.

The systematic review does not provide a ‘one size fits all’ template that can be easily applied to all areas of policy research. We need to develop the approach and there are a number of ways in which the traditional experiences of social science researchers can contribute to the development of systematic review methodology. Particular strengths of these traditions include experience of involving users in the research process, working with policy makers to identify research questions and combining methods to explore complex issues.

The way forward

It will only be by adapting the systematic review to fit our purposes that policy researchers will be able to develop approaches to research synthesis that serve the evidence based policy and practice agenda. One approach is through learning-by-doing and a number of reviews are now appearing that seek to use systematic review methodologies to explore non-clinical policy questions. For example, the team at the Department for Education and Skills’ EPPI center are providing support and training to review groups carrying out systematic research in the field of education and researchers at Cranfield University are developing the methodology by carrying out systematic reviews of management literature.

In the UK Centre we are also carrying out projects to further methodological development. For example, a key challenge in public policy research synthesis is to identify exactly what is meant by ‘good quality’ evidence. The social care evidence base is both elaborate and contested. It includes research knowledge from a wide range of methodological traditions as well as the everyday wisdom and experience of many stakeholders, including service users. There is no ‘gold standard’ of evidence, and considerable dispute about the merits of particular research methodologies or ‘ways of working’. SCIE has commissioned the Centre to explore the types and quality of knowledge in social care and specifically to

• identify and classify types of social care knowledge; and
• develop the ways of assessing their quality that will be acceptable to a wide spectrum of sometimes conflicting opinion.

Policy into practice

There is, of course, more to the evidence based approach that systematic reviews. Getting the evidence into policy and practice is equally important, and the ESRC Centre has a number of ‘nodes’ working on these difficult issues. For example, the research team at City University and Barnardos is working on improving the use and implementation of research by child care practitioners through developing ‘evidence nuggets’, based on the findings of systematic reviews. At the University of St Andrews, researchers are focusing on the ways in which research knowledge is utilized, modeling the process of utilization in policy and practice, and gathering evidence on what works in improving evidence use.

So are there ‘valuable lessons’ to be drawn from the evidence-based medicine movement? Yes and no. The overall aim of the Centre and its nodes is to develop evidence based policy making both by drawing on the progress made in evidence based medicine and by tapping in to what is unique and ‘special’ about developments in other areas of public policy making. We are also keen to promote a wider understanding of what it really means to make evidence a key tool in the process of policy development and implementation. To make this happen the center is progressing its methodological work alongside training and development activities to promote evidence based approaches to policy and practice. These activities are aimed at both individuals planning to commission or conduct reviews and wider research and policy audiences. In taking this work forward, we hope to help achieve Professor Smith’s vision of a society that is ‘more rather than less evidence based’.

Annette Boaz us a Senior research Fellow, ESRC UK Centre for Evidence Based Policy and Practice, Queen Mary, University of London.