• Perspectives
• Science and Society - From Monologue to Dialogue
• New approaches to public and stakeholder engagement and consultation
• Evaluation
• The need for representative samples in quantitative research
• Project budgets

Print version

Share |

HomePSP perspectivesEvaluation

Evaluation

There is no magic formula for evaluation. Evaluation is said to be “very important”, but some find it “very difficult”. People tend to be frightened of evaluation because they see it as a test and a threat. In essence though, evaluation is about seeing whether a policy, programme or project has had the desired effect and drawing out lessons for the future, not just of a specific policy or programme but also more broadly. To construct an evaluation strategy requires careful thought about the objectives of the policy or programme to be evaluated.

At the project or programme level, evaluation is a process that takes place before, during and after the programme or project. It includes looking at the quality of the content, the process through which the programme or project is realised and the impact of the project or programme on the target. It requires baselines to be set, quality criteria and thresholds to be determined and an understanding of where to find, and how to collect, relevant data and analyse it in a meaningful way. Knowing what, if anything, has changed as a result of a programme or project is not enough. It is also important to know why something has changed and how a programme or project can be improved.

The purposes of an evaluation should be to:

• establish whether the objectives of a programme or project have been met
• identify whether anything has changed as a result of the programme or project (often termed summative evaluation)
• identify how the programme or project could have been more effective
• identify whether there have been unintended outcomes and what these were; and
• support the development of a programme or project as part of the research and development process (sometimes known as formative evaluation).

There is often confusion between monitoring and evaluation data. This can be because both can be gathered in similar ways. In essence, monitoring is about counting things and ensuring the programme or project is on track. Evaluation is about the impact of a programme or project and ensuring it is well designed to make the maximum impact.

There are three roles for an evaluation:

• to support the development of a policy, programme or project (Formative evaluation);
• to ensure it is done better next time (Evaluation of the processes); and
• to assess the final impact of the policy, programme or project (Summative evaluation).

Setting aims and objectives

The first thing to do is to clarify the aim(s) of the policy, programme or project. Next there need to be objectives set– these are the things that have to happen for the aims to be achieved.

Policies and programmes will be evaluated against their objectives. Setting objectives is an art. Objectives should be SMART, that is:

• Specific;
• Measurable;
• Achievable;
• Relevant; and
• Time-bound.

Each objective should be all of these, and they inter-relate. For example, an objective might not be sufficiently specific to be measurable, so it can never be clear to what extent it was achieved. Similarly, an objective set with a timescale that is beyond the scope of the programme to measure is not relevant. Considering each element of SMART provides a process for establishing whether or not an objective is appropriate and can be used in evaluation.

In setting objectives the two main pitfalls are: setting objectives that are important, but which can’t be measured; and setting objectives that are measurable, but of little relevance.

Formative evaluation

Some people call formative evaluation market research or research and development. It supports the development of the policy, programme or project and tests ideas, concepts and pilots on representatives of the potential beneficiaries or audience. The emphasis at this stage is likely to be on discussion-based tools (qualitative research). Qualitative input at this stage can be crucial to understanding how to change a policy, programme or project to improve its functionality and appeal. To make improvements, it’s no good knowing people didn’t like it unless there is an understanding of what they didn’t like and why.

Evaluation of processes

Evaluating the process of managing and running a policy or programme will provide lessons for next time. When there are financial pressures on organisations, this assessment of processes can help to streamline management and identify the value of the administrative function. This supports both formative and summative evaluations.

Summative evaluation

Summative evaluation is the type of evaluation that people are most familiar with and is about looking at the impact of a policy or programme. There might be an emphasis on numerical data but depth of understanding can also be important at this summative stage. Qualitative data can be crucial in explaining what lies behind the numbers.

Benchmarking and baselines

It sounds obvious but if a policy or programme is trying to change something, then baseline data on the state of affairs before the policy or programme was launched provides something to measure against. Benchmarking against other policies or programmes can also help assess the impact of a particular policy or programme. To measure change, quantitative data is needed.

PSP has produced a guide for Research Councils UK on how to evaluate science communication projects but the basic ideas hold true for evaluating other types of projects and programmes. Evaluation: Practical Guidelines the full guide can be found at: www.rcuk.ac.uk

Site by Livewire