Evaluation
There is no magic formula for evaluation. Evaluation is said to be “very important”, described as “very difficult” and often not done very well. People tend to be frightened of evaluation because they see it as a test and a threat. In essence though, evaluation is about seeing whether a policy or programme has had the desired effect and drawing out lessons for the future, not just of a specific policy or programme but also more broadly. To construct an evaluation strategy, requires careful thought about the objectives of the policy or programme to be evaluated.
Evaluation is a process that takes place before, during and after a project. It includes looking at the quality of the content, the delivery process and the impact of the project or programme on the target groups. It requires baselines to be set, quality criteria and thresholds to be determined and an understanding of where to find, and how to collect, relevant data and analyse it in a meaningful way. Knowing what, if anything, has changed as a result of a project is not enough. It is also important to know why something has changed and how a project can be improved.
The objectives of an evaluation should be to:
- establish whether the objectives of a project or programme have been met
- identify whether anything has changed as a result of the project or programme (often termed summative evaluation)
- identify how the project could have been more effective
- identify whether there have been unintended outcomes and what these were; and
- support the development of a project as part of the research and development process (sometimes known as formative evaluation).
There is often confusion between monitoring and evaluation data. This may well arise because they can both be gathered in similar ways. In essence monitoring is about counting things and ensuring your project is on track. Evaluation is about the impact of a project and ensuring it is well designed to make the maximum impact.
Setting aims and objectives
The first thing to do is to clarify the aim(s) of the programme or policy. Next there need to be objectives set– these are the things that have to happen for the aims to be achieved.
Policies and programmes will be evaluated against their objectives. Setting objectives is an art. Objectives should be SMART, that is:
- Specific;
- Measurable;
- Achievable;
- Relevant; and
- Time-bound.
Each objective should be all of these, and they inter-relate. For example, an objective might not be sufficiently specific to be measurable, so it can never be clear to what extent it was achieved. Similarly, an objective set with a timescale that is beyond the scope of the programme to measure is not relevant. Considering each element of SMART provides a process for establishing whether or not an objective is appropriate.
In setting objectives the two main pitfalls are: setting objectives that are important, but which can’t be measured; and setting objectives that are measurable, but of little relevance.
There are three roles for an evaluation:
- To support the development of a policy or programme (Formative evaluation)
- To ensure it is done better next time (Evaluation of the processes); and
- To assess the final impact of the policy or programme (Summative evaluation).
Formative evaluation
Some people call formative evaluation market research or research and development. It supports the development of the policy or programme and tests ideas, concepts and pilots on representatives of the audience. The emphasis at this stage is likely to be on discussion-based tools (qualitative research). Qualitative input at this stage can be crucial in understanding how to change a policy or programme to improve its appeal. To make improvements, it’s no good knowing people didn’t like it unless there is an understanding of what they didn’t like and why.
Evaluation of processes
Evaluating the process of managing and delivering a policy or programme will provide lessons for next time.
Summative evaluation
Summative evaluation is the type of evaluation that people are most familiar with and is about looking at the impact of a policy or programme. There might be an emphasis on numerical data but depth of understanding can also be important at this summative stage. Qualitative data can be crucial in explaining what lies behind the numbers.
Benchmarking and baselines
It sounds obvious but if a policy or programme is trying to change something, then baseline data on the state of affairs before the policy or programme was launched provides something to measure against. Benchmarking against other policies or programmes can also help assess the impact of a particular policy or programme. To measure change, quantitative data is needed.