Evaluation of the National Network of Science Learning Centres

FINAL REPORT

January 2008
The GHK Consortium
GHK, Edcon Ltd, People Science and Policy Ltd and TNS Social Research

Executive Summary

1 Introduction and Evaluation Approach

This report presents the findings of the Evaluation of the National Network of Science Learning Centres (SLCs). The evaluation was commissioned by the Department for Children, Schools and Families (DCSF) and the Wellcome Trust, and undertaken by a consortium led by GHK Consulting in collaboration with Edcon, People Science and Policy Ltd and TNS Social Research between May 2006 and December 2007.

The aim of the evaluation was to assess and report on the progress of the SLC initiative against its short and medium term objectives. The study explored:

  • The impact of the initiative on science educators1 and their pupils/students,
  • Its impact on the culture of continuous professional development (CPD), and
  • The operation of the Network – in terms of effectiveness/contribution to impacts.


The evaluation consisted of four separate ‘Components’, each comprising a series of tasks as summarised below.

  • Component A: Scoping/Initial Evaluation Study (May to September 2006) – including visits to each Regional and the National Centres, interviews with staff, regional partners and educators, and interviews with 26 national stakeholders.
  • Component B: Initial Survey of Science Educators (July to December 2006) - including a telephone survey of 1,400 educators, and telephone interviews with 100 educators yet to receive SLC provision.
  • Component C: Educational Impact Study (June 2006 to July 2007) – featuring the observation of 22 SLC courses, and follow-up contacts with participants and schools to identify impact.
  • Component D: Early Impact Study (September to December 2007): including a survey of 5,000 educators using SLC services (the Early Impact Survey), return visits to the Centres, and follow-up interviews with national stakeholders.

Individual reports were produced for each study Component, which were consolidated in an Interim report (submitted in September 2006) and the final report.

2 The Science Learning Centre Initiative

The aims of the SLC initiative are multi-layered, ultimately to ensure that pupils experience a science education that equips them for their future lives and promotes a continued interest in science for further study or as a career option. The vision for the initiative is for science education to inspire pupils by providing exciting, intellectually stimulating and relevant experiences. This long term aim is supported by a series of short and medium term aims, including enhancing educators’ professional skills by introducing contemporary scientific ideas, providing training in effective teaching approaches and offering experience of modern scientific techniques.

The initiative is underpinned by two key principles – first that the SLC network is greater than the sum of its parts, and second that it offers educators continued access to high quality professional development services rather than discrete courses. However, this form of CPD demands the commitment of educators and their employers, and represents a cultural change for many. Changing CPD culture is likely to be the greatest challenge for the initiative, with key issues including:

  • Limited teacher exposure to science-specific, CPD on a systematic basis to meet their individual development needs;
  • The adequacy of school-based appraisal systems for assessing teacher performance and identifying development needs;
  • Difficulties for teachers in identifying and accessing suitable CPD services; and
  • The financial and time constraints which mitigate against participation in such professional development activities.

3 The Regional and National Centres

The initiative is delivered through a network of nine Regional Centres and the National Science Learning Centre (NSLC) at York. The Regional Centres and the NSLC display considerable differences in terms of funding, structures and delivery models. The Regional Centres are funded by DCSF and the NSLC by the Wellcome Trust, and were established as independent entities which have come together as a network.

3.1 The National Science Learning Centre (NSLC)

The NSLC is housed on the science campus at the University of York, with on-site facilities that allow provision to have a residential component. The NSLC focuses on providing multi-day courses with on-site accommodation, to which it recruits on a national basis, and commonly uses Impact Rewards and bursaries to subsidise educator attendance. With over 20 FTE staff, the Centre also acts as a ‘hub’ for the Regional Centres, including providing national-level marketing, managing the SLC portal and providing a resource centre for the Regional Centres.

3.2 The Regional Centres

The Regional Centres were given a considerable degree of freedom to develop their structures, partnerships and service offers. Seven are located within Higher Education Institutions, with one in a science centre and another in a specially built building. All received funding to develop or modify premises for classroom and practical sessions, including up-to-date laboratory and IT facilities. The Regional Centres have extended their reach by delivering services on an outreach basis, using a range of off-site delivery options including specially equipped ‘mini-Centres’ and partner or school/college premises.
The Regional Centres are underpinned by ‘core’ partnerships commonly featuring Higher Education Institutions (HEIs), which act as lead partners in all but two cases. Other core partners include: representatives of SETPOINTS and the Association for Science Education (ASE), providers of science CPD, Local Authorities and other organisations with a science focus – such as research facilities, museums and science and technology focussed companies. The Regional Centres have also established partnerships and collaborative arrangements with a range of regional and national organisations – often focused on service delivery and course development.

The Regional Centres differ considerably from the NSLC in terms of their:

  • Staffing - each Regional Centre had fewer than 10 FTE staff, in some cases considerably fewer, compared to the 20 plus FTEs at the NSLC.
  • Delivery models – with the Regional Centres providing mainly single day courses, compared to the multi-day, residential courses at the NSLC.
  • The delivery models followed by the Regional Centres share many similarities, which in addition to delivering outreach services were also characterised by the balance between internal and external delivery, and balance between ‘core’ programme and more ‘bespoke’ provision.

3.3 Progress and Achievements

The early stages of the evaluation identified how the National and Regional Centres were moving from a ‘development’ to a ‘consolidation’ phase – having established themselves, recruited staff and delivered their first programmes of provision. A series of challenges facing the Centres were also identified, including:

  • Having to raise awareness and establish themselves as key providers of science CPD – which often required greater effort than was initially anticipated;
  • Stimulating demand for science-specific CPD – in the context of competition for school budgets and teacher time, and other barriers to participation – and extending market penetration and educator participation;
  • Establishing effective working relationships with key partners, stakeholders and providers – addressing concerns over competition and service duplication, and embedding the Centres in the regional and national CPD infrastructure;
  • Achieving financial sustainability – by increasing educator participation and market penetration, while retaining the SLC mission of providing longitudinal CPD and cutting edge services.
    Early indications of the quality and appropriateness of SLC services were positive, with high levels of customer satisfaction being reported and the Interim report concluding that the elements were in place to provide positive impacts for educators. The latter stages of the study explored provision quality in more depth through the observation of a sample of SLC provision, and found that the courses observed exemplified good practice in teaching and learning.

In the 2006/7 year the Centres described making few major changes to their staffing structures and operational models, although in some cases they had reviewed their strategic aims and purposes on the basis of their initial experience. Some of the Centres reported how the involvement of individuals with specialisms in the primary, secondary and post-16 sectors had been helpful, including helping break into markets which had remained closed previously.

Despite facing challenges initially at least, the Regional Centres had made good progress in establishing regional and local partnerships and collaborative arrangements, and in so doing helped counter concerns over competition and duplication. Local Authority relationships emerged as particularly important – especially with science advisors and Secondary National Strategy consultants – and a variety of collaborative activities were identified including joint service delivery, sign-posting provision and recruiting Local Authority staff to advisory groups. It was recognised that, even where Centres had been particularly active, work remains to be done to establish good relations across all regions and Authorities.

Performance in 2006/7 was also positive, with all the Centres exceeding their targets for educator throughput, delivering over 13,000 educators days, and also offering a range of events and conferences for science educators. However, high levels of course cancellation continued to be reported across the Network, with some Centres reporting cancellation rates of 50% or above.
Finally, questions were raised over the extent to which the SLC portal is being used to its full potential. The portal functions range from on-line course booking to supporting CPD by linking discrete pieces of provision, and although its use by the Centres had increased over the study period, it was still not commonly used, for example, to provide performance reports across the Network. At the same time, portal use by educators using SLC services was limited, with a series of access and functionality issues being reported. Questions remain over whether portals can add value to CPD provision, and if so how this can be achieved.

4 The SLC Network

As well as being established as independent Centres, the evaluation also identified a series of initial tensions influencing the SLCs’ ability to work collaboratively. These included an emphasis on throughput targets, concerns over competition, difficulties in delivering courses across the Centres and the inability to share management information. However, the latter stages of the evaluation found that considerable progress had been made to establish a formalised ‘Network’, based on the realisation of the benefits of collaborative activity and the development of a Network strategy.
The Network has clarified its aims and objectives, and the means by which they will be achieved, through the development of a Network strategy. The aim of the strategy is to “Improve the quality of science teaching and learning through effective continuing professional development for all those involved in science education”, which is supported by four strategic objectives:

  1. To supply effective, high quality CPD to increase science educators’ awareness, understanding and confidence by updating their subject knowledge, pedagogical skills and appreciation of careers from science;
  2. To stimulate demand for CPD and embed it as an intrinsic and ongoing part of science educators’ professional life;
  3. To work strategically with partners in the science education community to coordinate and improve access to CPD at the regional and national levels; and
  4. To establish network cohesion, coordination and direction to operate efficiently and effectively.

The Network also has an Operational Plan, featuring a series of operational objectives showing how it will contribute to delivery across all four strategic objectives.
Key components of the Network structure include:

  • The Network Project Director – this post was created as a 12 month secondment to support network development, and as a central coordinating point for contacts with external partners and stakeholders.
  • The Directors’ Group – this group meets monthly and co-ordinates policy and practice across the network. It also retains overall responsibility for ensuring the network quality assurance system is being followed, and agrees an annual action plan of activities to be taken forward across a series of working groups.
  • Network working groups - there are currently six working groups, which were reviewed following the recommendations of the Interim report, namely: the Primary Development, Secondary Development, Post 16 Development, Communications, Learning and Technology and the Impact and Research groups. The working groups comprise SLC staff and are usually chaired at Director level, and are tasked to take forward specific actions to implement an annual action plan on a ‘task and finish’ basis.

The achievements of the Network in 2006/7 included an increased level of collaborative activity both between the individual Centres and with external partners, the increased exchange of good practice and the joint development of provision. This included bidding and delivering a series of national CPD projects, with a value of over £1.8 million, for funders including DCSF, the Royal Society of Chemistry and others.

Each Centre participates in the Network through attendance at meetings, working groups, collaborative activities and Network projects. Benefits from their participation included enhanced profile and credibility, the provision of a ‘strategic voice’ for lobbying and consultation, and the provision of peer support and the exchange of ideas and good practice. While commitment to the Network was high across the Centres, some faced challenges in finding the time to contribute to the level they would have liked.

The national stakeholders interviewed also considered that the Network was becoming more recognisable as an entity, and in many cases was now easier to engage with. However, as the following section describes, more remains to be done before the Network is viewed externally as a coherent and consistent whole.
5 The National Stakeholder Perspective

A series of 26 national stakeholders were interviewed at the start and end of the study, to explore their perceptions of the SLCs, the level and nature of engagement with them, and any change in these over time. The stakeholders represented a range of organisations, either from the education world with responsibility for CPD (where science was one subject amongst many) or the scientific world (where science education was one subject amongst many).

The first interviews identified a range of opinions of the initiative, which while broadly supportive included more negative views amongst the educational representatives, some of whom felt they had been excluded from the initiative. The second interviews were more positive, largely due to positive feedback from teachers on the quality of SLC provision and the Government’s commitment to the initiative, although some negative views were still expressed.

Individuals working on Government education initiatives described developing ways to work with the Centres, although some were less positive, largely as they feel the out-of-school model of provision is not the most effective. This issue was identified in the first interviews, although concerns had diminished as the stakeholders became aware of the Centres’ outreach activities and the inclusion of longitudinal components in their provision. The professional bodies remained supportive of the initiative, despite some having practical difficulties engaging with the ten different Centres. There was also the view that the Regional Centres were continuing to look for a role, were yet to position themselves in the regional CPD infrastructure, and needed to do more to link with Local Authorities and the Secondary National Strategy.

From the stakeholder perspective, the formalisation of the Network and the Network Project Director post had had a considerable impact on their relations with the initiative. The provision of a single contact point was highly valued, although others noted that some courses are still not available nation-wide. Indeed, provision was not seen as being consistently available across the country, and that a common course offer with a degree of local variation was required.

Views on the quality and appropriateness of SLC services were positive, based largely on reports from participating educators, although concerns continued to be raised over perceived variations in service quality across the Network. The second interviews also found fewer concerns over competition at the national level, and more of an acceptance that the SLCs are here to stay and any conflicts must be worked through. Stakeholders were, however, unclear about the course development process in particular, leading some to consider that the process is ad hoc and not well planned. The levels of SLC course cancellation were also raised, and influenced some stakeholders’ views of the planning of provision.

6 The Impact of the SLC Initiative

One of the key aims of the study was to identify the extent to which the SLC initiative was beginning to have an impact on educators, pupils and students and schools and colleges. The evaluation concluded that the SLCs are having positive impacts on all three areas, and identified a series of key factors which influence their achievement.
Evidence of impact was drawn mainly from three sources – examples cited by the Centres, the findings from the Early Impact Survey, and the results of the Educational Impact Study. Most Centres could only provide anecdotal examples, and while some had evaluated individual pieces of provision none had attempted to assess their impact more widely. While there are plans for a common approach to impact assessment across the Network, the Centres considered that their high levels of customer satisfaction and repeat business meant their services were having positive effects.
The Early Impact Survey and Educational Impact Study respectively provided a quantitative and a more qualitative, judgement-based insight into the impact of the Centres. Their findings suggest that:

  • Impacts on individual educators were the most commonly identified impacts, with most respondents to the Early Impact Survey reporting impacts on their:
    • Personal motivation – reported by over eight out of ten educators overall;
    • Subject knowledge – reported by almost eight out of ten educators; and
    • Confidence in the classroom – reported by almost six out of ten educators.
    The Educational Impact Study featured the observation of provision and follow-up with participating educators, allowing impacts to be explored in detail. Here gains in personal knowledge and skills represented ‘first stage’ impacts, which while valuable may not be sustained in the long term. ‘Second stage’ impacts on pedagogy, management and leadership capabilities, and on longer term professional development, were also identified. In these cases courses with a longitudinal component were more likely to impact than single day provision.
  • Impacts in the classroom and on pupils were reported by four out of five and three out of four respondents to the Early Impact Survey respectively. The impacts resulted from access to more engaging teaching resources, new ideas for lessons and practical sessions – with the provision of materials for use in the classroom being highly valued. The Educational Impact Study also provided examples of classroom impacts, including ‘second stage’ impacts on pedagogy which were likely to have a longer term influence.
  • Impacts on schools and other educators were reported less frequently, although two thirds of respondents to the Early Impact Survey reported impact on their schools overall and over half reported specific impacts on colleagues, whole school objectives and school development plans.

Many respondents to the Early Impact Survey also considered that further impacts were likely to result from their SLC involvement, but that these would take time to be realised. There is an important role for the Centres to play in ensuring that potential benefits are achieved, through the provision of continued support to embed change.

Impacting on the wider culture of science CPD remains a challenge for the Centres, although some positive change was suggested. At the regional level examples were offered by the Centres and their partners of the increased take-up of CPD by individual schools, suggesting some positive change from the ‘bottom up’. Similarly, the increased willingness of regional partners and stakeholders to engage with the initiative may also suggest a positive cultural shift. The national stakeholder interviews also suggested the beginnings of a shift in views on the importance of science-specific CPD, and the increased acceptance of and engagement with the SLC initiative. However the question remains over whether the Centres are working with a sufficient number of schools and colleges to influence CPD culture more widely.

A series of factors were identified which influenced the likelihood of positive impacts from SLC provision, including:

  • A strong longitudinal component to provision – either through multiple or linked days of provision, or a wrap-round model featuring pre- and post-course tasks;
  • The identification of expected learning outcomes in advance of attending provision - and acting on this information to tailor provision as practicable;
  • The use of pre- and post-course activities – and the completion of any preparatory work prior to course attendance; and
  • Post-course follow-up by the Centres.

Many of these factors were part of the delivery models followed by the Centres, although they were not applied on a consistent basis across the Network.

7 Conclusions and Recommendations

7.1 Conclusions

The evaluation concluded that good progress has been made towards the short and medium term aims of the SLC initiative, with the Regional and National Centres making a positive contribution to the professional skills of science educators which are impacting on pupils’ classroom experiences and learning. Particular achievements in the first two full academic years of operation include:

  • Establishing the Centres as operational entities – including setting up premises, administrative systems and recruiting staff.
  • Raising awareness of the initiative and the individual Centres regionally and nationally – increasingly through recommendations from service users.
  • Establishing the SLC Network – moving from ten independent Centres to a Network characterised by high levels of commitment and collaboration.
  • Delivering high quality CPD services with the potential for impact – with each Centre exceeding their throughput targets in 2006/7.
  • Establishing positive relationships with regional and national stakeholders – leading to a range of regional and national collaborative working arrangements.

The study concluded that the development of partnership and collaborative arrangements at the national and regional levels represent the way forward for the Network, supported by a clear and shared view of the Centres’ strategic purpose and uniqueness compared to other providers.

The Impact of the Initiative

The study found that the Centres are having positive impacts on the educators they work with, their colleagues in schools and colleges and the pupils they teach. Although these impacts cannot yet be quantified across the Network, the Early Impact Survey and Educational Impact Study identified impact was being achieved:

  • For individual educators - including improved motivation, subject knowledge and confidence in the classroom; and higher level impacts on pedagogy, management and leadership capabilities.
  • In the classroom and on pupils – including impacts on pupils’ learning, interest, motivation and achievement, and impacts on pedagogy which were likely to have a longer term influence.
  • For schools and other educators – although reported less frequently, impacts were reported on participating schools overall, and more specifically on colleagues, whole school objectives and school development plans.

Impacts on the culture of science-specific CPD were less easy to gauge, although the fieldwork and national stakeholder interviews suggested some positive developments at the regional and national levels. This area of impact remains the greatest challenge for the SLCs, and one they are unlikely to address in isolation.

The educators also commonly considered that further impacts were likely to result from their involvement with the Centres, but that these would take time to be realised. The challenge for the Network is to extend their influence by scaling up delivery, and taking further steps to ensure impacts result for participating educators and schools.

The Operation of the Network

The development of the SLC Network and the Network strategy represent major successes and provide a solid platform for further development. While early days in implementation, the strategy presents an opportunity to address outstanding operational issues, develop common approaches and allow the Centres to speak with a single voice.
The evaluation found that the Centres are becoming recognised as providers of high quality CPD, which was confirmed through the observation of sessions and the views and experiences of partners, stakeholders and participants. Satisfaction with SLC services is very high, and provision is commonly seen as being better or at least as good as previous experiences.
While no single model of provision was found to be inherently better than another in the context of their different objectives and intended learning outcomes, the inclusion of a longitudinal component was found to increase the likelihood of impact. While the Regional and National Centres’ delivery models featured provision of different durations, the Regional Centres increasingly offered single day courses, including pre- and post-course activities. However, these and other approaches to maximising impact were not used on a consistent basis across the Network. Other areas where common and consistent approaches would be beneficial include the SLC portal, the collection and reporting of management information and the assessment of impact.

7.2 Recommendations

The recommendations focus on the further development and success of the initiative, and include:

Strategic Recommendations/Recommendations for Funders and Policy Makers

  • Build on successes in integrating SLC services with other local, regional and national partners, strategies and approaches – for example the successful relationships between some SLCs and local Secondary National Strategy staff.
  • Emphasise the role of the SLCs in supporting change in the culture of CPD generally in schools and influencing the demand side. Seek to secure ring-fenced CPD budgets for schools.
  • Establish a research and analysis team at the NSLC, to provide a central research capacity to support the implementation of Network strategy. The main function of this team would be to explore supply and demand issues to inform programme planning, and future SLC strategy, nationally and regionally.
  • Based on the work of the research and analysis team, the Centres with their national, regional and local partners should identify a minimum CPD requirement for all science teachers.
  • As part of the implementation of the Network strategy, establish a set of key performance indicators (KPIs) which encapsulate the objectives of the initiative and can be collected, analysed and reported through the portal.
  • Implement a common approach to assessing impact across the Network, using the Impact Framework while recognising its limitations.

For the SLC Network

  • Make the Network Project Director post permanent, and create new posts to support bidding for Network projects and coordinating their delivery.
  • Explore the options for continuing and formalising the roles of the current Primary, Secondary and Post-16 ‘sector leads’.
  • Map provision against the minimum science CPD requirement at the regional and local levels, working with partners to identify gaps and areas of demand. Based on this establish registers of provision, to allow the Centres to provide a comprehensive referral function for science CPD.
  • Extend the longitudinal elements of SLC provision as appropriate to the objectives and intended outcomes of individual courses - including using pre and post course materials to facilitate impact, and offering linked or suites of provision to promote a more continuous approach to CPD.
  • Continue and extend efforts to engage a broader range of local, regional and national stakeholders with the Network and SLC initiative more broadly, ensuring capacity is in place to do so effectively.

The funders should commission an independent, external review of the SLC portal to establish the requirements of the Centres and how these can be addressed. Following this, the Network should take steps to ensure that the portal is used on a more consistent basis.

Full report available at www.wellcome.ac.uk

_______________________________________

1 Throughout the study the term ‘science educator’ has been used to include heads of science, science teachers and technicians in secondary schools and sixth form colleges/FE; science co-ordinators, teachers and assistants in primary schools; and teachers and assistants in special schools and pupil referral units.