Building an evaluation framework for Australian science and maths outreach programs in schools

Outreach programs in Science, Technology, Engineering and Maths (STEM) have proliferated nationally over the past decade. This has largely occurred in response to declines in student enrolment and engagement in these fields at both a secondary and tertiary level. STEM outreach programs are often resource intensive, however, they have been praised for achieving an array of positive outcomes ranging from improving student attitudes and enthusing teachers to dispelling negative stereotypes and changing student enrolment and career intentions. To date much of the evidence of such outcomes has been limited to anecdotal or only vaguely appropriate market-based output indicators. This research considered the role of program impact evaluation within the performance management cycle of STEM outreach programs. A survey of STEM outreach program providers was used to identify the common styles, reach, evaluation and collaboration approaches of Australian programs and a need was identified to encourage adequate objectives-driven evaluation across the field of STEM outreach. A range of impact evaluation methods were applied to investigate the potential impacts of two case study outreach programs while simultaneously exploring the value of the various evaluative approaches in the context of STEM outreach. The evaluation approaches used included student questionnaires (pre/post and expost), stakeholder focus groups, interviews, web analysis and researcher observations. Case study impact evaluations were undertaken for The Science and Engineering Challenge and The Questacon Tenix Maths Squad. Both programs were found to have the potential to positively influence student self reports on their attitudes towards science, engineering and/or maths. For both programs students with more negative attitudes prior to program participation reflected greater positive impacts from participation. Additionally students and other stakeholders reported that program participation improved students' self-confidence, teamwork abilities, understanding and knowledge of potential careers and in some cases students' subject enrolment and career intentions. A mixed-methods approach to STEM program evaluations was found to provide the most comprehensive understanding of program impact, while ex-post surveys exaggerated program impact through a lack of distinction between change and affirmation outcomes. In the expanding field of STEM outreach it is increasingly important for programs to utilise impact evaluation to maximise their efficiency, effectiveness and accountability. The Australian Commonwealth Government has recently stated, in relation to STEM outreach and engagement, that, "inspiration is too important to leave to chance" (DIISR, 20 I 0). If this is the case, then this report recommends that evaluation of program success is too important to leave to perception.