In 2020 we completed a major outcomes evaluation of sevelpostdoctoral fellowship programmes for MBIE. The results will be online at: https://www.mbie.govt.nz/science-and-technology/science-and-innovation/funding-information-and-opportunities/funding-agencies/royal-society-of-new-zealand/2020-fellowships-impact-evaluation/

Programme evaluation is a crucial part of making and keeping projects and services effective.

There are three main types of evaluation, and Pukeko Research has used them all.

Formative evaluation involves working alongside agencies as they set up new projects or services. The formative approach looks into every aspect of the set-up, including the goals, business plans, organisational matters, function and early results. A range of stakeholders and clients are interviewed. Formative recommendations may be made while the evaluation is continuing, through an interim process, to help adjust the settings of the service, or at the end. Key issues include whether the service has the capacity and capability to meet its goals. An example:

Formative evaluation of the Together Programme (2011-12)

Once a week after school, volunteer senior students from St Peter’s College, Epsom, hop into a minibus and drive down to Rongomai Primary School in Otara, where they work with and mentor students who have a family member in prison. The evaluation is being done in two parts – after six months and after twelve months. After six months we found that each agency or school had its own goals for the programme. It was difficult to set up individual mentoring and a group mentoring approach has evolved. The first evaluation is available here.

 Process evaluation measures the processes and outcomes of the project or service against its goals. It involves a mixture of qualitative analysis and quantitative results. Key issues include whether the service meets its goals, what goals it does meet, and whether it needs to be modified or restructured. An example of what we consider to be a best practice approach can be seen in the example below.

 Evaluation of the Hub Project fund for Ako Aotearoa

This process evaluation found that the hub project fund met a range of goals, include some unexpected ones. A number of recommendations were made to tighten up and improve elements of the service. Ako Aotearoa published the executive summary and the recommendations, plus the agency’s response to these. We consider this to be an example of good practice in working with evaluations.

Outcomes evaluation is the final approach to be considered here. This form uses qualitative and quantitative data to assess the success factors and impact of a programme or service. Often the issues are quite complex, especially as programmes often have perverse or unexpected effects, which may be positive, negative or both. The example below was of this kind.

 Evaluation of the Bright Futures Scholarships

 In this evaluation, the question arose of whether scholarship programmes were cutting across those already offered by tertiary organisations. We found that the Top Achiever Doctoral Scholarship was an important leader in the field, but it was obvious that many stakeholders did not see its values in that role. The scheme has since been disbanded.