Date and time: Wednesday 12th June 2019, 9am to 5pm (registration from 8.30am)
Title: Evaluation and Value for Money: An approach using rubrics and mixed methods
Location: Menzies School of Health Research, Charles Darwin University, Casuarina campus, Ellengowan Drive, Casuarina NT 0810
Facilitators: Julian King & Kate McKegg – Kinnect group, NZ
Register online by: 4 June 2019
Fees (GST inclusive): Members $440, Non-members $605, Student member $220, Student non-member $302.50
An AES and Menzies School of Health Research partnership event
About the workshop
It is important for good resource allocation, accountability, learning and improvement that policies, programs and other initiatives undergo rigorous evaluations of value for money (VfM). Many evaluators, however, lack training and experience in this area. This workshop offers evaluators a set of techniques to address that gap. These techniques build on approaches that will already be familiar to many evaluators, although there are also many evaluators and commissioners for whom these techniques will also be new. Drawing on a decade of the Kinnect group’s experience using rubrics for evaluation, together with Julian King’s doctoral research on evaluation and value for money, this workshop offers an approach to VfM that is rigorous, built on sound theory, and practical to use.
This participatory workshop provides practical guidance, underpinned by a sound theoretical framework, for evaluating VfM. The approach focuses on unpacking a process of explicit evaluative reasoning (using rubrics) that can be applied to VfM or any other evaluation, and the use of mixed methods (including economic evaluation where feasible and appropriate).
A sequence of steps will be shared, that when followed, help evaluators and commissioners develop and use context-specific definitions of good performance and VfM. These definitions provide a systematic framework to ensure the evaluation is aligned with the policy, program or initiative’s design, collects and analyses the right kind of evidence, draws sound conclusions, and answers VfM questions. Exercises and examples will be shared during the workshop.
The workshop includes an introductory overview of economic methods of evaluation (e.g. cost-benefit analysis), including considerations for determining when it is appropriate and feasible to use economic methods in a VfM assessment. It will not provide detailed instruction in the design and implementation of economic evaluations. There are courses already on offer that focus on economic methods of evaluation..
Learning Outcomes and Objectives
Participants will learn:
- How to frame an evaluative question about VfM;
- How to develop criteria (dimensions of VfM), and standards (levels of VfM) to provide a transparent basis for making evaluative judgements;
- How to combine multiple sources of evidence including quantitative and qualitative evidence, to support nuanced evaluative judgements;
- How to incorporate economic evaluation within a VfM framework (where feasible and appropriate), without limiting the analysis to economic methods alone;
- How to interpret mixed methods evidence, using rubrics, to make sound evaluative judgements; and
- How to present a clear and robust performance story, guided by the criteria and standards.
This workshop aligns with competencies in the AES Evaluator’s Professional Learning Competency Framework. The identified domains are:
- Domain 2 – Evaluation theory
- Domain 3 – Culture, stakeholders and context
- Domain 4 – Research methods and systematic inquiry
- Domain 7 – Evaluation Activities
Who should attend?
The training is designed for evaluators and commissioners of evaluation at all levels, from novice to expert. The training is suitable for evaluators and commissioners who want to learn about rubrics generally, as well as those who want to learn how to apply rubrics to value for money specifically.
About the facilitators
Julian King and Kate McKegg are highly experienced evaluators, presenters and workshop facilitators. The content in this workshop, updated for 2019, draws on a decade of training workshops on evaluative reasoning and rubrics. Similar VfM training has recently been delivered at the European Evaluation Society (EES) Conference 2018, a workshop for UK Evaluation Society members in 2018, Africa's Financial Sector Development Network Conference in 2017, a workshop for Australian Evaluation Society (AES) members at the Northern Institute, 2017, and the Aotearoa New Zealand Evaluation Association (ANZEA) Conference in 2016. An introduction to rubrics was presented by the Kinnect Group at several international conferences in 2018, including the Australasian Evaluation Society Conference in Tasmania.
Julian King is the director of Julian King & Associates Ltd and a member of the Kinnect Group. His PhD, titled Evaluation and Value for Money, develops a model for combining evaluative reasoning with economic evaluation. The model has been used globally to evaluate complex and hard-to-measure programs and policy reforms. It is taught as part of the University of Melbourne’s online Master of Evaluation program. In the development sector Julian teamed up with Oxford Policy Management to write a VfM Guide1 and has applied the approach in diverse settings spanning public financial management, governance, international trade, market development, climate change, health, education, and social development, among others. Closer to home, the Kinnect group has used the approach in large-scale evaluations such as the Māori and Pacific Education Initiative and the Sustainable Farming Fund. Publications and examples are available at www.julianking.co.nz/vfi
Kate McKegg is the director of The Knowledge Institute Ltd and a member of the Kinnect Group as well as an indigenous led collective Tuakana Teina, based in the Waikato region of New Zealand. Kate has worked in evaluation, evaluation capacity building, research, policy and public sector management since the late 1980s.
Kate has been applying rubrics in her evaluation work for many years, in government and non-government settings, including complex, developmental and indigenous contexts. With the other members of the Kinnect Group, a guide to evaluation was developed in 2018. This guide highlights the use of rubrics for evaluative reasoning and is available on the Kinnect website http://kinnect.co.nz/evaluation-building-blocks-a-guide/
1 King, J. & OPM (2018). OPM’s approach to assessing value for money – a guide. Oxford: Oxford Policy Management Ltd.