Date and time: Thursday 20th February 2020, 9am to 5pm (registration from 8.30am)
Title: Evaluation and Value for Money: An approach using rubrics and mixed methods
Location: Canberra CBD
Facilitators: Julian King
Register online by: 13 February 2020
Fees (GST inclusive): Members $484, Non-members $665, Student member $260, Student non-member $350
It is important for good resource allocation, accountability, learning and improvement that policies, programs and other initiatives undergo rigorous evaluations of value for money (VfM). Many evaluators, however, lack training and experience in this area. This workshop offers evaluators a set of techniques to address that gap. These techniques build on approaches that will already be familiar to many evaluators, although there are also many evaluators and commissioners for whom these techniques will also be new. Drawing on a decade of the Kinnect group’s experience using rubrics for evaluation, together with Julian King’s doctoral research on evaluation and value for money, this workshop offers an approach to VfM that is rigorous, built on sound theory, and practical to use.
This participatory workshop provides practical guidance, underpinned by a sound theoretical framework, for evaluating VfM. The approach focuses on unpacking a process of explicit evaluative reasoning (using rubrics) that can be applied to VfM or any other evaluation, and the use of mixed methods (including economic evaluation where feasible and appropriate).
A sequence of steps will be shared, that when followed, help evaluators and commissioners develop and use context-specific definitions of good performance and VfM. These definitions provide a systematic framework to ensure the evaluation is aligned with the policy, program or initiative’s design, collects and analyses the right kind of evidence, draws sound conclusions, and answers VfM questions. Exercises and examples will be shared during the workshop.
The workshop includes an introductory overview of economic methods of evaluation (e.g. cost-benefit analysis), including considerations for determining when it is appropriate and feasible to use economic methods in a VfM assessment. It will not provide detailed instruction in the design and implementation of economic evaluations. There are courses already on offer that focus on economic methods of evaluation..
Learning Outcomes and Objectives
Participants will learn:
- How to frame an evaluative question about VfM;
- How to develop criteria (dimensions of VfM), and standards (levels of VfM) to provide a transparent basis for making evaluative judgements;
- How to combine multiple sources of evidence including quantitative and qualitative evidence, to support nuanced evaluative judgements;
- How to incorporate economic evaluation within a VfM framework (where feasible and appropriate), without limiting the analysis to economic methods alone;
- How to interpret mixed methods evidence, using rubrics, to make sound evaluative judgements; and
- How to present a clear and robust performance story, guided by the criteria and standards.
This workshop aligns with competencies in the AES Evaluator’s Professional Learning Competency Framework. The identified domains are:
- Domain 2 – Evaluation theory
- Domain 3 – Culture, stakeholders and context
- Domain 4 – Research methods and systematic inquiry
- Domain 7 – Evaluation Activities
Who should attend?
The training is designed for evaluators and commissioners of evaluation at all levels, from novice to expert. The training is suitable for evaluators and commissioners who want to learn about rubrics generally, as well as those who want to learn how to apply rubrics to value for money specifically.
About the facilitator
Julian King is the director of Julian King & Associates Ltd and a member of the Kinnect Group. His PhD, titled Evaluation and Value for Money, develops a model for combining evaluative reasoning with economic evaluation. The model has been used globally to evaluate complex and hard-to-measure programs and policy reforms. It is taught as part of the University of Melbourne’s online Master of Evaluation program. In the development sector Julian teamed up with Oxford Policy Management to write a VfM Guide1 and has applied the approach in diverse settings spanning public financial management, governance, international trade, market development, climate change, health, education, and social development, among others. Closer to home, the Kinnect group has used the approach in large-scale evaluations such as the Māori and Pacific Education Initiative and the Sustainable Farming Fund. Publications and examples are available at www.julianking.co.nz/vfi
1 King, J. & OPM (2018). OPM’s approach to assessing value for money – a guide. Oxford: Oxford Policy Management Ltd.