Evaluation Study or Project Award

Queensland Government’s Office of the Commonwealth Games, Department of Innovation, Tourism Industry Development, and the Commonwealth Games; and the evaluation team of Mark Douglas, Robert Grimshaw, Nicolette Pavlovski, Sean Conway, Kelly Reynolds, Joanne Ryan and Meghan Purcell – for the Evaluation and Monitoring Framework for the Embracing 2018 Legacy Program

The Queensland Government’s Embracing 2018 Legacy Program aims to ensure the Queensland community realised lasting benefits from hosting the 2018 Commonwealth Games. The reach of this event extends beyond the Games host and event cities to provide measurable outcomes for Queensland and Australia. An Evaluation and Monitoring framework for this program has evolved since 2013 and is designed to guide implementation of the Embracing 2018 Legacy Program and assess its outcomes over a 10-year period.

The assessors were impressed by the evaluation’s sound use of evaluation theory and approaches, including the ability to incorporate emergent findings into a results’ framework and ensure ongoing connections between projects. There was evidence of strong and sustainable connections being developed; for example, with the evaluation team building on work started in Glasgow with the 2014 Commonwealth Games, and their commitment to developing a framework that can be used by future hosts of the Commonwealth Games. The assessors noted the evaluation team’s commitment to publicly share the methods behind their framework. 

Evaluation Policy and Systems Award

The Victorian Department of Health and Human Services (DHHS) Centre for Evaluation and Research

The DHHS Centre for Evaluation and Research led the development and implementation of evaluation policy and systems in this large Victorian state government department. It played a pivotal role in transforming the evaluation culture of the department through policy guidance, advice and support, design and delivery, training and knowledge translation.

The Centre for Evaluation and Research demonstrated excellence in applying theory-based principles and approaches to developing and implementing evaluation policy and systems. The judges were impressed with the high-quality evaluation resources and pragmatic support that departmental staff have access to, including tools and tailored advice. The nomination included good evidence of stakeholder satisfaction with the team’s work. The judges were also impressed with the significant contribution of the team to transferring knowledge across the department and broader community and public sectors. Overall, the judges considered this Award winner as an exemplary case of evaluation policy and systems contributing to service delivery and improving community wellbeing, including among those most vulnerable to the risks of poorer outcomes. 

Community Development Award 

Palmerston/ Tiwi Island Communities for Children (C4C) Participatory Evaluation conducted by the Communities for Children (C4C) Local Committee, Pandanus Evaluation & Planning Services (Nea Harrison) and the Australian Red Cross (Rachel Dunne) 

The Award judges noted that the work was well-thought through, extremely thorough and comprehensive. A local committee, the elders and young women were all actively involved in the project in a strong collaboration with the facilitator of the evaluation. The nomination indicated an understanding of how an evaluation design contributes to community development goals and how the use of community development processes enables the accomplishment of the evaluation.

Areas that stand out as exemplary and examples of good evaluative practice in this evaluation include: 
- culturally appropriate evaluation design and methodology
- strong community engagement at all stages of the evaluation from inception to conclusion, including closure and reporting back to the community 
- a developmental approach working sensitively and in sympathy with the local community 
- outcomes resulting in sustainable benefits for local participants (e.g. capacity building) 
- use of advanced techniques –logic framework, quality rubrics, the community report.

Indigenous Evaluation Award

Aboriginal Evidence Building Partnership Pilot
ARTD Consultants Simon Jordan, Sue Leahy, Ruby Leahy Gatfield, Kieran Sobels, Christine Eastman, Stephanie Quail, Imogen Williams, Holly Kovac
Aboriginal Evidence Building Team, Their Futures Matter: Nattlie Smith, Sharon Macleod
Coffs Harbour Aboriginal Community Centre Inc (Abcare): Garry Mathews, Greg Bennett, Trent Matthews
Tirkandi Inaburra Cultural and Development Centre: Matt Watts, Damien Thorne, Beverley Tucker, Department of Communities, and Justice Troy Mott, Department of Education

This evaluation was conducted by a team committed to enabling Indigenous self-determination and empowerment from the outset. This was due in large part to the Indigenous leadership within the team combined with the skill and experience of the other team members in this context.

The evaluation demonstrated how collaborative and participatory approaches, together with a strong partnership model, enabled and paid respect to the roles and cultural imperatives of two organisational partners involved, as well as to the intended objectives of the NSW government agency involved. Overall, the judges concluded that this Award winner has made a strategic contribution to fostering a culture of evidence building and evaluation capacity for the ongoing monitoring and review of the Aboriginal services sector in NSW. The judges are confident that the work will lead to increased benefits for the two organisations’ service recipients, who are Aboriginal families whose children are at risk of removal, and Aboriginal boys aged 12–15 years who are seeking to develop their resilience and life skills. 

Evaluation Publication Award (Caulley Tulloch Award)

Samantha Abbato – for 'The case for evaluating process and worth: evaluation of a programme for carers and people with dementia' 

This is a book chapter published in the most recent volume of the prestigious series Advances in Program Evaluation, edited by Trisha Greenhalgh & Saville Kushner. Its thesis is the utility of the case study approach as a major component of a mixed-methods evaluation. What makes the chapter worthy of the award is the author's careful analysis and demonstration of the role of case studies in mixed methods evaluations, including the way she contrasts them with the quantitative methods more frequently used in the sector covered by the project (community health/dementia). This chapter will be particularly valuable for people new to evaluation, or to those coming from a quantitative background who wish to gain an understanding of the role of case study research in evaluation. The nomination explicitly and convincingly addresses the specified filters and the seven specified criteria of excellence.  

Public Sector Evaluation Award

DFAT Office of Development Effectiveness – for the Evaluation of the Australian NGO Cooperation Program 

This evaluation displays a highly professional and rigorous approach to public sector evaluation planning, and subsequent practice in data collection, analysis and synthesis, as well as determination of findings and recommendations. It is notable in the participatory manner in which it included industry stakeholders in the major stages of conduct of the evaluation and built acceptance and ownership in both its process and product, including useful recommendations. Importantly, with respect to the Award's assessment criteria, these qualities gave considerable impetus to new policy and program initiatives in line with the recommendations made.

Emerging New Talent Award

Alexandra (Lex) Ellinson

The judges were impressed by the high level of professional competence achieved by Lex, which has been developed within four years of evaluation experience. She holds a Senior Consultant role in ARTD and is an active contributor to the AES as a member and secretary of the Advocacy and Alliances Committee, as well as a member of the Realist SIG.

In the projects Lex has worked on or led, she has demonstrated a breadth of knowledge and skill essential for a high quality evaluator. Her evaluation reports demonstrate ability to engage with different methodologies including quasi-experimental designs, cost analysis, qualitative methods and monitoring methodologies. Lex has colleagues who seek her out for discussions about ethics, particularly in the areas of cultural sensitivity and people experiencing trauma. The nomination provides evidence of exemplary stakeholder engagement, particularly in areas involving youth and very positive working relationships with Aboriginal community elders.

Outstanding Contribution to Evaluation Award

Jess Dart

Jess is a recognised leader in evaluation with over 25 years of experience in the collaborative design and evaluation of programs that seek to bring about a more equitable and just society. The judges were impressed by this nominee’s sustained application of authentic inclusivity, and her high level of ethical standards so clearly evidenced through her practice. The judges noted this nominee’s skill in being able to combine deep evaluation knowledge and theoretical understandings with straightforward communication. She has undertaken more than 30 external evaluations and overseen over 120 evaluations in countries around the world. The judges noted the high level of professionalism and stakeholder satisfaction evidenced in this nominee’s work. As an ‘evaluation entrepreneur’, the judges were impressed by this nominee’s capacity to constantly scan the horizon for where evaluation is headed, and to create fresh approaches and techniques. The judges were impressed by her pioneering work on no less than five innovative approaches.

The judges were impressed by this nominee’s other contributions to evaluation knowledge. She is a contributor to evaluation textbooks, an author of refereed journal articles including publications in the American Evaluation Journal and New Directions in Evaluation, and a prolific trainer who has reached more than 1,000 participants, including many AES members.

Jess has been an AES member since 1997 and a Board member since 2014 in the role of treasurer; and was involved in the aes18 International Evaluation Conference as convenor. The judges acknowledged this nominee’s significant contribution to the AES.