Welcome to the AES Blog
Australasia has some excellent evaluators. More than that, we have an evaluation community full of ideas and a willingness to share. The AES has long provided a place for us to come together, at regional events and the annual conference, to develop our community together. Now we’re taking it online! The new AES blog will be a space for AES members – both new and experienced – to share their perspectives, reflecting on their theory...
If you have an idea, please contact us on blog@aes.asn.au. Please also view our blog guidelines.
From transactional to relational evaluation commissioning: the why, the what and the how
By Eleanor Williams and Skye Trudgett
Most people who have worked in evaluation have some kind of picture in their head of what it feels like when an evaluation partnership is going well. There are some common threads – a clear scope of work which has been well communicated, well-mapped aims and desired outcomes for the program, and easily-accessed data to allow for evaluative judgements to be made. The relationship between commissioner and evaluator might be described as professional but friendly with regular and frank communication flowing in both directions. Everyone involved in the evaluation – including those delivering and receiving the policy or program – have the opportunity to genuinely contribute to the evaluation and benefit from its findings.
If this dream state is relatively easy to describe, why don't we experience it more often? Why does the relationship between commissioner and evaluator more often feel transactional rather than relational? This blog seeks to briefly outline why it would be beneficial to address this issue, what it might look like if we did, and how we might get there.
Why is a relational approach helpful in evaluation commissioning?
If you stand right back, effective evaluation that is working as intended should help everyone involved in policies and programs to be curious, learn about how their policies and programs and being experienced and what outcomes they achieving. If this is done well, evaluation then supports decision-makers to understand these experiences and outcomes to inform their choices to keep, adapt or cease the policy or program going forward. This is unfortunately not a quick, easy or cheap thing to achieve. Good evaluation requires good planning, a learning mindset and trusted relationships that allow people to honestly share their experiences in a range of formats.
Evaluation as a discipline is often thought of in one of two ways: either more like auditing or more like continuous learning. From my experience, most people who choose to work in evaluation are hoping to work on the latter – they hope to be learning partners who help organisations better understand if programs are working as intended, for whom and in what circumstances. This is not to say that auditing does not play an important role in ensuring that organisations are accountable for their commitments. But it is to say that evaluators tend to aspire to going beyond this to answering a broader range of questions than 'did we do what we said we would do?' and helping organisations more deeply understand 'are the things we are doing working well and for whom?' This requires a trusted advisory relationship where everyone involved feels safe and supported.
What would a relational approach to evaluation commissioning look like in practice?
In my discussions with other evaluators, most people can describe how they would like to work with commissioners but more importantly what they do not want these relationships to look like. In some ways it is easier to start by describing what evaluators do not want! So let me briefly describe the transactional approach that many evaluators are keen to avoid.
A transactional approach at its core treats evaluation like it is an off-the-shelf product that can be scoped and priced with little input from the commissioners. A transactional procurement provides little information about the policy or program and hopes that the evaluator, even with limited data, will be able to magically provide an answer about whether the policy or program is working within a short period of time, and at a low, fixed price. The evaluator will only ask questions through a formal process and cannot speak directly to anyone involved in the policy or program which could interfere with the objective, merit-based procurement process.
However, the reality of a relational advisory relationship is that it is more like a good conversation or a dance. Under a relational approach, the commissioner might describe what they are hoping for, the advisor asks questions and offers up suggestions, and there is a detailed back and forth engagement until a clear and sensible scope is reached. This engagement would be respectful and acknowledge that both parties bring knowledge and expertise to the task which means that together they will produce a robust and context-appropriate approach to the evaluation. There may be some aspects of the task that need to emerge over time, once more information has been gathered from the other relevant parties to the evaluation – the local communities involved and any organisations involved in delivering services. It may take time to determine what is a reasonable timeframe for services to mature, change to occur and outcomes to emerge.
Given the many emergent aspects of the evaluation, any contractual arrangement would include flexibility or staging to allow for appropriate scoping as the evaluation develops. It would allow for subject-matter experts, including people with local community knowledge and experience in culturally appropriate data collection, to be engaged at key points in the process. This would also allow for the time required for data analysis to be determined once the exact nature and scope of the data became clear.
Nothing described as a relational approach to commissioning is necessarily precluded by government procurement arrangements but it is non-standard. Standard procurements approaches tend towards the transactional approach described above and for good reason – there are many goods and services that government procures that can absolutely be efficiently purchased in this way. Evaluation is not one of them.
How could we achieve more relational evaluation commissioning?
Moving to a more relationship evaluation approach largely falls to the commissioners of evaluation but there are parts of this transition that require both technical and cultural change from both sides.
Technical change – starting the 'boring revolution?'1
Government procurement is never the most scintillating topic but it is where we should start if we are going to achieve better and more relational evaluation commissioning. Those of us working inside government need to think harder about how we want our evaluations to look and our evaluation partners to engage. Then we need to work our what we need to do to enable a different outcome from our standard transactional ways. This means working closely with our procurement teams to work through what is possible and it means spending the time getting our documentation and processes right to communicate how we hope to work.
This is the 'boring revolution'! Taking our boring but important processes and working hard to make them deliver better engagements. In good news, it has been done before and there are examples of good practice to share. Three immediate suggestions emerged from my recent discussions on this topic which are all relatively easy to implement: share more information on the evaluation up front; consider how to reduce burden in responding to your request for quote processes; and consider breaking down engagements into stages to allow more flexibility in evaluation design and delivery.
The first opportunity is for government to be more transparent up-front which would allow evaluators to make a more informed decision about whether they are the right partner for the work at hand. There are many ways that this can be achieved which include providing more information within any request for quote and the price range for the procurement; and arranging one or more information sessions where evaluators can attend and ask questions about scope. These are small changes but they have the potential to save a lot of time for all parties involved and allow for a more transparent engagement up front.
The second opportunity is to provide a two-stage request for quote process to save providers from completing lots of documentation if they are unlikely to be successful. In practice this may look like a brief EoI process that helps the commissioner to shortlist providers so that only the more suitable providers complete a full scoping of the work. Many evaluators will be familiar with the high level of time and effort it takes to deliver a comprehensive project plan and quote and it can be very disappointing if this work is wasted.
And finally, a more structural option is to think about procuring each stage of work so it can be more accurately scoped and also allows for a change of provider, or additional evaluators to be brought in for key sections of the work. For many evaluations it is hard to know up front how much data collection will be required and how long analysis will take, particularly for multi-year or complex evaluation projects. By breaking the work down into multiple phases, it makes not only the scoping and quoting phase easier but allows for evaluators and commissioners to make more informed choices at each phase of the work, rather than be locked into a potentially unhappy partnership for multiple years.
These are only a few of the technical solutions that commissioners could consider and hopefully a boring revolution could flush out several more. But in order to achieve the technical change, we also need a cultural and mindset change about how to engage.
Cultural change – a collective endeavour
Finally, we all have an opportunity to change how we engage between the public sector, the evaluation community and academia to see ourselves more broadly as partners in trying to generate better evidence and improve policies, programs and services for the benefit of the community. This mindset shift calls for us to see each other as part of a broader team where we all play distinct roles. We all have a role to play in building the kinds of relationships that can support relational practice and not to see ourselves as being on different teams.
It is important to note that this can still be well managed within an ethical framework, where there are clear lines of accountability and independence that ensure evaluation findings are robust and manage issues around bias, values, respect and impartiality. The Australian Evaluation Society's Guidelines for Ethical Conduct of Evaluations, Code of Ethics and Cultural Safety Framework all help to describe good practice on this front.
Cultural change can be slow but we are hopeful that the level of interest in working new ways, and particularly the generous spirit of the evaluation community, can be harnessed to drive improvements in how we work. By prioritising trust and collaboration, we can support a better environment for collective learning and an evidence base that better serves our communities.
It is important to note that this can still be well managed within an ethical framework, where there are clear lines of accountability and independence that ensure evaluation findings are robust and manage issues around bias, values, respect and impartiality. The Australian Evaluation Society's Guidelines for Ethical Conduct of Evaluations, Code of Ethics and Cultural Safety Framework all help to describe good practice on this front.
Cultural change can be slow but we are hopeful that the level of interest in working new ways, and particularly the generous spirit of the evaluation community, can be harnessed to drive improvements in how we work. By prioritising trust and collaboration, we can support a better environment for collective learning and an evidence base that better serves our communities.
| Case study: Australian Human Rights Commission and the evaluation of the Wiyi Yani U Thangani (Women's Voices) Project |
| The Wiyi Yani U Thangani project is a multi-year initiative set out to capture what Aboriginal and Torres Strait Islander women and girls consider strengths, challenges and aspirations for change. In seeking an external partner to support the evaluation of Wiyi Yani U Thangani, the Australian Human Rights Commission (AHRC) took an innovative approach to procurement that allowed for multi-stage process to build a relationship between the AHRC and the evaluators that provides a model for others to consider. This relational approach began before the initial contract commenced in order to build trust and understanding between the commissioners and the evaluators. The five stages of engagement played out as follows: Pre-contract: Potential and existing partners were invited to a summit event with some scholarships provided to support attendance. Some very light touch volunteer support was provided by the potential partner at the event. This allowed for the evaluators to start to understand the approach, system and the support provided to women, girls and gender diverse Mob involved. Initial small contract for workshop: Evaluators were engaged under a small initial contract of less than $10k focused on a face-to-face workshop. This allowed the commissioners and evaluators to get to know each other and "test the vibe". The workshop involved sharing meals together and connecting. This engagement allowed the evaluation engagement to start with relationship-building. More substantial contract to develop evaluation framework: A subsequent contract was developed (less than $100k) which focused on key deliverables that prepared the understanding, monitoring, evaluation and learning (UMEL) framework and ensured a deep understanding of the work undertaken to date. Trust is now well established and the relationship began working to build the level of understanding of the evaluator. The commissioners shared deep knowing with the evaluators. Multi-year contract to establish change agenda and strategy: The final multi-year contract (under $200k) was developed to establish and implement the UMEL strategy and change agenda, grounded in the deep knowing build through relationships. This supported the ongoing work of the client to build continuous learning into their approach. Ongoing relationship: Continued support and connections between the commissioner and evaluator but now as people in relationship with trust and respect. This element is not paid work, it is relational and involves lateral love and care-giving. |
The blog builds on a panel discussion from the Australian Evaluation Conference on the topic of transactional to relational commissioning. Our sincere thanks to the other panellists from this discussion for their insightful input: George Argyrous from the Paul Ramsay Foundation, Kristy Hornby from Grosvenor Performance Group, Jo Norman from the Victorian Department of Health's Centre for Evaluation and Research Evidence and Luke Craven.
If you are interested in joining the boring revolution, a great place to start is by signing up for the Australian Public Sector Evaluation Network: https://www.aes.asn.au/special-interest-groups/aspen
About the authors:
Eleanor Williams is Managing Director, Australian Centre for Evaluation; Skye Trudgett is CEO/Founder, Kowa Collaboration and Adjunct Lecturer UNSW.
----------------
Endnote: 1. With thanks to Luke Craven for coining the term "boring revolution".