by Sarah Oxford

The Australian Evaluation Society recently held FestEVAL21, a week of online activities to celebrate evaluation. This year’s event included more than 1,100 registrants with attendance of about 3,000 people across 23 sessions. The engaging session topics ranged from themes addressing practical application such as capability and capacity building in evaluation to encouraging evaluator reflection and behaviour change. FestEVAL gave us ample opportunities to begin this work by platforming experts with diverse life experiences in evaluation.

The topics that caught my interest were the ones that made me look inwards and ask, what can I do to lead evaluations with inclusive and anti-racist practices? As evaluators a lot of our work involves exploring human behaviour change. But what happens when our behaviour and the systems that we work in and through need to change? How can we improve ourselves, our evaluation practices and the systems at large?

A provocation to decolonise evaluation

During the FestEVAL launch, First Nations woman and evaluator Skye Trudgett discussed that the space in evaluation between theoretical and practical application is where systemic racism rears its ugly head. Skye explained that the commonly discussed importance of self-determination and applying First Nations ways of “knowing, doing and being” initiated in the early stages of evaluation with goodwill frequently falls flat when the actual work begins.

Why is it so outrageous that an Aboriginal woman would stand up today and say my path here so far has been really hard and lonely and that is not good enough?
– Skye Trudgett 

From Skye's experiences, racism arises in this space when there is a misalignment of values, which are then subjected to processes and systems underpinned by the dominant cultural hegemony. Here, First Nations ideas and practices are driven to align with methodological concepts created and sustained by the dominant White culture. As such, she argued, there is a continual push for First Nations people to do evaluation in the “right or White way”.

Why is it so outrageous that I should suggest that patriarchal colonialism continues to show its face within evaluations of our First Nations practice?
– Skye Trudgett

Building on her experiences conducting evaluations within a system that prioritises the dominant White culture, she made a strong call to action, asking those who are uncomfortable to ask themselves ‘why’ and to reflect on their own positionality. Next, she asked those interested in eradicating racism to dig into that interest because interest is a core ingredient to social change.

In addition to challenging evaluators to be better, Skye suggested practical and tested ways to tighten the space between theory and application.

A good place to begin education and engagement is the AES First Nations Cultural Safety Framework. This important document was launched during FestEVAL. The authors are well known facilitators Sharon Gollan and Kathleen Stacey; the project was led by the AES Indigenous Culture and Diversity Committee with a cross-cutting First Nations project team. The Framework will provide crucial guidance to evaluators seeking to undertake their work in a manner that is culturally safe across all phases of the evaluation process. The  AES First Nations Cultural Safety Framework can be viewed via https://www.aes.asn.au/first-nations-evaluators

Other suggestions included:  

• Listening
• Employing people from community who hold significant knowledge about local processes and contacts and pay them the same as non-local consultants
• Sharing power and leaving decision making with the community
• Spending time in the community and focusing on building social relationships, not outputs

In addition to extensive efforts in community, she highlighted exemplary non-racist and Indigenous-focused work for AES members to learn from:

• National Centre of Indigenous Excellence for applying a flexible approach
• The NSW Department of Community and Justice for establishing an Aboriginal Knowledge Circle
• Seed Mob for creating a “knowing doing being” impact framework and demonstrating what success looks like.

An open discussion about ‘the space’ and allyship

The session Re-imagining multicultural evaluation - taking our practice to the next level facilitated by Eva Sarr, Professor Rodney Hopson, Dr Katrina Bledsoe and Dr Jane Davidson thematically built on Skye’s provocation. This session, designed as an interactive open discussion, encouraged participants interested in engendering social change to engage in dialogue and share their experiences and ideas.

The session focused on four key questions:

• What are the strengths of multicultural evaluation in the areas where you work?
• What are the challenges of multicultural evaluation in the areas where you work?
• What are our aspirations for multicultural evaluation?
• What do you think you need to equip you to undertake evaluations that authentically respond to local cultural contexts?

Through dialogue, the panellists defined multicultural evaluation as a network of like-minded individuals who integrate cultural consideration and context into theory, measures and practice, and as a result, the evaluation is culturally responsive and promotes equity, diversity and inclusion. Participants then joined the discussion.

I think one key thing for me is that multicultural evaluation needs to not just use a good, culturally responsive approach/process, but what is valuable to and valued by different communities need to be infused into what and how we evaluate.
- Panellist

Participants openly discussed challenges when conducting evaluations. One example included negotiating conflicting goals between the funder interested in quick outputs and the evaluator wanting to apply participatory approaches to better serve the community.

It’s often a trade-off of the priorities of the funder and the community being evaluated. Ensuring that the values of the evaluated community are prioritised or at least equally considered is important - often doesn't happen!
– Participant

Donors' demands for quick results contrast with equitable approaches. Participants argued the importance of setting expectations and pushing back when donors' needs do not align with community values.

It's about setting expectations - the pressure from decision makers above for quick evaluations/timeframes are sometimes based on previous experience or a lack of push back (sometimes from fear of repercussions/reputational risk). In the space of behavioural science in the justice sector in [location], we've been pushing back on this by showing the value of observations, and the importance of spending time and investment to be immersed in the communities that we are wanting to work with to truly understand their context. We try to get those on the ground who know their communities best to be part of the engagement and co-design of solutions to ensure that those voices (conflicting or not) are not lost.
– Participant

Another issue addressed was conducting evaluations in highly diverse settings where groups of people have conflicting needs. Fostering equity across and among people and groups creates challenges for many evaluators.

One of the most challenging and important elements in multicultural evaluation must be the recognition of intersectionality - gender, sexuality, age, disability, economic status - all of which are so challenging in working on communication and data collection in the COVID pandemic, etc. Important not to assume “community leaders” are always reflecting the experience of the community “led” - important in relation to gender and age at the very least.
– Participant

To assist with this challenge, one participant suggested Mathea Roorda’s tool that looks at the various interest groups participating in the evaluation. Other suggestions from the panellists included using the AES First Nations Cultural Safety Framework, finding culturally appropriate allies, adopting the principle of ‘do no harm’ and paying attention to nuance (e.g. generational family change, intersections across identity, diversity within a community). While making these suggestions, panellists openly acknowledged that trying to understand the complexity of context, tension and nuance is a significant task and very challenging, and there is much room for future conversations and learnings on the subject.

Many strategies were discussed as pathways forward for re-imaging multicultural evaluation. For example, a participant suggested focusing on learning local value systems and working to understand the evaluation through the lens of the local communities, as well as focusing on listening, sharing and making decisions with intention. Intersectionality and equity were also discussed. Intersectionality refers to the ways in which different aspects of a person’s identity can expose them to overlapping forms of discrimination and marginalisation. Participants suggested intersectionality should be applied at the fore of evaluation. Equity is when fairness is considered, and conditions allow each individual to reach their full potential. For example, the evaluator should use data collection methods to enable everyone to contribute equally if literacy levels vary among participants. All themes discussed are highlighted below.

I was drawn to these sessions because, as a white person, I often reflect on what good ‘allyship’ means and my privilege as an evaluator who reports and presents findings on people’s experiences. These sessions reminded me that this is hard but essential work, and as evaluators, we must keep these conversations going and take the time to commit to self-reflection and improving our evaluation practices.

----------------

Dr Sarah Oxford is a Senior Research Officer (Evaluation) at the Institute for Safety, Compensation and Recovery Research (ISCRR), Monash University

• @SarahWOxford
• https://www.linkedin.com/in/sarahoxford/