Share your ideas
AES Blog

Welcome to the AES Blog

Australasia has some excellent evaluators. More than that, we have an evaluation community full of ideas and a willingness to share. The AES has long provided a place for us to come together, at regional events and the annual conference, to develop our community together. Now we’re taking it online! The new AES blog will be a space for AES members – both new and experienced – to share their perspectives, reflecting on their theory... If you have an idea, please contact us on blog@aes.asn.au. Please also view our blog guidelines.

Everyday ethics challenges for evaluators

Ethics_blog_cover3

By Squirrel Main, Eleanor Williams, Kristy Hornby, Mandy Charman 

Everyday ethical challenges for evaluators

Ethics are part and parcel of any evaluation journey. Every evaluator, at some point, will face tricky situations where they'll need to balance ethical principles with practical decisions. While formal ethics processes usually revolve around consent and transparency in evaluation design, the real challenges often pop up beyond that. Evaluators frequently work in environments full of volatility, uncertainty, complexity, and ambiguity (known as 'VUCA' conditions), which means we need to stay flexible and responsive throughout the entire evaluation process to keep things on track.

A lot of the discussions about ethics in evaluation in Australia seem to centre on meeting the requirements of the National Statement on Ethical Conduct in Human Research and the needs of Human Research Ethics Committees (HRECs). But the less glamorous, day-to-day ethical challenges we face when we're actually designing, delivering, and reporting on evaluations tend to get less attention. This post provides early thinking on a basic framework for evaluators who want to think about their own ethical decision-making and kickstart a conversation about whether more guidance, advice, or support is needed.

Why do ethical challenges crop up?

We like to think of high-quality evaluation as independent, unbiased, and rigorous. But let's be honest – it's not always that simple. Evaluations are often participatory, they require us to listen to a lot of different voices, and sometimes we need to juggle the interests of various stakeholders. It's not unusual to find ourselves in situations where our independence is tested, where we have to decide whose perspective to prioritise, or where we debate what truly makes for 'rigorous' evidence.

These situations often lead to ethical decisions, but one of the biggest challenges evaluators seem to face is pressure from commissioners who want to manipulate the evaluation process or results. That's where things can get really sticky. 

How common are ethical challenges?

At the 2024 Australian Evaluation Society Conference, the authors ran a quick poll during a panel discussion, asking attendees how often they run into ethical issues. Seventy-one people responded, and the results were interesting. Good news first – most people (55%) said they rarely come across ethical challenges (1-5 times a year). However, 45% of respondents admitted they encounter them more frequently, with four people saying they deal with ethical dilemmas more than 20 times a year! 


We also asked people to share the kinds of ethical issues they've faced (anonymously, of course) and how they handled them. Here's what we learned:

  • Engaging with First Nations communities
    A recurring theme was the lack of meaningful engagement with First Nations people in evaluations. For ethical evaluation, it's crucial to include their perspectives and ensure the work brings real benefits to their communities.
  • Integrity and accuracy
    Misrepresenting findings, cherry-picking data, or pushing for only positive outcomes can seriously damage the credibility of an evaluation. Sticking to your interpretation of the facts – even when it's uncomfortable – is key to ethical practice.
  • Power imbalances
    Power dynamics often skew evaluation results, especially when stakeholders try to pressure evaluators into changing findings. Maintaining independence and resisting external pressure is crucial for ethical evaluation.
  • Transparency and stakeholder involvement
    Evaluations need to be transparent, with stakeholders kept informed and involved in decision-making. This can help avoid ethical problems like the suppression of negative findings or selective reporting.


The responses to these challenges were just as varied as the challenges themselves. Some people worked through them by discussing or negotiating outcomes. Others built trust with stakeholders or escalated the issue to senior leaders. A few even walked away from the work altogether! Our favourite response, though, was someone who said they coped by 'writing cathartic ranting blogs under an anonymous pen name.' This variety shows how wide-ranging ethical challenges can be, as are our responses as evaluators. 

How can we define the severity of ethical challenges?

Ethical challenges come in all shapes and sizes, and they don't fit neatly into categories. That said, it can be helpful to think about them in terms of severity. During the AES panel, we cheekily broke things down into three levels of severity — 'the good, the bad, and the ugly.' Based on the responses from the audience, we have now expanded that to four levels (with more professional names):

  • Minor
    Low-impact breaches that are unintentional and isolated, with full transparency.
    Example: Minor oversights in reporting or small errors that do not materially affect the outcome or requests for small wording changes from evaluation commissioners.
  • Moderate
    Negligent or unintentional issues with moderate impact and partially transparent processes.
    Example: Excluding a group of stakeholders from consultations by accident but addressing the issue later.
  • Major
    These breaches might be either intentional or negligent, and they can range from isolated incidents to systemic issues.
    Example: Cherry-picking evaluation participants or changing findings to suit political needs.
  • Critical
    High-impact breaches that are systemic and intentional, often with opaque processes.
    Example: Manipulating data to falsely claim a program's success or to cover up program failure.


While this blog will not dive into fully defining each level, the examples above show what might fall into each category and the types of responses these challenges warrant. 

How should we respond to ethical callenges?

Now that we've got our four categories, let's talk about how to respond to these ethical dilemmas. Based on what came up at the AES Conference, here's a quick guide to the kinds of responses that might be appropriate for each level:

Minor
These are the smaller, unintentional breaches. Think of things like a minor conflict of interest or a few selective suggestions for who should be consulted or interviewed. These can usually be resolved through an open conversation with the commissioner to iron things out.

Moderate
Here, we're talking about issues with a bigger impact, but still mostly unintentional. Maybe the scope of the evaluation was defined in a way that shines a more positive light on the results, or perhaps a vulnerable group was unintentionally excluded. In these cases, it might be time to escalate the issue to someone higher up or ask for a written explanation of any requested changes to keep things transparent.

Major
These are the kinds of breaches that involve deliberate manipulation or negligence – things like cherry-picking participants or changing findings to fit political needs. In these cases, it is essential to stand firm and not allow any changes that would manipulate the evaluation findings or outcome. If things get too sticky, it might even be time to consider terminating the contract.

Critical

These are the really serious breaches with high impacts, like manipulating data to falsely claim success for a program or to cover up program failure. When it comes to these situations, it is probably best to walk away from the project and, in extreme cases, consider taking legal action. 

How do we refine our framework? 

To get better at dealing with these challenges, it is useful to think about the different dimensions of ethical issues. We have started to play around with some dimensions that might help categorise the severity of these problems:

Impact on stakeholders
Who is affected, and how bad is it? For example, if the issue has the potential to harm specific populations, like First Nations people, or breaches the rights of program participants, that's clearly high impact. If only a small group is affected, the impact might be moderate, and if there's little visible harm, it could be low impact.

Intent and motivation
Was the breach intentional, negligent, or accidental? If there's clear intent to manipulate findings, that is obviously more severe. If it was unintentional but careless, like forgetting to seek ethical clearance, it is still an issue, but less so.

Scope of the breach
Is the problem systemic, affecting the whole evaluation, or is it a one-off issue? Systemic issues, like pressure from higher-ups to avoid awkward findings across multiple evaluations, are more serious than isolated data collection errors.

Transparency and accountability

How open is the process? Is there full transparency and accountability, or are key details hidden from stakeholders? The less transparency, the bigger the ethical issue. 

Want to join the conversation? 

If this topic resonates with you, or you've got feedback on the categories or examples we've shared, Grosvenor is looking to dig deeper into the frequency and severity of these ethical issues, in partnership with the Australian Centre for Evaluation, The University of Melbourne's Assessment and Evaluation Research Centre, the Centre for Excellence in Child and Family Welfare's Outcomes Practice Evidence Network (OPEN) and the Paul Ramsay Foundation. We are keen to hear more about what people are experiencing out in the field. If you'd like to participate in a follow-up survey or case study interview, we would love you to complete this survey:https://www.surveymonkey.com/r/everydayethics2025
We also need your help in distributing this survey as far and wide across the Australian evaluation community as possible – please share amongst your networks and help us get the word about this important exploratory research out there! Survey closes end of May 2025. 

-------------------------------------
The blog builds on a panel discussion from the Australian Evaluation Conference on the topic of 'Navigating ethics dilemmas when evaluating for government: the good, the bad and the ugly'. This blog is the product of panellists from this discussion: Eleanor Williams, from the Australian Centre for Evaluation, Kristy Hornby from Grosvenor Performance Group, Mandy Charman from the Outcomes, Practice and Evidence Network as well as Squirrel Main from the Paul Ramsay Foundation and Amy Gullickson from the University of Melbourne who have been passionate contributors to subsequent conversations.

The AES produced Guidelines for the Ethical Conduct of Evaluations and an AES Code of Ethics which can be accessed at https://www.aes.asn.au/evaluation-resources/ethical-guidelines

If you are interested in further discussions around evaluating for government, a great place to start is by signing up for the Australian Public Sector Evaluation Network: https://www.aes.asn.au/special-interest-groups/aspen

The New South Wales Public Sector Commission also hosts an Ethical Behaviour Community of Practice: https://www.comprac.nsw.gov.au/communities-of-practice/ethical-behaviour

Recommended further reading

Creating an enabling environment for culturally re...

ABOUT US   |    CONTACT US   |    LEGALS
Search site
© 2023 Australian Evaluation Society Limited 
ABN 13 886 280 969 ACN 606 044 624
Address: 425 Smith Street, Fitzroy, Victoria, 3065, Australia

We acknowledge the Australian Aboriginal and Torres Strait Islander peoples of this nation. We acknowledge the Traditional Custodians of the lands in which we conduct our business. We pay our respects to ancestors and Elders, past and present. We are committed to honouring Australian Aboriginal and Torres Strait Islander peoples’ unique cultural and spiritual relationships to the land, waters and seas and their rich contribution to society.