by Renée Madsen
Regionally-based evaluators – those living and working outside major cities – are a vital part of the evaluation ecosystem. They bring the benefits of evaluation to areas where essential services can be thinly spread and under pressure to deliver the best possible results with limited resources. Regionally-based evaluators ensure that evaluation is accessible to those who would not otherwise be able to engage with evaluation expertise, and we represent the profession in areas it would not otherwise reach.
With increasing numbers of people moving from capital cities to regional areas, regional evaluators with their unique perspectives and experiences will become ever more important in pushing the boundaries of what evaluation can achieve, and ensuring that the profession adds value for all Australians, regardless of where they live.
However, at the moment regionally-based evaluators make up less than 15% of all AES members. What does this mean for our profession’s ability to give regional communities the same access to evaluation as major cities? And how can we increase the capacity of our profession to truly understand and involve regional communities in evaluation?
I’m an evaluator based in Townsville, North Queensland. I have been evaluating a diverse range of programs for 20 years, and I’ve been an AES member for almost 10 of those. I believe in building the strengths of regional areas, and the power of evaluation to create positive change, and like many regional evaluators, I love combining the two.
Evaluating a strategic Townsville beach. Source: Renée Madsen
How many regional AES members are there?
For the purposes of this article, the definition of regional is anywhere outside a major city, as defined by the ABS Remoteness Structure. The map below shows the number of regional AES members in Australia.
Number of regional AES members. The ABS Remoteness Structure considers all areas of the Northern Territory and Tasmania as regional. Source: Renée Madsen and Michelle Wightwick.
Regionally-based evaluators are only a small proportion of AES members. For example, Victoria has the largest number of AES members at 356, but less than 10% of these members are based outside a major city. This trend is repeated across all states and territories (except NT and Tas). Overall, only 14.6% of all AES members are regional.
With such a small number of us living outside major cities, can our profession give regional communities the same access to and standard of evaluation as major cities? Given the cost of travel and the need to understand local context, can regional areas enjoy equitable access to evaluation expertise? And do we, as a profession, have the capacity and the commitment to truly empower and involve regional communities in evaluation?
The value of regionally-based evaluators
Our numbers may be small, but our contribution is big! Here are just a few ways that regional evaluators add value to the profession:
We go the extra mile - sometimes literally.
Conference attendance and professional development can involve multi-leg flights and/or long car journeys, and all the expense that comes with those. Without the benefit of local networking events, we take the initiative and reach out to find other practitioners using social media or Google, or ask someone to send us the notes from seminars that we couldn’t attend before everything went online due to COVID-19. (Shoutout to the AES Qld Regional Committee for sending me seminar notes over the years!)
We’re also highly driven to learn extra skills to expand our toolkit. With limited access to technical specialists, who are generally based in major cities, we often become a ‘jack of all trades’, choosing to learn specialist techniques and approaches ourselves so we can implement them with regional communities that would not receive the benefit of them otherwise.
We know how things are done in our local community.
Every evaluator knows that stakeholder engagement can make or break an evaluation. Regional practitioners understand the local stakeholders and their relationships, where the landmines are, and the likely touchpoints for collaboration. Mistrust of government programs - and anyone associated with them - runs deep in some places. This is particularly relevant for evaluators, as much of our work is evaluating government-funded programs.
More Aboriginal and Torres Strait Islander people live in regional areas than capital cities, and locally-based evaluators are more likely to have helpful contacts, understand the lay of the land, and know how our local First Nations brothers and sisters prefer to be invited to share their knowledge and experiences.
We’re masters at making evaluation relevant.
One of the great rewards of being a practitioner in the regions is bringing evaluation to people who have never used it before. In places where ‘government’ and ‘head office’ are often a long way away, regional evaluators become very good at explaining what evaluation is and making it relevant. We work across a wide range of scales and industries, from multi-million-dollar programs to small volunteer projects and everything in between, involving government, researchers, industry bodies, technical specialists and community groups.
We are practical, flexible, creative and resourceful
There’s never a dull moment when you’re a regionally-based evaluator. I’ve facilitated an evaluation discussion with graziers in an outback pub garden with no walls to stick up my trusty butchers paper (the horror!) while the pub’s resident dog wandered around snuffling our table scraps. I’ve explained to bemused volunteer conservation groups that the funding body needs them to report on whether they used corflute or cardboard tree guards for high-level evaluation purposes. (No, I’m not making that one up.) Working outside of corporate office environments teaches us to be highly flexible and resourceful.
We’re also experts at adapting evaluation to diverse communities and being creative in low technology settings. Instead of a workshop in a central location with lovely catering and ipads all round, we’re more likely to be standing in the middle of a field or on the phone at night, competing for attention with events like mustering; or scribbling diagrams on the back of a coaster as we talk with community members. We’ll cobble together elements of different approaches to find new and practical ways to use evaluation and add value to the communities we live and work in. Everyone deserves good evaluation!
As a regional evaluator, you may be subject to intense scrutiny. Source: Renée Madsen
What I’d love to see as a regional evaluator:
Get in touch and share your thoughts….
I would love to hear from evaluators who live and work outside metropolitan areas. What’s it like for you as a regionally-based evaluator? What do you think about the future of evaluation outside major cities?
For my ‘big city’ colleagues - much has been made of society’s capacity to make changes to established practices in the wake of COVID-19. Will you do anything differently in connecting and collaborating with your regional colleagues?
Renée Madsen is Principal Consultant at Create and Evaluate, a group facilitation and evaluation consultancy in Townsville, North Queensland. Connect with Renée on LinkedIn or visit www.createandevaluate.com.au . Renée would like to thank her fellow regional evaluators Dr Julie Funnell, Ms Barbara Colls and Ms Donna Turner for their contributions to this article.
by AES Relationships Committee
The changing context
The global scale and speed of disruption caused by the COVID-19 pandemic is unprecedented in our lifetimes. The pathway to recovery and management of COVID-19 is expected to be complex and challenging, with significant long-term implications for individuals, organisations, governments and the country.
The coordinated national response in Australia has so far been successful because the best available data and evidence has significantly influenced decision-making. The evidence-informed approach that has served us well to-date remains equally critical going forward.
During the pandemic, many public sector initiatives and supports have been designed, adjusted or expanded to assist individuals, households and businesses to survive and adapt. Some services have been interrupted or halted. As restrictions lift, consideration will need to be given to which adjustments are maintained.
The AES considers that sound data collection and analysis should be built into the establishment of any new or adapted initiatives to maximise the value of evaluation. Evaluation can also support the development of new initiatives and support service redesign activities.
Evidence and evaluation play an important role
Evaluation – and evaluative thinking – remains central in offering systematic review of new and changing initiatives, and to pre-empt potential unintended consequences. It can be undertaken across the policy and program life-cycle to:
Evaluators are adapting their approaches
To effectively deliver on existing work, evaluators have adapted their approaches to meet physical distancing requirements. Although service clients and stakeholders may feel harder to reach, digital platforms are enabling connections across traditional geographic and social boundaries.
Evaluators are able to continue their work by:
The AES recommends that monitoring, evaluation and evidence continue wherever possible to support post-pandemic recovery and review.
This statement has been prepared by AES members for AES members to support discussions about why evaluation has particular relevance and value during the pandemic, and how evaluations may be adapted.
by Anthea Rutter
Sue Funnell was one of the early trail blazers in evaluation methods. By her estimate, Sue has been in the profession for over 43 years. Over this time, she has held a number of roles in evaluation, including as the director of her own consulting company. She was a founding member of the AES, had two terms as President, was chair of the Awards Committee, and a presenter and trainer.
I first came across Sue in the 90s when she detailed her approach to program logic. We were also on the AES Board together for a few years. Sue has had a huge influence on the practice of evaluation, so I was very interested to find out how she came into the field.
I joined the Centre for Research and Measurement in evaluation, NSW Department of Education in the 70s. It was my first job after finishing a Psychology Honours degree. I started a part time Master’s degree in measurement and evaluation led by Ralph Straton at the University of Sydney and then received an Educational Research and Development Centre scholarship to the University of Illinois in the US. My project was in measurement, but I arrived there to find a hotbed of evaluators: Stake, Hastings and House, amongst others. This consolidated my interest in evaluation.
From that initial focus on measurement and her reputation as a leader in the development of approaches to program logic, what have emerged as Sue’s main areas of interest?
Mainly programs that achieve their results through behaviour change, such as educational and advisory programs and regulatory initiatives. I’m also interested in helping evaluators and commissioners to develop a sound description and understanding of the evaluand, so that they can identify appropriate evaluation questions.
A career as long as Sue’s is bound to have challenges, what have been the main ones?
I reckon balancing clients’ needs and expectations, particularly relating to time horizons on the one hand and my commitment to quality on the other. Also commissioners of evaluations constantly change and, with this, comes changes in their demands on a particular evaluation.
Another challenge is the speed of change in the policy context, which would be greater now! It would appear that people are more interested in short-term initiatives and results than in longer-term strategic approaches.
As well as challenges, a good career has its highlights – I asked Sue about hers.
Working with Bryan Lenne in the Program Evaluation Unit in the NSW Public Service Board was a game changer. This started me on the path to enhancing program logic approaches, providing a tool to get managers to think about their programs and how to ‘connect the dots’. I’ve received lots of positive feedback as well as criticism of my approach to program logic. I’ve honed the approach over time, and this culminated co-writing Purposeful Program Theory: effective use of theories of change and logic models with Patricia Rogers.
As well as this, setting up my own successful company in 1992 and, for 25 years, I was working across a wide range of policy areas and a wide range of jurisdictions and levels: local, state, federal, international, NGOs.
As you’d expect, Sue’s approach to evaluation has had a number of influences, among these:
- Working in the Program Evaluation Unit in the NSW Public Service Board with Bryan Lenne
- Ernie House’s early (and continuing) work on Social Justice
- Hatry’s work on Comparison is the Name of the Game
- Undertaking meta evaluations, particularly to do with the evaluation function in organisations
- The Joint Committee Standards on Utility, Feasibility, Accuracy and Propriety
- Patton’s work on Utilisation-Focused Evaluation
- Locally, material coming out of different levels of government around program budgeting, in particular the concepts of appropriateness, effectiveness and efficiency.
Sue also gave an honest appraisal of the strengths and challenges in the growth and development of the practice of evaluation.
In the early days, evaluation was a fledgling field trying to define itself. There was much greater emphasis on evaluation models, such as Stufflebeam’s Decision Making model and Stake’s Responsive evaluation. I doubt these days whether current evaluators think much about or use models. Perhaps this happens in academia, but I doubt whether they play a great role for practising evaluators.
Evaluation has been strengthened by becoming multi-disciplinary recognising the need to draw on many fields. A more nuanced understanding of what is gold standard has developed. Amongst evaluators, what is gold standard is what is fit for purpose. Importantly, applying program logic is neutral with respect to choice of methodology to address evaluation questions. However, from time to time, there is a push, especially from Government, for RCTs to be the only gold standard.
There has, over the years, been a constant tension between relative emphasis on monitoring and performance indicators on the one hand and evaluation studies on the other. There has also been a frequent re-badging of performance information and evaluation approaches by state and federal government, often with nothing or little new added!
There has been greater participation in evaluation by large companies (such as the big four). A lot is done in the name of evaluation that might more accurately be called management review.
When I asked Sue the main skills or competencies evaluators need to have or develop to keep pace with emerging trends, her first thought was that she had been out of evaluation for a while. But, on reflection, she had some key insights.
Fleet-footedness and adaptability, while minimising compromises to quality is important.
We can also make greater use of secondary data and possibly rely less on primary data. Social media has probably become a greater source of secondary information, but evaluators need to have the competencies to assess that information over time and draw on a wide range of social media sources, so that they are not influenced unduly by a particular social bubble.
Beyond the skills we need, I asked Sue what issues evaluators ought to be thinking as about and seeking to resolve in the next decade?
If evaluators want to contribute to worthwhile social changes, then they need to actively address social justice issues and take some stance. This raises the question of whether evaluators should become more socially activist and perhaps one way to do this is to move a bit away from evaluating individual programs to evaluating how well government and society are addressing issues. For example, in relation to domestic violence, what has been done in this area, how well is it working and what can be done? How can we do it and how can we evaluate it? However, a vexed question is ‘ who pays for it?’. I don’t have the answer to that.
by Eleanor Williams
COVID-19 has, for many, been a time of adaptation and creation of a new sense of normality. As we move away, gratefully, from local crisis management, we have the opportunity to reflect on not only our own resilience through this time, but what we have learned and how we have adapted through adversity.
Eleanor Williams from the Centre for Evaluation and Research Evidence, Victorian Dept of Health and Human Services and the Australian Public Sector Evaluation Network shares her reflections on Evaluation Adaptation through COVID-19.
COVID-19 has brought about strange new ways of working. Evaluation teams across Victoria, including ours, are adapting to working remotely, away from colleagues, comfort zones and familiar professional places and spaces. As we adjust to navigating these new evaluation environments and experiences, I have really valued the opportunity to share perspectives with others through blogs, online forums and webinars.
I’ve been reflecting on a number of challenging questions, many of which are focussed around how evaluation can provide maximum value when key systemic and organisational decisions are having to be made quickly and reactively to rapidly emerging and changing situations.
COVID-19 forces big policy questions onto the table as we all try to identify how this crisis impacts our services; how it can be best mitigated; what can stay the same and what needs to swiftly evolve to meet projected changes in patterns of need across health and human services.
In our context evaluators are trying to work out how we best adapt to the changes happening around us. Should we be holding steady and proceeding with existing project work? Can we feasibly do this? Are the agencies we are working with able to focus on evaluation at the moment? Do we adapt our focus instead to new knowledge priorities and to make space for emerging demands?
Our team has had to cope with losing staff to emergency response secondments, bringing about a sudden re-prioritisation of projects. At the same time new COVID-related evaluation requests are coming in daily and our remaining team members are running a series of rapid evaluations to provide fast evidence to decision-makers about the impact of service and practice changes that have emerged during Covid-19.
It has been a balancing act of trying to retain the integrity of longer term and larger evaluations, while downscaling to manage additional demands and depleted team numbers. And we ask ourselves, can we really deliver robust and insightful findings within extra tight timeframes and in a rapidly changing landscape? And cutting across it all, we ask ourselves how we can hold onto our reflective practice principles to use these times of challenge and change as learning opportunities?
Adapting our practice
Rapid but thoughtful adaptation has been key to the resilience required to navigate these times. Not only have we had to find ways to work without the face-to-face engagement which has always been central to evaluation capacity building, data collection and delivery of findings, we have also had to individually adapt to fast changes to our team member’s roles and availability.
It is testimony to the versatile skillsets of evaluators that our team members have been deployed into not only rapid evidence reviews for public health emergency responses, but front line data collection, working phonelines to provide emergency information to the public, even acting as concierges for hotels being used for quarantine purposes.
In parallel, team members learnt how to run program logic and investment logic mapping sessions through Microsoft Teams becoming, like so many, over-night experts in video-conferencing and telephone interviews. And alongside this we shared the challenges of the world’s professionals now forced to work in shared space with partners, children and pets; with laptops propped up on books at kitchen tables or in cramped bedrooms in share houses.
What will we take forward?
As life returns to “new normal” in the coming months we will be alongside the rest of our communities in dealing with the backlog of work and life that had to be put aside during the crisis – at the same time as rising to the new demands of 2020-21’s recovery phase.
Breaks in data collection and unexpected, unforeseen changes in the ways that human services are used in the community will pose challenges for reliable evaluation findings. In particular it will be difficult to separate the internal effectiveness of programs and projects from the impact of COVID-19.
While there will be no easy answers, there are some emerging upsides as well. The world has discovered that it is possible to be less reliant on the face-to-face engagement, which opens up the opportunity for major efficiencies of time and resources devoted to logistical coordination of face-to-face focus groups, workshops and interviews. As evaluators we can start to imagine a life with less time and money spent on function rooms, hire cars and accommodation. Information could be only a click away through effective use of Zoom, Skype or Teams or any number of online platforms. These experiences can show us that distance need not be an obstacle to engagement with stakeholders across state, country and even the world.
It is these conversations, whether facilitated by AES or in our daily lives, that will support our profession to adapt and deliver the best possible evidence and insights in the emerging new environment.