Evaluation in the COVID-19-shaped “new normal”
by Eleanor Williams
COVID-19 has, for many, been a time of adaptation and creation of a new sense of normality. As we move away, gratefully, from local crisis management, we have the opportunity to reflect on not only our own resilience through this time, but what we have learned and how we have adapted through adversity.
Eleanor Williams from the Centre for Evaluation and Research Evidence, Victorian Department of Health and Human Services and the Australian Public Sector Evaluation Network shares her reflections on Evaluation Adaptation through COVID-19.
COVID-19 has brought about strange new ways of working. Evaluation teams across Victoria, including ours, are adapting to working remotely, away from colleagues, comfort zones and familiar professional places and spaces. As we adjust to navigating these new evaluation environments and experiences, I have really valued the opportunity to share perspectives with others through blogs, online forums and webinars.
I’ve been reflecting on a number of challenging questions, many of which are focussed around how evaluation can provide maximum value when key systemic and organisational decisions are having to be made quickly and reactively to rapidly emerging and changing situations.
COVID-19 forces big policy questions onto the table as we all try to identify how this crisis impacts our services; how it can be best mitigated; what can stay the same and what needs to swiftly evolve to meet projected changes in patterns of need across health and human services.
In our context evaluators are trying to work out how we best adapt to the changes happening around us. Should we be holding steady and proceeding with existing project work? Can we feasibly do this? Are the agencies we are working with able to focus on evaluation at the moment? Do we adapt our focus instead to new knowledge priorities and to make space for emerging demands?
Our team has had to cope with losing staff to emergency response secondments, bringing about a sudden re-prioritisation of projects. At the same time new COVID-related evaluation requests are coming in daily and our remaining team members are running a series of rapid evaluations to provide fast evidence to decision-makers about the impact of service and practice changes that have emerged during Covid-19.
It has been a balancing act of trying to retain the integrity of longer term and larger evaluations, while downscaling to manage additional demands and depleted team numbers. And we ask ourselves, can we really deliver robust and insightful findings within extra tight timeframes and in a rapidly changing landscape? And cutting across it all, we ask ourselves how we can hold onto our reflective practice principles to use these times of challenge and change as learning opportunities?
Adapting our practice
Rapid but thoughtful adaptation has been key to the resilience required to navigate these times. Not only have we had to find ways to work without the face-to-face engagement which has always been central to evaluation capacity building, data collection and delivery of findings, we have also had to individually adapt to fast changes to our team member’s roles and availability.
It is testimony to the versatile skillsets of evaluators that our team members have been deployed into not only rapid evidence reviews for public health emergency responses, but front line data collection, working phonelines to provide emergency information to the public, even acting as concierges for hotels being used for quarantine purposes.
In parallel, team members learnt how to run program logic and investment logic mapping sessions through Microsoft Teams becoming, like so many, over-night experts in video-conferencing and telephone interviews. And alongside this we shared the challenges of the world’s professionals now forced to work in shared space with partners, children and pets; with laptops propped up on books at kitchen tables or in cramped bedrooms in share houses.
What will we take forward?
As life returns to “new normal” in the coming months we will be alongside the rest of our communities in dealing with the backlog of work and life that had to be put aside during the crisis – at the same time as rising to the new demands of 2020-21’s recovery phase.
Breaks in data collection and unexpected, unforeseen changes in the ways that human services are used in the community will pose challenges for reliable evaluation findings. In particular it will be difficult to separate the internal effectiveness of programs and projects from the impact of COVID-19.
While there will be no easy answers, there are some emerging upsides as well. The world has discovered that it is possible to be less reliant on the face-to-face engagement, which opens up the opportunity for major efficiencies of time and resources devoted to logistical coordination of face-to-face focus groups, workshops and interviews. As evaluators we can start to imagine a life with less time and money spent on function rooms, hire cars and accommodation. Information could be only a click away through effective use of Zoom, Skype or Teams or any number of online platforms. These experiences can show us that distance need not be an obstacle to engagement with stakeholders across state, country and even the world.
It is these conversations, whether facilitated by AES or in our daily lives, that will support our profession to adapt and deliver the best possible evidence and insights in the emerging new environment.