Share your ideas
AES Blog

Welcome to the AES Blog

Australasia has some excellent evaluators. More than that, we have an evaluation community full of ideas and a willingness to share. The AES has long provided a place for us to come together, at regional events and the annual conference, to develop our community together. Now we’re taking it online! The new AES blog will be a space for AES members – both new and experienced – to share their perspectives, reflecting on their theory... If you have an idea, please contact us on blog@aes.asn.au. Please also view our blog guidelines.

Systems Thinking in the Sunshine State

STblog1

by AES QLD Committee Members

Evaluators in the AES network are increasingly being challenged to apply evaluative thinking, methods and tools to innovative, emergent, place-based or otherwise complex initiatives. These initiatives often seek to achieve improvements not only in individuals and institutions, but in the systems that hold 'wicked' societal problems in place. The desired systems-level outcomes are often difficult to define, predict and measure and can change and evolve as the implementing organisations learn which strategies are most effective in reaching their goal.

In response, a recent issue of the AES QLD regional committee's newsletter focussed on resources, methods and mindsets to support members to in evaluating  complex systems change initiatives. Here are the take-outs.

Evaluating complex systems initiatives in a nutshell

Contributed by AES QLD Committee Member Kiri Dicker

STblog1

A complex system is a system in which there are multiple, interrelated parts and in which a perfect understanding of these parts does not automatically convey a perfect understanding of the whole system's behaviour. [i] Systems are everywhere—they can be ecological, mechanical, organisational, political or cultural.[ii] A system’s overall purpose or goal is achieved through the actions and STblog2interactions of its components.[iii] In other words, the complexity of the system is integral to its effective functioning, which only occurs when all components are performing optimally (within themselves and in relation to each other).

It is important to remember that complex systems can rarely be described as a single system, rather that there are ‘systems within systems.’[v] [ii] These systems are open systems, meaning they interact with other systems, have permeable boundaries, and are affected by their external environment (often described as the context).[v]

Photo by John Barkiple on Unsplash

In the public health sector, systems have multiple programs, policies, agencies, or institutions at the national, state, or local level with the common goal of achieving better outcomes for children, youth, adults, or families.[iv] Programs that seek to improve the way that systems function are known as systems initiatives. These initiatives are inherently complex because they involve multiple interventions at different levels of a system, which are connected in ways that are often unclear and difficult to predict.

Coffman describes system initiatives as those that:

…involve multiple programs and players and feature outcomes at multiple levels (individual, family, community, and state)…they tackle difficult deep-rooted problems such as gaps in services and outcomes based on race, income, culture, and language. Finally, all efforts to improve systems are long-term efforts that evolve over time in response to the political, economic, and social contexts around them.[ii]

Systems initiatives pose unique challenges for evaluators because they:

  • explicitly seek to achieve outcomes at multiple levels across a defined system
  • have multiple components (activities), that are interrelated and interdependent
  • work with multiple actors and organisations and, therefore, have multiple components to evaluate
  • are often deliberately iterative, seeking to test new ideas to figure out what works, and intend on evolving throughout the implementation timeframe, without establishing a ‘fixed’ model.

For these reasons, systems initiatives cannot be evaluated in the same ways as projects or programs operating in simple contexts. For example, a key characteristic of complex systems is that they are constantly changing and evolving, and so they are often unpredictable. Therefore, the development of a comprehensive evaluation plan that overly specifies the evaluation’s key questions, evaluation approach, design, data collection and analysis methods, timeline, and budget may not be particularly helpful for evaluating complex initiatives or initiatives in complex environments.[v]

[i] Miller, John H., and Scott E. Page (1 January 2007). Complex adaptive systems: an introduction to computational models of social life.
[ii] Coffman, J, ‘A framework for evaluating systems initiatives’, Build Strong Foundations for our Youngest Children, 2017, pg. 3.
[iii] Ross, D., ‘Process engineering: A necessary step to a better public health system. Information Knowledge Systems Management’, 2019, Vol. 8, pp. 299–309.
[iv] Leischow, S., Best, A., Trochim, W., Clark, P., Gallagher, R., Marcus, S., & Matthews, E, (2008). Systems thinking to improve the public’s health. American Journal of Preventive, 35(2S), S196–S203.
[v] Ling, T, ‘Evaluating complex and unfolding interventions in real time’, Evaluation 18(1), 2012, 79–91


Unleashing your systems thinking superpowers

Contributed by AES QLD Committee Member Lewê Atkinson

Every system is perfectly designed to get the results it gets - Edward Demming

With bright eyes and a deep conviction, Zazie Tolmer responded to a question about her next steps after completion of the Children and Youth Area Partnerships (CYAPs) Project, the subject of her presentation at aes18, by saying:

I am not sure really…now that I have been exposed to this way of thinking, I don’t think I can ever go back to the way I was…

She had unleashed her systems thinking superpowers. 
 
Systems Thinking is an old (circa 1950s) yet newly rediscovered and higher orientation to life (see this article applying a Buddhist world view to traditional evaluation methodologies). It is a better, more natural, and holistic view of living systems (such as individuals, teams, and organisations) as they try to STblog3survive and thrive in today’s dynamic environment. Systems thinking is a habit of mind.  Fortunately, a lot like languages, it is easier to grasp at an early age.  Indeed, the Waters Centre for systems thinking has a mission of imparting the 14 habits of systems thinking from K-12 in the USA.
 
My professional career spans more than 30 years now.  I have experienced more of an “awakening” rather than an “unleashing” of my own systems thinking superpowers.  My top 4 superpowers, upon which my professional practice is founded, include: seeing systems as a whole and the parts within them; seeing the “big picture” of a system; seeing situations, experiences, viewpoints, and more, through the eyes of others; and seeing how understanding mental models can deepen our understanding of system structure.   This blog post demonstrates the use of these superpowers to help scientists and researchers to identify “hidden” barriers to translation of their discoveries into positive impact within our communities.
Photo by Josh Hild on Unsplash

You are made of stuff that is as old as the planet, one third as old as the universe, though this is the first time that those atoms have been gathered together such that they think that they are you - Frank Close, Particle Physicist

Join GAST – the Global Association of Systems Thinking to stay engaged.

Readers may be interested in the upcoming AES workshop: Workshop: Systems Evaluation Theory – Practice & Implementation (Online 10 December 2020), facilitated by Ralph Renger, Lewê Atkinson and Brian Keogh.


Useful resources for evaluating complexity

Getting up to speed on evaluating complex systems initiatives can be overwhelming, so we have made it easy by curating this list of useful resources to get your started:

  • Social impact consultancy FSG has produced a series of highly useful frameworks for thinking about systems change and evaluating complexity. These include:
  • A framework for evaluating systems initiatives developed by Julia Coffman for the Build Initiative is still a good read, despite being over 10 years old. It also includes advice on developing a high-level theory of change for a systems initiative.
  • Clear Horizon has produced a number of resources for public use that are suitable for evaluating complex initiatives, include:

 

The AES Fellows: More than the sum of their indivi...
Alan Woodward: Applying knowledge to improve servi...

ABOUT US   |    CONTACT US   |    LEGALS
Search site
© 2023 Australian Evaluation Society Limited 
ABN 13 886 280 969 ACN 606 044 624
Address: 425 Smith St VIC 3065, Australia

We acknowledge the Australian Aboriginal and Torres Strait Islander peoples of this nation. We acknowledge the Traditional Custodians of the lands in which we conduct our business. We pay our respects to ancestors and Elders, past and present. We are committed to honouring Australian Aboriginal and Torres Strait Islander peoples’ unique cultural and spiritual relationships to the land, waters and seas and their rich contribution to society.