by AES QLD Committee Members

Evaluators in the AES network are increasingly being challenged to apply evaluative thinking, methods and tools to innovative, emergent, place-based or otherwise complex initiatives. These initiatives often seek to achieve improvements not only in individuals and institutions, but in the systems that hold 'wicked' societal problems in place. The desired systems-level outcomes are often difficult to define, predict and measure and can change and evolve as the implementing organisations learn which strategies are most effective in reaching their goal.

In response, a recent issue of the AES QLD regional committee's newsletter focussed on resources, methods and mindsets to support members to in evaluating  complex systems change initiatives. Here are the take-outs.

Evaluating complex systems initiatives in a nutshell

Contributed by AES QLD Committee Member Kiri Dicker

A complex system is a system in which there are multiple, interrelated parts and in which a perfect understanding of these parts does not automatically convey a perfect understanding of the whole system's behaviour. [i] Systems are everywhere—they can be ecological, mechanical, organisational, political or cultural.[ii] A system’s overall purpose or goal is achieved through the actions and interactions of its components.[iii] In other words, the complexity of the system is integral to its effective functioning, which only occurs when all components are performing optimally (within themselves and in relation to each other).

It is important to remember that complex systems can rarely be described as a single system, rather that there are ‘systems within systems.’[v] [ii] These systems are open systems, meaning they interact with other systems, have permeable boundaries, and are affected by their external environment (often described as the context).[v]

Photo by John Barkiple on Unsplash

In the public health sector, systems have multiple programs, policies, agencies, or institutions at the national, state, or local level with the common goal of achieving better outcomes for children, youth, adults, or families.[iv] Programs that seek to improve the way that systems function are known as systems initiatives. These initiatives are inherently complex because they involve multiple interventions at different levels of a system, which are connected in ways that are often unclear and difficult to predict.

Coffman describes system initiatives as those that:

…involve multiple programs and players and feature outcomes at multiple levels (individual, family, community, and state)…they tackle difficult deep-rooted problems such as gaps in services and outcomes based on race, income, culture, and language. Finally, all efforts to improve systems are long-term efforts that evolve over time in response to the political, economic, and social contexts around them.[ii]

Systems initiatives pose unique challenges for evaluators because they:

• explicitly seek to achieve outcomes at multiple levels across a defined system
• have multiple components (activities), that are interrelated and interdependent
• work with multiple actors and organisations and, therefore, have multiple components to evaluate
• are often deliberately iterative, seeking to test new ideas to figure out what works, and intend on evolving throughout the implementation timeframe, without establishing a ‘fixed’ model.

For these reasons, systems initiatives cannot be evaluated in the same ways as projects or programs operating in simple contexts. For example, a key characteristic of complex systems is that they are constantly changing and evolving, and so they are often unpredictable. Therefore, the development of a comprehensive evaluation plan that overly specifies the evaluation’s key questions, evaluation approach, design, data collection and analysis methods, timeline, and budget may not be particularly helpful for evaluating complex initiatives or initiatives in complex environments.[v]

[i] Miller, John H., and Scott E. Page (1 January 2007). Complex adaptive systems: an introduction to computational models of social life.
[ii] Coffman, J, ‘A framework for evaluating systems initiatives’, Build Strong Foundations for our Youngest Children, 2017, pg. 3.
[iii] Ross, D., ‘Process engineering: A necessary step to a better public health system. Information Knowledge Systems Management’, 2019, Vol. 8, pp. 299–309.
[iv] Leischow, S., Best, A., Trochim, W., Clark, P., Gallagher, R., Marcus, S., & Matthews, E, (2008). Systems thinking to improve the public’s health. American Journal of Preventive, 35(2S), S196–S203.
[v] Ling, T, ‘Evaluating complex and unfolding interventions in real time’, Evaluation 18(1), 2012, 79–91

Unleashing your systems thinking superpowers

Contributed by AES QLD Committee Member Lewê Atkinson

Every system is perfectly designed to get the results it gets - Edward Demming

With bright eyes and a deep conviction, Zazie Tolmer responded to a question about her next steps after completion of the Children and Youth Area Partnerships (CYAPs) Project, the subject of her presentation at aes18, by saying:

I am not sure really…now that I have been exposed to this way of thinking, I don’t think I can ever go back to the way I was…

She had unleashed her systems thinking superpowers. 
 
Systems Thinking is an old (circa 1950s) yet newly rediscovered and higher orientation to life (see this article applying a Buddhist world view to traditional evaluation methodologies). It is a better, more natural, and holistic view of living systems (such as individuals, teams, and organisations) as they try to survive and thrive in today’s dynamic environment. Systems thinking is a habit of mind.  Fortunately, a lot like languages, it is easier to grasp at an early age.  Indeed, the Waters Centre for systems thinking has a mission of imparting the 14 habits of systems thinking from K-12 in the USA.
 
My professional career spans more than 30 years now.  I have experienced more of an “awakening” rather than an “unleashing” of my own systems thinking superpowers.  My top 4 superpowers, upon which my professional practice is founded, include: seeing systems as a whole and the parts within them; seeing the “big picture” of a system; seeing situations, experiences, viewpoints, and more, through the eyes of others; and seeing how understanding mental models can deepen our understanding of system structure.   This blog post demonstrates the use of these superpowers to help scientists and researchers to identify “hidden” barriers to translation of their discoveries into positive impact within our communities.
Photo by Josh Hild on Unsplash

You are made of stuff that is as old as the planet, one third as old as the universe, though this is the first time that those atoms have been gathered together such that they think that they are you - Frank Close, Particle Physicist

Join GAST – the Global Association of Systems Thinking to stay engaged.

Readers may be interested in the upcoming AES workshop: Workshop: Systems Evaluation Theory – Practice & Implementation (Online 10 December 2020), facilitated by Ralph Renger, Lewê Atkinson and Brian Keogh.

Useful resources for evaluating complexity

Getting up to speed on evaluating complex systems initiatives can be overwhelming, so we have made it easy by curating this list of useful resources to get your started:

• Social impact consultancy FSG has produced a series of highly useful frameworks for thinking about systems change and evaluating complexity. These include:

• The Water of Systems Change framework
• 10 propositions for evaluating complexity
• This systems mapping toolkit.

• A framework for evaluating systems initiatives developed by Julia Coffman for the Build Initiative is still a good read, despite being over 10 years old. It also includes advice on developing a high-level theory of change for a systems initiative.

• The recently updated 'Developing and evaluating complex interventions' by the British Medical Research Council also offers some good advice.

• Clear Horizon has produced a number of resources for public use that are suitable for evaluating complex initiatives, include:

• A place-based evaluation framework and associated toolkit produced in partnership with The Australian Centre for Social Innovation and Collaboration for Impact
• The Significance Instances of Policy and Systems Change (SIPSI) technique. This think piece on how to develop a theory of change for a complex systems initiative.