Scope: How transferable is the understanding of system dynamics across different complex problems? That is, if we gain experience in managing a complex system, can we expect to be well prepared to transfer this expertise to the management of a different system?

Why? Answering these questions has important and direct implications for the use of computer modelling in managing complex ecosystem and social problems: it can provide experimental evidence of its effectiveness or otherwise, and may eventually lead to the development of training methods suited to these complex tasks.

Outcome: The results of the experiments are contained inBoschetti, Hardy, Grigg & Horwitz, 2011, Can we learn how complex systems work?, Emergence: Complexity and Organization, 13, 4, 47-62.

This approach relies on one assumption and one expectation (among others); the assumption is that modelling is useful to improve system understanding; the expectation is that by gaining modelling expertise our understanding broadens and our capacity to transfer the knowledge to related, but different, problems improves.

The purpose of the project is to generate some evidence for or against the above claims. We use tools ranging from numerical modelling, to psychological questionnaires and description and analysis of mental cognitive models to address these questions.

Related work. This project was inpired by Dietrich Dörmer's The Logic of Failure and by the work of John Sterman, Linda Sweeney and Erling Moxnes. In different ways, they all highlight the difficulty that people (even highly trained) encounter in understanding the basic principles of system dynamics. Most important, their work stresses the impact that lack of understanding of basic system dynamics can have on policy making, both at the level of the people with the power to take decisions and at the level of the general public who needs to decide whether to support them. 

The approach. We work on two assumptions. The first one is that even ‘experts’ have trouble understanding simple dynamical concepts, like stocks & flows. This assumption is based on the work of John Sterman, Linda Sweeney and Erling Moxnes and has been informally confirmed by our experiments with a number of researches with various backgrounds via the use of our Pen & Paper Test. The second assumption is that a complex system can not be satisfactorily managed without a proper understanding of stocks & flows. In other words, understanding stocks & flows is necessary, but not sufficient, to the management of complex dynamical systems.

We developed a set of three Toy Models, written in Netlogo. The Toy Models are designed to train the user on stocks & flows problems of increasing complexity:

• Toy Model 1 includes one stock, one inflow and one outflow
• Toy Model 2 includes two stocks, two inflows and one outflow
• Toy Model 3 includes two stocks, two inflows and one outflow with feedback.

We ask whether users trained on three Toy Models perform better at using a complex computer model than a control group. As complex computer model we chose the Chocolate Factory designed by Dietrich Dörmer. The performance on the Chocolate Factory depends on a proper management of stock & flows as well as on several other factors. Importantly, the Chocolate Factory simulates a process which depends on many sub-processes, parameters and feedbacks. This not only represents a considerable increase in dynamical complexity compared to the Toy Models, but also forces the user to work with insufficient information, uncertain data, possible surprises and under a certain level of frustration, phenomena which are nicely described and analysed in The Logic of Failure.

The question we ask from our experiment is: do subjects trained on the Toy Models show an improvement in the way they manage stocks & flows in the Chocolate Factory, compared to a control group?

The experiment. The experiment will be carried out on 7-8 August 2009 at the School of Natural Sciences, Edith Cowan University, Western Australia, in collaboration with Associate Professor Pierre Horwitz.

Experimental design. Below you can find the experiment design, schedule and supporting material.

Summary of results. 28 subjects completed both ‘Pen & Paper’ Tests (Questionnaire 1 and 5). Unfortunately, only 8 of them took part in the modelling part of the experiment. Consequently, these results can not be considered statistically reliable. However, from an ‘anecdotal’ point of view they give some indications which we summarise here and which may be worthwhile following up on.

• From the experiment with computer models (8 students) we speculate that:
  1. the training group managed stocks & flows better than the control group in the complex task (KS test significance 0.1). This was assessed by checking how they controlled the balance between raw material, storage, spoilage and production in the Chocolate Factory.
  2. the training group performed slightly better in the overall complex task; however the difference is not significant (KS test significance 0.5). This was assessed by evaluating the overall economic return in the Chocolate Factory. This can be explained by noticing that managing stock & flows is necessary, but not sufficient, to perform well in a complex problem.
  3. Personality type (as analyses via Questionnaire 2) predicts reasonably well performance on both the Toy Models and the Chocolate Factory.
  4. Personality type effect is less pronounced after training on the Toy Models.
  5. No correlation is found between self-confidence and self-esteem and performance.
     
• From the '2 pen & paper' questionnaires (25 students) we noticed that:
  1. There was a marked improvement between Questionnaire 1 and Questionnaire 5. (They ask an analogous question but reversed in sign and cast into a CO2 vs fishery scenario).
  2. This improvement is NOT due to the 8 students who participated in the modelling component of the experiment, since they improved as much as the ones who did not. Also, no lecture was given in between the 2 questionnaires. A possible interpretation is that stock & flows test 2 was 'easier' or 'more intuitive'. Either this was due to the reversal in sign between the flows in the 2 questions, or because the fish example is considered easier than the CO2 one.

• Experience with modelling usually requires a considerable amount of time, while we are hoping to detect an improvement in the management of stocks & flows after approx one hour of training only. Although this training is specifically designed for the task at hand, in general we would expect that gaining knowledge on system dynamics which is reliably transferable will require considerably more time. A follow up experiment may want to address this issue.
• A proper evaluation of performance on the Chocolate Factory may require many more iterations than possible within the 6 hours allowed in our experiment. This is particularly true within the context of the The Logic of Failure work, in which one of the purposes of the exercise is to challenge cognitive abilities, by forcing the subject to confront uncertainty as well as the magnitude of the task. Due to time limitation, in our experiment we aim to detect an improvement in the management of stock & flows, within the larger context of the management of the overall factory. In other words, we focus on detecting technical improvements rather than more general cognitive learning.
• Implicit in our approach is an attempt to break down complex dynamics into constitutive components, stocks & flows being one of them. This is even more true for our ‘Pen & Paper’ Test, in which we also include feedbacks and other social issues. To a Complex System Science ‘purist’ this may be reminiscent of ‘reductionism’ and consequently be seen as a wrong approach or a contradiction, if not a heresy. We prefer to abstain from this issue. For the purpose of this experiment, the question is whether breaking complexity into components is helpful for learning purposes and hopefully our results will suggest a partial answer.
• It is debatable to what extent the Chocolate Factory can be considered a ‘complex’ model. From a cognitive perspective we can probably say it is so: subjects are faced with uncertainty, with an overwhelming amount of information to process and with many parameters to control at once; from this point of view, a Chocolate Factory run recreates several real world challenges. From a dynamical perspective it can be argued that running a factory may be less ‘complex’ than managing an ecological or social system. With more time and resource it would be useful to run a similar experiment employing a wider range of ‘microwords’.

Supplementary Material

References

• Boschetti, Hardy, Grigg & Horwitz, 2011, Can we learn how complex systems work?, Emergence: Complexity and Organization, 13, 4, 47-62.
• Cronin, M.A., Gonzalez, C. and Sterman, J.D., 2009. Why don't well-educated adults understand accumulation? A challenge to researchers, educators, and citizens. Organizational Behavior and Human Decision Processes., 108:116-130.
• Dorner, D., 1996. The Logic Of Failure: Recognizing And Avoiding Error In Complex Situations Metropolitan Books, New York.
• Ericsson, A., 1993. The Role of Deliberate Practice in the Acquisition of Expert Performance. Psychological Review, 100:363-406.
• Hogan, K. and Thomas, D., 2001. Cognitive Comparisons of Students' Systems Modeling in Ecology. Journal of Science Education and Technology, 10:319-345.
• Moxnes, E., 1998. Overexploitation of renewable resources: the role of misperceptions. Journal of Economic Behavior and Organization, 37:107–127.
• Moxnes, E., 2000. Not only the tragedy of the commons: misperceptions of feedback and policies for sustainable development. System Dynamics Review, 16:325–348.
• Moxnes, E. and Saysel, A.K., 2009. Misperceptions of global climate change: information policies. Climatic Change, 93:15-37.
• Sterman, J.D., 2008. Risk Communication on Climate: Mental Models and Mass Balance. Science, 322:532-533.
• Sterman, J.D. and Sweeney, L.B., 2002. Cloudy skies: assessing public understanding of global warming. System Dynamics Review, 18:207-240.
• Sterman, J.D. and Sweeney, L.B., 2007. Understanding public complacency about climate change: adults’ mental models of climate change violate conservation of matter. Climatic Change, 80:213-238.
• Sweeney, L.B. and Sterman, J.D., 2000. Bathtub dynamics: initial results of a systems thinking inventory. System Dynamics Review, 16:249-286.
• Sweeney, L.B. and Sterman, J.D., 2007. Thinking about systems: student and teacher conceptions of natural and social systems. System Dynamics Review, 23:285-311.