Practice Bridge

Try try again: Lessons learned from success and failure in participatory modeling

Authors: Eleanor J. Sterling,Moira Zellner,Karen E. Jenni,Kirsten Leong,Pierre D. Glynn,Todd K. BenDor,Pierre Bommel,Klaus Hubacek,Antonie J. Jetter,Rebecca Jordan,Laura Schmitt Olabisi,Michael Paolisso,Steven Gray

Abstract

Participatory Modeling (PM) is becoming increasingly common in environmental planning and conservation, due in part to advances in cyberinfrastructure as well as to greater recognition of the importance of engaging a diverse array of stakeholders in decision making. We provide lessons learned, based on over 200 years of the authors’ cumulative and diverse experience, about PM processes. These include successful and, perhaps more importantly, not-so-successful trials. Our collective interdisciplinary background has supported the development, testing, and evaluation of a rich range of collaborative modeling approaches. We share here what we have learned as a community of participatory modelers, within three categories of reflection: a) lessons learned about participatory modelers; b) lessons learned about the context of collaboration; and c) lessons learned about the PM process. First, successful PM teams encompass a variety of skills beyond modeling expertise. Skills include: effective relationship-building, openness to learn from local experts, awareness of personal motivations and biases, and ability to translate discussions into models and to assess success. Second, the context for collaboration necessitates a culturally appropriate process for knowledge generation and use, for involvement of community co-leads, and for understanding group power dynamics that might influence how people from different backgrounds interact. Finally, knowing when to use PM and when not to, managing expectations, and effectively and equitably addressing conflicts is essential. Managing the participation process in PM is as important as managing the model building process. We recommend that PM teams consider what skills are present within a team, while ensuring inclusive creative space for collaborative exploration and learning supported by simple yet relevant models. With a realistic view of what it entails, PM can be a powerful approach that builds collective knowledge and social capital, thus helping communities to take charge of their future and address complex social and environmental problems.

Keywords: Participatory modeling Collaborative modeling Stakeholder engagement Planning Environmental management 
DOI: http://doi.org/10.1525/elementa.347
 Accepted on 27 Jan 2019            Submitted on 14 Mar 2018
Domain Editor-in-Chief: Anne R. Kapuscinski; University of California, US
Associate Editor: Kim A. Locke; Geographical Sciences, University of Maryland, US
Associate Editor: Mark Borsuk; Duke University, US


Introduction

Participatory Modeling (PM) is becoming increasingly common in environmental planning and conservation due to advances in cyberinfrastructure and to greater recognition of the importance of engaging a diverse array of stakeholders in decision making. We initiated this reflective article at the first of a series of workshops on PM, sponsored by the National Socio-Environmental Synthesis Center (SESYNC). The goal of the workshop series is to organize and consolidate scholarship around the practice of PM. PM encompasses the use of a broad range of modeling approaches in various forms of collaboration among practitioners, academics, and other stakeholders who engage in a purposeful learning process that elicits and formalizes the implicit and explicit knowledge of participants to support decision-making and action. Our knowledge community has learned important lessons from their experiences, successes, and failures. Given our experience, we firmly believe that PM has the potential to help communities guide themselves toward more positive futures. We offer these lessons learned in the hope that they will guide practitioners as they assist communities with using meaningful, appropriate, and illuminating modeling tools in participatory processes.

The reflections in this paper emerge from the authors’ over 200 years of cumulative and diverse experience conducting PM processes with communities concerned with different issues. Some examples of our work can be found in Table 1, at www.participatorymodeling.org, and in Gray et al., 2018.

Table 1

Examples of authors’ diverse experience conducting PM processes. DOI: https://doi.org/10.1525/elementa.347.t1

How to…: Citation(s)

Address flooding using decentralized green infrastructure Zellner et al., In Press
Ensure the sustainability of groundwater supplies Zellner et al., 2012
Protect biodiversity and ecosystem functions, and understand drivers of bushmeat trade Nayaki et al., 2014
Model the social and economic impacts of climate change on coastal resources Gray et al., 2014
Provide decision-support for wildlife managers in the Pacific Htun et al., 2016
Support conservation planning in relation to invasive species management Gray et al., 2017
Enact with farmers water resource planning at times of drought Douglas et al., 2016
Address land use conflicts and manage trade-offs of a range of ecosystem services Hubacek et al., 2009; Schmitt Olabisi et al., 2017

Our collective interdisciplinary background allowed for reflection on a rich range of collaborative modeling approaches, incorporating insights and experiences with environmental modeling, spatial analysis, urban and regional planning, psychology, anthropology, computer science, social science, and economics. The reflections in this paper are also informed by prior summary and review articles. These prior articles, covering multiple participatory studies, describe commonalities, articulate different overarching structures for the PM processes, describe lessons learned, and propose principles for effective PM (e.g., Voinov and Brown Gaddis, 2008; Voinov and Bousquet, 2010; Oteros-Rozas et al., 2015; Voinov et al., 2016; Gray et al., 2018). This paper focuses less on summarizing previous PM projects, and more on what we have learned about the challenges of implementing PM, especially in creating meaningful stakeholder and modeler partnerships within the complex social and political contexts where PM is most needed.

These reflections and lessons learned were generated and organized as follows. During our first workshop (February 2016), each participant wrote a summary of the key lessons that they had learned in their PM research and practice. We used inductive logic (Charmaz and Belgrave, 2007) to identify patterns arising from compiled data and grouped them into the three key themes described below. We built on this synthesis through several iterations, drawing from diverse literature to provide the foundation for this paper.

We have not previously shared these lessons widely because their articulation is frequently extraneous to traditional academic scholarship. We are often discouraged from publishing about failures (Becu et al., 2007). Our intent herein is to share our experience beyond advancing the technological dimension of modeling. We seek to encourage, and maybe even inspire others to embrace the uncertainty and messiness inherent to PM through lessons we have learned via many trials, successes, and even more errors, very much like any modeling process (Railsback and Grimm, 2012). It is our hope that sharing these lessons will help other practitioners skip some of the more painful learning steps we ourselves have worked through, and more effectively build the powerful collective knowledge and social capital that can emerge from PM processes.

Figure 1 illustrates a common evolution of assumptions and practices in PM, where researchers, who are eager to put their skills and knowledge to good use in supporting decisions, move from a strictly technical perspective towards full embrace of the partnership perspective. Each puzzle piece depicted in Figure 1 is necessary but not sufficient in describing how best to achieve innovative solution-building and action. For example, a common initial assumption is that providing decision-makers with the “right” information (e.g., more accurate/precise data, an understanding of interaction effects) and the “right” tools (e.g., models that accurately represent processes, interactions, and provide meaningful results), is sufficient for them to solve their problems. This is not enough to lead to improved decisions, actions, and effective solutions, possibly because decision-makers distrust modeling processes and tools that exclude their involvement.

Figure 1 

Progression of researchers’ evolving assumptions and practices in Participatory Modeling. Researchers move from a strictly technical perspective towards full embrace of the partnership perspective. Light blue boxes are evolving assumptions; blue boxes are challenges researchers face when acting on those assumptions, leading to new actions based on the assumptions. Each puzzle piece is necessary but not sufficient in describing how best to achieve innovative solution-building and action, building towards a fuller set of foundations for effective application. DOI: https://doi.org/10.1525/elementa.347.f1

Faced with falling short of their goals, researchers add increasingly more nuance to their approaches, moving next to an emphasis on communicating in the “right” ways (e.g., clearly translating and presenting information and models to both decision-makers and stakeholders), and when that too fails, moving towards more inclusion and participation. Researchers may assume that engaging selected stakeholders in data collection, analysis, and modeling will facilitate ownership and make stakeholders feel empowered and informed. When anticipated outcomes do not occur, researchers then turn to the need for diversity to enhance innovation and representation. Conflict may arise in engaging multiple stakeholders. In response, while keeping in mind previous assumptions and responses, the researchers focus on social processes, towards an understanding that attending to emotions, transparency of models and assumptions, power dynamics, and tradeoffs are key elements of effective PM.

This evolution takes place in individuals as well as across whole fields where initial technocratic approaches are enhanced by and eventually embrace more inclusive approaches. We have seen similar evolutions in risk management (Fischhoff, 1995), in probabilistic decision analysis (Spetzler et al., 2016), and in stakeholder engagement (Sterling et al., 2017). Recognition of greater complexity leads to consideration of processes as well as outcomes; more inclusive public input and consultation approaches, including possibilities for civic engagement and co-management (Leong et al., 2009, 2012); and a “commitment to action” based on the entire process (Leong et al., 2011; Spetzler et al., 2016).

We have grouped our reflections into three main categories regarding lessons learned: a) how modelers need to engage in PM, b) how to adapt to the social and political context of collaboration, and c) how to set up the PM process itself. Our reflections may be of most interest to modelers and researchers but we also share these lessons learned for facilitators and communities who may be interested in working in this arena.

Lessons learned about the role of participatory modelers

“Participatory modelers,” in the context of this paper, are individuals who facilitate, organize, and develop models for PM projects. A diverse set of skills, including prior content knowledge, facilitation skills, and technical modeling skills are needed to ensure that the PM process and resulting models are useful and helpful to the participating communities. It is a rare individual who possesses all of these skills; it is often better to consider a collaborative team as “the” participatory modeler (Prell et al., 2007). Here we report the skills that we have found to be most important for the team:

  1. Content knowledge coupled with openness to learn from local experts: A participatory modeler needs to be able to speak intelligently with stakeholders about the problem being modeled, and thus needs to have or develop background knowledge regarding the subject matter(s) related to the site, problem, or issue to be modeled. In addition to academic knowledge, important content knowledge comes from learning directly from the stakeholders about their unique experience with the complex problem to be addressed, and about their concerns, needs, and aspirations. When modelers come in as “outside experts,” all too often the result is alienation rather than collaboration (Hoch et al., 2015; Zellner et al., In Press). Recognizing the value and importance of local content knowledge means that modelers focus less on their own expertise and more on learning from local stakeholders. As an example, in the context of a collaborative stormwater management park project in a low-income community (Zellner et al., 2018), project leaders developed an ice-breaker activity to explain how to use causal loop diagramming (CLD) to lay out the complex interactions between flooding and economic development. The issue of traffic congestion was chosen to illustrate how one could draw the links among factors that reinforce congestion or alleviate it. As project leaders developed the exercise with key stakeholders in preparation for a larger community workshop, they learned from participants that traffic congestion was not an issue in their community, so they could not relate to the topic or to the CLD activity. The team then switched to a housekeeping and chores problem that most people could relate to, and which proved to be very effective in illustrating how community members could use CLD to link the problem they were facing with their collective aspirations. This unlocked and completely transformed the planning process for the project. The community’s CLD was built on local knowledge, beyond what the modelers could contribute.
  2. Skills to liaise effectively and build relationships with stakeholders: Participatory modelers need skills in identifying and selecting stakeholders (Reed et al., 2009, 2013a), and in building relationships and, consequently, credibility with stakeholders. These include strong listening skills accompanied by humility and patience. A good listener learns more than an expounder, and deep listening (not interrupting, allowing for people to express their ideas in their preferred way) and patience are critical in developing trusting relationships. Furthermore, the recognition that stakeholders’ expertise is equally valid to the modelers’ provides a solid foundation for mutual trust and respect, with the understanding that modelers are, themselves, another type of stakeholder (Barreteau et al., 2003; Zellner, 2008).
  3. Facilitation skills to elicit effective stakeholder engagement: Effective stakeholder engagement includes ensuring stakeholders communicate with each other. Often it means managing conflict. The facilitation process (Kaner, 2014) elicits participants’ needs, priorities, and understanding of the system, and fosters empathy among people who may hold different worldviews (Zellner, 2008; Hovmand, 2014). This may require opportunities for participants to “step into other people’s shoes”. For example, we saw increased understanding of a complex system and empathy with other stakeholders when we coached stakeholder groups to discuss mental model representations by other groups and to brainstorm solutions that would be acceptable to these groups (Singer et al., 2017). Professional facilitation can be useful in such cases, but just as with “outside experts” with content knowledge, it may not always benefit the engagement process. Community facilitators can be effective partners in the participatory modeling team (Hovmand, 2014).
  4. Ability to recognize, extract, and communicate a conceptual model from discussions with and among stakeholder groups: It is not a simple task to identify common themes from group discussions and translate them into a conceptual model of the system. It is equally difficult to make that conceptual model transparent and tractable in a way in which stakeholders recognize their inputs and gain intellectual ownership of the model. The modeling techniques need to fit the problem characteristics, and the model needs to be understandable and usable for the people who contributed to building it (Borshchev and Filippov, 2004; Martin and Schlüter, 2015). Modelers must strive to ensure their own views and favored methods do not drive the model development. As one example, an initial effort to develop an extension to a National Park in Thailand failed when modelers focused on using a spatial distribution model to demarcate boundaries. Applying different modeling techniques (including role-playing games and agent-based modeling) helped move negotiations to a more successful, innovative, and integrative discussion of solutions based on resource access and multi-functional space (Barnaud et al., 2013).
  5. Ability to recognize how one’s own intuitions can serve as biases in the PM process: Reflective thinking and self-awareness can help us assess the limits of our expertise and recognize the subjectivity in our own knowledge and judgements (Glynn, 2017). Such awareness generally improves the outcomes of our decisions (Kruger and Dunning, 1999; Wu and Dunning, 2017). Reflective thinking can help us recognize our biases, beliefs, heuristics, and values that often drive our efforts and the decisions that we make (Voinov et al., 2014; Glynn et al., 2017). For example, one of the authors collaboratively developed a flooding model to help communities design green infrastructure scenarios, where the modelers had represented flooding as a specific depth of accumulated water (Zellner et al., In Press). In collaborative design workshops with stakeholders, participants noted that there were, in fact, different definitions of flooding, depending on who and what areas were affected. The modeling team then included in the PM interfaces an ability for users to define the depth of water that constituted flooding for them (Zellner, pers. obs.). Another author realized that her own bias towards an older, male participant in a leadership role caused her to interpret his emphasis on responsibility and single-point decision making as authoritarian and power hungry. However, in the context of the stakeholder group—a group of wildland firefighters—it was a sign of caution and a risk management approach. Without this realization, important insights would have been lacking from the PM process.
  6. Ability to measure the success of the PM process: Building a model collectively is not the final goal of a PM process. Rather, the social learning that model development and use supports, and the actions that learning informs are most important. PM efforts must be evaluated, both formatively and summatively, and results shared (Hoch et al., 2015; Milz et al., 2017; Radinsky et al., 2017). Also, it is critical that modelers participate in measuring and communicating how PM helped communities to make better decisions.

Lessons learned about the social and political context of collaboration

Stakeholder groups are not monolithic; they have varying organizational structures, multiple and competing objectives, and they often evolve over time. Individual stakeholders cannot represent an entire group; stakeholders and stakeholder groups are dynamic and cross-scalar. Multiple manifestations of a “community” exist within any particular geographic region. Communities can be defined, for instance, by political, religious, genealogical/familial/clan, or by resource management oversight characteristics. Setting manageable boundaries for selection of groups to participate in a modeling initiative, and identifying in which stages different stakeholders should participate, is crucial (Prell et al., 2007; Reed et al., 2013a, 2013b, 2017). Some stakeholders need to be engaged from the very beginning (Reed et al., 2009; Cormier-Salem, 2014; Sterling et al., 2017). Yet, trying to engage every stakeholder may mean spending a lot of time reaching consensus but not creating a lot of clarity on action (Büscher and de Beer, 2011; Sterling et al., 2017). Experience, context, and clear stakeholder engagement strategies help to overcome these challenges (Sterling et al., 2017).

Navigating power asymmetries is a tricky ethical and logistical issue. On the one hand, when significant power differentials exist, inviting all stakeholders (a “neutral posture”) may simply exacerbate initial power asymmetries if less powerful participants are less able to contribute (Kritek, 2002). On the other hand, excluding or separating the more powerful participants (a “non-neutral posture”) to empower the weaker actors engenders questions regarding how people are chosen (Barnaud and Van Paassen, 2013) and potentially weakens the usefulness of the results of the PM process.

Understanding the social and political context within which one is operating is key. For instance, some stakeholders may have specific expectations about roles and relationships based on previous experiences. Others may have little or no experience in working with modelers on planning and decision-making, and may need extra assistance in learning how to participate effectively. Still others may have engagement fatigue. Governance structures differ around the world, and those structures and norms may change the real and perceived benefits of participating in a PM process. Some groups may be uncomfortable with group decision-making, preferring to defer to figures of authority. And, at times, groups are set on the decisions that they want to make, and, while they may be open to a collaborative engagement, they are resistant to new ideas. Our lessons from and about operating within a variety of social and political contexts include:

  1. In some cultures, knowledge is not shared freely: Researchers from a western science perspective often assume that people share knowledge, and share it in the same way across cultures. In some cultures, knowledge is considered to be freely available to all. In other cultures, knowledge is held more closely, and may not be shared with others until they have earned the right to learn something (van Kerkhoff, 2013; Matsui, 2015). Knowledge ownership has implications for who participates in various aspects of the process, for how information circulates, what information is available for use in the modeling processes, and if and how that information is used. Knowledge that is not represented in the PM process has little to no chance of informing the resulting decisions. Explicit criteria for a culturally appropriate process of knowledge sharing and use can help ensure these concerns are considered (Conley and Moote, 2003), even if not represented directly in models. As an example, when working in the Pacific, researchers often ask participants to specify who should have access to knowledge used in the model. This could include restricting to those who provide the information and the modelers only, adding other knowledge holders in a particular domain (for instance other fisherpeople), adding other community members but no one beyond the community, and so on (Sterling pers. obs.).
  2. Community co-leads are crucial partners in navigating social and political context: Frequently, it helps to identify a small group of advisors/co-leads from the focal community who can help you navigate the local social and political setting (Hovmand, 2014; Zellner et al., In Press). Selecting appropriate co-leads is a challenge of its own, and it is important to explore systematically who would best fill these roles, and who has appropriate knowledge and trust of the community. As an example, while working in Solomon Islands, one of the authors found that some people who have knowledge that might be of most interest to researchers may be reluctant to engage in a leadership role. In contrast, some of those who were interested in assuming that role in the process lacked the trust of the community (Sterling pers. obs.). It took deep knowledge of the community members and their relationships to determine whom to engage.
  3. Economic or policy gains may not be the (only) benefits participants care about: Participatory modelers and conveners may assume that their goals for the process are shared by those who engage with them. More specifically, they may assume that reaching a policy decision, implementing an action, and delivering economic gains to the community are important to everyone. This is not universally the case. In our work, we have found that some participants may be guided more by social norms, such as wanting to fit in or enhance their stature in a group or a community. Others may be motivated by a desire to contribute in a positive way to another community or to the next generation or to leave behind a meaningful legacy. Some may desire to preserve the present or the past, to act against the potential disappearance of a culture or way of life, or against the potential extinction of a species, or against the degradation of an environment or ecosystem (Sterling et al., 2017). Some might participate to be at the table, gain first-hand information, or try to influence participants towards their point of view. Some participants may be motivated by all of these factors. It is important for participatory modelers to recognize that the PM process is embedded in this social and political context, and that focusing too narrowly on economic or policy questions may shortchange the stakeholder community.
  4. Many people care deeply about their social standing, which can affect their participation: PM is a process that naturally engenders group dynamics. Group dynamics (such as “groupthink,” posturing and differentiation, attention seeking, and following a confident leader) can sometimes impede critical thinking and independent perspectives that are essential to improving knowledge and management of an issue (Forsyth, 2014). In situations where group dynamics are very strong and or are poorly managed, the social standing of individuals who think differently from the group may be threatened by speaking counter to the group, and they may reduce their participation. Alternatively, people might only share what they think the researcher or leader wants to know. For other individuals, social standing may be increased by being a strong voice in the group. Unmanaged concerns about social standing and power imbalances within a group may lead to a less representative process or model than intended (Hoch et al., 2015; Zellner et al., In Press).
  5. Methods used, and the way in which they are used, can significantly influence how engaged stakeholders feel: Not everyone likes structure, or “being taught.” An over-reliance on capturing input in writing (e.g., on flip charts) and on categorization can hamper data collection in societies where storytelling, narrative, and oral information exchange is the norm. Similarly, sharing information using graphs and pie charts can be off-putting for some stakeholders while being very helpful for others. Visual depictions of future scenarios accompanied by vignettes or narratives can be a powerful tool for eliciting discussion, but may also lead people to think of the particular scenarios described as the most likely or only possible futures (Spiegelhalter et al., 2011). Open and active engagement in collaborative design is crucial to tailoring the PM into a process that the community owns, masters, legitimizes, and ultimately uses to guide their actions (Zellner et al., In Press). In Solomon Islands, for instance, use of art-based facilitation elicited responses from community members who might not otherwise have shared perspectives (McCarter et al., 2018).
  6. Conscious and unconscious biases can affect processes, interactions, and models: PM processes can threaten existing social and political order, and the existing social and political order can threaten the integrity and usefulness of PM. Not everyone is open to learning that their preferred decision may fail. The reluctance can affect the models themselves: people may select data to fit their pre-existing mental model (Tversky and Kahneman, 1974) or cultural model or decision, or to seek confirmation of their hypotheses (Anderson et al., 1980; Nickerson, 1998; Sébastien and Bauler, 2013; Glynn et al., 2017). This reluctance can affect the acceptance and implementation of PM results: people may show more distrust when outputs do not match their expectations (Hoch et al., 2015). Many innate prioritizations unconsciously affect the way we make decisions, conduct science, and manage ourselves, our communities, and our environments and resources. Recognizing these prioritizations improves PM (Hämäläinen and Alaja, 2008; Glynn, 2014, 2017; Hämäläinen, 2015; Morewedge et al., 2015; Voinov et al., 2016; Glynn et al., 2017).

Lessons learned about the PM process

Managing the participation process in PM is as important as managing the model building process. Models within a collaborative process can function as “boundary objects” (Star and Griesemer, 1989; Harvey and Chrisman, 1998), providing a means to bridge ideas across disciplines and participants’ perspectives, and thus promoting learning through collaboration (Akkerman and Bakker, 2011; Zellner et al., 2012). PM processes help stakeholders bring to light assumptions, causes, solutions, and values of which they may have only subconscious awareness. This provides opportunities for users to make ideas visible and open for discussion, negotiation, and revision, and supports constructive discourse. PM models also allow individual and collective cognition to be externalized and made explicit, by mentally offloading difficult tasks into an environment (e.g., computer screens and notebooks) where thinking can be organized and discussed (Bart, 1995; Zellner, 2008). Furthermore, because modeling forces us to explicitly formalize diverse knowledges, ideally individuals coming from different backgrounds should be able to communicate in this shared workspace (van der Leeuw, 2004).

Robust, meaningful, and impactful PM is a huge effort, especially if compared to a more traditional top-down or authoritative decision-making process. It takes time, funding, and iterative engagement to build the relationships and trust among the participants, tools, and process (particularly how to navigate conflict and tradeoffs), in addition to building the models themselves. That said, once the iterative process is established, further collective innovation and action may come more easily and quickly. Participants become used to thinking collaboratively with modeling tools. Slowing down to engage in such processes paradoxically gets communities to where they want to be, faster (Zellner and Campbell, In Press).

Previous experiences with collaborative processes shape participant expectations of PM, however. Community trust in such processes can be eroded if conveners of a past collaborative process did not take their input seriously, or if members are afraid to voice their opinions, perspectives, and knowledge in public. Some processes become driven more by the modelers and by the needs of a quantitative model than the interests and needs of the community. This can establish an expectation that the modelers are “in charge” and lead to less innovation and participation from the community in model development and use.

Lessons we have learned about how to manage these issues include:

  1. PM is not always the right approach: Evaluating the effectiveness of PM is inherently complex, and there is no universal agreement about its benefits. Perhaps especially because we are participatory modelers, it is crucial to consider explicitly whether the benefits of PM are relevant and sufficiently valuable for the problems being considered to be worth the time and effort PM requires. For example, in cases of emergency one might want to follow a top-down approach rather than having extensive discussions. If the social and political context will not allow true decision-making authority to go to a participatory group, then it might not be a wise use of time to engage in PM.
  2. Participants’ needs, rights, and relationships must be respected and protected: Free, prior, and informed consent (FPIC) is a principle mandated by international human rights standards that is fundamental to ethical engagement (FAO, 2016). This principle ensures that there is no coercion to stakeholder engagement, that stakeholders have the opportunity to consent or deny engagement prior to commencement of activities, and that they are informed regarding the details of the engagement, including but not limited to scale, scope, purpose, implications, and potential impacts including risks. However, obtaining FPIC can be challenging. Sometimes the process of informed consent raises suspicion, making participants wary of signing documents and prompting questions about risks involved in participation. In the initial stages of collaboration, it is necessary to build trust and explain how informed consent means participants should have the freedom to act and speak without concern about what others outside of the process might think of them. It is important to be aware of power dynamics that favor consent and to allow for critical inquiry and dissent (Forsyth, 2014).
  3. The most informative results arise from fearless and playful collaborative modeling: The first model is rarely the best model, and one of the underlying motivations for PM is the belief that we get better models, and better and more useful results by engaging the full knowledge of the community. It takes time to build the collaboration and confidence in model conclusions and model limitations (Zellner and Campbell, 2015). Confidence increases as understanding increases, and open, iterative modeling where all questions are explored with an open mind increases that understanding. Processes that create an environment where mistakes are acceptable and even desirable for learning and innovation help build confidence. Art and improvisation exercises can be especially useful to build and connect different pieces, overcome the fear of trial-and-error, create relationships among people who do not know each other well, or strengthen relationships among those who do not trust each other (Pink, 2006).
  4. Stakeholder expectations may exceed the scope of the PM process: Communication is key among researchers and between researchers and other stakeholders and participants. It is particularly important to understand and communicate clearly about the process, what the outcomes will be, what products will result from the process, and how they will be shared or used. When stakeholders have expectations that cannot be met, they may disengage from the process, and may be more reluctant to engage in future collaborative processes. For instance, when working with herders in Kenya, Reid et al. (2016) suggested that external organizers be clear regarding their ability to meet a request. Especially important is addressing those situations where researchers do not have expertise and therefore need to reach out to others for relevant resources or to co-generate new knowledge. It is useful to iteratively state, revise, and refine expectations throughout the process, to ensure everyone knows what they are doing and why. This can help maintain a clear definition of the kind of outcomes that are expected, and increases the likelihood of acceptance of the outcomes.
  5. Conflict management may be necessary: Processes that focus on positions in negotiations often mask the underlying needs people have but cannot articulate (Susskind et al., 1999). A good process will help people identify their motivations, which in some conflict situations might be related to feelings of being disrespected or undervalued, or to disparities in social power among participants (Sturm and Antonakis, 2015; Simpson et al., 2015). Listening to how stakeholders talk about an issue or learning about their relationships with each other can help identify other factors that may be driving disagreements, such as history, identity conflicts, or deep value differences (Madden and McQuinn, 2014). Respectful listening to, and acknowledgment of, opposing points of view can be helpful in resolving differences and conflicts and in fostering innovation, especially with power dynamics in mind (Barnaud and Van Paassen, 2013; Minter et al., 2014). Learning and trust-building, for both scientists and communities, is a critical outcome of collaborative processes, including PM (Susskind et al., 1999; Conley and Wondolleck and Yaffee, 2000; Moote, 2003; Gray et al., 2017).
  6. Exploration and compromise may be more valuable than consensus: PM generally involves multiple sources of knowledge stemming from diverse knowledge systems, where differing opinions and proposed strategies are common. One knowledge system should not be used to validate or invalidate information from another system. Rather, credible, salient, and legitimate knowledge (as defined by Cash et al., 2003) from within a knowledge system can be brought to the table and the congruences and differences explored (Tengö et al., 2014). This often involves balancing rigor and flexibility in what constitutes “evidence” (Sterling et al., 2017). Rather than attempting to minimize differences when they arise, it is important to recognize how differences shed light on diverse parts of a system or on the varying tradeoffs across solutions (Zellner et al., In Press). Attempts to reach general agreement/consensus across differences can, at times, completely invalidate a set of values or knowledge. True differences can be overcome by focusing on mutually valued goals, or by working with these differences to assess what participants give up and how to compensate for it.

Conclusions

PM is a powerful approach for addressing complex social and environmental problems. While it holds great promise, it can also come at a high cost: it is difficult to gather adequate skills, funds, and participants’ time, and it takes time to gradually build strong relationships between scientists, community partners, and public agencies. It may require that actors with decision-making responsibilities cede some of their authority to a group and process over which they have little control. Based on the lessons presented above, we conclude our reflection with some cross-cutting recommendations.

Members of a project team should reflect on their abilities with regards to our lessons learned on the role of participatory modelers. All participatory modelers need to be aware of how to support relationship building among participants, and design and use tools accordingly. If critical skills are missing, the project team should first build its own capacity and may consider recruiting project partners with the needed expertise. Successful PM efforts create an inclusive environment that supports participants with differing values, ideas, and priorities. Sharing (time, viewpoints, stories, common challenges, food, small slices of life, etc.) helps to develop these relationships and understanding. Stakeholders know more about the problem, obstacles and opportunities, and the community than do modelers from outside the community. Provide time for people to understand the purpose of the modeling, but more importantly, for participants to share their wants and needs. Modelers are a support to them, to help them achieve their goals and aspirations. Find ways to “check in” regularly and in different ways to see how learning is progressing in individuals and across the collaboration, and how stakeholders are perceiving the collaboration and the issues in general (Bennett, 2016). This includes structured observations of the dialogue and deliberations occurring around the PM tools, and what actions participants have collectively agreed to support. Power dynamics frequently influence how people from different backgrounds interact. Try to understand the motivations underlying participants’ behavior, and explicitly acknowledge and legitimize those motivations. Note that researchers are all part of the process, not outsiders taking objective notes from afar, so modelers should also expose their motivations and reflect on their biases.

Whatever tools and techniques are used, make sure to work with a subset of the stakeholders to design the PM with them. Not only will the PM process (tools, location and time, facilitation and deliberation setups) thus be comfortable to users, but also the non-modeler participants become advocates and facilitators of these tools, building trust in the process, and easing the transition towards self-reliance and appropriate tool use to inform decision-making. Provide feedback to stakeholders on model development and, to groups who were not involved in its development, on prior model use and outcomes. Recruit and, if necessary, train different kinds of facilitators (technical/modeling, community) to help manage the social interactions among community participants, between modelers and participants, and between users and the tools developed and used (Hovmand, 2014).

Consider how to deal with dissonance between what participants initially expect and what the model suggests; if the gap is too wide, it can be difficult for participants to embrace the results and make effective decisions. This discomfort may be compounded by the fact that uncertainty is inherent in any complex problem that is the focus of the PM activity, and is often reflected in the modeling process and outcomes. Participants may equate such uncertainty to ignorance, and thus distrust the modeling effort and its insights, instead of incorporating the uncertainty into the design of robust policies that can be effective in an uncertain world (Zellner 2008). Scaffolding between expectations and results needs to be built into the PM process, in order to address confirmatory bias (Hoch et al., 2015; Zellner et al., In Press).

Keep models simple, relevant, and tractable, distilled to key decision points of importance to the participants (Zellner et al., In Press). The purpose of PM is for stakeholders to participate in the analysis of the complex issues they face, to gain insights relative to their roles in problems and in their solutions, to harness their collective creativity in designing solutions, and to examine and deliberate about the various tradeoffs with appropriate information. If models become intractable or irrelevant “black boxes” that do not adequately represent and make it easy for stakeholders to examine assumptions, goals, and values, then that possibility is lost.

Finally, PM involves the community in understanding and modeling what are typically complex human, social, and natural systems. Although a project may “wrap up,” achieving its decision-specific goals, the community and those complex systems continue to exist and evolve. A valuable result of PM would be to foster ongoing dialogue and collaborative analysis that is adaptive to the inevitable surprises brought about by complex problems and systems. Theoretical frameworks, case studies, and individual experiences with PM and other forms of collaborative governance are signaling what some have identified as a “new governance era” (e.g., Leong et al., 2011). Yet, these approaches are not fully integrated into agency policies and practices (Leong et al., 2011; Zellner and Campbell, 2015) despite notable international efforts and commitments such as the Local Agenda 21 coming out of the UN Rio Declaration on Environment and Development, the European Union’s (EU) Aarhus Convention on Access to Information, Public Participation in Decision-Making and Access to Justice in Environmental Matters, and the EU’s Water Framework Directive. Having overcome the initial hurdles of setting up a PM process, it is worth investing in collaborative stakeholder networks to facilitate ongoing learning and exchanges from peer to peer, outside of PM efforts. Long-term relationships with stakeholders can carry across many complex projects, transferring the ways of thinking across projects and building the capacity to effectively deal with such complexity in different cases.

Our hope is that these lessons are useful to new practitioners who venture on this path, with a realistic view of what PM entails, recommendations of steps to take to ensure a productive process, and the enthusiasm and confidence in this approach as a support for a new form of collaborative governance to grapple with difficult and persistent problems.

Acknowledgements

Co-authors of this paper are U.S. Government employees, therefore: Any use of trade, product, or firm names in this publication is for descriptive purposes only and does not imply endorsement by the U.S. Government. We thank Nadav Gazit for comments, technical edits, and graphic design, Don Rosenberry and Collin Lawrence for U.S. Geological Survey technical reviews, and Amanda Sigouin for editing support. We appreciate the helpful comments from two outside reviewers and the thoughtful guidance from the editors.

Funding information

This work was supported by the National Socio-Environmental Synthesis Center (SESYNC) under funding received from the National Science Foundation DBI-1052875. The material is also based upon work supported by the National Science Foundation (NSF) under Grants No. EF-1427091, 1427453, and 1444184. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. In addition, this research was conducted by the Assessing Biocultural Indicators Working Group supported in part by SNAPP: Science for Nature and People Partnership, a collaboration of The Nature Conservancy, the Wildlife Conservation Society and the National Center for Ecological Analysis and Synthesis (NCEAS) at the University of California, Santa Barbara.

Competing interests

The authors have no competing interests to declare.

Author contributions

Contributed to conception and design: EJS, MZ, KEJ, KL; Contributed to acquisition of data: All authors; Contributed to analysis and interpretation of data: EJS, MZ, KEJ, KL, PDG; Drafted and/or revised the article: All authors; Approved the submitted version for publication: EJS, MZ.

References

  1. Akkerman, SF and Bakker, A. 2011. Boundary crossing and boundary objects. Rev Educ Res 81(2): 132–169. DOI: 10.3102/0034654311404435

  2. Anderson, CA, Lepper, MR and Ross, L. 1980. Perseverance of social theories: The role of explanation in the persistence of discredited information. J Pers Soc Psychol 39(6): 1037. DOI: 10.1037/h0077720

  3. Barnaud, C, Le Page, C, Dumrongrojwatthana, P and Trébuil, G. 2013. Spatial representations are not neutral: Lessons from a participatory agent-based modelling process in a land-use conflict. Environmental Modelling & Software 45: 150–159. DOI: 10.1016/j.envsoft.2011.11.016

  4. Barnaud, C and Van Paassen, A. 2013. Equity, power games, and legitimacy: Dilemmas of participatory natural resource management. Ecology and Society 18(2): 21. DOI: 10.5751/ES-05459-180221

  5. Barreteau, O, Antona, M, D’Aquino, P, Aubert, S, Boissau, S, Bousquet, F, Daré, W, Etienne, M, Le Page, C and Mathevet, R. 2003. Our companion modelling approach. JASSS-J Artif Soc S 6(1): n.p.

  6. Bart, J. 1995. Acceptance criteria for using individual-based models to make management decisions. Ecol Appl 5(2): 411–420. DOI: 10.2307/1942032

  7. Becu, N, Neef, A, Schreinemachers, P and Sangkapitux, C. 2007. Participatory computer simulation to support collective decision-making: Potential and limits of stakeholder involvement. Land Use Policy 25(4): 498–509. DOI: 10.1016/j.landusepol.2007.11.002

  8. Bennett, NJ. 2016. Using perceptions as evidence to improve conservation and environmental management. Conserv Biol 30(3): 582–592. DOI: 10.1111/cobi.12681

  9. Borshchev, A and Filippov, A. 2004. From system dynamics and discrete event to practical agent based modeling: Reasons, techniques, tools. The 22nd International Conference of the System Dynamics Society, July 25–29. Oxford, England.

  10. Büscher, B and de Beer, E. 2011. The contemporary paradox of long-term planning for social-ecological change and its effects on the discourse-practice divide: evidence from Southern Africa. J Environ Plann Man 54(3): 301–318. DOI: 10.1080/09640568.2010.506075

  11. Cash, DW, Clark, WC, Alcock, F, Dickson, NM, Eckley, N, Guston, DH, Jäger, J and Mitchell, RB. 2003. Knowledge systems for sustainable development. Proceedings of the national academy of sciences 100(14): 8086–8091. DOI: 10.1073/pnas.1231332100

  12. Charmaz, K and Belgrave, LL. 2007. Grounded Theory. In: Ritzer, G (ed.), The blackwell encyclopedia of sociology 4: 25. Oxford, UK: John Wiley & Sons, Ltd. DOI: 10.1002/9781405165518.wbeosg070.pub2

  13. Conley, A and Moote, MA. 2003. Evaluating collaborative natural resource management. Society and Natural Resources 16(5): 371–386. DOI: 10.1080/08941920309181

  14. Cormier-Salem, M-C. 2014. Participatory governance of Marine Protected Areas: A political challenge, an ethical imperative, different trajectories. Senegal case studies. SAPIENS 7(2).

  15. Douglas, EM, Wheeler, SA, Smith, DJ, Overton, IC, Gray, SA, Doody, TM and Crossman, ND. 2016. Using mental-modelling to explore how irrigators in the Murray–Darling Basin make water-use decisions. J Hydrol: Reg Stud 6: 1–12. DOI: 10.1016/j.ejrh.2016.01.035

  16. FAO. 2016. Free Prior and Informed Consent: An indigenous peoples’ right and a good practice for local communities. Manual for Project Practitioners . FAO. Available at: http://www.fao.org/3/a-i6190e.pdf.

  17. Fischhoff, B. 1995. Risk perception and communication unplugged: Twenty years of process. Risk Analysis 15(2): 137–145. DOI: 10.1111/j.1539-6924.1995.tb00308.x

  18. Forsyth, DR. 2014. Group Dynamics . 6th ed. Belmont, CA: Wadsworth Cengage Learning 34.

  19. Glynn, PD. 2014. W(h)ither the oracle? Cognitive biases and other human challenges of integrated environmental modeling. In: Ames, DP, Quinn, NWT and Rizzoli, AE (eds.), 7th Intl Congress on Env Modelling and Software . San Diego, USA: International Environmental Modelling and Software Society.

  20. Glynn, PD. 2017. Integrated environmental modelling: Human decisions, human challenges. Geol Soc London Spec Publ 408(1): 161–182. DOI: 10.1144/SP408.9

  21. Glynn, PD, Voinov, AA, Shapiro, CD and White, PA. 2017. From data to decisions: Processing information, biases, and beliefs for improved management of natural resources and environments. Earth’s Future 5(4): 356–378. DOI: 10.1002/2016EF000487

  22. Gray, S, Gagnon, A, Gray, S, O’Dwyer, B, O’Mahony, C, Muir, D, Devoy, RJN, Falaleeva, M and Gault, J. 2014. Are coastal managers detecting the problem? Assessing stakeholder perception of climate vulnerability using Fuzzy Cognitive Mapping. Ocean Coast Manage 94: 74–89. DOI: 10.1016/j.ocecoaman.2013.11.008

  23. Gray, S, Jordan, R, Crall, A, Newman, G, Hmelo-Silver, C, Huang, J, Novak, W, Mellor, D, Frensley, T, Prysby, M and Singer, A. 2017. Combining participatory modelling and citizen science to support volunteer conservation action. Biol Conserv 208: 76–86. DOI: 10.1016/j.biocon.2016.07.037

  24. Gray, S, Voinov, A, Paolisso, M, Jordan, R, BenDor, T, Bommel, P, Glynn, P, Hedelin, B, Hubacek, K, Introne, J, Kolagani, N, Laursen, B, Prell, C, Schmitt Olabisi, L, Singer, A, Sterling, E and Zellner, M. 2018. Purpose, processes, partnerships, and products: Four Ps to advance participatory socio-environmental modeling. Ecol Appl 28(1): 46–61. DOI: 10.1002/eap.1627

  25. Hämäläinen, RP. 2015. Behavioural issues in environmental modelling–The missing perspective. Environ Modell Softw 73: 244–253. DOI: 10.1016/j.envsoft.2015.08.019

  26. Hämäläinen, RP and Alaja, S. 2008. The threat of weighting biases in environmental decision analysis. Ecol Econ 68(1–2): 556–569. DOI: 10.1016/j.ecolecon.2008.05.025

  27. Harvey, F and Chrisman, N. 1998. Boundary objects and the social construction of GIS technology. Environ Plann A 30(9): 1683–1694. DOI: 10.1068/a301683

  28. Hoch, C, Zellner, M, Milz, D, Radinsky, J and Lyons, L. 2015. Seeing is not believing: Cognitive bias and modelling in collaborative planning. Plan Theor Pract 16(3): 319–335. DOI: 10.1080/14649357.2015.1045015

  29. Hovmand, PS. 2014. Community based system dynamics . 1st ed. New York, USA: Springer-Verlag. DOI: 10.1007/978-1-4614-8763-0

  30. Htun, H, Gray, SA, Lepczyk, CA, Titmus, A and Adams, K. 2016. Combining watershed models and knowledge-based models to predict local-scale impacts of climate change on endangered wildlife. Environ Modell Softw 84: 440–457. DOI: 10.1016/j.envsoft.2016.07.009

  31. Hubacek, K, Beharry, N, Bonn, A, Burt, T, Holden, J, Ravera, F, Reed, M, Stringer, L and Tarrasón, D. 2009. Ecosystem services in dynamic and contested landscapes: The case of UK uplands. In: Winter, M, Lobley, M (eds.), What is land for? The Food, Fuel and Climate Change Debate , 167–188. London, UK: Earthscan.

  32. Kaner, S. 2014. Facilitator’s Guide to Participatory Decision-Making , 3rd ed. Hoboken, New Jersey: Jossey-Bass. 363. (Jossey-Bass Business & Management Series).

  33. Kritek, PB. 2002. Negotiating at an uneven table: Developing moral courage in resolving our conflicts . 2nd ed. Hoboken, New Jersey: Jossey-Bass. (Jossey-Bass Health Series).

  34. Kruger, J and Dunning, D. 1999. Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol 77(6): 1121. DOI: 10.1037/0022-3514.77.6.1121

  35. Leong, KM, Decker, DJ, Lauber, TB and Chase, L. 2012. Stakeholders as beneficiaries of wildlife management. In: Decker, DJ, Riley, SJ and Siemer, WF (eds.), Human dimensions of wildlife management , 2nd ed., 26–41. Baltimore, USA: The Johns Hopkins University Press.

  36. Leong, KM, Decker, DJ, Lauber, TB, Raik, DB and Siemer, WF. 2009. Overcoming jurisdictional boundaries through stakeholder engagement and collaborative governance: Lessons learned from white-tailed deer management in the U.S. In: Andersson, K, Eklund, E, Lehtola, M and Salmi, P (eds.), Beyond the rural-urban divide: Cross-continental perspectives on the differentiated countryside and its regulation 360: 221–247. Bingley, UK: Emerald Publishing Group.

  37. Leong, KM, Emmerson, DP and Byron, R. 2011. The new governance era: Implications for collaborative conservation and adaptive management in department of the interior agencies. Hum Dimens Wildl 16(4): 236–243. DOI: 10.1080/10871209.2011.585436

  38. Madden, F and McQuinn, B. 2014. Conservation’s blind spot: The case for conflict transformation in wildlife conservation. Biol Conserv 178: 97–106. DOI: 10.1016/j.biocon.2014.07.015

  39. Martin, R and Schlüter, M. 2015. Combining system dynamics and agent-based modeling to analyze social-ecological interactions—An example from modeling restoration of a shallow lake. Front Environ Sci 3(66): 1–15. DOI: 10.3389/fenvs.2015.00066

  40. Matsui, K. 2015. Problems of defining and validating traditional knowledge: A historical approach. The International Indigenous Policy Journal 6(2). Available at: https://ir.lib.uwo.ca/iipj/vol6/iss2/2. DOI: 10.18584/iipj.2015.6.2.2

  41. McCarter, J, Sterling, E, Jupiter, S, Cullman, G, Albert, S, Basi, M, Betley, E, Boseto, D, Bulehite, E, Harron, R, Holland, P, Horning, N, Hughes, A, Jino, N, Malone, C, Mauli, S, Pae, B, Papae, R, Rence, F, Revo, O, Taqala, E, Taqu, M, Woltz, H and Filardi, CE. 2018. Biocultural approaches to developing well-being indicators in Solomon Islands. Ecology and Society 23(1). DOI: 10.5751/ES-09867-230132

  42. Milz, D, Zellner, M, Hoch, C, Radinsky, J, Pudlock, K and Lyons, L. 2017. Reconsidering scale: Using GIS to inform spatial planning talk. Planning Practice & Research , 1–18.

  43. Minter, T, van der Ploeg, J, Pedrablanca, M, Sunderland, T and Persoon, GA. 2014. Limits to indigenous participation: The Agta and the northern sierra Madre Natural Park, the Philippines. Hum Ecol 42(5): 769–778. DOI: 10.1007/s10745-014-9673-5

  44. Morewedge, CK, Yoon, H, Scopelliti, I, Symborski, CW, Korris, JH and Kassam, K. 2015. Debiasing decisions: Improved decision making with a single training intervention. Policy Insights from the Behavioral and Brain Sciences 2(1): 129–140. DOI: 10.1177/2372732215600886

  45. Nayaki, A, Gray, S, Lepczyk, C, Skibins, J and Rentsch, D. 2014. Local-scale dynamics and local drivers of bushmeat trade. Conserv Biol 28(5): 1403–1414. DOI: 10.1111/cobi.12316

  46. Nickerson, RS. 1998. Confirmation bias: A ubiquitous phenomenon in many guises. Rev Gen Psychol 2(2): 175. DOI: 10.1037/1089-2680.2.2.175

  47. Oteros-Rozas, E, Martín-López, B, Daw, TM, Bohensky, EL, Butler, JR, Hill, R, Martin-Ortega, J, Quinlan, A, Ravera, F, Ruiz-Mallén, I, Thyresson, M, Mistry, J, Palomo, I, Peterson, GD, Plieninger, T, Waylen, KA, Beach, D, Bohnet, IC, Hamann, M, Hanspach, J, Hubacek, K, Lavorel, S and Vilardy, S. 2015. Participatory scenario planning in place-based social-ecological research: Insights and experiences from 23 case studies. Ecol Soc 20(4): 32. DOI: 10.5751/ES-07985-200432

  48. Pink, D. 2006. A whole new mind: Why right-brainers will rule the future. New York, USA: Penguin Group.

  49. Prell, C, Hubacek, K, Reed, M, Quinn, C, Jin, N, Holden, J, Burt, T, Kirby, M and Sendzimir, J. 2007. If you have a hammer everything looks like a nail: Traditional versus participatory model building. Interdiscipl Sci Rev 32(3): 263–282. DOI: 10.1179/030801807X211720

  50. Radinsky, J, Milz, D, Zellner, M, Pudlock, K, Witek, C and Lyons, L. 2017. How planners and stakeholders learn with visualization tools: Using learning sciences methods to examine planning processes. J Environ Plann Man 60(7): 1296–1323. DOI: 10.1080/09640568.2016.1221795

  51. Railsback, SF and Grimm, V. 2012. Agent-based and individual-based modeling: A practical introduction. Princeton, USA: Princeton University Press.

  52. Reed, MS, Graves, A, Dandy, N, Posthumus, H, Hubacek, K, Morris, J, Prell, C, Quinn, CH and Stringer, LC. 2009. Who’s in and why? A typology of stakeholder analysis methods for natural resource management. J Environ Manage 90(5): 1933–1949. DOI: 10.1016/j.jenvman.2009.01.001

  53. Reed, MS, Hubacek, K, Bonn, A, Burt, TP, Holden, J, Stringer, LC, Beharry-Borg, N, Buckmaster, S, Chapman, D, Chapman, PJ, Clay, GD, Cornell, SJ, Dougill, AJ, Evely, AC, Fraser, EDG, Jin, N, Irvine, BJ, Kirkby, MJ, Kunin, WE, Prell, C, Quinn, CH, Slee, B, Stagl, S, Termansen, M, Thorp, S and Worrall, F. 2013a. Anticipating and managing future trade-offs and complementarities between ecosystem services. Ecol Soc 18(1): 5. DOI: 10.5751/ES-04924-180105

  54. Reed, MS, Kenter, J, Bonn, A, Broad, K, Burt, T, Fazey, IR, Fraser, EDG, Hubacek, K, Nainggolan, D, Quinn, CH, Stringer, LC and Ravera, F. 2013b. Participatory scenario development for environmental management: A methodological framework illustrated with experience from the UK uplands. J Environ Manage 128: 345–362. DOI: 10.1016/j.jenvman.2013.05.016

  55. Reed, MS, Vella, S, Challies, E, de Vente, J, Frewer, L, Hohenwallner-Ries, D, Huber, T, Neumann, RK, Oughton, EA, del Ceno, JS and van Delden, H. 2017. A theory of participation: What makes stakeholder and public engagement in environmental management work? Restor Ecol . DOI: 10.1111/rec.12541

  56. Reid, R, Nkedianye, D, Said, M, Kaelo, D, Neselle, M, Makui, O, Onetu, L, Kiruswa, S, Ole Kamuaro, N, Kristjanson, P, Ogutu, J, BurnSilver, SB, Goldman, MJ, Boone, RB, Galvin, KA, Dickson, NM and Clark, WC. 2016. Evolution of models to support community and policy action with science: Balancing pastoral livelihoods and wildlife conservation in savannas of East Africa. P Natl Acad Sci USA 113(17): 4579–4584. DOI: 10.1073/pnas.0900313106

  57. Schmitt Olabisi, L, Liverpool-Tasie, S, Rivers, L, Ligmann-Zielinska, A, Du, J, Denny, R, Marquart-Pyatt, S and Sidibé, A. 2017. Using participatory modeling processes to identify sources of climate risk in West Africa. Environment Systems and Decisions 32: 23. DOI: 10.1007/s10669-017-9653-6

  58. Sébastien, L and Bauler, T. 2013. Use and influence of composite indicators for sustainable development at the EU-level. Ecol Indic 35: 3–12. DOI: 10.1016/j.ecolind.2013.04.014

  59. Simpson, JA, Farrell, AK, Orina, MM and Rothman, AJ. 2015. Power and social influence in relationships. In: Mikulincer, M and Shaver, PR (eds.), APA Handbook of personality and social psychology: Interpersonal relations 3. Washington, DC, USA: American Psychological Association. DOI: 10.1037/14344-015

  60. Singer, A, Jetter, A, Ellsworth, L, Gray, S, Zhang, P and Oussama Laraichi, M. 2017. Policy scenarios for fire-adapted communities: Understanding stakeholder risk perceptions in Ashland, Oregon. Project Report . Portland, USA: Portland State University. Available at: www.mentalmodeling.org.

  61. Spetzler, C, Winter, H and Meyer, J. 2016. Decision quality: Value creation from better business decisions. Hoboken, USA: John Wiley & Sons. DOI: 10.1002/9781119176657

  62. Spiegelhalter, D, Pearson, M and Short, I. 2011. Visualizing uncertainty about the future. Science 333(6048): 1393–1400. DOI: 10.1126/science.1191181

  63. Star, SL and Griesemer, JR. 1989. Institutional ecology, translations’ and boundary objects: Amateurs and professionals in Berkeley’s Museum of Vertebrate Zoology, 1907–39. Soc Stud Sci 19(3): 387–420. DOI: 10.1177/030631289019003001

  64. Sterling, EJ, Betley, E, Sigouin, A, Gomez, A, Toomey, A, Cullman, G, Malone, C, Pekor, A, Arengo, F, Blair, M, Filardi, C, Landrigan, K and Porzecanski, AL. 2017. Assessing the evidence for stakeholder engagement in biodiversity conservation. Biol Conserv 209: 159–171. DOI: 10.1016/j.biocon.2017.02.008

  65. Sturm, RE and Antonakis, J. 2015. Interpersonal power: A review, critique, and research agenda. J Manag 41(1): 136–163. DOI: 10.1177/0149206314555769

  66. Susskind, L, McKearnan, S and Thomas-Larmer, J. 1999. The Consensus Building Handbook: A Comprehensive Guide to Reaching Agreement. Thousand Oaks, USA: SAGE Publication Ltd.

  67. Tengö, M, Brondizio, ES, Elmqvist, T, Malmer, P and Spierenburg, M. 2014. Connecting diverse knowledge systems for enhanced ecosystem governance: The multiple evidence base approach. Ambio 43(5): 579–591. DOI: 10.1007/s13280-014-0501-3

  68. Tversky, A and Kahneman, D. 1974. Judgment under uncertainty: Heuristics and biases. Science 185(4157): 1124–1131. DOI: 10.1126/science.185.4157.1124

  69. Van Der Leeuw, SE. 2004. Why model? Cybernet Syst 35(2–3): 117–128. DOI: 10.1080/01969720490426803

  70. van Kerkhoff, L. 2013. Knowledge governance for sustainable development: A review. Challenges in Sustainability 1(2): 82–93. DOI: 10.12924/cis2013.01020082

  71. Voinov, A and Bousquet, F. 2010. Modelling with stakeholders. Environ Modell Softw 25(11): 1268–1281. DOI: 10.1016/j.envsoft.2010.03.007

  72. Voinov, A and Brown Gaddis, EJ. 2008. Lessons for successful participatory watershed modeling: A perspective from modeling practitioners. Ecol Model 216(2): 197–207. DOI: 10.1016/j.ecolmodel.2008.03.010

  73. Voinov, A, Kolagani, N, McCall, MK, Glynn, PD, Kragt, ME, Ostermann, FO, Pierce, SA and Ramu, P. 2016. Modelling with stakeholders–next generation. Environ Modell Softw 77: 196–220. DOI: 10.1016/j.envsoft.2015.11.016

  74. Voinov, A, Seppelt, R, Reis, S, Nabel, JE and Shokravi, S. 2014. Values in socio-environmental modelling: Persuasion for action or excuse for inaction. Environ Modell Softw 53: 207–212. DOI: 10.1016/j.envsoft.2013.12.005

  75. Wu, K and Dunning, D. 2017. Hypocognition: Making sense of the landscape beyond one’s conceptual reach. Rev Gen Psychol 1(2): 25–35. DOI: 10.1037/gpr0000126

  76. Wondolleck, JM and Yaffee, SL. 2000. Making collaboration work: Lessons from innovation in natural resource management . Washington, DC, USA: Island Press.

  77. Zellner, ML. 2008. Embracing complexity and uncertainty: The potential of agent-based modeling for environmental planning and policy. Plan Theor Pract 9(4): 437–457. DOI: 10.1080/14649350802481470

  78. Zellner, ML and Campbell, SD. 2015. Planning for deep-rooted problems: What can we learn from aligning complex systems and wicked problems? Plan Theor Pract 16(4): 457–478. DOI: 10.1080/14649357.2015.1084360

  79. Zellner, ML and Campbell, S. n.d. Planning with(in) complexity: Pathways to extend collaborative planning, incremental planning, and Big Data with complex systems modelling. In: de Roo, G, Yamu, C and Zuidem, C (eds.), Research handbook on planning and complexity . Cheltenham, UK: Edward Elgar Publishing. In Press.

  80. Zellner, ML, Keller, J and Massey, D. 2018. Robbins Renewal and Resiliency Project Management. Report to the Chicago Community Trust, Chicago, IL: University of Illinois at Chicago.

  81. Zellner, ML, Lyons, LB, Hoch, CJ, Weizeorick, J, Kunda, C and Milz, DC. 2012. Modeling, learning, and planning together: An application of participatory agent-based modeling to environmental planning. URISA Journal 24(1): 77–93.

  82. Zellner, ML, Lyons, L, Milz, D, Shelley, J, Hoch, C, Massey, D and Radinsky, J. n.d. Participatory complex systems modeling for environmental planning: Opportunities and barriers to learning and policy innovation. In: Porter, WF, Zhao, J, Schmitt Olabisi, L and McNall, M (eds.), Innovations in collaborative modeling . East Lansing, USA: Michigan State University Press. In Press.