Start Submission

Reading: Linking knowledge to action in ocean ecosystem management: The Ocean Modeling Forum


A- A+
Alt. Display

Policy Bridge

Linking knowledge to action in ocean ecosystem management: The Ocean Modeling Forum


Tessa B. Francis ,

Puget Sound Institute, University of Washington Tacoma, Tacoma, Washington, US
X close

Phillip S. Levin,

The Nature Conservancy of Washington, Seattle, Washington; University of Washington, Seattle, Washington, US
X close

Andre E. Punt,

University of Washington, Seattle, Washington, US
X close

Isaac C. Kaplan,

Northwest Fisheries Science Center, National Marine Fisheries Service, National Oceanic and Atmospheric Administration, Seattle, Washington, US
X close

Anna Varney,

Northwest Fisheries Science Center, National Marine Fisheries Service, National Oceanic and Atmospheric Administration, Seattle, Washington, US
X close

Karma Norman

Northwest Fisheries Science Center, National Marine Fisheries Service, National Oceanic and Atmospheric Administration, Seattle, Washington, US
X close


The sustainable use of global marine resources depends upon science-based decision processes and systems. Informing decisions with science is challenging for many reasons, including the nature of science and science-based institutions. The complexity of ecosystem-based management often requires the use of models, and model-based advice can be especially difficult to convert into policies or decisions. Here, we suggest five characteristics of model-based information and advice for successfully informing ocean management decision-making, based on the Ocean Modeling Forum framework. Illustrated by examples from two fisheries case studies, Pacific sardines Sardinops sagax and Pacific herring Clupea pallasii, we argue that actionable model-based output should be aspirational, applicable, parsimonious, co-produced, and amplifying. 

Knowledge Domain: Ocean Science
How to Cite: Francis, T.B., Levin, P.S., Punt, A.E., Kaplan, I.C., Varney, A. and Norman, K., 2018. Linking knowledge to action in ocean ecosystem management: The Ocean Modeling Forum. Elem Sci Anth, 6(1), p.83. DOI:
 Published on 20 Dec 2018
 Accepted on 17 Nov 2018            Submitted on 02 Mar 2018
Domain Editor-in-Chief: Jody W. Deming; Department of Biological Oceanography, University of Washington, US
Associate Editor: Julie E. Keister; School of Oceanography, University of Washington, US


Managing the Earth’s marine resources is an increasingly complex task. Demands on ocean resources mount as a growing human population identifies more uses for the services and benefits provided by the world’s oceans, and builds the capacity to access them (Crowder et al., 2006; Halpern et al., 2015). Supported by technological advances, the areal extent of human activities and impacts has expanded steadily over the past half century (Merrie et al., 2014). In addition, the number of countries engaging in the extraction of ocean resources is increasing. This increase is especially prevalent in previously inaccessible waters outside of national jurisdiction, further adding to the complexity of the ocean management landscape (Merrie et al., 2014). Considerable effort is being devoted to devising management programs and strategies that are sufficiently lithe and responsive to these increasing demands (Foley et al., 2010; Ban et al., 2014; Mengerink et al., 2014; Lewison et al., 2015). In addition, the increasing emphasis on ecosystem-based management (EBM) of the oceans broadens the decision landscape beyond traditional ocean resource management to incorporate multiple ecosystem stressors and endpoints ranging from toxic pollutants to food web dynamics to human wellbeing (National Oceanic and Atmospheric Administration [NOAA], 2006; Ehler and Douvere, 2009; European Commission, 2009; Council on Environmental Quality, 2010; National Ocean Council, 2013; International Council for the Exploration of the Sea [ICES], 2014). This structural and thematic complexity requires a suite of technical tools to support decision-making, including holistic, system-wide understanding of impacts across multiple objectives, and evaluation of tradeoffs among objectives (Levin et al., 2009; Lewison et al., 2015).

The complexity of decisions in ocean ecosystem management has been met with the development of increasingly complex analytical tools, and models are often required to describe system dynamics and to evaluate the consequences of potential management actions (Levin et al., 2009; Punt et al., 2016a). End-to-end models have been developed to address this challenge (Collie et al., 2009; Fulton et al., 2011; Moloney et al., 2011; Plagányi et al., 2014), although these models are often data-hungry and computationally intensive, creating inefficiencies and long waiting periods for model development of this type (Ascough et al., 2008; Kaplan et al., 2018).

A challenge remains, in successfully linking sophisticated analytical tools to the complex decision landscape, in part owing to major gaps that exist between the communities of technical tool developers and decision makers (McNie, 2007; Cook et al., 2013), including potentially different perspectives on the usefulness of technical information in decision-making (Kunseler et al., 2015). The challenge to link knowledge to action for effective natural resource management is now well recognized (Knight et al., 2008; Weaver et al., 2014; Clark et al., 2016b; Beier et al., 2017; Wall et al., 2017). Knowledge that is useful to management is salient (relevant to management needs), credible (authoritative and trusted), and legitimate (developed within a process that accounts for all stakeholder perspectives) (Cook et al., 2013; Kunseler et al., 2015; van Voorn et al., 2016). Recognition of the need to make science usable has led to efforts to improve science communication (Fischhoff and Scheufele, 2014, and references therein; Young et al., 2014), to explicitly engage in the process of knowledge co-production or collection (Jacobs et al., 2016; Reid et al., 2016), and to increase support for interdisciplinary research groups (Levin et al., 2016), but few of those efforts translate to management action. Meaningfully linking scientific findings to policy may be done most successfully by science-policy intermediaries, boundary organizations, or “knowledge brokers” (Cook et al., 2013; Bednarek et al., 2016, 2018; Clark et al., 2016b).

One approach to addressing the challenges posed by modeling complex systems with high degrees of uncertainty is to compare across multiple models that differ in their level of complexity, structure, assumptions, and parameterization (Verburg et al., 2008; Steinacher et al., 2010; Fulton et al., 2011). This approach was the foundation of the successful Energy Modeling Forum, formed at Stanford University in 1976, which has a focus on investigating the structure and function of multiple models relevant to individual energy policy topics (Huntington et al., 1982). Multi-model approaches offer many advantages, including the ability to compare the consequences of hypotheses, the ability to identify areas of agreement and divergence among model outputs, collaboration within the modeling community, and continued model development (Weller et al., 2013). They also take different forms, each intended to improve forecasts, reduce the uncertainty associated with a single model approach, and/or broaden the scope of evaluation beyond that available for a single model. Multi-model approaches include ensemble modeling (Tebaldi and Knutti, 2007; Reichler and Kim, 2008; Ruiz et al., 2014; Krishnamurti et al., 2016; Shin et al., 2017) and model coupling (Reichler and Kim, 2008; Bever and MacWilliams, 2013; Boomer et al., 2013; Krishnamurti et al., 2016), approaches that have also been applied in evaluating the impacts of climate change on marine ecosystems and fisheries worldwide (Monier et al., 2013; Niiranen et al., 2013; Henson et al., 2017).

To address the challenge of linking model-based advice to action, the Ocean Modeling Forum (OMF) was developed to address specific ocean-ecosystem management issues using multi-model approaches. The OMF brings together models and modelers, decision makers and/or managers, subject and system experts, and other experts and stakeholders, in ad hoc working groups of 15–20 individuals to jointly develop usable model-based advice in support of specific ocean management issues. The OMF framework has been tested on two inaugural case studies, focused on social-ecological system impacts of forage fish harvest in the California Current and Northeast Pacific ecosystems: Pacific sardine (Sardinops sagax), and Pacific herring (Clupea pallasii). These test cases have led to the development of guidance for generating actionable, model-based knowledge. Based on these test cases, we identify and discuss below five primary conditions or characteristics of model-based knowledge for effectively linking it to management of the world’s oceans: the knowledge is aspirational, applicable, parsimonious, co-produced, and amplifying.

Making model-based knowledge matter

Here we describe characteristics that make model-based knowledge “ready for action,” i.e., useful in a decision space. We use the word “knowledge” not only to represent specific model results and outputs, but also to generally reflect the scholarship generated through the process described herein of developing model-based advice. We suggest that this knowledge has various characteristics that reflect the attitudes, motivations, and approaches of the groups involved in its production. However, we argue that not only do the attitudes, motivations, and approaches of the people involved in knowledge production matter, but the knowledge, advice, and outputs produced also need to share the characteristics we identify and describe below.

1. Knowledge is aspirational

Aspirational means of lofty purpose, or reaching for the sky, such as aspirational goals. Ocean managers and policymakers have increasingly signaled their lofty desire to pursue ecosystem-based management of ocean resources over the last decades, in Europe (ICES, 2014) and the United States (Witherell, 2003; Council on Environmental Quality, 2010; Pacific Fishery Management Council, 2014a). While there is widespread agreement on the desire and need for EBM, the implementation and operationalization of EBM and of ecosystem-based fisheries management (EBFM) are both complex and challenging (NOAA, 2006; ICES, 2014; Link and Browman, 2017). Frameworks for organizing EBM and EBFM, such as the Integrated Ecosystem Assessment (IEA) approach (Levin et al., 2009), have been developed and are being deployed globally (NOAA, 2006; Levin et al., 2013; ICES, 2014; Harvey et al., 2017a; Levin et al., 2018). Yet, very few concrete fishery management actions or harvest rules in place explicitly account for impacts on ecosystem components (Skern-Mauritzen et al., 2016; Harvey et al., 2017b), such as dependent predator species, or dependent human communities. In most cases, explicit EBFM remains aspirational. Here, we argue that knowledge to inform EBFM must also be aspirational, by encompassing ecosystem components and processes.


The Pacific Fishery Management Council (PFMC), the agency responsible for management of fisheries in federal waters along the US West Coast, has aspirational objectives for the management of Pacific sardines (and other coastal pelagic species) that include providing adequate forage for dependent species (Pacific Fishery Management Council, 2016a). Achieving this objective requires understanding which species are dependent on forage species, and what abundance of forage is adequate to support their populations. It also requires understanding how current management strategies affect the abundance of forage fish available to dependent species in space and time, and how forage-dependent species respond to different levels of forage biomass available. Evaluating such impacts of management strategies requires incorporating details related to population dynamics of the predators and the prey, including other available forage species, migration/movement, interannual and seasonal climate variation, and oceanographic influences (Sydeman et al., 2017).

To address the aspirational goals of the PFMC, the OMF Pacific sardine working group evaluated the impacts on sardine-dependent predators of the harvest rule implemented in regulation to set catch limits for the US fishery using multiple models that, taken together, allow consideration of such ecosystem impacts. The current harvest control rules take into account environmental influences via a temperature-dependent exploitation rate parameter. To the conceptual model for Pacific sardine used in the previous evaluations of harvest control rules (Pacific Fishery Management Council, 2013), the OMF working group added several ecosystem-level components that could be added to any future control rule revision, including food web interactions, spatial distribution and movement of predators and prey, seasonal and interannual climate variation, and bottom-up forcing using a suite of models of varying levels of complexity (Figure 1).

Figure 1 

Conceptual model of the Pacific sardine system. Ecosystem components (white ovals) and the models of varying complexity (colored boxes) employed by the Ocean Modeling Forum’s Pacific sardine working group, each of which incorporated different parts of the ecosystem. Single species model (SSM), the sardine operating model used to estimate impacts of harvest rules on sardine biomass (Hurtado-Ferro and Punt, 2014); Ecopath, a highly-resolved, forage fish-centric version of a food-web model used to identify highly dependent sardine predators (Koehn et al., 2016); MICE, a model of intermediate complexity for ecosystem assessment built to evaluate the impacts of sardine harvest on predators (Punt et al., 2016b); Atlantis, an end-to-end ecosystem model of the California Current Ecosystem (Kaplan et al., 2017). DOI:

2. Knowledge is applicable

For model results to be truly useful in decision-making, they need to be applicable, that is, what they produce should match the expectations of the management decision process, which sits within a larger governance system. Decision-making processes that influence the management of natural resources have their own cultures, processes, and timelines. Governance systems can be generally classified along a gradient from top-down control to community-based, and each governance archetype is associated with a general approach to using science and scientific advice (Acheson, 2006). Models most likely to be used are typically initially designed – or adjusted – to address the specific management question at hand. Modeling results are more likely to be actionable if they provide outputs that can be readily applied to decisions (Wall et al., 2017). Examples of this dependence on context include requests for specific research initiatives within US Fishery Ecosystem Plans (Dereynier, 2014), and application of social science within reviews of fishing quota programs (Gulf of Mexico Fishery Management Council, 2013; Pacific Fishery Management Council and National Marine Fisheries Service, 2010). This context can only be navigated by direct collaboration between modelers, decision makers, and stakeholders.

Collaborative efforts to address specific management actions should provide the opportunity to weigh tradeoffs among ecosystem goals, and model outputs that are situated in a tradeoff framework will be more applicable to decisions than those offering only one way forward (Link, 2010). An example is the use of a management strategy evaluation framework, which involves using a stakeholder process to identify objectives and potential management strategies. This approach determines which models and other tools are available to evaluate the alternative strategies, the outcomes of which can be compared for their performance against stated objectives, so long as standards for model performance against stated objectives are identified (Punt, 2017). Where objectives have been identified, scenarios can be designed to generate outputs that are linked to these objectives so that the multi-objective performance of alternative management actions can be compared (e.g., Fulton et al., 2014; Punt et al., 2016a). Thus, the knowledge is applicable to decisions related to achieving objectives.


To achieve its management objective of providing adequate forage for dependent species, the PFMC has the power to set limits on the U.S. harvest of Pacific sardines. The decision to be made, therefore, is how much sardine to harvest in U.S. waters. The ecological impacts of this decision relate to the long-term sustainability of sardines and the predators that depend on them. The applicability of models relevant to this problem was guided by a committee of the PFMC (PFMC, 2013), whose members reviewed the models existing before the OMF case study, found them inadequate, and provided guidance on specifications for new models that would be required for policy applications.

To support management objectives under the current harvest control rule, the OMF focused on existing harvest levels by evaluating impacts on predators under ranges of sardine biomass estimated using a model of intermediate complexity for ecosystem assessment (MICE; Plaganyi et al., 2014). The MICE projects population dynamics of: Pacific sardine, Northern anchovy Engraulis mordax, and “other forage”; predators brown pelican Pelecanus occidentalis and California sea lion Zalophus californianus; and three fisheries (Canada, United States, and Mexico) in a spatially-, temporally- and age-explicit framework with environmental stochasticity driving sardine recruitment (Punt et al., 2016b). The models explored impacts of current sardine management on forage fish predators using model-predicted sardine biomass potentially resulting from the current harvest control rules. The final harvest limit for the US fishery for sardine is the minimum of the acceptable biological catch (the exploitation rate corresponding to maximum sustainable yield, HMSY, multiplied by the estimate of the biomass of sardine aged 1 year and older, reduced by a buffer to account for scientific uncertainty) and the harvest guideline (HG). The latter is calculated using the formula:


where B1+ is the estimate of the biomass of sardine aged 1 year and older, and 0.87 is a factor to account for the assumed proportion of sardine in US waters. The value of HMSY depends on sea surface temperature collected by the CalCOFI program (Punt et al., 2016b).

To complement the MICE, we applied a multi-model approach (Kaplan et al., 2018) to evaluate the broader ecosystem-level impacts of the range of sardine abundances projected by the MICE under the current control rule (Table 1). This multi-model approach (Figure 2) involved application of a simple food web model (Koehn et al., 2016) as well as a full ecosystem Atlantis model (Kaplan et al., 2017). The food web model allowed calculation of indicators of prey importance in the food web and predator dependence upon those prey groups. The multi-model approach allowed consideration of how a broader set of ecological and economic objectives responds to alternative levels of sardine abundance (Figure 2), and of how model parameterization and structural uncertainty influence results. In the case of Atlantis and the MICE, economic performance was measured simply, in terms of sardine catch.

Table 1

Sardine biomasses resulting from realistic management scenarios using two ecosystem models. DOI:

MICE Atlantisc

Sardine biomass categories by range of model predictionsa (tons) Proportion of model runsb Sardine biomass (tons) Multiplier of fishing mortality

2.8–121 million 0.42 3.7 million 0.0
1.4–2.8 million 0.16 1.9 million 0.5
628,000–1.4 million 0.12 960,000 1.0
171,000–628,000 0.10 290,000 2.0
26,000–171,000 0.07 50,000 4.0
0–26,000 0.13 3,000 8.0

a Biomass outputs by the MICE are the results of 100,000 stochastic realizations using the existing harvest rule (Punt et al., 2016b); results are binned into biomass categories.

b Relative proportion or frequency of biomass categories.

c To achieve sardine biomasses that also fell within the MICE biomass categories, the deterministic Atlantis model (Kaplan et al., 2017) was forced by constant sardine fishing rates, yielding biomasses based on the fishing rate multipliers.

Figure 2 

Trade-offs among multiple sardine ecosystem objectives explored by multiple models. Kaplan et al. (2018) combined the MICE and Atlantis model results to evaluate effects of declines in sardine abundance on predators and the ecosystem. Here, nodes on the seven axes represent multiple objectives evaluated by the MICE (“Pelican” axis) or the Atlantis model (all other axes). Colored lines are different levels of sardine abundance (see Table 1), which lead to the scores for objectives indicated on each axis. All objectives are standardized to a range of 0–1. Scores for all biomass levels are equal to 1 for Sea lions, Pelagic feeding seabirds, and Piscivores. Scores for all but the two lowest biomass levels are equal to 1 for Pelicans from the MICE. The scores for the two lowest biomass levels are equal to 0 for Sardines. DOI:

Improved socio-economic metrics can also provide context for the ecological and fishery outputs from the Atlantis model or the MICE, as part of this multi-model approach (Figure 3). One such suite of metrics, coastal community vulnerability indices, are generalized socioeconomic vulnerability metrics for human communities that rely on the marine ecosystem. These metrics have had broad application for defining the risks to human communities from exogenous shocks such as disasters (hurricanes, oil spills, fishery collapses) and have been employed in the US Northeast and Southeast (Jacob et al., 2012; Jepson and Colburn, 2013), Alaska (Himes-Cornell and Kasperski, 2015), at the US national level (, and in Mexico (Morzaria-Luna et al., 2014). The Community Social Vulnerability Index is derived from seven community-level social and economic indices of community characteristics: personal disruption, poverty, population composition, housing characteristics, labor force structure, natural resource labor force, and wealth and education. These methods are being applied to fishery management nationally (Gulf of Mexico Fishery Management Council, 2013) and have been identified as applicable tools for the California Current.1 Thus, one step toward the aspirational goal of assessing social impacts of changing harvests (e.g., as projected by ecosystem models) is to consider those harvests in the context of social and economic vulnerability metrics familiar to fishery decision makers.

Figure 3 

Social vulnerability composite score for communities with substantial sardine landings compared to other communities. Solid bars indicate social vulnerability composite scores for communities with substantial sardine landings, based on the outputs from Atlantis and the MICE from the OMF working group. Open bars indicate social vulnerability measures of five other communities, ranging from the community with lowest vulnerability measure (El Granada) to the highest vulnerability measure (Neah Bay). Communities with substantial sardine landings are those that contain ports with annual average sardine landings of >1000 metric tons for 2004–2013 (Pacific Fisheries Information Network, 2016). The ports of San Pedro and Terminal Island are contained within Los Angeles. CDP indicates census designated place. See Appendix S1 for detailed methods. DOI:

3. Knowledge is parsimonious

Bigger is not always better. The shift from single-species management to ecosystem-based management has been accompanied by a parallel shift in focus from single-species models and assessments to multi-species and ecosystem models. While the advent of supercomputing coupled with the desire to address complex, ecosystem-level questions has afforded increasing opportunity to develop complex ecosystem models – as of the writing of this paper, 28 Atlantis ecosystem models are in use and an additional 16 are in development (E. Fulton, personal communication) – building such models is time-, resource-, and data-intensive. The added complexity in ecosystem-scale models confers many benefits, including the ability to incorporate a broad range of ecosystem components and evaluate impacts of management scenarios across a suite of objectives. However, there are multiple challenges associated with that added complexity, including competing hypotheses about ecosystem dynamics, increasing need for data, and highly uncertain predictions (Fulton et al., 2003). Investing in the development of such models may be appropriate and necessary for the breadth of management questions of the modern era, but parsimonious models are often better suited to addressing specific problems at hand, or providing tactical advice, which may be preferred (Hannah et al., 2010). For ecosystem-level projects, deploying multiple models of varying levels of complexity, each suited to a different purpose and audience, can be useful (Fulton et al., 2015). In addition, models containing high numbers of parameters and outputs, not all of which are relevant to the question at hand, increase parameter uncertainty and may impede the dialogue between resource users, managers and modelers (Collie et al., 2016), a fact that can erode trust in the efficacy of models in multi-stakeholder processes. Given limitations on time and energy, a prudent investment approach may be to couple simpler, more parsimonious models with more complex ones, including using qualitative models and conceptual models as components of complex EBM efforts (Harvey et al., 2016).


In the case of the OMF Pacific sardine working group, the relevant management objective is to “provide adequate forage for dependent species” (Pacific Fishery Management Council, 2016a), and four models were employed to address this objective. The simplest model in terms of its ecosystem components was a single-species model developed to evaluate the impacts of existing sardine harvest rules on fishery- and sardine-specific objectives, such as mean sardine biomass, and probabilities of falling above or below certain biomass thresholds (Figure 1; Hurtado-Ferro and Punt, 2014). This model included the influence of an environmental variable, temperature, on recruitment. The next most complicated model was the static food web model constructed specifically for California Current forage fish with high trophic resolution (Koehn et al., 2016). This model was used to identify predators most likely to be highly dependent upon sardines and therefore potentially impacted by sardine harvest, and to identify forage species in addition to sardine that are important prey in the California Current.

The MICE model described above was intentionally constructed to be a parsimonious approach to addressing the particular questions at hand. This approach is advantageous in evaluating the impacts of management actions because MICE are tactical; i.e., they can be structured to target specific management questions or objectives, they contain parameters that can be adjusted by fitting model outputs to observations, and they are not very intensive computationally. Fast computation times (on the order of hours versus days) mean that MICE are well-suited to exploration of a range of specific uncertainties via scenarios and across multiple replicates associated with stochasticity or process/observation error (Plagányi et al., 2014). In this case the MICE expanded upon the single-species model by incorporating climate, oceanography, predators and other fisheries. Climatic and oceanographic factors were accounted for in the MICE by modelling the slope at the origin of the stock-recruitment relationship as a function of an oceanographic driver. The function of the driver is to generate simulated variation in sardine abundance similar in magnitude to that observed in sardine scales contained in sediment core samples from the southern California Bight, prior to exploitation. Atlantis end-to-end ecosystem model (Kaplan et al., 2017) was conducted, following its initial review by an external panel of the Pacific Fishery Management Council (2014b). This “end-to-end” Atlantis model included food web dynamics, oceanography, biogeochemistry, and fisheries (Figure 1), and the update refined the parameterization to explore sardine ecology and management questions.

4. Knowledge is co-produced

For many years, scientists aiming to produce usable knowledge have followed the “loading dock” model, wherein scientific data are produced and results are published in the peer-reviewed literature, where they remain (somewhat) available to anyone interested in applying them to management. The potential consequence of this approach is that decision makers, when faced with a decision and seeking technical input, can find that scientific outputs are mismatched to the decision point. Increasing attention is being paid to replacing these standard practices with a process of co-production of knowledge: the co-identification of research questions and outputs between scientists and decision makers or managers, informed by known decision points and available, relevant scientific tools; and the extended, iterative progress towards shared vision and action (Cash et al., 2006; Clark et al., 2016b; Nel et al., 2016). Research should be designed and conducted in consideration of possible linked management actions to overcome the gaps between knowledge production or research on the one hand, and action or management on the other (Jarvis et al., 2015). In this way, the ensuing technical inquiry can be shaped to target a specific decision: outputs match management-relevant variables, timeframes are relevant for decision landscapes, and points of entry for technical information into the policy process are identified. Likewise, through direct collaboration with scientists, decision makers can bracket their needs by what information can be produced, within what timeframe, and bearing what level of uncertainty.

Knowledge co-production often also includes stakeholders, whose participation can influence the data or knowledge used for technical analyses, and who will weight the potential outcomes during the decision process. Furthermore, as the impacts of decisions on multiple ecosystem components are weighed against each other, expertise across multiple disciplines is required within the group. Natural scientists can be joined, for example, by social scientists to assist in defining and measuring the human dimensions of ecosystems, such as social vulnerability (Figure 3), or by economists to describe and evaluate the influence of supply chains on the system; a multitude of additional examples exist. Often, the use of knowledge brokers or “boundary spanners” can be employed (Bednarek et al., 2016). Knowledge co-production is a vital component of actionable science, though barriers to applying the resulting outputs to actions still exist (Weichselgartner and Kasperson, 2010).


Motivated by the idea that knowledge co-production improves the usefulness of science, the OMF convenes working groups comprised of modelers, managers, empiricists and stakeholders to address single issues of ocean management concern. By including managers or decision makers along with modelers and other experts, the management questions can be framed together, the models can be structured together to ensure that they will address the relevant decision(s), and other stakeholders can influence what information is used in the modeling activities. The working groups are diverse in their membership, and convene groups of individuals who otherwise would not have the opportunity to collaborate.

Quantifying co-production is a challenge. One approach is to measure the diversity of the stakeholders participating in the knowledge production activity. To identify the diversity of the OMF network of collaborators, we collected data from the OMF Pacific herring working group participants on their collaboration networks, as defined by their publications. The Pacific Herring working group of 20 individuals included academic ecologists, food web and stock assessment modelers, anthropologists, political scientists, economists, indigenous tribal elders, First Nations scientists, resource management agency scientists, fishing industry executives, and commercial fishermen.

We conducted a bibliographic network analysis, mapping the 1,039-person co-authorship network based on the publication records (in Web of Science) of working group members, prior to jointly producing manuscripts as part of the working group (Figure 4). Our analysis showed the dominance of individuals publishing in the life sciences, as well as the existence of many interdisciplinary participants publishing in life and social sciences. We also found multiple nodes (individuals) and clusters of individuals that, prior to participation in the OMF, were disconnected from a central, highly connected aggregation of individuals. Those disconnected nodes illustrate the impact of the OMF: the map shows disconnected individuals and their networks as separate from the rest of the OMF network. A shared publication links all isolated nodes into the network (Supplementary Figure S1), and the newly created network includes not only a diverse set of stakeholders, but individuals and their collaborators who were previously not linked, despite having expertise on the same topic, Pacific herring. The new network formed after a joint publication resulted in an increase in the number of authors in the network, from 1,039 to 1,044, and more author-to-author links in the group, from 5,894 to 6,097. In addition, the total number of links (the number of individual manuscript-based links between pairs of authors) increased from 8,049 to 8,259. Thus, the OMF broadened the network, linking previously disconnected individuals, and increased the number of links between individuals and the average between-person links in the network, strengthening ties within the network.

Figure 4 

OMF Herring working group network. Network of coauthors based upon all Web of Science-listed publications of the OMF Herring working group members. Each circle represents an individual; circle size indicates number of links (i.e., shared publications), and color represents the research area that is the focus of the publication, based on Web of Science categorizations. Gray circles indicate a publication that encompasses more than three research areas. Distance between circles is relatedness, based upon publications: the more shared publications or co-authors, the closer are two circles. Lines represent shared publications. Clusters are formed by individual people closely related by shared publications. A total of 5,894 links exist in the network. See Appendix S1 for detailed methods. DOI:

Opportunities exist for knowledge co-production to improve ocean management efforts, as most relevant research, ranging from fisheries management to human sense of place, is currently conducted in isolated silos of disciplinary homogeny versus in cross-disciplinary groups. To illustrate the potential for improvement, we surveyed the members of the OMF Pacific Herring and Pacific Sardine working groups (see Appendix S1 for methods) to identify the ocean management issues that could most benefit from multi-model approaches and knowledge co-production. We then conducted a network analysis of the 1,000 most connected authors from the 500 most recently published papers in the top nine ocean management issues identified (Figure 5). The analysis shows a core of individuals publishing in two or more themes that are disconnected from clusters of other closely-linked individuals publishing in single themes, especially fisheries, eutrophication, and governance. In other words, interdisciplinary (i.e., working across topics) researchers are largely disconnected from individuals working only within a single discipline or topic. Many scientists are not exposed to interdisciplinary research opportunities. Furthermore, within topics, or individual ocean management themes, individual tightly-connected groups are very distant and disconnected from each other. This disconnect may occur between research groups of different disciplines, such as social and natural scientists, or of physicists and environmental scientists, who conduct research in the same field but not often with each other. The existence of clusters, plus individuals who are wholly disconnected from everyone else, represents opportunity for increased co-production of knowledge across disciplinary silos to inform ocean decision-making.

Figure 5 

Network analysis of individuals publishing in the top nine OMF case study opportunities. Each circle represents an individual researcher publishing in one or more topic. Clusters are formed by individual people closely related by shared publications. Circle color indicates publication topic; circle size indicates number of links (shared publications). Distance between circles indicates degree of relatedness, according to the bibliographic records (connecting lines have been omitted for clarity). Clusters of circles are closely-related groups of people, as indicated by their publications. Group A includes climate change, habitat loss, toxics; Group B includes governance, fisheries, eutrophication, cumulative impacts, and marine spatial planning. See Appendix S1 for detailed methods. DOI:

5. Knowledge is amplifying

The final characteristic of model-based knowledge that is ready for action is that it amplifies existing empirical information, and information derived from individual models, through cross pollination among multiple modelers and between modelers and other experts. At its foundation, a multi-model approach is an extension of the general scientific strategy of multiple hypotheses (Chamberlin, 1890), and a suite of tools has been developed in response to this strategy for evaluating among competing hypotheses (Burnham and Anderson, 2004; Burnham et al., 2011; Grueber et al., 2011). The power of bringing multiple models together is not only in reducing uncertainty by varying model complexity and breadth, but also in modelers learning from each other through iteration, in more effective engagement with stakeholders, and in being able to address both tactical and strategic concerns (Fulton et al., 2015). The multi-model process provides richness about how the system should be modeled. Dialog among modelers also daylights individual biases and distortions in model building, which leads to being a better modeler.

Additional wisdom is also gained by engaging with experts with extensive experience in the ecosystem, or with the focal species, who serve to represent knowledge about the ecosystem. Models can amplify such system understanding. Developing models through iterative interaction with knowledge holders can improve models by identifying which system dynamics are most likely to impact the issues at hand and which should be prioritized for inclusion in the models, what additional sources of data and analyses can be used to inform the models, how to validate model outputs, etc. (Fulton et al., 2015). Models can amplify ideas and different types of knowledge, including traditional knowledge, arising from other stakeholders and sources. The value of traditional knowledge, including traditional ecological knowledge, in ecological understanding and resource management is widely recognized; combining traditional knowledge with scientific approaches can amplify the power of each individual form of knowledge (Huntington, 2000; Moller et al., 2004), and incorporating traditional knowledge into models can effectively inform conservation and management (Butler et al., 2012; Polfus et al., 2014).


In the OMF Pacific sardine working group, diets of predators in two models, the MICE (Punt et al., 2016b) and the Atlantis ecosystem model (Kaplan et al., 2017), were informed by the highly resolved Ecopath food web model of the California Current ecosystem (Koehn et al., 2016). Data collection and initial model parameterization for the Atlantis model and the food web model were a collaborative effort (with authors shared across papers). This effort did not necessarily force parallel interpretation of data sources or parallel model structure, but provided some efficiency to the efforts and some common language before the models were brought to broader participants. The outputs of the MICE, sardine abundance, were also used to structure the Atlantis model outputs, allowing for an apples-to-apples comparison of model predictions by managers. Ecopath information was directly passed to the MICE, including being used to prioritize the predators in the MICE. The MICE and Ecopath results both led to realization of the importance of anchovy, which was highlighted in an Atlantis manuscript (Kaplan et al., 2017).

The Herring working group also employed models that amplified messages and findings from other models and sources of knowledge. The Herring working group evaluated the performance of alternative management actions against social, ecological and economic objectives in a management strategy evaluation framework, using an operating model of herring population dynamics internally linked to a stock assessment model. This stock assessment model was structured to mimic a separately developed, novel, spatially-structured stock assessment model (Punt et al., 2018). These two models were developed through consultation among modelers within the OMF. Another model evaluated the consequences of harvest under scenarios of socially learned migration behavior in herring, the presence of which was informed by traditional ecological knowledge (MacCall et al., 2018), amplifying existing knowledge about herring not previously incorporated into models to investigate the impacts of management decisions.

Development of the Ecopath, MICE, and Atlantis models involved extensive synthesis of existing empirical and modeling studies, but also additional expert knowledge generated within the Ocean Modeling Forum. For example, improvements during the OMF informed the effort to represent sardine specifically in the California Current Atlantis model, disaggregating them from other forage fish species, as had been the approach in the earlier versions of the model (Horne et al., 2010; Kaplan et al., 2013). This disaggregation made the model more relevant for the present case study, while addressing improvements to ecosystem models that were requested by the Pacific Fishery Management Council (2013) and an external review panel (Pacific Fishery Management Council, 2014b; Kaplan et al., 2017). The representation of sardine included refining the diet information, first taken from individual-based models contributed by OMF members (Rose et al., 2015), then modified based on expert knowledge within the Forum that suggested reliance on filter feeding and higher consumption of phytoplankton by adult sardines (Figure 6), also supported by studies from other ecosystems (van der Lingen et al., 2006, 2009). Another example of including expert knowledge from the OMF involved parameterization of sardine migration patterns in both Atlantis and the MICE based on unpublished modeling of a working group member (summarized in Punt et al., 2016b, their Appendix A.1.1.1 and Figures A.2 and A.3). These examples are consistent with the role of expert knowledge within approaches to ecosystem-based fishery management (Smith et al., 2007).

Figure 6 

Diets of juvenile and adult sardine in Atlantis ecosystem model. Discussion during the Ocean Modeling Forum first centered around parameterizing Atlantis using diets from a recent individual based model (Rose et al., 2015; bottom panels), with subsequent addition of phytoplankton to the diets based on expert opinion (top panels). Results are from the first year of a 50-year Atlantis model run. DOI:

Impacts of action-ready knowledge

We have illustrated five characteristics of model-produced knowledge that make such knowledge suitable for use in ocean management and decision-making. What, then, have been the impacts of such model-produced advice? Quantifying impact is one of the greatest challenges facing practitioners working broadly at the science-policy boundary (Lyall et al., 2004; Meagher and Lyall, 2013). While the holy grail for applied scientists has long been direct evidence that their science is being used to improve concrete outcomes, such as (for conservation scientists) more habitat protected, improved water quality, or increased fish biomass, the direct linking of science to such impacts is difficult and rare. In addition to these direct instrumental impacts of research on policy and practice decisions (Meagher and Lyall, 2013), a broader suite of impacts of research on decisions can be measured. These include conceptual impacts, where research changes thinking or raises consciousness, and capacity-building, meaning education, training, or other forms of collaboration. Other processes can also be impacted: attitudes can be changed, especially towards knowledge exchange; and enduring connectivity, especially between researchers and knowledge users, can be established (Meagher and Lyall, 2013).

As illustrated above, the Sardine working group focused on developing parsimonious models (e.g., a MICE) that aspired to address ecosystem and especially food web impacts of sardine harvest, and that were applicable to management (i.e., evaluated impacts of the existing sardine harvest rule on the food web). These models may ultimately have instrumental impacts, i.e., changes to harvest practices to support sardine-dependent predators, but those impacts are likely to occur over longer periods of time. In the meantime, we can track conceptual impacts as the decision-making body considers the models in its deliberations: the subgroup of the PFMC focused on sardines reported its intention to track and use these models to examine the impacts of harvest on the food web (Pacific Fishery Management Council, 2016b), and a modeling update was subsequently requested by PFMC and presented by an OMF working group member (Pacific Fishery Management Council, 2016c).

The Herring working group, multi-stakeholder and interdisciplinary, co-produced knowledge related to the consequences of herring fisheries for social, economic, and ecological objectives, and amplified traditional knowledge in the formulation of models to evaluate potential outcomes of management decisions. This process has impacted attitudes about knowledge exchange, as noted by Harvey Kitka, a member of the Sitka Tribe of Alaska’s Herring Committee and a member of the Herring working group:

“This group has been very interesting. I and my group, especially the Herring Committee, we figured for the longest time that if we can get traditional knowledge and science to work together, it’ll probably be the best thing that’s ever happened.”

Before the Herring working group was formed, exchange of information used in decision-making about herring and herring fisheries was limited between fisheries management agencies and traditional knowledge holders, which was a source of conflict between these groups (Hume, 2015; Welch, 2015; Woolsey, 2015). Testimony from Herring working group members demonstrated that participation in the co-production of model-based knowledge served to change attitudes about the potential for various types of knowledge to be used in models and improved collaborations, opening pathways for enduring connections (J. Cleary, personal communication, S. Dressel, personal communication). Tracking impacts of knowledge and research remains a challenge for those working at the boundary between science and its use, and future efforts should focus on this broader suite of impact types.


Operationalizing ocean management for multiple benefits is challenging and requires a flexible approach (Dickey-Collas, 2014). Ocean resources are often shared among multiple sectors, and governance of large marine ecosystems is complex and heterogeneous (Crowder et al., 2006; Mahon et al., 2010), making the successful application of knowledge in decision spaces difficult. Considerable effort has been applied to developing guidance for improving the use of science in natural resource management and decision-making, including identifying features of actionable science (Cook et al., 2013; Kirchhoff et al., 2013; Clark et al., 2016b), intermediaries for bridging the knowledge-action divide (Bednarek et al., 2016, 2018; Clark et al., 2016a) and developing how-to guides for the co-production of knowledge (Beier et al., 2017). Such thinking can be extended to encompass the challenges and opportunities for linking model-based knowledge to ocean management. The Ocean Modeling Forum has developed one framework, described here, for linking model-based advice to ocean management decision-making; there are likely others (e.g., Lynam et al., 2016), though a full review is beyond the scope of the current paper. Unique to the OMF approach is its working group structure, flexibility to various decision settings (i.e., not designed for application to a single governance system), multi-stakeholder approach, and inclusion of decision makers as working group members. Particularly pressing is the need to connect climate science to local decision makers, and work continues to identify indicators and tools that can help to meet this need (Mauger and Kennard, 2017; Wall et al., 2017). We have determined five elements of actionable knowledge useful for linking model-based outputs to ocean management issues: knowledge is aspirational, parsimonious, applicable, co-produced, and amplifying. Such a framework can also complement large-scale investigations into data gaps and modeling needs, as has been done for terrestrial biodiversity modeling (Urban et al., 2016). Linking science to decision-making is a primary challenge for the conservation and sustainability of ocean resources, and indeed natural resources worldwide. Investing effort in ensuring that marine research is well positioned to have impact, while energy- and time-intensive, should be a primary consideration for ocean scientists.

Supplemental file

The supplemental file for this article can be found as follows:



We thank two anonymous reviewers, C Harvey, and J Samhouri for their insightful and helpful comments on earlier drafts of this manuscript. We acknowledge the contributions by the members of the OMF Pacific sardine and Pacific herring working groups, as well as the guidance of our advisory board members. We particularly appreciate Kai Lee for his vision and support in the genesis of the Ocean Modeling Forum, and its development thus far.

Funding information

The Ocean Modeling Forum is a program of the University of Washington College of the Environment and in collaboration with The Nature Conservancy of Washington. The Ocean Modeling Forum and TBF are supported by the David & Lucile Packard Foundation and the Pew Charitable Trusts. Atlantis model development benefited greatly from a grant to ICK from the NOAA Ocean Acidification Program and National Centers for Coastal Ocean Science. Contributing support for this work was provided by the National Oceanic and Atmospheric Administration’s Integrated Ecosystem Assessment (NOAA IEA) Program. This paper is NOAA IEA program contribution #2018.

Competing interests

The authors have no competing interests to declare.

Author contributions

  • Contributed to conception and design: TBF, PSL, AEP
  • Contributed to acquisition of data: TBF, PSL, AEP, IK, KN, AV
  • Contributed to analysis and interpretation of data: TBF, PSL, AEP, IK, KN, AV
  • Drafted and/or revised the article: TBF, PSL, AEP, IK
  • Approved the submitted version for publication: TBF, PSL, AEP, IK, KN, AV


  1. Acheson, JM. 2006. Institutional failure in resource management. Annu Rev Anthropol 35(1): 117–134. DOI: 10.1146/annurev.anthro.35.081705.123238

  2. Ascough, J, Maier, H, Ravalico, J and Strudley, M. 2008. Future research challenges for incorporation of uncertainty in environmental and ecological decision-making. Ecol Model 219(3): 383–399. DOI: 10.1016/j.ecolmodel.2008.07.015

  3. Ban, NC, Bax, NJ, Gjerde, KM, Devillers, R, Dunn, DC, Dunstsan, PK, Hobday, AJ, Maxwell, SM, Kaplan, DM, Pressey, RL, Ardron, JA, Game, ET and Halpin, PN. 2014. Systematic conservation planning: A better recipe for managing the high seas for biodiversity conservation and sustainable use. Conserv Lett 7(1): 41–54. DOI: 10.1111/conl.12010

  4. Bednarek, AT, Shouse, B, Hudson, CG and Goldburg, R. 2016. Science-policy intermediaries from a practitioner’s perspective: The Lenfest Ocean Program experience. Sci Publ Policy 43(2): 291–300. DOI: 10.1093/scipol/scv008

  5. Bednarek, AT, Wyborn, C, Cvitanovic, C, Meyer, R, Colvin, RM, Addison, PFE, Close, SL, Curran, K, Farooque, M, Goldman, E, Hart, D, Mannix, H, McGreavy, B, Parris, A, Posner, S, Robinson, C, Ryan, M and Leith, P. 2018. Boundary spanning at the science–policy interface: The practitioners’ perspectives. Sustain Sci 13: 1175–1183. DOI: 10.1007/s11625-018-0550-9

  6. Beier, P, Hansen, LJ, Helbrecht, L and Behar, D. 2017. A how-to guide for coproduction of actionable science. Conserv Lett 10(3): 288–296. DOI: 10.1111/conl.12300

  7. Bever, AJ and MacWilliams, ML. 2013. Simulating sediment transport processes in San Pablo Bay using coupled hydrodynamic, wave, and sediment transport models. Mar Geol 345: 235–253. DOI: 10.1016/j.margeo.2013.06.012

  8. Boomer, KMB, Weller, DE, Jordan, TE, Linker, L, Liu, Z-J, Reilly, J, Shenk, G and Voinov, AA 2013. Using multiple watershed models to predict water, nitrogen, and phosphorus discharges to the Patuxent Estuary. J Am Water Resour Assoc 49(1): 15–39. DOI: 10.1111/j.1752-1688.2012.00689.x

  9. Burnham, KP and Anderson, DR. 2004. Multimodel inference: Understanding AIC and BIC in model selection. Sociol Method Res 33: 261–304. DOI: 10.1177/0049124104268644

  10. Burnham, KP, Anderson, DR and Huyvaert, KP. 2011. AIC model selection and multimodel inference in behavioral ecology: Some background, observations, and comparisons. Behavioral Ecology and Sociobiology 65: 23–35. DOI: 10.1007/s00265-010-1029-6

  11. Butler, J, Tawake, A, Skewes, T, Tawake, L and McGrath, V. 2012. Integrating traditional ecological knowledge and fisheries management in the Torres Strait, Australia: The catalytic role of turtles and dugong as cultural keystone species. Ecol Soc 17(4). DOI: 10.5751/ES-05165-170434

  12. Chamberlin, TC. 1890. The method of multiple working hypotheses. Science 15: 92–96. (reprinted in Science 148: 754–759 [1965]). DOI: 10.1126/science.148.3671.754

  13. Clark, WC, Tomich, TP, van Noordwijk, M, Guston, D, Catacutan, D, Dickson, NM and McNie, E. 2016a. Boundary work for sustainable development: Natural resource management at the Consultative Group on International Agricultural Research (CGIAR). Proc Natl Acad Sci USA 113(17): 4615–4622. DOI: 10.1073/pnas.0900231108

  14. Clark, WC, van Kerkhoff, L, Lebel, L and Gallopin, GC. 2016b. Crafting usable knowledge for sustainable development. Proc Natl Acad Sci USA 113(17): 4570–4578. DOI: 10.1073/pnas.1601266113

  15. Collie, JS, Botsford, LW, Hastings, A, Kaplan, IC, Largier, JL, Livingston, PA, Plagányl, É, Rose, KA, Wells, BK and Werner, FE. 2016. Ecosystem models for fisheries management: Finding the sweet spot. Fish Fish 17(1): 101–125. DOI: 10.1111/faf.12093

  16. Collie, JS, Gifford, DJ and Steele, JH. 2009. End-to-end foodweb control of fish production on Georges Bank. ICES J Mar Sci 66(10): 2223–2232. DOI: 10.1093/icesjms/fsp180

  17. Cook, CN, Mascia, MB, Schwartz, MW, Possingham, HP and Fuller, RA. 2013. Achieving conservation science that bridges the knowledge–action boundary. Conserv Biol 27(4): 669–678. DOI: 10.1111/cobi.12050

  18. Council on Environmental Quality. 2010. Final Recommendations of the Interagency Ocean Policy Task Force. Washington, D. C.: Council on Environmental Quality. Available at: Accessed 1 September 2010.

  19. Crowder, LB, Osherenko, G, Young, OR, Airamé, S, Norse, EA, Baron, N, Day, JC, Douvere, F, Ehler, CN, Halpern, BS, Langdon, SJ, McLeod, KL, Ogden, JC, Peach, RE, Rosenber, AA and Wilson, JA. 2006. Sustainability – Resolving mismatches in US ocean governance. Science 313(5787): 617–618. DOI: 10.1126/science.1129706

  20. Dereynier, YL. 2014. US fishery management councils as ecosystem-based management policy takers and policymakers. Coast Manage 42(6): 512–530. DOI: 10.1080/08920753.2014.964678

  21. Dickey-Collas, M. 2014. Why the complex nature of integrated ecosystem assessments requires a flexible and adaptive approach. ICES J Mar Sci 71(5): 1174–1182. DOI: 10.1093/icesjms/fsu027

  22. Ehler, CN and Douvere, F. 2009. Marine Spatial Planning: A step-by-step approach toward ecosystem-based management. Intergovernmental Oceanographic Commission and Man and the Biosphere Programme 53. Paris, France: UNESCO.

  23. European Commission. 2009. An integrated Maritime Policy for the European Union. Brussels, Belgium: European Commission. DOI: 10.3000/19770677.L_2011.321.eng

  24. Fischhoff, B and Scheufele, DA. 2014. The science of science communication II. Proc Natl Acad Sci USA 111: 13583–13584. DOI: 10.1073/pnas.1414635111

  25. Foley, NN, Halpern, BS, Micheli, R, Armsby, MH, Caldwell, MR, Crain, CM, Prahler, E, Rohr, N, Sivas, D, Beck, MW, Car, MH, Crowder, LB, Duffy, JE, Hacker, SD, McLeod, KL, Palumbi, SR, Peterson, CH, Regan, HM, Ruckelshaus, MH, Sandifer, PA and Steneck, RS. 2010. Guiding ecological principles for marine spatial planning. Mar Pol 34(5): 955–966. DOI: 10.1016/j.marpol.2010.02.001

  26. Fulton, EA, Boschetti, F, Sporcic, M, Jones, T, Little, LR, Dambacher, JM, Gray, R, Scott, R and Gorton, R. 2015. A multi-model approach to engaging stakeholder and modellers in complex environmental problems. Environ Sci Policy 48: 44–56. DOI: 10.1016/j.envsci.2014.12.006

  27. Fulton, EA, Link, JS, Kaplan, IC, Savina-Rolland, M, Johnson, P, Ainsworth, C, Horne, P, Gorton, R, Gamble, RJ, Smith, ADM and Smith, DC. 2011. Lessons in modelling and management of marine ecosystems: The Atlantis experience. Fish Fish 12(2): 171–188. DOI: 10.1111/j.1467-2979.2011.00412.x

  28. Fulton, EA, Smith, ADM and Johnson, CR. 2003. Effect of complexity on marine ecosystem models. Mar Ecol Prog Ser 253: 1–16. DOI: 10.3354/meps253001

  29. Grueber, CE, Nakagawa, S, Laws, RJ and Jamieson, IG. 2011. Multimodel inference in ecology and evolution: Challenges and solutions. J Evol Biol 24: 699–711. DOI: 10.1111/j.1420-9101.2010.02210.x

  30. Gulf of Mexico Fishery Management Council. 2013. Red Snapper Individual Fishing Quota Program 5-year Review. Tampa FL: Gulf of Mexico Fishery Management Council.

  31. Halpern, BS, Frazier, M, Potapenko, J, Casey, KS, Koenig, K, Longo, C, Lowndes, JS, Rockwood, RC, Selig, ER, Selkoe, KA and Walbridge, S. 2015. Spatial and temporal changes in cumulative human impacts on the world’s ocean. Nat Commun 6: 7. DOI: 10.1038/ncomms8615

  32. Hannah, C, Vezina, A and John, MS. 2010. The case for marine ecosystem models of intermediate complexity. Prog Oceanogr 84: 121–128. DOI: 10.1016/j.pocean.2009.09.015

  33. Harvey, CJ, Garfield, N, Williams, G, Andrews, K, Barceló, C, Barnas, K, Bograd, S, Brodeur, R, Burke, B, Cope, J, deWitt, L, Field, J, Fisher, J, Greene, C, Good, T, Hazen, E, Holland, D, Jacox, M, Kasperski, S, Kim, S, Leising, A, Melin, S, Morgan, C, Munsch, S, Norman, K, Peterson, WT, Poe, M, Samhouri, J, Schroeder, I, Sydeman, W, Thayer, J, Thompson, A, Tolimieri, N, Varney, A, Wells, B, Williams, T and Zamon, J. 2017a. Ecosystem Status Report of the California Current for 2017: A Summary of Ecosystem Indicators Compiled by the California Current Integrated Ecosystem Assessment Team (CCIEA). U.S. Department of Commerce, NOAA Technical Memorandum NMFS-NWFSC-139 . DOI: 10.7289/V5/TM-NWFSC-139

  34. Harvey, CJ, Kelble, CR and Schwing, FB. 2017b. Implementing “the IEA”: Using integrated ecosystem assessment frameworks, programs, and applications in support of operationalizing ecosystem-based management. ICES J Mar Sci 74: 398–405. DOI: 10.1093/icesjms/fsw201

  35. Harvey, CJ, Reum, JCP, Poe, MR, Williams, GD and Kim, SJ. 2016. Using conceptual models and qualitative network models to advance integrative assessments of marine ecosystems. Coast Manage 44(5): 486–503. DOI: 10.1080/08920753.2016.1208881

  36. Henson, SA, Beaulieu, C, Ilyina, T, John, JG, Long, M, Séférian, R, Tjiputra, J and Sarmiento, JL. 2017. Rapid emergence of climate change in environmental drivers of marine ecosystems. Nat Commun 8: 9. DOI: 10.1038/ncomms14682

  37. Himes-Cornell, A and Kasperski, S. 2015. Assessing climate change vulnerability in Alaska’s fishing communities. Fish Res 162: 1–11. DOI: 10.1016/j.fishres.2014.09.010

  38. Horne, PJ, Kaplan, IC, Marshall, KN, Levin, PS, Harvey, CJ, Hermann, AJ and Fulton, EA. 2010. Design and Parameterization of a Spatially Explicit Ecosystem Model of the Central California Current. NOAA Tech . Memo. NMFS-NWFSC-104, 1–140.

  39. Hume, M. 2015 Month: Apr Day: 1 . Heiltsuk First Nation claims victory over disputed herring fishery. The Globe and Mail.

  40. Huntington, HG, Weyant, JP and Sweeney, JL. 1982. Modeling for insights, not numbers: The experiences of the Energy Modeling Forum. OMEGA The Intl J of Mgmt Sci 10(5): 449–462. DOI: 10.1016/0305-0483(82)90002-0

  41. Huntington, HP. 2000. Using traditional ecological knowledge in science: Methods and applications. Ecol Appl 10: 1270–1274. DOI: 10.1890/1051-0761(2000)010[1270:UTEKIS]2.0.CO;2

  42. Hurtado-Ferro, F and Punt, AE. 2014. Revised analyses related to Pacific sardine harvest parameters: Report to Pacific Fishery Management Council. 40.

  43. International Council for the Exploration of the Sea. 2014. Implementing the ICES Strategic Plan 2014–2018 . Copenhagen, Denmark: International Council for the Exploration of the Sea.

  44. Jacob, S, Weeks, P, Blount, B and Jepson, M. 2012. Development and evaluation of social indicators of vulnerability and resiliency for fishing communities in the Gulf of Mexico. Mar Policy 29(10): 16–22.

  45. Jacobs, K, Level, L, Buizer, J, Addams, L, Matson, P, McCullough, E, Garden, P, Saliba, G and Finan, T. 2016. Linking knowledge with action in the pursuit of sustainable water-resources management. Proc Natl Acad Sci USA 113(17): 4591–4596. DOI: 10.1073/pnas.0813125107

  46. Jarvis, R, Borrelle, S, Breen, BB and Towns, D. 2015. Conservation, mismatch and the research–implementation gap. Pacific Conservation Biology 21(2): 105–107. DOI: 10.1071/PC14912

  47. Jepson, M and Colburn, LL. 2013. Development of social indicators of fishing community vulnerability and resilience in the US Southeast and Northeast regions. NOAA Technical Memorandum NMFS-F/SPO-129 . Department of Commerce, National Oceanic and Atmospheric Administration.

  48. Kaplan, IC, Francis, TB, Punt, AE, Koehn, LE, Curchitser, E, Hurtado-Ferro, F, Johnson, KF, Lluch-Cota, SE, Sydeman, WJ, Essington, TE, Taylor, N, Holsman, K, MacCall, AD and Levin, PS. 2018. A multi-model approach to understanding the role of Pacific sardine in the California Current food web. Mar Ecol Prog Ser . DOI: 10.3354/meps12504

  49. Kaplan, IC, Koehn, LE, Hodgson, EE, Marshall, KN and Essington, TE. 2017. Modeling food web effects of low sardine and anchovy abundance in the California Current. Ecol Model 359: 1–24. DOI: 10.1016/j.ecolmodel.2017.05.007

  50. Kirchhoff, CJ, Lemos, MC and Dessai, S. 2013. Actionable knowledge for environmental decision making: Broadening the usability of climate science. Annu Rev Environ Resour 38(1): 393–414. DOI: 10.1146/annurev-environ-022112-112828

  51. Knight, AT, Cowling, RM, Rouget, M, Balmford, A, Lombard, AT and Campbell, BM. 2008. Knowing but not doing: Selecting priority conservation areas and the research-implementation gap. Conserv Biol 22(3): 610–617. DOI: 10.1111/j.1523-1739.2008.00914.x

  52. Koehn, LE, Essington, TE, Marshall, KN, Kaplan, IC, Sydeman, WJ, Szoboszlai, AI and Thayer, JA. 2016. Developing a high taxonomic resolution food web model of the California Current ecosystem to assess the trophic position of forage fish and their predators. Ecol Model 335: 87–100. DOI: 10.1016/j.ecolmodel.2016.05.010

  53. Krishnamurti, TN, Kumar, V, Simon, A, Bhardwaj, A, Ghosh, T and Ross, R. 2016. A review of multimodel superensemble forecasting for weather, seasonal climate, and hurricanes. Rev Geophys 54(2): 336–377. DOI: 10.1002/2015RG000513

  54. Kunseler, EM, Tuinstra, W, Vasileiadou, E and Petersen, AC. 2015. The reflective futures practitioner: Balancing salience, credibility and legitimacy in generating foresight knowledge with stakeholders. Futures 66: 1–12. DOI: 10.1016/j.futures.2014.10.006

  55. Levin, PS, Breslow, SJ, Harvey, CJ, Norman, KC, Poe, MR, Williams, GD and Plummer, ML. 2016. Conceptualization of social-ecological systems of the California Current: An examination of interdisciplinary science supporting ecosystem-based management. Coast Manage 44(5): 397–408. DOI: 10.1080/08920753.2016.1208036

  56. Levin, PS, Essington, TE, Marshall, KN, Koehn, LE, Anderson, LG, Bundy, A, Carothers, C, Coleman, F, Gerber, LR, Grabowski, JH, Houde, E, Jensen, OP, Möllmann, C, Rose, K, Sanchirico, JN and Smith, ADM. 2018. Building effective fishery ecosystem plans. Mar Policy 92: 48–57. DOI: 10.1016/j.marpol.2018.01.019

  57. Levin, PS, Fogarty, MJ, Murawski, SA and Fluharty, D. 2009. Integrated ecosystem assessments: Developing the scientific basis for ecosystem-based management of the ocean. PLoS Biol 7(1): e1000014. DOI: 10.1371/journal.pbio.1000014

  58. Levin, PS, Kelble, CR, Shuford, RL, Ainsworth, C, deReynier, Y, Dunsmore, R, Fogarty, MJ, Holsman, K, Howell, EA, Monaco, ME, Oakes, SA and Werner, F 2013. Guidance for implementation of integrated ecosystem assessments: A US perspective. ICES J Mar Sci 71: 1198–1204. DOI: 10.1093/icesjms/fst112

  59. Lewison, RL, Hobday, AJ, Maxwell, S, Hazen, E, Hartog, JR, Dunn, DC, Briscoe, D, Fossette, S, O’Keefe, CE, Barnes, M, Abecassis, M, Bograd, S, Bethoney, ND, Bailey, H, Wiley, D, Andrews, S, Hazen, L and Crowder, LB. 2015. Dynamic ocean management: Identifying the critical ingredients of dynamic approaches to ocean resource management. Bioscience 65(5): 486–498. DOI: 10.1093/biosci/biv018

  60. Link, JS and Browman, HI. 2017. Operationalizing and implementing ecosystem-based management. ICES J Mar Sci 74(1): 379–381. DOI: 10.1093/icesjms/fsw247

  61. Lyall, C, Bruce, A, Firn, J, Firn, M and Tait, J. 2004. Assessing end-use relevance of public sector research organisations. Res Policy 33: 73–87. DOI: 10.1016/S0048-7333(03)00090-8

  62. Lynam, CP, Uusitalo, L, Patrício, J, Piroddi, C, Queirós, AM, Teixeira, H, Rossberg, AG, Sagarminaga, Y, Hyder, K, Niquil, N, Möllmann, C, Wilson, C, Chust, G, Galparsoro, I, Forster, R, Veríssimo, H, Tedesco, L, Revilla, M and Neville, S. 2016. Uses of innovative modeling tools within the implementation of the Marine Strategy Framework Directive. Front Mar Sci 3. DOI: 10.3389/fmars.2016.00182

  63. MacCall, AD, Francis, TB, Punt, AE, Siple, MC, Armitage, DR, Cleary, JS, Dressel, SC, Jones, RR, Kitka, H, Lee, LC, Levin, PS, McIsaac, J, Okamoto, DD, Poe, M, Reifenstuhl, S, Schmidt, JO, Shelton, AO, Silver, JJ, Thornton, TF, Voss, R and Woodruff, J. 2018. A heuristic model of socially learned migration behaviour exhibits distinctive spatial and reproductive dynamics. ICES Journal of Marine Science . DOI: 10.1093/icesjms/fsy091

  64. Mahon, R, Fanning, L, McConney, P and Pollnac, R. 2010. Governance characteristics of large marine ecosystems. Mar Pol 34(5): 919–927. DOI: 10.1016/j.marpol.2010.01.016

  65. Mauger, GS and Kennard, HM. 2017. Integrating Climate Resilience in Flood Risk Management: A Work plan for the Washington Silver Jackets: A report for FEMA. University of Washington, Seattle, WA, USA: Climate Impacts Group. DOI: 10.7915/CIG7MP4WZ

  66. McNie, EC. 2007. Reconciling the supply of scientific information with user demands: An analysis of the problem and review of the literature. Environ Sci Policy 10(1): 17–38. DOI: 10.1016/j.envsci.2006.10.004

  67. Meagher, L and Lyall, C. 2013. The invisible made visible: Using impact evaluations to illuminate and inform the role of knowledge intermediaries. Evid Policy 9: 409–418. DOI: 10.1332/174426413X14818994998468

  68. Mengerink, KJ, Van Dover, CL, Ardron, J, Baker, M, Escobar-Briones, E, Gjerde, K, Koslow, JA, Ramirez-Llodra, E, Lara-Lopez, A, Squires, D, Sutton, T, Sweetman, AK and Levin, LA. 2014. A call for deep-ocean stewardship. Science 344(6185): 696–698. DOI: 10.1126/science.1251458

  69. Merrie, A, Dunn, DC, Metian, M, Boustany, AM, Takei, Y, Elferink, AO, Ota, Y, Christensen, V, Halpin, PN and Osterblom, H. 2014. An ocean of surprises – Trends in human use, unexpected dynamics and governance challenges in areas beyond national jurisdiction. Global Environ Change 27: 19–31. DOI: 10.1016/j.gloenvcha.2014.04.012

  70. Moller, H, Berkes, F, Lyver, P and Kislalioglu, M. 2004. Combining science and traditional ecological knowledge: Monitoring populations for co-management. Ecol Soc 9(3). DOI: 10.5751/ES-00675-090302

  71. Moloney, CL, St John, MA, Denman, KL, Karl, DM, Köster, FW, Sundby, S and Wilson, RP. 2011. Weaving marine food webs from end to end under global change. J Marine Syst 84(3): 106–116. DOI: 10.1016/j.jmarsys.2010.06.012

  72. Monier, E, Scott, JR, Sokolov, AP, Forest, CE and Schlosser, CA. 2013. An integrated assessment modeling framework for uncertainty studies in global and regional climate change: The MIT IGSM-CAM (version 1.0). Geosci Model Dev 6(6): 2063–2085. DOI: 10.5194/gmd-6-2063-2013

  73. Morzaria-Luna, HN, Turk-Boyer, P and Moreno-Baez, M. 2014. Social indicators of vulnerability for fishing communities in the Northern Gulf of California, Mexico: Implications for climate change. Mar Pol 45: 182–193. DOI: 10.1016/j.marpol.2013.10.013

  74. National Oceanic and Atmospheric Administration. 2006. Evolving an ecosystem approach to science and management through NOAA and its partners. National Oceanic and Atmospheric Administration .

  75. Nel, JL, Roux, DJ, Driver, A, Hill, L, Maherry, AC, Snaddon, K, Petersen, CR, Smith-Adao, LB, Van Deventer, H and Reyers, B. 2016. Knowledge co-production and boundary work to promote implementation of conservation plans. Conserv Biol 30: 176–188. DOI: 10.1111/cobi.12560

  76. Niiranen, S, Yletyinen, J, Tomczak, MT, Blenckner, T, Hjerne, O, MacKenzie, BR, Muller-Karulis, B, Neumann, T and Meier, HEM. 2013. Combined effects of global climate change and regional ecosystem drivers on an exploited marine food web. Glob Change Biol 19(11): 3327–3342. DOI: 10.1111/gcb.12309

  77. Pacific Fisheries Information Network (PacFIN). Data retrieval dated Month: January Day: 2 , 2016, Portland, Oregon: Pacific States Marine Fisheries Commission (

  78. Pacific Fishery Management Council. 2013. Draft Report of the Pacific Sardine Harvest Parameters Workshop. Portland Oregon U.S.A.: Pacific Fishery Management Council

  79. Pacific Fishery Management Council. 2014a. Managing our Nation’s Fisheries 3: Advancing Sustainability. Portland, Oregon, USA: Pacific Fishery Management Council.

  80. Pacific Fishery Management Council. 2014b. Atlantis Model for the California Current: Report of Methodology Review Panel Meeting. Portland, Oregon, USA: Pacific Fishery Management Council.

  81. Pacific Fishery Management Council. 2016a. Coastal Pelagic Species Fishery Management Plan. Portland, Oregon, U.S.A.: Pacific Fishery Management Council.

  82. Pacific Fishery Management Council. 2016b. Coastal Pelagic Species Management Team report on Northern Anchovy stock assessment and management measures. Month: November 2016. Portland Oregon U.S.A.: Pacific Fishery Management Council.

  83. Pacific Fishery Management Council. 2016c. Coastal Pelagic Species Management Team Meeting Minutes. Month: October 2017. Portland Oregon U.S.A.: Pacific Fishery Management Council.

  84. Pacific Fishery Management Council and National Marine Fisheries Service. 2010. Rationalization of the Pacific Coast Groundfish Limited Entry Trawl Fishery; Final Environmental Impact Statement Including Regulatory Impact Review and Initial Regulatory Flexibility Analysis. Portland, OR: Pacific Fishery Management Council. Month: June 2010.

  85. Plagányi, ÉE, Punt, AE, Hillary, R, Morello, EB, Thébaud, O, Hutton, T, Pillans, RD, Thorson, JT, Fulton, EA and Smith, AD. 2014. Multispecies fisheries management and conservation: Tactical applications using models of intermediate complexity. Fish Fish 15(1): 1–22. DOI: 10.1111/j.1467-2979.2012.00488.x

  86. Polfus, JL, Heinemeyer, K, Hebblewhite, M, Taku River Tlingit First Nation. 2014. Comparing traditional ecological knowledge and western science woodland caribou habitat models. J Wildlife Manage 78: 112–121. DOI: 10.1002/jwmg.643

  87. Punt, AE. 2017. Strategic management decision-making in a complex world: Quantifying, understanding, and using trade-offs. ICES J Mar Sci 74(2): 499–510. DOI: 10.1093/icesjms/fsv193

  88. Punt, AE, Butterworth, DS, de Moor, CL, De Oliveira, JAA and Haddon, M. 2016a. Management strategy evaluation: Best practices. Fish Fish 17: 303–334. DOI: 10.1111/faf.12104

  89. Punt, AE, MacCall, AD, Essington, TE, Francis, TB, Hurtado-Ferro, F, Johnson, KF, Kaplan, IC, Koehn, LE, Levin, PS and Sydeman, WJ. 2016b. Exploring the implications of the harvest control rule for Pacific sardine, accounting for predator dynamics: A MICE model. Ecol Model 337: 79–95. DOI: 10.1016/j.ecolmodel.2016.06.004

  90. Punt, AE, Okamoto, DK, MacCall, AD, Shelton, AE, Armitage, DR, Cleary, JS, Davies, IP, Dressel, SC, Francis, TB, Levin, PS, Jones, RR, Kitka, H, Lee, LC, McIsaac, JA, Poe, MR, Reifenstuhl, S, Silver, JJ, Schmidt, JO, Thornton, TF, Voss, R and Woodruff, J. 2018. When are estimates of spawning stock biomass for small pelagic fishes improved by taking spatial structure into account? Fish Res 206: 65–78. DOI: 10.1016/j.fishres.2018.04.017

  91. Reichler, T and Kim, J. 2008. How well do coupled models simulate today’s climate? Bull Amer Meteorol Soc 89(3): 303–311. DOI: 10.1175/BAMS-89-3-303

  92. Reid, RS, Nkedianye, D, Said, MY, Kaelo, D, Neselle, M, Makui, O, Onetu, L, Kiruswa, S, Kamuaro, NO, Kristjanson, P, Ogutu, J, BurnSilver, SB, Goldman, MJ, Boone, RB, Galvin, KA, Dickson, NM and Clark, WC. 2016. Evolution of models to support community and policy action with science: Balancing pastoral livelihoods and wildlife conservation in savannas of East Africa. Proc Natl Acad Sci USA 113(17): 4579–4584. DOI: 10.1073/pnas.0900313106

  93. Rose, KA, Fiechter, J, Curchitser, EN, Hedstrom, K, Bernal, M, Creekmore, S, Haynie, A, Ito, S, Lluch-Cota, S, Megrey, BA, Edwards, CA, Checkley, D, Koslow, T, McClatchie, S, Werner, F, MacCall, A and Agostini, V. 2015. Demonstration of a fully-coupled end-to-end model for small pelagic fish using sardine and anchovy in the California Current. Prog Oceanog 138: 348–380. DOI: 10.1016/j.pocean.2015.01.012

  94. Ruiz, D, Brun, C, Connor, SJ, Omumbo, JA, Lyon, B and Thomson, MC. 2014. Testing a multi-malaria-model ensemble against 30 years of data in the Kenyan highlands. Malar J 13: 14. DOI: 10.1186/1475-2875-13-206

  95. Shin, DW, Baigorria, GA, Romero, CC, Cocke, S, Oh, JH and Kim, BM. 2017. Assessing crop yield simulations driven by the NARCCAP regional climate models in the southeast United States. J Geophys Res-Atmos 122(5): 2549–2558. DOI: 10.1002/2016JD025576

  96. Skern-Mauritzen, M, Ottersen, G, Handegard, NO, Huse, G, Dingsør Stenseth, NC and Kjesbu, OS. 2016. Ecosystem processes are rarely included in tactical fisheries management. Fish Fish 17(1): 165–175. DOI: 10.1111/faf.12111

  97. Smith, ADM, Fulton, EJ, Hobday, AJ, Smith, DC and Shoulder, P. 2007. Scientific tools to support the practical implementation of ecosystem-based fisheries management. ICES J Mar Sci 64(4): 633–639. DOI: 10.1093/icesjms/fsm041

  98. Steinacher, M, Joos, F, Frolicher, T, Bopp, L, Cadule, P, Cocco, V, Doney, SC, Gehlen, M, Lindsay, K and Moore, JK. 2010. Projected 21st century decrease in marine productivity: A multi-model analysis. Biogeosciences 7(3). DOI: 10.5194/bg-7-979-2010

  99. Sydeman, WJ, Thompson, SA, Anker-Nilssen, T, Arimitsu, M, Bennison, A, Bertrand, S, Boersch-Supan, P, Boyd, C, Bransome, NC, Crawford, RJM, Daunt, F, Furness, RW, Gianuca, D, Gladics, A, Koehn, L, Lang, JW, Logerwell, E, Morris, TL, Phillips, EM, Provencher, J, Punt, AE, Saraux, C, Shannon, L, Sherley, RB, Simeone, A, Wanless, RM, Wanless, S and Zador, S. 2017. Best practices for assessing forage fish fisheries-seabird resource competition. Fish Res 194(Supplement C): 209–221. DOI: 10.1016/j.fishres.2017.05.018

  100. Tebaldi, C and Knutti, R. 2007. The use of the multi-model ensemble in probabilistic climate projections. Philos T Roy Soc A 365(1857): 2053–2075. DOI: 10.1098/rsta.2007.2076

  101. Urban, MC, Bocedi, G, Hendry, AP, Mihoub, JB, Pe’er, G, Singer, A, Bridle, JR, Crozier, LG, De Meester, L, Godsoe, W, Gonzalez, A, Hellmann, JJ, Holt, RD, Huth, A, Johst, K, Krug, CB, Leadley, PW, Palmer, SC, Pantel, JH, Schmitz, A, Zollner, PA and Travis, JM. 2016. Improving the forecast for biodiversity under climate change. Science 353(6304). DOI: 10.1126/science.aad8466

  102. van der Lingen, CD, Bertrand, A, Bode, A, Brodeur, R, Cubillos, L, Espinoza, P and Friedland, K. 2009. Trophic Dynamics. In: Checkley, DM, Alheit, J, Oozeki, Y and Roy, C (eds.), Climate Change and Small Pelagic Fish , 112–157. Cambridge, UK: Cambridge University Press. DOI: 10.1017/CBO9780511596681.009

  103. van der Lingen, CD, Hutchings, L and Field, JG. 2006. Comparative trophodynamics of anchovy Engraulis encrasicolus and sardine Sardinops sagax in the southern Benguela: Are species alternations between small pelagic fish trophodynamically mediated? Afr J Mar Sci 28(3–4): 465–477. DOI: 10.2989/18142320609504199

  104. van Voorn, GAK, Verburg, RW, Kunseler, EM, Vader, J and Janssen, PHM. 2016. A checklist for model credibility, salience, and legitimacy to improve information transfer in environmental policy assessments. Environ Modell Softw 83: 224–236. DOI: 10.1016/j.envsoft.2016.06.003

  105. Verburg, PH, Eickhout, B and van Meijl, H. 2008. A multi-scale, multi-model approach for analyzing the future dynamics of European land use. Ann Regional Sci 42(1): 57–77. DOI: 10.1007/s00168-007-0136-4

  106. Weaver, CP, Mooney, S, Allen, D, Beller-Simms, N, Fish, T, Grambsch, AE, Hohenstein, W, Jacobs, K, Kenney, MA, Lane, MA, Langner, L, Larson, E, McGinnis, DL, Moss, RH, Nichols, LG, Nierenberg, C, Seyller, EA, Stern, PC and Winthrop, R. 2014. From global change science to action with social sciences. Nat Clim Change 4(8): 656–659. DOI: 10.1038/nclimate2319

  107. Weichselgartner, J and Kasperson, R. 2010. Barriers in the science-policy-practice interface: Toward a knowledge-action-system in global environmental change research. Global Environ Change 20(2): 266–277. DOI: 10.1016/j.gloenvcha.2009.11.006

  108. Welch, C. 2015 Month: Feb Day: 12 . Fighting Over Herring—the Little Fish That Feeds Multitudes. National Geographic .

  109. Weller, DE, Benham, B, Friedrichs, M, Najjar, R, Paolisso, M, Pascual, P, Shenk, G and Sellner, K. 2013. Multiple models for management in the Chesapeake Bay. STAC Publication 14-004(14-004), 37.

  110. Witherell, D. 2003. Managing our nation’s fisheries: Past, present and future. In: Witherell, D (ed.). Washington, D. C.: Fisheries management in the United States.

  111. Woolsey, R. 2015. Month: Feb Day: 25 . Individual, commercial fishing interests clash in board testimony. Sitka, Alaska: KCAW radio.

  112. Young, JC, Waylen, KA, Sarkki, S, Albon, S, Bainbridge, I, Balian, E, Davidson, J, Edwards, D, Fairley, R, Margerison, C, McCracken, D, Owen, R, Quine, CP, Stewart-Roper, C, Thompson, D, Tinch, R, Van den Hove, S and Watt, A. 2014. Improving the science-policy dialogue to meet the challenges of biodiversity conservation: Having conversations rather than talking at one-another. Biodivers Conserv 23(2): 387–404. DOI: 10.1007/s10531-013-0607-0

comments powered by Disqus