Domain Editor-in-Chief: Donald R. Zak; School of Natural Resources & Environment, University of Michigan, Ann Arbor, Michigan, United States
Associate Editor: Julian D. Olden; School of Aquatic & Fishery Sciences, University of Washington, Seattle, Washington, United States

Koch et al. (2015) used structured expert judgment (Aspinall and Cooke, 2013) to quantify the uncertainty in performance of a range of stormwater control measures (SCMs) in two watersheds of contrasting physiography (the Piedmont and Coastal Plain provinces of the Chesapeake Bay basin). Their analysis of expert knowledge concluded that there is wide uncertainty in SCM performance, that rainfall is the primary source of variability in SCM performance, and that there is an order of magnitude greater export of nitrogen (per unit area of land) from Piedmont watersheds than from Coastal Plain watersheds.

As one of the experts whose judgment was elicited for the study (expert 9), I have concerns about the way information on uncertainty of N retention estimates was used in the study. In particular, two of the experts whose estimates were ultimately deemed of little information value (4 and 9), pointed to information that either a) casts doubt over the accuracy of the authors’ ‘known’ answers to calibration questions (which are used to assess the reliability of experts’ answers) for the Coastal Plain watershed, or b) if the data are accurate, raises important questions about the performance of the conventional stormwater drainage network in that watershed. The aim of this comment is to explore these concerns and re-assess the conclusion of large differences in export from urban watersheds between the Coastal Plain and Piedmont provinces. I raise two issues with the study: 1) the treatment of uncertainty; and 2) a large apparent inconsistency in the rainfall and discharge estimates provided for the Coastal Plain watershed. This re-assessment is important because it has large implications for the most appropriate approaches to stormwater management in the two provinces.

Each expert’s assessment was assigned a weight based on their estimates of 5th, 50th and 95th percentile of N export from 11 storms (6 Piedmont, 5 Coastal): by the range of uncertainty and by whether the known value fell within their estimate range. The methods by which known values were determined were not described in the paper, nor were estimates of uncertainty provided. Typically, load estimation has three sources of quantifiable uncertainty: a) error in chemical analysis of samples (which is likely to be small); b) sampling error in capturing the variability in N concentrations during the storm; and c) hydrographic measurement error (e.g. uncertainty in the rating curve). Like many estimates of physical phenomena, these estimates can be combined and propagated to indicate the range of known uncertainties (Bertrand-Krajewski and Muste, 2008). If a study using expert knowledge was aiming to quantify uncertainty, it would thus seem more appropriate to calibrate experts’ assessments of uncertainty against known uncertainty. A more continuous assessment of experts’ reliability (than the binary assessment of whether the known value fell within the experts’ range) could be achieved by assessing the degree to which each expert’s range overlapped with the known range of uncertainty.

The experts were provided with hydrographic data without information on the quality of the rating curves for each gage (Appendix S3, Koch et al. 2015). They were not provided information on measured N concentrations. My re-assessment of variability in estimates among the experts suggests that information on the quality of hydrographic information in particular is critical to a robust assessment of this work.

Four experts provided what were regarded as highly informative answers to the calibration questions for the Piedmont watershed (Q1–5, and 12; Appendix S6, Koch et al. 2015): the known value fell within the estimated range for all 6 questions for experts 1, 4, and 8, and for 5 of 6 for expert 9; and experts 8 and 9 had the lowest ranges of uncertainty. However, the estimates of experts 4 and 9 were ultimately rated lowly because their estimates of N export from the Coastal Plain watershed were an order of magnitude higher than the known values (Q33–37; Appendix S6, Koch et al. 2015).

The discrepancy in reliability of experts 4 and 9 between the 2 watersheds was not explored in detail by Koch et al. (2015), and no explanation of these experts’ reasoning was provided. In Appendix S5, Koch et al. (2015) noted that most experts were concerned about the 15-km distance between the rain and stream gages that were the source of Coastal Plain rainfall and discharge information. However, uncertainty arising from the distant rain gage cannot explain the much higher median estimates of experts 4 and 9. Rather, the large difference between their estimates and the known values (and the estimates of most other experts) must result from interpretations of flow volumes arising from inconsistencies in the provided rainfall and discharge data. I therefore present here the reasoning I used to identify such inconsistencies, and ultimately chose not to use the provided hydrographic data to calculate my N-load estimates.

The Coastal Plain watershed, which is in Severna Park, Maryland, draining to the North Cypress Branch, has a total area of 1.5 km2 of which 0.8 km2 is covered by impervious surfaces. Most of the impervious surfaces of the watershed are serviced by stormwater drainage pipes, including a central commercial area encompassing ∼0.35 km2 that is almost 100% impervious (Fig. 15, Appendix S3, Koch et al. 2015). Almost all of the indicated stormwater pipes drain directly to either of the 2 tributaries of the North Cypress Branch (Fig. 15, Appendix S3, Koch et al. 2015), and Google Street view images show that conventional ‘curb-and-channel’ roads draining to stormwater pits are the norm throughout the watershed. The SCMs of the watershed (including an engineered section of the northern tributary) intercept runoff from a small proportion of the watershed’s impervious surfaces (<0.1 km2). Although it is possible that some of the buildings of the catchment drain informally, permitting their runoff to infiltrate into soils, available information suggests that at least 50% (and likely much more) of the impervious surfaces drain to stormwater pipes, which are designed to deliver stormwater runoff to the stream without opportunity for infiltration into the watershed soils.

The discharge at the flow gage during each storm should therefore consist of at least 50% of the volume of runoff generated from impervious surfaces of the watershed together with some contribution of subsurface flows and potentially runoff from pervious surfaces during large events. In other words, impervious runoff should be no more than twice the observed discharge from the watershed.

However, if rainfall over the watershed equaled that recorded at the gage, runoff from impervious surfaces would be 7-50 times greater than the measured discharge volume in all 4 large storms considered in the paper, and 3-5 times higher in the 2 small events (Fig. 1). That the measured discharge for the largest event (155 mm) was only 15% of estimated impervious runoff is particularly surprising given that such a large event would likely have saturated the soils of the coastal plain (Markewich et al., 1990), giving rise to pervious runoff in addition to impervious runoff. The probability that the observed discharge values for the 6 events could be explained by spatial variation in rainfall was extremely small. For 4 events, the observed discharge was substantially less than the minimum estimated impervious runoff from 1000 simulations accounting for spatial variation in rainfall records, and for 2 it was near the minimum (Fig. 1; see Methods, below).

doi: 10.12952/journal.elementa.000077.f001.
Figure 1.  

Differences in reported discharge and likely stormwater runoff in 6 events.

The reported discharge (gray triangles) for 6 events in the Coastal Plain watershed considered by Koch et al. (2015). Gray squares show the runoff volume from impervious surfaces of the watershed assuming the rainfall recorded at the 15-km distant gage equaled rainfall on the watershed. Boxplots (including outliers as open circles) show the distribution of 1000 simulations of impervious runoff for each event assuming spatial variability in rainfall as reported by Baigorria (2007).

I therefore deemed the hydrographic data unreliable, and based my estimates of N export from the Coastal Plain watershed on the rainfall records. If my assessment was correct, then an important potential source of uncertainty in estimates of N export not considered by Koch et al. (2015) is hydrographic measurement error, and their conclusion of much lower N export per unit area from the Coastal Plain is incorrect.

Another explanation could be that the discharge data is correct, and large volumes of water exfiltrate into watershed soils either into an aquifer or to waters downstream of the gage. I consider this unlikely, but if it were the case, then the stormwater drainage system of the study watershed loses a large proportion of its flows before discharging to the stream: certainly a failure to perform as designed. However, if this unlikely behavior were the case, it has important implications for stormwater management design in watersheds with similar soils.

This analysis highlights the importance of stormwater drainage infrastructure in driving the hydrology of streams in urban catchments. While studies such as that of Utz et al. (2011) have demonstrated lower N yields from Coastal Plain watersheds compared to Piedmont watersheds—which Koch et al. (2015) attributed to the greater capacity for subsurface flows in Coastal Plain soils—it is important to consider the degree to which stormwater drainage systems prevent infiltration into the soils. Differences in soil properties between regions may drive a reduced reliance by civil engineers on piped drainage in sandier soils, and is likely to make the implementation of infiltration systems easier. However, it is difficult to conceive of a mechanism to explain such large differences in N export from conventionally drained urban watersheds from the two regions as deduced by Koch et al. (2015) if the drainage system successfully quarantines impervious runoff from watershed soils.


Impervious runoff was estimated for each storm and for daily rainfall data assuming a 1-mm initial loss before impervious surfaces generate runoff (Boyd et al., 1993). The volume of impervious runoff in ML therefore equals rainfall in mm minus 1, multiplied by the impervious area in km2.

To assess if spatial variation in rainfall could account for discrepancies between rainfall and discharge, I estimated likely uncertainty in the rain gage estimates. Daily rainfall records from gauges in south-east USA 0–50 km apart were correlated with R = 0.4–0.7, when combined by month (Baigorria et al., 2007). Using a 5-y daily rainfall record for Annapolis (2000–2004), I determined that normally distributed estimates of rainfall (mean = measured rainfall, standard deviation = 18) showed similar correlations with the recorded data to those reported by Baigorria et al. (2007: see Text S1). Selecting from this distribution, I generated for each event, 1000 rainfall estimates for the watershed, and then calculated the impervious runoff volume for the calculated rainfall. However, the 6 events were not random samples: each gave rise to a rapid rise and fall in discharge typical of rapidly conveyed urban stormwater (Figs. 18–25 Appendix S3, Koch et al. 2015). Thus none of the events could represent an occasion where rain was recorded at the gage, but not enough fell in the watershed to initiate impervious runoff. I therefore removed all simulations with <2 mm of rain, and resampled the distributions.


© 2015 Walsh. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Supplemental material

Text S1.
R script containing all data and calculations reported in this comment.
File Type: R
File Size: 0.01 MB

Competing interests

The author was one of the experts interviewed by Koch et al. (2015). He is funded by a research grant by Melbourne Water, a corporation owned by the Government of Victoria, with statutory responsibilities for the management of rivers and other waterways of the Melbourne region. They fund “public good” research to further the environmental sustainability of their operations.

Funding information

The author is funded by Melbourne Water through the Melbourne Waterway Research-Practice Partnership.