Coverage and magnitude of UCL errors

In repeated trials, an appropriate 95% UCL should exceed or “cover” the true mean 95% of the time. In practice, we never know how well a 95% UCL has performed in terms of coverage because the true mean is unknown. However, in simulation studies we have the opportunity to repeatedly evaluate a theoretical DU for which the mean is known and compute many UCLs.

For positively skewed distributions (e.g., lognormal), the mean of the ISM samples will underestimate the population mean >50% of the time, whereas the 95% UCL will typically underestimate <5% of the time. Accordingly, coverage is defined in this context as the percentage of the simulations for which the 95% UCL actually exceeds the true DU mean. As an example, Table 4-2 gives selected results for a simulation with 5000 trials where the mean and 95% UCL were calculated by sampling a lognormal distribution with mean = 100 and SD = 200. The 95% UCL for each trial was based on a Chebyshev equation applied to sample statistics for 3 replicates of 30 increments. The values from the UCL column are then compared to the true mean of 100. If the design were built to theoretically have 95% confidence that the true mean was less than the calculated UCL, then the ideal result from the 5000 iterations would be to find approximately 5% (i.e., 250 of 5000) of the UCL values are below the true mean. Figure 4-4 shows a histogram of the 5000 UCL values from this simulation where the y-axis represents the fraction of total iterations in each bin. In this example, the UCL histogram shows that approximately 5% of the UCL values are below the true mean. This exercise shows that the UCL coverage for this simplified scenario met the design criteria. It is interesting to note that the grand mean of 3 replicates underestimated the true mean nearly 60% of the time (in contrast to the 95% UCL underestimating the mean only about 5% of the time), exemplifying why the UCL is often used to protect against underestimation of the true mean.

Table 4-2. Example of UCL simulations
Trial Mean UCL RPD
1 64.7 85.0 –15%
2 61.7 102.7 2.7%
3 100.7 105.2 5.2%
4 90.8 107.0 7.0%
4999 96.1 215.3 115.3%
5000 253.2 855.0 755.0%
RPD = [(UCL 100)/100] 100%.

Histogram of calculated Chebyshev UCL values using 5000 trials of a lognormal distribution (mean = 100, SD = 200), 30 increments, and 3 replicates.
Figure 4-4. Histogram of calculated Chebyshev UCL values using 5000 trials of a lognormal distribution (mean = 100, SD = 200), 30 increments, and 3 replicates. The "true mean" (100) is exceeded in approximately 95% of the trials.

The optimal methodology for calculating a UCL should provide adequate coverage of the mean and produce a UCL that is not unduly large. The magnitude of difference between the UCL and the true mean can be expressed as the RPD defined as follows:

RPD = [(95% UCL – μ)/μ] × 100%

As shown in Table 4-2, RPD may be negative or positive depending on whether or not the UCL exceeds the true mean. RPD may be calculated for all UCL results or can be calculated for those UCLs that fall above (RPDA) and below (RPDB) the true site mean, separately. When used for just those UCLs that fall below the site mean, the RPDs reveal the magnitude of the potential underestimation. This calculation is particularly informative in situations where the coverage does not meet the specified 95% criteria.

Figure 4-5 illustrates examples of RPDA and RPDB for simulations using lognormal distributions with CV = 1 and CV = 4. Each simulation represents 5000 trials using 30 increments (m) and 2, 3, 5, or 7 replicates (r). Results for both the Chebyshev UCL and Student’s-t UCL are given side by side. Error bars represent the 5th and 95th percentile RPD values, and the point in the center corresponds to the median. For example, for CV = 1 and r = 3, the Chebyshev UCL generally exceeds the true mean by less than 50% and underestimates by less than 10%. The deviation of the UCL using Student’s-t is slightly lower for the overestimates and comparable for the underestimates. For CV = 4 and r = 3, the magnitude of the deviations increases for both the Chebyshev UCL (95th percentile RPDAof 214% and RPDB of –23%) and Student’s-t UCL (95th percentile RPDA of 160% and RPDB of –25%). Information on coverage and RPD ranges can be combined to yield the following observations:

The Student's-t UCL and Chebyshev UCL provide estimates of the mean that, even for highly variable distributions, generally exceed the true mean by no more than 200% or underestimate the mean by no more than 25%.

  • Even for distributions with high variance (e.g., CV = 4, r = 3), the 95% UCL using either Chebyshev or Student’s-t equations can be expected to yield values that exceed the true mean by no more than 150%–200% and underestimate by less than 25%;
  • Student’s-t UCL more frequently underestimates the true mean than does the Chebyshev UCL.
  • The magnitude of the underestimate (RPDB) will be comparable; however, the magnitude of the overestimate (RPDA) will be greater for the Chebyshev UCL.

Figure 4-5. Range of overestimation (RPDA) and underestimation (RPDB) of 95% UCLs using Chebyshev and Student's-t calculation methods for ISM simulations with lognormal distributions (CV = 1 and CV = 4), 30 increments and 27 replicates. Error bars represent 5th and 95th percentiles of 5000 trials.

ISM replicates tend to produce UCLs with smaller RPDA and RPDB than a corresponding data set of discrete samples. This desirable quality of ISM is due to the physical averaging of the individual increments. Therefore, ISM UCL values may provide reasonably reliable estimates of the site mean even when the desired 95% coverage is not achieved but RPDB is minimal. It is unlikely that one 95% UCL method excels at all performance metrics. In addition, performance can vary depending on site characteristics. Method selection requires balancing the importance of each metric.

In general, all other conditions being the same, as the number of increments and replicates increases, the error is expected to decrease. This decrease in the standard error will be reflected by an improvement in bias, the coverage and RPD of the UCL. The influence of these components of the sampling design varies depending on characteristics of the population sampled (e.g., magnitude of DH, single or multiple populations) and the sampling method (e.g., systematic random sampling, random sampling with grid, or simple random sampling). The central concept governing the optimization of the sampling design is that while initial increases in the number of replicates and increments improve estimation, there are diminishing returns with increasing numbers of samples. At some point, increasing the number of samples is unlikely to yield an appreciable improvement in either the coverage of the UCL or the magnitude of the over/underestimate of the UCL as indicated by the RPD calculations.

5 Note that this concept is completely separate and unrelated to that of spatial "coverage" as applied to areal representativeness of samples taken over a DU.