4.4.3.3 Site-to-background comparisons

For background screening approaches, summary statistics from discrete sample results (representing individual site measurements) are not directly comparable to summary statistics from ISM sample results, which represent mean estimatesA common element of most site investigations is the comparison of the contaminant distribution in volumes of soil collected from a DU to the distribution in soil collected from a suitable background or reference area. In some cases, regional background values that represent upper-bound estimates may have been derived or endorsed by regulatory entities. Since these types of background values are derived from discrete samples, their information content is sufficiently different from that of an ISM sample to preclude a direct comparison of summary statistics. For example, a set of discrete sample results provides a measure of the distribution of concentrations in relatively small volumes of soil throughout the DU, whereas a set of ISM samples provides measure of the distribution of mean concentrations, each of which is an estimate of the population mean for the entire DU. Therefore, the SDs estimated from the samples represent very different properties of the contaminant distribution. Regional background levels are typically based on an upper-bound statistic, such as an upper percentile or an upper tolerance limit (UTL, i.e., a UCL for a percentile). The objective is to establish a threshold for point-by-point comparisons to each individual (discrete) site result. If no site result exceeds the threshold, one can be reasonably confident that the distribution is not elevated with respect to background. Similar to the discussion of comparisons with numerical action levels, it may not be possible to satisfy decision objectives with ISM when a numerical threshold is intended for comparison to discrete observations (i.e., maximum concentrations in small volumes) rather than estimates of average concentrations. Discrete and ISM data sets have different characteristics, and statistical procedures for comparing DU ISM data with discrete background data, and vice versa, have not been well established.

An alternative background screening approach is to use hypothesis testing to compare the distributions, rather than screening against an upper-bound statistic. This alternative is often used because it is well established that there is a high likelihood with point-by-point screening that one or more site exceedances will be observed by random chance even if the distributions are exactly the same. Furthermore, the error rate increases with increasing numbers of samples for the site. The hypothesis testing approach allows for localized exceedances so long as the difference in the means (or upper tails) is not statistically significant.

For this document, comprehensive simulation studies were not conducted to evaluate the statistical performance of background comparison tests for ISM results (i.e., small number of samples, moderate asymmetry). Since tests are robust to moderate violations of assumptions of normality and equal variance, the fact that formal distribution testing cannot be conducted (see Section 4.1.1) is not expected to be a major limitation for background screening with ISM data. Instead, the two key challenges for ISM are achieving the desired statistical power of the tests (i.e., likelihood of detecting differences in the populations that exist) due to small number of samples and the inability to evaluate upper tails of the underlying distributions. Section 7.2.4 provides a detailed discussion of the assumptions associated with different hypothesis tests, highlighting why results of statistical tests can be misleading when the background and site data sets have fewer than five observations each. In addition, decision errors may be affected if the samples are collected with different sampling designs, including different number of increments/replicates, different sample masses, DU volume, sampling protocols, depth intervals, and sampling patterns. Therefore, the results of hypothesis tests applied to ISM data sets should be interpreted with caution until these limitations can be more thoroughly studied. If formal statistical tests are not used, simple graphical analysis (e.g., dot plots grouping ISM results by study area) may be informative as a semi-quantitative method for comparing background and site distributions. For background comparisons, graphical evaluations are preferred over formal statistical tests (e.g., hypothesis tests) because the performance of hypothesis tests has not been evaluated for small sample sizes (number of replicates) expected with most ISM sampling designs.

Comparison of site ISM data to background discrete data using either hypothesis testing or UTLs is not recommended because the variance is represented differently in ISM and discrete sampling. Comparison of an ISM estimate of the mean to a discrete sample collected from soil representing background is likely to lead to decision errors in which one incorrectly concludes that the contaminant distribution on site is consistent with background conditions.