7.3 Assessment of Error

It is desirable to seek quantitative information on the potential magnitude of error in ISM data when using those data to make decisions. In all environmental sampling, two basic types of error are produced:

  • error associated with the collection of sample(s) in the field
  • error associated with the processing and analysis of those sample(s)

The ISM approach, which includes both field and laboratory steps, is intended to minimize the potential error and produce more technically defensible data by specifying the targeted volume and the parameter to be estimated and collecting, subsampling, and processing the sample(s) in accordance with the recommendations of sampling theory. An important component of this process is, to the extent possible, to assess the errors generated in the sampling and analysis scheme from beginning to end (i.e., from collection of soil in the field through the production of an analytical result).

Replicate ISM samples collected from each DU in the field provide a measure of total sampling and analysis error.

One means of evaluating ISM data is through comparison of the results of replicates, both as taken in the field and in the laboratory. As discussed in Section 5.3.5, field replicates consist of separate ISM samples taken by the field team from the same area (SU or DU). They are not field splits—they are collected and processed as separate samples. Laboratory replicates are samples taken from a single ISM sample, usually in the laboratory. They can be taken from the bulk ISM sample at a number of points during sample processing, depending on the process step(s) being evaluated. Replicates taken at the beginning of laboratory processing of the bulk ISM sample are used to evaluate potential overall error resulting from laboratory processing and analysis.

Three or more ISM samples are needed to calculate a defensible 95% UCL.

Use and interpretation of replicate data depends in part on the decision mechanism being applied. For example, field replicate data allow calculation of 95% UCL values needed for Decision Mechanism 3 and allow statistical comparison of site and background results using hypothesis testing in Decision Mechanism 4. Decision Mechanisms 5–7 can also rely on 95% UCL values calculated from field replicates.

The precision of ISM data can be quantified from replicate ISM sample results.

Replicate data can also be used to calculate an RSD, which is used to evaluate the precision of the data. RSD is a measure of reproducibility of estimates of the mean provided by replicates. Just as the sample mean and standard deviation are estimates of the corresponding population parameters, the sample RSD is an estimate of the ratio of the population parameters. It provides a measure of the total error associated with the data, although not necessarily the accuracy of the estimate. To calculate appropriate statistics, at least three field replicate samples are needed. Ideally, the project team then designates one of these replicates for separation into laboratory replicates. Replicate RSD data are intended to quantify the total error of the measurement system and attribute that error to either field sampling or laboratory procedures.

The total error is estimated based on the field replicate RSDs. Laboratory error can also be estimated based on the laboratory replicate RSDs. The field sampling component of error can then be estimated by subtracting the laboratory error from the total error. Therefore, the collection of field and laboratory replicates allows the error to be attributed to either the laboratory or the field sampling processes.

High RSD values for the laboratory component indicate potential problems with laboratory subsampling of the bulk ISM sample or other sources of analytical error. In this situation, the source(s) of laboratory error should be investigated and resolved.

High RSD values for the field component can have different implications depending on the decision mechanism being applied. For example, a high RSD (e.g., exceeding 30%–35%) from field replicates, but with acceptable RSDs from laboratory replicates, strongly suggests a substantial degree of heterogeneity in the DU contaminant concentrations. For Decision Mechanism 2, where a simple average of the replicates is used to derive the average concentration, this situation represents a problem. It means that estimates provided by the individual ISM replicates are quite variable and that the estimate of the average for the DU they provide may be unreliable. If the results are close enough to an action level that decision errors are possible, resampling with an increased number of increments may be used to reduce error. For Decision Mechanism 3, potential error created by heterogeneous concentrations is handled through calculation of the 95% UCL. Simulation studies discussed in Section 4 show, that with appropriate choice of 95% UCL method, conservative estimates of the mean to satisfy sampling objectives for this decision mechanism can be obtained despite high RSD values. This principle applies as well to other decision mechanisms where a 95% UCL is calculated.

A low RSD is not an indication that the mean is accurate or that the 95% UCL exceeds the population mean unless the distribution can be reasonably assumed to be relatively homogeneous.

A low RSD indicates that the field replicates are providing reproducible estimates of the average and generally triggers no additional steps to refine the estimate. However, it must be recognized that RSD is a measure of precision, not accuracy (see Section 4 for addition discussion of these concepts). Thus, an estimate of the average from replicates with a low RSD is not necessarily close to the actual mean. The opportunity for significant error is greatest when the DU is relatively heterogeneous and the replicates by chance give similar results. Unless information on heterogeneity of contaminants within the DU is available, it is difficult to judge whether this situation may have occurred and consequently the degree to which a low RSD should be reassuring. This is certainly an issue for the simple average of replicate data in Decision Mechanism 2. It is also an issue for Decision Mechanism 3 and others where a 95% UCL is calculated. Simulation studies discussed in Section 4 have shown that the UCL does not always ensure that a conservative estimate of the mean is obtained when the RSD is low. That is, when the RSD is low, the mean can be underestimated even by a 95% UCL. In short, a low RSD from field replicates offers information on the reliability of the estimate of average only when the contaminant distribution within the DU is known, or can be confidently assumed, to be relatively homogeneous.

For Decision Mechanism 6, replicates are often collected from a fixed percentage of DUs; however, the selection and number of DUs from which field and laboratory replicates are collected is not a simple matter—there is no one size fits all approach. Therefore, the number of DUs from which replicates are collected must be determined using site-specific considerations. Simply relying on a fixed percentage and arbitrary decisions to select which DUs will have replicates is ill advised.

If budgetary considerations limit the number of samples, field and laboratory replicates should be collected from those DUs that will provide the most useful information. Knowledge of source areas and areas likely to have high or low concentrations should be used to make deliberate choices. If there is a choice between a DU with anticipated high concentrations (i.e., above the action level) vs. one with low concentrations (i.e., close to the action level), the DU with concentrations closest to the action level should be selected for replicate samples. The closer contaminant concentration gets to the action level, the more important replicate statistics are in making a decision. Detection limit may also be a consideration in some situations. DUs with detectable concentrations provide more information than DUs were concentrations cannot be measured.

It is advisable to collect field and laboratory replicates from DUs that are believed to have different characteristics in terms of contaminant distribution, contaminant concentration, sampling design, or sample matrix. When less than 100% of DUs have replicate samples, the RSD (same as CV) from one or more DUs can be applied to similar DUs, subject to the limitations described for Decision Mechanism 6 above. If different sources of contamination or different release mechanisms are identified, field and laboratory replicates should be collected from each different DU. Furthermore, other factors that may influence the number of DUs with replicates are significantly different soil types that could cause different contaminant distributions and/or sample preparation efficiencies and different numbers of increments in ISM samples.