A.1.2.2 Criticism of the M-3 simulations

The primary criticism of the M-3 simulations focuses on whether the simulations are sampling from the same population as the one from which the "true" mean is being estimated. Two aspects of this concern are described below.

Inaccessible locations. A DU with inaccessible locations still has soil in those locations, and the soil has some characteristics, including concentration levels of the analyte of interest. These inaccessible locations can be handled several different ways in practice. The sampling team can try to avoid placing proposed sampling locations in inaccessible locations, they can ignore those locations if selected, or they can develop a scheme, much like in the simulations in Section A.5, to take nearby soils if the exact location is inaccessible.

It is fairly intuitive that if the sampling team gathers soils from areas near the inaccessible sample locations, they won’t get exactly the same results as for the locations that were selected (by simple random sampling or other random within grid or other method), but they will get something nearby that will generally be similar to the soils in the inaccessible location. Of course, there are times when this will not be quite true. For example, if the contamination is due to aerial dispersion and there is a building that has been in place since before the beginning of the period of contamination, the soils under that building are not likely to be well represented by the soils near the building. Nonetheless, depending on the CSM, it is often a reasonable sampling approach to collect nearby soils with the expectation that they will provide an acceptable surrogate for the actual selected sampling location.

The intent of the sampling exercise discussed here is to determine the average concentration of a particular analyte across the DU. The simulations in this appendix all have that basic goal in mind. The nice thing about simulations is that, unlike in field studies, we know the "truth." That is, the actual mean concentration across the DU is known in these simulations. With this knowledge, we can look at the outcomes of various simulated approaches, compare them to the known characteristics of the DU, and determine how well the sampling method works for each simulated DU.

To calculate the true mean of the DU, the approach is usually simple. If the simulations are from a probability distribution (PD), the mean is defined by that distribution. If the simulations are from a map with different concentration values assigned to locations on the map, then the concentrations of all locations should be summed and divided by the total number of locations on the map.

The challenge comes when discrete points are spread over a map and not all points on a map are of equal size or there are areas with multiple points or no points (i.e., the points are not distributed evenly and completely across the map). In these cases, even though it is a simulation, the true mean of the entire DU is not actually known, and some method has to be agreed on for estimating it. Focusing only on the maps of DUs with areas that are inaccessible for sampling, an approach to defining the true mean must be defined. Is it reasonable to include only those locations for which the concentration is known and call that the true mean? To do so would be to ignore that there are many areas of the site where the true concentrations are not known. It is certainly possible to estimate the mean at the site by just including the known concentration values. But, if the true mean is estimated that way, then the sampling method would have to use the same approach (i.e., avoid the inaccessible areas) to have results that can be compared to the estimated true mean. If instead, the concentrations across the inaccessible areas are included in the estimate of the true mean (by using the concentrations of their nearest neighbors as reasonable surrogates for them, for example), then the sampling methodology will again have to use a similar approach for the results to be compared meaningfully to the estimate of the true mean. The project team must determine which population is actually of interest (the soils across the entire DU or only those soils that are accessible) and then work from that population throughout their simulations.

If the method used to estimate the true mean does not match the simulation sampling method, then it is not appropriate to compare the simulation results to the estimated true mean. In the case where the true mean is estimated using only the known concentration values and the simulations impute values from near neighbors rather than avoid the inaccessible locations, the simulation results will be estimating the bias that is created by having that mismatched logic. Essentially, what that would be looking at is the following:

  • The true mean, estimated only from known concentrations without any input for the areas of the site that are inaccessible, is equivalent to assuming that the mean of the concentrations in the inaccessible areas is equal to the mean of the known concentrations.
  • The simulated mean, calculated using nearest neighbors when inaccessible sampling locations are selected, is equivalent to assuming that the concentrations in the inaccessible areas are most similar to their nearest neighbors.
  • Comparing this estimated true mean to these simulated results would be a way to assess the bias that occurs if the inaccessible soils are assumed to be most similar to their neighboring soils, but in fact they are just the same, on average, as the soils from across the entire DU and not more similar to those soils nearest to them.

For this reason, the results of the simulations in Section A.5 should not be considered indicative of the implications of ISM sampling of bulk materials but rather an indication of why it is very important to be sure that the population of interest is well defined and the sampling strategy is carefully designed to be sure it is capturing the information that it is intended to capture.

Equal Sample Support. To define sample support in a simulation, some type of unit size must be defined. In actual applications sample support is often thought of in terms of mass. The simulations could define sample support as a 2-D area or include a mass characteristic for each location.

The simulations in Section A.5 discuss the use of increments with the “same sample support.” They then say that they use the method of including the number of points within a specified distance of that location (2-D area). However, the documentation does not explicitly state what is done with the points found within the specific distance. Ignoring the issue in the previous paragraphs, the sampling process in Section A.5 identifies a location then searches for all discrete points within a specified distance and defines those points to be included in an increment that then is included in a larger ISM sample. In the A.5 simulations, it is assumed that equal 2-D area is equivalent to "same sample support."

As increments are collected across the sites in Section A.5, some increments can have 1 point included and others can have 10, 20, or more points included in one increment. The concentration level of each point is averaged to provide one concentration value for the increment. This average of the point values (irrespective of how many points were used to make the increment average) is then included with the other increments, assumed to be of equal support, into the ISM sample. Yet the method to define the true mean is based on a raw average of all points thrown on the map.

The two-stage averaging that does not keep track of the number of points per increment is an additional factor that creates a bias from the assumed "true" mean. This bias is not a result of ISM—it is only a function of the algorithm applied in the simulation process. The underlying assumption of equal sample support is a noble goal, but as with the previously stated concern, the actual simulation routines produce results that are indicative of the logic used in the simulations and not of ISM sampling of bulk materials.