Sampled to Death? The Rise and Fall of Probability Sampling in Archaeology

Edward B. Banning

After a heyday in the 1970s and 1980s, probability sampling became much less visible in archaeological literature as it came

under assault from the post-processual critique and the widespread adoption of “full-coverage survey.” After 1990, published

discussion of probability sampling rarely strayed from sample-size issues in analyses of artifacts along with plant and animal

remains, and most textbooks and archaeological training limited sampling to regional survey and did little to equip new gen-

erations of archaeologists with this critical aspect of research design. A review of the last 20 years of archaeological literature

indicates a need for deeper and broader archaeological training in sampling; more precise usage of terms such as “sample”;

use of randomization as a control in experimental design; and more attention to cluster sampling, stratified sampling, and non-

texto y el entrenamiento arqueológico limitaron el muestreo al estudio regional e hicieron poco para equipar a las nuevas

generaciones de arqueólogos con este aspecto crítico de diseño de la investigación. Un resumen de los últimos 20 años de

literatura arqueológica indica la necesidad de una formación arqueológica más profunda y amplia en el muestreo, el uso

más preciso de términos como “muestra”, el uso de la aleatorización como control en el diseño experimental y una mayor

atención al muestreo conglomerado, muestro estratificado, y muestreo no espacial tanto en capacitación como en

investigación.

Palabras clave: muestreo de probabilidad, estadísticas, prospección, historia de la arqueología, pedagogía arqueológica

R

ecently, I was asked to write a contribu-

tion on spatial sampling in archaeology

book. Given that sampling theory is a critical

aspect of research design and control for bias,

this puzzled and concerned me.

There can be legitimate reasons not to employ

probability sampling. What rattled me when I

tried to find those case studies is the possibility

that many archaeologists are neglecting probabil-

ity sampling for the wrong reasons.

Here, I explore some possible reasons for this

neglect before offering some suggestions for

American Antiquity 86(1), 2021, pp. 43–60

Copyright © 2020 by the Society for American Archaeology. This is an Open Access article, distributed under the terms of the

Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use,

distribution, and reproduction in any medium, provided the original work is properly cited.

doi:10.1017/aaq.2020.39

43

Sampling entails two important concepts: (1) the

population, or set of phenomena—such as sites,

lations that consist of spaces, such as excavation

squares, because we cannot enumerate popula-

tions of sites, artifacts, or “ecofacts” that we

have not yet surveyed or excavated. A “sampling

frame” is a list of the population’s “elements” or

members, or a grid or map for identifying the set

of spatial elements in a spatial population. Sam-

ple size is just the number of elements in the sam-

ple, whereas sampling fraction is the sample size

divided by the number of elements in the whole

of whole populations impractical or unethical.

We also recognize that taphonomic factors can

distance our sample of a “fossil assemblage”

still farther from a “deposited assemblage” that

may be our real population of interest (Holtzman

1979; Meadow 1980). The question is, How con-

fident should we be about inferences based on a

small subset of a population?

venience” or “oppo rtunistic” samples are just

superior information for some purpose, such as

excavation areas selected for their probability

of yielding a long, stratified sequence. Samples

such as these are not flawed, for certain purposes

at least, but they entail the risk that they may not

be “representative” of the population of interest.

In other words, the sample’s characteristics

might not be very similar to the characteristics

of the whole population. A nonrandom differ-

ence between the value of some population char-

population so that we can estimate a parameter

on the basis of a statistic. This always involves

some randomness. The classic probability sam-

pling strategies include simple random sampling

with replacement, in which every “element” of

the population—whether an artifact, bone frag-

ment, space, or volume—has an equal probabil-

ity of selection at each and every draw from the

population. This is like picking numbers from a

hat but then replacing them so that some ele-

sampling, in which we randomly select the first

element and then all the others are strictly deter-

mined by a “spacing rule.” For example, we

might organize artifacts in rows, randomly select

one of the first four artifacts by rolling a die

(ignoring 5 and 6), and then take every fourth

artifact in sequence to yield a 25% sampling frac-

tion. Stratified sampling involves dividing the

population into subpopulations (“strata”) that dif-

fer in relevant characteristics before sampling

of these basic ones.

Sample elements need not be spatial

(Figure 2), but the fact that archaeologists can

rarely specify populations of artifacts or “eco-

facts” in advance often forces them to employ

cluster sampling. Cluster samples occur when-

ever the population of interest consists of items

such as artifacts, charcoal, or bone fragments,

but the population actually sampled is a spatial

one, such as a population of 2 × 2 m squares

accessibility—could complicate sampling

and much of the literature it inspired strongly

privilege sampling in regional surveys, with less

attention to sampling sites, assemblages, or arti-

facts (but see Orton 2000).

cate the next generation of archaeologists (Ragir

1967; Watson et al. 1971). Most focused on the

basic spatial sampling designs. Generally lacking

was discussion of when probability sampling

was appropriate and how to defi ne populations

(Ambrose 1967), and artifact analysis (Cowgill

1964) also appeared. There were more general

reviews (Mueller 1975b;O’Brien and Lewarch

1979; Redman 1974) and desktop simulations

of sampling designs (Mueller 1974; Plog

1976). Sampling soon saw application outside

North America (e.g., Cherry et al. 1978;Mac-

Donald et al. 1979; Redman and Watson 1970),

and the number of articles in American Antiquity

that discussed or used probability sampling grew

error (Nance 1981), or sampling shell middens

(Campbell 1981).

During the 1970s, much research still used

purposive sampling or ignored this sampling

wave. Even authors who did not embrace sam-

pling, however, tended to be somewhat apolo-

getic, offering caveats about their samples’

usefulness or describing their research as

preliminary.

The Fall of Probability Sampling in

or mentions of sampling in the statistical sense

peaked about 1980 in the former and 1990 in

the latter, then declined, albeit with some recov-

ery in the late 1990s and again in the last few

years, never returning to pre-1990 levels. What

these graphs do not reveal is that articles prior

to 1985 tend to be about sampling, while most

after 1990 just mention having used some kind

of random or systematic sample, usually without

presenting any details. Those few about sam-

ment, especially as research in statistics devel-

oped considerably after 1970 (e.g., Orton

2000:11; Thompson and Seber 1996).

Remarkably, a common claim of the 1990s

is that some pattern “is highly unlikely to result

from sampling error or random chance” (e.g.,

Falconer 1995:405), despite relying on small

or non-probabilistic samples. Other authors

acknowledge bias in their data but go on to

analyze them as though they are unbiased, or

flaws in then-recent archaeological sampling—

including arbitrary sampling fractions, the sup-

pression of standard error through impractically

small sample elements, and the ignoring of

prior information—foreshadowed this decline.

Hole (1980:232), however, was not criticizing

sampling per se, just its misapplications. Even

though some authors judged her critique as

extreme (Nance and Ball 1986; Scheps 1982),

advocated abandoning probability sampling sug-

The Post-Processual Critique

During the 1980s, attacks on sometimes pseudo-

scientific or dehumanizing examples of New

Archaeology engendered an anti-science rhetoric

that may have made probability sampling a vic-

tim. Shanks and Tilley led this attack by arguing

that mathematical approaches entail assumptions

that theory is value free, and that “categories of

analysis are necessarily designed to enable

certain calculations to be made” (1987:57).

ever, is not really the use of samples but the use

of inadequate ones (Sørensen 2017:106). This is

not a problem with science; it just underscores

the need for better sampling.

paradigm still base inferences on samples and

analyze data as though they represent something

more than the sample itself. Even Shanks and

Tilley (1987:173–174) explicitly used stratified

sampling in their analysis of beer cans and

Korinthian pots” (1999:40). This is an opportun-

istic sample of “all complete pots known” to

Shanks, but he expects them to represent popula-

tions of artifacts and the people who made them,

such as the pottery of “archaic Korinth” (Shanks

1999:2, 9, 10, 151). He further generalizes about

the “early city state,” an even more abstract popu-

lation (Shanks 1999:210

–213), and wonders if

his sample is “somewhat biased” for some pur-

science archaeologists still use statistical reason-

ing and sampling. Their anti-science rhetoric,

however, may still have had a chilling influence

on explicit archaeological sampling.

The “Full-Coverage” Program

When Fish and Kowalewski (1990) published

The Archaeology of Regions, “full-coverage

survey” had already begun to trend. Its premise

that small samples are an inadequate basis for

some kinds of research is undeniable (Banning

1990:254; Kintigh 1990:238). “Full coverage”

site size, the proportion of small sites, and the

rank-size statistic—because they underrepresent

small, unobtrusive sites and artifact densities

unless their practitioners correct for this (Cowgill

1990:252–258). None of the surveys in The

Archaeology of Regions did so, however.

decrying sampling at the 1993 Theoretical

Archaeology Group conference, “almost every-

body was against sampling” (Kamermans

Certain misconceptions also discouraged interest

in sampling. Binford (1964) had proclaimed that

sampling requires populations of equal-sized

spatial units. Many archaeologists found that

arbitrary grids, especially of squares, were rarely

practical or useful because their boundaries did

Biases could result from the nature of these

proportion of decorated pottery, but it might be

acceptable for estimating the proportions of tem-

per recipes in pottery fabrics, for example.

Bayesian exchangeability has any role in archae-

ologists’ attitudes to probability sampling. Few

texts on archaeological analysis even mention

exchangeability (Banning 2000:88; Buck et al.

1996:72–74; Orton 2000:21). One other does,

but without naming it (Drennan 2010:88–92).

chaeological training (cf. Cowgill 2015; Thomas

1978:235, 242). A publication on teaching

archaeology in the twenty-first century (Bender

and Smith 2000) ignores sampling design, or

even statistics more generally, except for a single

mention of sampling as a useful skill (Schulden-

rein and Altschul 2000:63). A proposed area

for reform of archaeological curriculum, “Funda-

English-language textbooks I could find that

cover archaeological methods, but it limits

multiple-edition books to the earliest and latest

editions I could access.

Renfrew and Bahn’s(2008:80–81) explica-

tion of major spatial sampling strategies is typ-

ical. After briefly mentioning non-probabilistic

sampling, they describe random, stratified

random, systematic, and systematic unaligned

sampling designs, all in spatial application.

rationale for strata or the need to verify that strata

are statistically different. As usual, sampling’s

justification is that “archaeologists cannot usu-

ally afford the time and money necessary to

investigate fully the whole of a large site or all

sites in a given region” (Renfrew and Bahn

2008:80), while they say that probability sam-

pling allows generalizations about a site or

region.

Most texts that do not specialize in quantita-

error, or nonspatial sampling. All but two only

present sampling in regional survey, and only

four mention alternatives to geometrical sample

elements. A few misrepresent sampling as

a means to find things, especially sites, rather

than to estimate parameters or test hypotheses

(e.g., Muckle 2014:99; Thomas 1999:127).

Seventeen more specialized texts provide a fuller

discussion of sampling, but they probably reach

smaller, more advanced audiences.