Data dredging

Data dredging is the term used to refer to the unscrupulous search for 'statistically significant' relationships in large quantities of data. This activity was formerly known in the statistical community as data mining, but that term is now in widespread use with a substantially different meaning, so the term data dredging is now used instead.

Conventional statistical procedure is to formulate a research hypothesis, (such as 'people in higher social classes live longer') then collect relevant data, then carry out a statistical significance test to see whether the results could be due to the effects of chance.

A key point is that one is not allowed to formulate the hypothesis as a result of seeing the data. If you want to work this way, you have to collect a data set, then partition it into two subsets, A and B. Subset A is held back and subset B is examined for interesting hypotheses. Once a hypothesis has been formulated it can be tested on subset A, since it was not used to construct the hypothesis.

Any large data set contains some chance features which will not be present in similar data sets, and to simply declare these as 'facts' is spurious. A typical example would be a TV marketing campaign intended to drive up sales. The campaign is run in one television area but not in another, which serves as a control group. Suppose that upon analysis it is found that sales in the treatment group are not significantly higher than in the control group. The analyst, fearful of telling the bad news to the sales director, analyses subgroups of the data and finds that sales did go up for left-handed Chinese males in the month of August, and the result is 'statistically significant'. This is then reported to the sales director in an attempt to offset the overall bad news.

It is important to realise that the alleged statistical significance here is completely spurious - significance tests do not protect against data dredging. You are testing a data set on which the hypothesis is known to be true, and that is therefore not a representative data set and any resulting significance levels are false.