Exploratory data analysis

Exploratory data analysis (EDA) is that part of statistical practice concerned with reviewing, communicating and using data where there is a low level of knowledge about its cause system . It was so named by John Tukey. Many EDA techniques have been adopted into data mining.

Tukey held that too much emphasis in statistics was placed on evaluating and testing given hypotheses (confirmatory data analysis ) and that the balance was in need of redressing in favour of using data to suggest hypotheses to test. In particular, confusion of the two types of analysis and employing them on the same set of data can lead to bias owing to the effect of testing hypotheses suggested by the data.

The objectives of EDA are to:

• Suggest hypotheses about the causes of observed phenomena
• Assess assumptions on which statistical inference will be based
• Support the selection of appropriate statistical tools and techniques
• Provide a basis for further data collection through surveys or experiments

The principle graphical tools used in EDA are:

• Box plot
• Histogram
• MultiVari chart
• Run chart
• Pareto chart
• Scatter plot
• Stem-and-leaf plot

The principle quantitative tools are:

• Median polish
• Letter values
• Resistant line
• Resistant smooth
• Rootogram

Bibliography

• Hoaglin, D C; Mosteller, F & Tukey, J W (Eds) (1985) Exploring Data Tables, Trends and Shapes ISBN 0471097764
• Hoaglin, D C; Mosteller, F & Tukey, J W (Eds) (1983) Understanding Robust and Exploratory Data Analysis ISBN 0471097772
• Tukey, J W (1977) Exploratory Data Analysis ISBN 0201076160
• Velleman, P F & Hoaglin, D C (1981) Applications, Basics and Computing of Exploratory Data Analysis ISBN 087150409X