# Exploratory Data Analysis

Before applying any machine learning model, it is essential to understand the
structure and properties of the data you are working with. Exploratory Data
Analysis (EDA) is the practice of summarizing, visualizing, and probing datasets
to uncover patterns, anomalies, and relationships that inform both feature
engineering and model selection.

This module focuses on the unique challenges that arise when the number of
features is large — a regime commonly encountered in chemical engineering applications
such as process data, spectroscopy, and materials science. We use two contrasting
datasets throughout: the **Dow Chemical distillation column dataset** (40 continuous
process variables) and the **MNIST handwritten digits dataset** (64 pixel-intensity
features), which together illustrate EDA strategies across a range of feature types
and scales.

## Topics

- **Topic 5.1 — High-Dimensional Data**: The curse and blessing of dimensionality,
  summary statistics, histogram grids, scatter plot matrices, and correlation heatmaps.

- **Topic 5.2 — Dimensionality Reduction**: Principal component analysis, t-SNE,
  and other techniques for projecting high-dimensional data to interpretable
  low-dimensional representations.

- **Topic 5.3 — Clustering**: k-means, hierarchical clustering, DBSCAN, and
  evaluation metrics for unsupervised grouping of data.

- **Topic 5.4 — Generative Models**: Gaussian mixture models and autoencoders
  for density estimation and latent-space representation.

## Datasets

The **Dow impurity dataset** (`impurity_dataset-training.xlsx`) contains time-stamped
sensor readings from a primary distillation column with 40 process variable inputs and
a product impurity target. The same dataset was introduced in Module 4; here the focus
shifts from data cleaning to exploratory analysis of the feature structure.

The **MNIST digits dataset** is a classic benchmark in machine learning, consisting
of 8×8 pixel images of hand-written digits 0–9. It is built into scikit-learn and
requires no download.