Data Management

Contents

Data Management#

Real-world data science is rarely limited by the choice of model — it is almost always limited by data quality. Before any regression or classification algorithm can be applied, the raw data must be collected, organized, cleaned, and stored in a form that supports efficient analysis.

This module covers the practical skills needed to manage data in chemical engineering contexts. We use the Dow Chemical distillation column dataset — a real industrial time-series with sensor readings, missing values, and outliers — as a running example throughout.

Topics#

  • Topic 4.1 — Data Organization: Pandas indexing and filtering, handling missing values, outlier detection, and efficient storage with HDF5.

  • Topic 4.2 — Online Data Access: Retrieving data from web APIs, parsing JSON responses, and working with databases programmatically.

Dataset#

The Dow impurity dataset (impurity_dataset-training.xlsx) contains time-stamped sensor measurements from a primary distillation column, including reflux flow rates, feed flows, and bed temperatures. The target variable y:Impurity is the product impurity level. This dataset was provided by Dow Chemical for educational use.

Contents