Loading data

We can load the data used in this chapter with the following function.  It's very similar to the function we used in chapter 2, however it's adapted for this dataset.

from sklearn.preprocessing import StandardScaler

def load_data():
 """Loads train, val, and test datasets from disk"""
 train = pd.read_csv(TRAIN_DATA)
 val = pd.read_csv(VAL_DATA)
 test = pd.read_csv(TEST_DATA)

 # we will use a dict to keep all this data tidy.
 data = dict()
 data["train_y"] = train.pop('y')
 data["val_y"] = val.pop('y')
 data["test_y"] = test.pop('y')

 # we will use sklearn's StandardScaler to scale our data to 0 mean, unit variance.
 scaler = StandardScaler()
 train = scaler.fit_transform(train)
 val = scaler.transform(val)
 test = scaler.transform(test)

 data["train_X"] = train
 data["val_X"] = val
 data["test_X"] = test
 # it's a good idea to keep the scaler (or at least the mean/variance) so we can unscale predictions
 data["scaler"] = scaler
 return data