Python Programming Tutorials

Handling Non-Numerical Data for Machine Learning

Welcome to the 35th part of our machine learning tutorial series. We've recently begun talking about clustering specifically, but in this tutorial we're going to be covering handling non-numeric data specifically, which is of course not clustering-specific.

The data that we're going to be working with is the Titanic Dataset.

For a brief overview of the data and values:

Pclass Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)
survival Survival (0 = No; 1 = Yes)
name Name
sex Sex
age Age
sibsp Number of Siblings/Spouses Aboard
parch Number of Parents/Children Aboard
ticket Ticket Number
fare Passenger Fare (British pound)
cabin Cabin
embarked Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)
boat Lifeboat
body Body Identification Number
home.dest Home/Destination

The main focus on this dataset is typically on the survival column. When using supervised machine learning, chances are, you will be training the data against the survival column as the classification. With clustering, however, we let the machine make the groups, and basically a label of its own. My first interest is if the groups are clearly related to any of the columns, especially the survival column. For our current tutorial, we're currently doing flat-clustering, which is where we tell the machine we want two groups, but later we'll also let the machine determine the number of groups.

For now, however, we're up against another issue. If we read this dataset into a pandas, we'll see that we get something like:

#https://pythonprogramming.net/static/downloads/machine-learning-data/titanic.xls
import matplotlib.pyplot as plt
from matplotlib import style
style.use('ggplot')
import numpy as np
from sklearn.cluster import KMeans
from sklearn import preprocessing, cross_validation
import pandas as pd

'''
Pclass Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)
survival Survival (0 = No; 1 = Yes)
name Name
sex Sex
age Age
sibsp Number of Siblings/Spouses Aboard
parch Number of Parents/Children Aboard
ticket Ticket Number
fare Passenger Fare (British pound)
cabin Cabin
embarked Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)
boat Lifeboat
body Body Identification Number
home.dest Home/Destination
'''

df = pd.read_excel('titanic.xls')
print(df.head())

   pclass  survived                                             name     sex  \
0       1         1                    Allen, Miss. Elisabeth Walton  female   
1       1         1                   Allison, Master. Hudson Trevor    male   
2       1         0                     Allison, Miss. Helen Loraine  female   
3       1         0             Allison, Mr. Hudson Joshua Creighton    male   
4       1         0  Allison, Mrs. Hudson J C (Bessie Waldo Daniels)  female   

       age  sibsp  parch  ticket      fare    cabin embarked boat   body  \
0  29.0000      0      0   24160  211.3375       B5        S    2    NaN   
1   0.9167      1      2  113781  151.5500  C22 C26        S   11    NaN   
2   2.0000      1      2  113781  151.5500  C22 C26        S  NaN    NaN   
3  30.0000      1      2  113781  151.5500  C22 C26        S  NaN  135.0   
4  25.0000      1      2  113781  151.5500  C22 C26        S  NaN    NaN   

                         home.dest  
0                     St Louis, MO  
1  Montreal, PQ / Chesterville, ON  
2  Montreal, PQ / Chesterville, ON  
3  Montreal, PQ / Chesterville, ON  
4  Montreal, PQ / Chesterville, ON  
   pclass  survived  name  sex      age  sibsp  parch  ticket      fare  \
0       1         1   110    0  29.0000      0      0     748  211.3375   
1       1         1   839    1   0.9167      1      2     504  151.5500   
2       1         0  1274    0   2.0000      1      2     504  151.5500   
3       1         0   284    1  30.0000      1      2     504  151.5500   
4       1         0   563    0  25.0000      1      2     504  151.5500   

   cabin  embarked  boat   body  home.dest  
0     52         1     1    NaN        173  
1     44         1     6    NaN        277  
2     44         1     0    NaN        277  
3     44         1     0  135.0        277  
4     44         1     0    NaN        277

The issue is, we've got non-numerical data here. The machine learning algorithm is going to require numbers. We can just drop the name column, it has no use to us. Should we drop the sex column? I don't think so, it seems like a pretty important column, especially given our knowledge of "women and children first." What about the cabin column? Might it have been a bit important where on the ship you were? I suspect so! Maybe slightly less valuable is where you embarked from, but, at this point, we already know we're going to have to handle for non-numerical data anyway.

There are many ways to handle for non-numerical data, this is just the method I personally use. First, you will want to cycle through the columns in the Pandas dataframe. For columns that are not numbers, you want to find their unique elements. This can be done by simply take a set of the column values. From here, the index within that set can be the new "numerical" value or "id" of the text data.

To begin:

def handle_non_numerical_data(df):
    columns = df.columns.values
    for column in columns:

Creating function, getting the columns, beginning to iterate through them. Continuing:

def handle_non_numerical_data(df):
    columns = df.columns.values
    for column in columns:
        text_digit_vals = {}
        def convert_to_int(val):
            return text_digit_vals[val]

        if df[column].dtype != np.int64 and df[column].dtype != np.float64:
            column_contents = df[column].values.tolist()
            unique_elements = set(column_contents)

Here, we've added an embedded function that converts the parameter value to whatever the value of that item (as a key) is from the text_digit_vals dictionary. We aren't using it just yet, but we're about to. Next, while we're iterating through the columns, we're going to ask if that column is not either an np.int64 or np.float64. If not, then we're going to convert the column to a list of its values, then we take the set of that column to get just the unique values.

def handle_non_numerical_data(df):
    columns = df.columns.values
    for column in columns:
        text_digit_vals = {}
        def convert_to_int(val):
            return text_digit_vals[val]

        if df[column].dtype != np.int64 and df[column].dtype != np.float64:
            column_contents = df[column].values.tolist()
            unique_elements = set(column_contents)
            x = 0
            for unique in unique_elements:
                if unique not in text_digit_vals:
                    text_digit_vals[unique] = x
                    x+=1

            df[column] = list(map(convert_to_int, df[column]))

    return df

Continuing along, for each of the unique elements we find, we create a new dictionary key that is that unique element, with a value of a new number. Once we've iterated through all of the unique values, we then use mapping to map the function we created before to the pandas column. Not sure what mapping is? Check out the Mapping a Function with Pandas tutorial.

Now we can add a couple final lines:

df = handle_non_numerical_data(df)
print(df.head())

Full code:

#https://pythonprogramming.net/static/downloads/machine-learning-data/titanic.xls
import matplotlib.pyplot as plt
from matplotlib import style
style.use('ggplot')
import numpy as np
from sklearn.cluster import KMeans
from sklearn import preprocessing, cross_validation
import pandas as pd

'''
Pclass Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)
survival Survival (0 = No; 1 = Yes)
name Name
sex Sex
age Age
sibsp Number of Siblings/Spouses Aboard
parch Number of Parents/Children Aboard
ticket Ticket Number
fare Passenger Fare (British pound)
cabin Cabin
embarked Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)
boat Lifeboat
body Body Identification Number
home.dest Home/Destination
'''

df = pd.read_excel('titanic.xls')
#print(df.head())
df.drop(['body','name'], 1, inplace=True)
df.convert_objects(convert_numeric=True)
df.fillna(0, inplace=True)
#print(df.head())

def handle_non_numerical_data(df):
    columns = df.columns.values
    for column in columns:
        text_digit_vals = {}
        def convert_to_int(val):
            return text_digit_vals[val]

        if df[column].dtype != np.int64 and df[column].dtype != np.float64:
            column_contents = df[column].values.tolist()
            unique_elements = set(column_contents)
            x = 0
            for unique in unique_elements:
                if unique not in text_digit_vals:
                    text_digit_vals[unique] = x
                    x+=1

            df[column] = list(map(convert_to_int, df[column]))

    return df

df = handle_non_numerical_data(df)
print(df.head())

Output:

   pclass  survived  sex      age  sibsp  parch  ticket      fare  cabin  \
0       1         1    1  29.0000      0      0     767  211.3375     80   
1       1         1    0   0.9167      1      2     531  151.5500    149   
2       1         0    1   2.0000      1      2     531  151.5500    149   
3       1         0    0  30.0000      1      2     531  151.5500    149   
4       1         0    1  25.0000      1      2     531  151.5500    149   

   embarked  boat  home.dest  
0         1     1        307  
1         1    27         43  
2         1     0         43  
3         1     0         43  
4         1     0         43

If the df.convert_objects(convert_numeric=True) is giving you deprecation warnings or errors, feel free to just comment it out. I usually keep it there to be absolutely explicit, but the dataframe *should* be read as numbers where there are numbers. For some reason, Pandas will seemingly randomly read some rows in columns as strings, despite even the strings being actual number. Makes no sense to me, so I just convert to numeric to be totally certain.

Okay great, so we've got numbers, and now we can continue along to do flat clustering with this data!

The next tutorial:

Practical Machine Learning Tutorial with Python Introduction
Regression - Intro and Data
Regression - Features and Labels
Regression - Training and Testing
Regression - Forecasting and Predicting
Pickling and Scaling
Regression - Theory and how it works
Regression - How to program the Best Fit Slope
Regression - How to program the Best Fit Line
Regression - R Squared and Coefficient of Determination Theory
Regression - How to Program R Squared
Creating Sample Data for Testing
Classification Intro with K Nearest Neighbors
Applying K Nearest Neighbors to Data
Euclidean Distance theory
Creating a K Nearest Neighbors Classifer from scratch
Creating a K Nearest Neighbors Classifer from scratch part 2
Testing our K Nearest Neighbors classifier
Final thoughts on K Nearest Neighbors
Support Vector Machine introduction
Vector Basics
Support Vector Assertions
Support Vector Machine Fundamentals
Constraint Optimization with Support Vector Machine
Beginning SVM from Scratch in Python
Support Vector Machine Optimization in Python
Support Vector Machine Optimization in Python part 2
Visualization and Predicting with our Custom SVM
Kernels Introduction
Why Kernels
Soft Margin Support Vector Machine
Kernels, Soft Margin SVM, and Quadratic Programming with Python and CVXOPT
Support Vector Machine Parameters
Machine Learning - Clustering Introduction
Handling Non-Numerical Data for Machine Learning
K-Means with Titanic Dataset
K-Means from Scratch in Python
Finishing K-Means from Scratch in Python
Hierarchical Clustering with Mean Shift Introduction
Mean Shift applied to Titanic Dataset
Mean Shift algorithm from scratch in Python
Dynamically Weighted Bandwidth for Mean Shift
Introduction to Neural Networks
Installing TensorFlow for Deep Learning - OPTIONAL
Introduction to Deep Learning with TensorFlow
Deep Learning with TensorFlow - Creating the Neural Network Model
Deep Learning with TensorFlow - How the Network will run
Deep Learning with our own Data
Simple Preprocessing Language Data for Deep Learning
Training and Testing on our Data for Deep Learning
10K samples compared to 1.6 million samples with Deep Learning
How to use CUDA and the GPU Version of Tensorflow for Deep Learning
Recurrent Neural Network (RNN) basics and the Long Short Term Memory (LSTM) cell
RNN w/ LSTM cell example in TensorFlow and Python
Convolutional Neural Network (CNN) basics
Convolutional Neural Network CNN with TensorFlow tutorial
TFLearn - High Level Abstraction Layer for TensorFlow Tutorial
Using a 3D Convolutional Neural Network on medical imaging data (CT Scans) for Kaggle
Classifying Cats vs Dogs with a Convolutional Neural Network on Kaggle
Using a neural network to solve OpenAI's CartPole balancing environment