Machine Learning: Classification

This How-To Guide demonstrates how to load a dataset, build a classification model, and perform predictions using that model.

Goal#

The goal of this how-to guide is to provide an introduction to Rel’s machine learning functionality. As one part of a larger series of machine learning how-to guides, this guide will focus on classification. Specifically, we will explore how to load a dataset, build a classification model, and perform predictions using that model.

Preliminaries#

We recommend that you also go through the CSV Import Guide and JSON Import and Export Guide, since they contain examples and functionality useful to understand how to appropriately load different kinds of data into the system.

Dataset#

For this how-to guide we will be using the Palmer Archipelago (Antarctica) penguin data. We will use a copy of the penguin dataset located in our public S3 bucket.

This is a multivariate dataset with instances of penguins together with their features. We will be using the penguins_size.csv file for our guide.

The dataset contains 344 instances of penguins from three species (classes), Adelie, Chinstrap and Gentoo. The Adelie species contains 152 instances of penguins, Chinstrap has 68, and Gentoo has 124.

For each instance within the dataset, in addition to the species, there are 6 features:

Our goal in this guide is to build a classifier to predict the species of the penguin, given its features.

Here is a sample of the first five lines of the penguins_size.csv file that we will be working with:

species,island,culmen_length_mm,culmen_depth_mm,flipper_length_mm,body_mass_g,sex
Adelie,Torgersen,39.1,18.7,181,3750,MALE
Adelie,Torgersen,39.5,17.4,186,3800,FEMALE
Adelie,Torgersen,40.3,18,195,3250,FEMALE
Adelie,Torgersen,NA,NA,NA,NA,NA
Adelie,Torgersen,36.7,19.3,193,3450,FEMALE
...

As you can see, there are certain instances of penguins where the data is not available (as denoted by NA in the example above). To address this, we will be performing some data cleaning over the loaded data as we will discuss in a bit.

Loading the Data#

Let’s begin building a classifier by loading the data from the file containing the penguin data. We can load the file using load_csv as follows:

def config[:path] = "s3://relationalai-documentation-public/ml-classification/penguin/penguins_size.csv"

def config[:schema, :species] = "string"
def config[:schema, :island] = "string"
def config[:schema, :culmen_length_mm] = "float"
def config[:schema, :culmen_depth_mm] = "float"
def config[:schema, :flipper_length_mm] = "float"
def config[:schema, :body_mass_g] = "float"
def config[:schema, :sex] = "string"

// insert transaction
def insert[:penguin_raw] = lined_csv[load_csv[config]]

Note, in the code above, we have specified the path to the file which is located in our public AWS S3 bucket. We used an s3:// url, which indicates a path to a public AWS bucket.

Additionally, we are reading the header from the file and we will use the header names as our feature names. Finally, we specified the schema of the imported file. Specifically, we indicated that the first two and last features (species, island, sex) are of type string, while the remaining (culmen_length_mm, culmen_depth_mm, flipper_length_mm, body_mass_g) are float. In this guide we will learn to predict the species feature.

Cleaning the Data#

As we discussed in the previous section, there are certain instances (or lines) in the dataset that we would need to clean up. One such instance was shown earlier, where all the values where set to NA. As a first step, Rel has already cleaned up these values for us. Since it wasn’t able to parse NA as float, these instances were stored as load_errors inside the penguin_raw relation:

penguin_raw:load_errors

As we can see from the file positions, there were two such lines having all of their features set to ‘NA’ in the dataset.

In addition to those errors, there are also a few lines where sex is defined as NA (8 in total) and one line where sex is defined as .. For the purpose of this guide we will drop all rows with an issue and we can get a clean dataset as follows:

def row_with_error(row) =
    penguin_raw:sex(row, "NA") or
    penguin_raw:sex(row, ".") or
    penguin_raw:load_errors(row, _, _)

def insert[:penguin] = column, row, entry... :
    penguin_raw(column, row, entry...) and not
    row_with_error(row)

The final penguin dataset looks as follows:

table[penguin]

Analyzing the Data#

We can easily visualize the data we just loaded in different ways. For example, let’s take a look at the distribution of male and female penguins by species:

def output = vegalite:plot[
    vegalite:bar[
        :species,
        { :aggregate, "count" },
        { :data, penguin; :color, :sex; }
    ]
]

Preparing the Data#

Once we have the data loaded, we need to transform the data in order to feed them into the machine learning models.

In general, we support a variety of machine learning models. The complete list of supported models can be found in the Machine Learning Library.

Most of these models require two relations:

• one containing the features to be used as inputs to train a model, and
• one containing the response (or target) variable (or, class, in our case) that we want to learn to predict.

To this end, we put the feature data in the features relation and the class data (that are currently read as strings) in the response_string relation. Note that in the current implementation of the Machine Learning Library, the relation from which you extract the features (i.e., penguin) needs to be an extensional database (EDB) relation. This is done by using insert earlier when you defined the penguin relation.

def features = penguin[col]
    for col in {
        :island; :culmen_length_mm; :culmen_depth_mm;
        :flipper_length_mm; :body_mass_g; :sex
    }

def response_string = penguin:species

We can easily get statistics about our features data using describe:

table[describe[features]]

and, of course, we can do the same for our response_string data:

table[(:response, describe_full[response_string])]

Here, we used describe_full because we have only one column in the response_string relation. Contrary to describe, describe_full provides statistics for the overall set of data rather than per feature.

Converting Class Names to Integers#

You will use an mlpack classifier, so you need to represent the response classes specifically as integers. You cannot use strings or floats to represent the classes.

To this end, you will first identify all the unique classes. You can get them using last:

def classes = last[response_string]

Next, we add numbers as an id for each class. We can do this using sort, which sorts the classes and we can use the ordering index as the class id:

def id_class = sort[classes]

id_class

In order to join with the relation response_string and get the ids, we need to swap the first and second columns. We can do this using transpose:

def class_id = transpose[id_class]

Note that transpose simply swaps the first and second columns, and is not to be confused with the typical matrix transposition. After we swap the columns, we can join with the response_string relation:

def response = response_string.class_id

Of course, we could have done all this in one step as follows:

def response = response_string.(transpose[sort[last[response_string]]])

Creating Training and Test Datasets#

In classification (as well as other machine learning approaches), we use a “training” dataset to learn a classification model and a “test” dataset to determine the accuracy of our model. In certain cases, we may also use a validation dataset for parameter tuning, but we will consider only training and test for the purposes of this how-to guide.

Because the penguin dataset is not already split into training and test sets, we will have to create these two datasets.

In the following, we split our data into training and test sets with a ratio of 80/20. We specify the splitting ratio and the seed in split_params. The splitting is done by mlpack_preprocess_split, which splits the keys in the two sets. Afterwards, we join them with the features and response so that we generate the corresponding training and test data sets:

def split_params = {("test_ratio", "0.2"); ("seed", "42")}

def data_key(:keys, k) = features(_, k, _)
def data_key_split = mlpack_preprocess_split[data_key, split_params]

def feature_train(f, k, v) = features(f, k, v) and data_key_split(1, k)
def feature_test(f, k, v) = features(f, k, v) and data_key_split(2, k)

def response_train(k, v) = response(k, v) and data_key_split(1, k)
def response_test(k, v) = response(k, v) and data_key_split(2, k)

The relation split_params specifies the exact splitting ratio between training and test sets. Note that the parameter name as well as the value need to be encoded as strings.

At this point, we can also add various checks to ensure that we have included all the instances from the original data set when we did the splitting in training and test. For example, we can check that the number of instances in training and test add up:

ic all_data() {
    count[feature_train] + count[feature_test] = count[features]
}

Or, we can more rigorously ensure that we have actually performed a split using all the available data:

ic all_features() {
    equal(features, union[feature_train, feature_test])
}

Building a Classifier#

In this guide, we will be using mlpack to create a decision tree classifier. The decision tree classifier of mlpack (as well as most of the other classifiers) can accept a set of optional parameters to tune the specific algorithm. The parameters for each classifier (aka hyper-parameters) are documented in the Machine Learning Library reference.

We set the hyper-parameters through a relation (we call it hyper_param here), as follows:

def hyper_param = {
    ("minimum_leaf_size", "10");
    ("minimum_gain_split", "1e-07")
}

Note that each classifier has its own parameters that you can find through the Machine Learning Library reference. Additionally, it is important to note that the parameters currently need to be passed as strings, similar to the example above. We can also pass no parameters to the classifier. In our example, we specified that we want the minimum number of instances in a leaf to be 10 and we set the minimum gain for node splitting to be 1e-07.

At this point, we are ready to build our classifier. We will use mlpack_decision_tree and specify the features for learning (i.e., the feature_train relation), the classes to learn to predict (i.e., the response_train relation), and the parameters:

def classifier = mlpack_decision_tree[
    feature_train,
    response_train,
    hyper_param
]

Now we have a trained classifier with the relation classifier, which represents the model we have learned.

Performing Predictions#

Our trained model classifier is now ready to make predictions. To make predictions, we have to use mlpack_decision_tree_predict, where we need to provide:

1. the trained ML model,
2. a relation with features similar to the one that was used for training, and
3. a number that indicates the number of keys used in the feature relation.

The information about the number of keys is necessary because it defines the arity of the relation with the features used to perform the predictions. In our case, we have only one key: the csv file position, which we carried over from the data loading step.

We can predict the penguin species using the training dataset:

def prediction_train = mlpack_decision_tree_predict[
    classifier,
    feature_train,
    1
]

We can, of course, also predict the penguin species of the unseen test dataset:

def prediction_test = mlpack_decision_tree_predict[
    classifier,
    feature_test,
    1
]

Let’s look at some predictions for the test dataset:

top[5, prediction_test]

Evaluating Our Model#

We can evaluate machine learning models using a variety of metrics. One popular way is the accuracy, which is defined as the fraction of the number of correct predictions over the total number of predictions.

We can compute the accuracy of the classifier model on the training dataset as follows:

def train_accuracy =
    count[pos : prediction_train[pos] = response_train[pos]] /
    count[response_train]

train_accuracy

Of course, what we really care about is the performance of our model on the test dataset:

def test_accuracy =
    count[pos : prediction_test[pos] = response_test[pos]] /
    count[response_test]

test_accuracy

We can also compute precision and recall (aka sensitivity) metrics for each class

def score_precision[c] =
    count[pos : prediction_test(pos, c) and response_test(pos, c)] /
    count[pos : prediction_test(pos, c)]

def score_recall[c] =
    count[pos : prediction_test(pos, c) and response_test(pos, c)] /
    count[pos : response_test(pos, c)]

and query them.

score_precision

score_recall

With precision and recall metrics at hand, we can also compute the F1 score for each class:

def score_f1[c] =
    2 * score_precision[c] * score_recall[c] /
    (score_precision[c] + score_recall[c])

We can then query them.

score_f1

Finally, we can compute the full confusion matrix (where actual is the actual class, or response, and predicted is the predicted class):

def confusion_matrix[predicted, actual] = count[
    x : response_test(x, actual) and prediction_test(x, predicted)
]

When we query for it, we get:

confusion_matrix

Note that count does not return 0 for an empty relation, which means that if no data record of class actual was predicted to be of class predicted then this pair does not appear in confusion_matrix. This reflects the fundamental principle that, in Rel, missing data (or NULL in SQL) is not explicitly stored or represented.

To assign a zero count to these missing values, we simply need to explicitly define that for any missing predicted-actual class pair, (predicted, actual), we want to assign a count of 0. This is done below with the left_override (<++) operator:

table[
    confusion_matrix[class_column.class_id, class_row.class_id] <++0
    for class_column in classes,
        class_row in classes
]

Here we also convert the integer class IDs back to their original class names and state that we want the relation to be displayed as a wide table.

Training Multiple Classifiers#

With Rel we can easily train and test multiple classifiers. Let’s consider an example.

We will train a set of classifiers on the same train and test datasets as before, but we will use a different set of hyper-parameters for each classifier. We will use a relation called hyper_param within a module called fine_tune to keep all the different hyper-parameter configurations:

module fine_tune
    def hyper_param = {
        ("Classifier 1", {("minimum_leaf_size", "10"); ("minimum_gain_split", "1e-07")});
        ("Classifier 2", {("minimum_leaf_size", "20"); ("maximum_depth", "3")});
        ("Classifier 3", {("minimum_leaf_size", "5"); ("maximum_depth", "0")});
    }
end

In hyper_param relation, we use an integer key (i.e, 1, 2, 3, ...) to identify each hyper-parameter configuration. This key will be useful to identify the classifiers from each configuration as well. We can now train multiple classifiers easily as follows:

module fine_tune
    def classifier[i] = mlpack_decision_tree[
        feature_train,
        response_train,
        hyper_param[i]
    ]
end

Note that the call to mlpack_decision_tree is the same as before, except we are iterating all the hyper-parameter configurations of the hyper_param.

We can now create predictions for each of the trained classifiers on the test set:

module fine_tune
    def prediction_test[i] = mlpack_decision_tree_predict[
        fine_tune:classifier[i],
        feature_test,
        1
    ]
end

And, as the next step, we can compute the precision for each classifier:

module fine_tune
    def score_precision(i, cl, score) =
        c = count[ pos :
            fine_tune:prediction_test(i, pos, id) and
            response_test(pos, id)
        ]
        and n = count[pos : fine_tune:prediction_test(i, pos, id)]
        and score = c/n
        and id = class_id[cl]
        from c, n, id
end

def output = table[fine_tune:score_precision]

Finally, we can plot the performance of each classifier over some specific metric. For example, we can show the precision of each classifier for each of the three classes as follows:

def precision_plot_data[:[], i] = {
    (:classifier_id, cid);
    (:class, cl);
    (:precision, pr)
}
from cid, cl, pr where sort[fine_tune:score_precision](i, cid, cl, pr)

def chart:data:values = precision_plot_data
def chart:mark = "bar"
def chart:width = 300
def chart = vegalite_utils:x[{
    (:field, "class");
}]

def chart = vegalite_utils:y[{
    (:field, "precision");
    (:type, "quantitative");
    (:axis, :format, ".3f");
}]

def chart:encoding:xOffset = { (:field, "classifier_id"); (:type, "nominal");}
def chart:encoding:color:field = "classifier_id"

def output = vegalite:plot[chart]

Based on the analysis of performance of multiple classifiers, Rel allows us to easily determine which classifier is expected to perform the best. As an example, let’s pick the classifier with the maximum precision on the test set over all classes:

def score_precision_overall[i] =
    count[pos : fine_tune:prediction_test[i, pos] = response_test[pos]] /
    count[fine_tune:prediction_test[i]]

def max_precision_classifier_id = argmax[score_precision_overall]
def max_precision = score_precision_overall[max_precision_classifier_id]

def output:classifier = max_precision_classifier_id
def output:precision = max_precision

Summary#

We demonstrated the use of a decision tree classifier on our penguin dataset. More specifically, we used mlpack_decision_tree, i.e., a decision tree classifier from mlpack. We support other classifiers as well. For example:

• mlpack_logistic_regression
• mlpack_linear_svm
• mlpack_random_forest

In addition to mlpack, we also support other machine learning libraries such as glm or xgboost and we have more coming.

It is important to note here that all of our machine learning models are specifically designed to have the same API. In this way, we can easily swap machine learning models (of similar type, i.e., classification models). In our example in this guide we can simply switch mlpack_decision_tree with mlpack_random_forest, change the hyper_params to the right parameters for mlpack_random_forest (or just leave it empty to use the defaults), and we now have a random forest classifier.

In addition to the machine learning models, the Machine Learning Library also has useful functionality for other tasks. For example, we can perform k-nearest-neighbor search on a relation through mlpack_knn or perform dimensionality reduction through kernel principal component analysis (KPCA) in a given dataset through mlpack_kernel_pca.

For a complete list of machine learning models and related functionality, see the Machine Learning Library.