Leave-one out cross validation python from scratch

Brinnae Bent, PhD

Jun 30, 2020

7 min read

I have received a number of requests for how to implement leave-one-person-out cross validation with random forests. I refined my methods into a function and published it on the Digital Biomarker Discovery Pipeline [DBDP]. If you want to implement this cross validation in your own work with random forests, please visit the repository on the DBDP GitHub. The ‘how-to’ of using the function is well documented and can be easily implemented with only one line in a number of projects.

To evaluate the performance of a model on a dataset, we need to measure how well the predictions made by the model match the observed data.

One commonly used method for doing this is known as leave-one-out cross-validation [LOOCV], which uses the following approach:

1. Split a dataset into a training set and a testing set, using all but one observation as part of the training set.

2. Build a model using only data from the training set.

3. Use the model to predict the response value of the one observation left out of the model and calculate the mean squared error [MSE].

4. Repeat this process n times. Calculate the test MSE to be the average of all of the test MSE’s.

This tutorial provides a step-by-step example of how to perform LOOCV for a given model in Python.

Step 1: Load Necessary Libraries

First, we’ll load the necessary functions and libraries for this example:

from sklearn.model_selection import train_test_split
from sklearn.model_selection import LeaveOneOut
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LinearRegression
from numpy import mean
from numpy import absolute
from numpy import sqrt
import pandas as pd

Step 2: Create the Data

Next, we’ll create a pandas DataFrame that contains two predictor variables, x1 and x2, and a single response variable y.

df = pd.DataFrame[{'y': [6, 8, 12, 14, 14, 15, 17, 22, 24, 23],
                   'x1': [2, 5, 4, 3, 4, 6, 7, 5, 8, 9],
                   'x2': [14, 12, 12, 13, 7, 8, 7, 4, 6, 5]}]

Step 3: Perform Leave-One-Out Cross-Validation

Next, we’ll then fit a multiple linear regression model to the dataset and perform LOOCV to evaluate the model performance.

#define predictor and response variables
X = df[['x1', 'x2']]
y = df['y']

#define cross-validation method to use
cv = LeaveOneOut[]

#build multiple linear regression model
model = LinearRegression[]

#use LOOCV to evaluate model
scores = cross_val_score[model, X, y, scoring='neg_mean_absolute_error',
                         cv=cv, n_jobs=-1]

#view mean absolute error
mean[absolute[scores]]

3.1461548083469726

From the output we can see that the mean absolute error [MAE] was 3.146. That is, the average absolute error between the model prediction and the actual observed data is 3.146.

In general, the lower the MAE, the more closely a model is able to predict the actual observations.

Another commonly used metric to evaluate model performance is the root mean squared error [RMSE]. The following code shows how to calculate this metric using LOOCV:

#define predictor and response variables
X = df[['x1', 'x2']]
y = df['y']

#define cross-validation method to use
cv = LeaveOneOut[]

#build multiple linear regression model
model = LinearRegression[]

#use LOOCV to evaluate model
scores = cross_val_score[model, X, y, scoring='neg_mean_squared_error',
                         cv=cv, n_jobs=-1]

#view RMSE
sqrt[mean[absolute[scores]]]

3.619456476385567

From the output we can see that the root mean squared error [RMSE] was 3.619. The lower the RMSE, the more closely a model is able to predict the actual observations.

In practice we typically fit several different models and compare the RMSE or MAE of each model to decide which model produces the lowest test error rates and is therefore the best model to use.

Additional Resources

A Quick Intro to Leave-One-Out Cross-Validation [LOOCV]
A Complete Guide to Linear Regression in Python