A Data Leakage mistake often made while using GridSearchCV / RandomizedSearchCV

Use of sci-kit learn pipeline allow us to fit only the train split while cross-validation thus avoiding data leakage while hyper-parameter tuning.

We all know about the importance of creating a separate train and test set to avoid data leakage. We use the statistics of the train data only to transform our data i.e. in the language of sci-kit learn for any data transformation we ‘fit’ only on train data and not on the test/validation data.

e.g.: if we want to standardize a given feature we calculate the mean and standard deviation of the train data only and use them to standardize the test/validation data.

I have noticed that people after doing the above transformation directly pass the transformed dataset to the GridSearchCV or RandomizedSearchCV which internally performs similar train-test(validation) cross-validation splits of the transformed data and it has no mechanism to calculate the statistic of the train split only and leave the test/validation split out.

i.e. ideally we should ‘fit_transform’ each training split and ‘transform’ the validation split for each iteration of the Cross Validation run.

We should use sci-kit learn pipeline along with GridSearchCV or RandomizedSearchCV (or any other Cross Validation) to avoid data leakage.

Instead of passing the estimator(model) we pass a pipeline which has the details of data transformations required along with the estimator/model to use. So we can avoid data leakage for each cross-validation step.

Consider a example without Pipeline:

Here we are using the make_column_transformer class to transform any data. You can also use ColumnTransformer. I am not using any dataset here as i just want to just explain the utility of the pipeline. You can refer to the model training notebook in this github repository for usage with other modelling functions.

from sklearn.compose import make_column_transformer
from sklearn.preprocessing import KBinsDiscretizer
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import PowerTransformer

# using make_column_transformer here that would transform any input data as
# per the given details like the first feature (given by index 0) will be KBinsDiscretize etc.

transformation_required = make_column_transformer(
    (KBinsDiscretizer(n_bins=4,strategy='quantile') , [0] ),
    (OneHotEncoder(handle_unknown='ignore') , [1]),
    (ResponseEncoding(), (2) ),
    (PowerTransformer(method='yeo-johnson',standardize=True) , (3,4,5))

 Like i have noticed in most code, the X_train is first transformed and then used inside GridsearchCV as shown below: 

from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import GridSearchCV

# usually i have seen X_train being fully transformed before any Cross Validation
# so every CV split will have data leakage

X_train = transformation_required.fit_transform(X_train.values,y_train)

# example gridsearchcv code
 
model = SGDClassifier(loss='log_loss', random_state=92)
params = {'sgdclassifier__alpha':[0.0001,0.005,0.001,0.05,0.01,0.1,0.5,1]} #hyperparameters

clf = GridSearchCV( model, params, cv=5, return_train_score=True, refit=False,
                    scoring={'recall','precision','f1'}, verbose=4)

clf.fit(X_train,y_train)

Here we are using a simple example of using the SGDClassifier. To fit the GridSearchCV we have passed the Training data only(X_train,y_train) which is transformed beforehand and the GridSearchCV has on information passed on what transformations are required as it is going to perform its own multiple splits and arrange them as train-test splits, each requiring its separate transformation to avoid data leakage.

Let’s look at an example with Pipeline:

Here we are using the make_pipeline class. You can also use the Pipeline class.

from sklearn.pipeline import make_pipeline

# creating pipeline that will first transform any input data and then apply the estimator.
pipe_lr = make_pipeline(
                        transformation_required ,
                        SGDClassifier(loss='log_loss', random_state=92)
                       )

params = {'sgdclassifier__alpha':[0.0001,0.005,0.001,0.05,0.01,0.1,0.5,1]} #hyperparameters

# here passing the pipeline as estimator so any cross-validation will pass the
# data to the pipeline which will fit_transform the train split only and avoid
# data leakage

clf = GridSearchCV( pipe_lr, params, cv=5, return_train_score=True, refit=False,
                    scoring={'recall','precision','f1'}, verbose=4)

# below X_train is the raw data (without any transformation) as pipeline will
# take care of that for each CV split

clf.fit(X_train,y_train) 

 
You can refer to the model training notebook in this github repo for usage of pipeline and for other hyper-parameter tuning technique like BayesSearchCV.

Thank you for reading!!

Comments

Post a Comment

Popular posts from this blog

vLLM Parameter Tuning for Better Performance

 vLLM Parameters     We all know that vLLM is fast and easy to use library for LLM inference and serving. We shall go through some parameters tuning to get better performance out of vLLM.   The vLLM Engine parameter we shall discuss are: -- max-num-batched-tokens -- max-model-len -- gpu-memory-utilization -- enable-prefix-caching -- enable-chunked-prefill -- enforce-eager max-model-len: TL;DR  as per your max token usage(input+output) - By default the max model length is the  max context length of the model you are using eg. for llama 3 8B instruct model it would be 8192 - If you can determine the max context length for your use case and if it is less than the max context length of your model, it is better to set this parameter to that value. - Along with preventing out of memory error while loading or using the model it will also help to set other parameters like max-num-batched-tokens and gpu-memory-utilization. - The value includes both input and output...
Read more

LLM Web Scraping - Webpage to LLM Friendly Text - Fully Open Source

LLM Web Scraping Webpage to LLM Friendly Text LLM are good with extracting data from texts. So to scrape any webpage we provide the webpage text to the LLM in such a format that it becomes easy to extract data from them.   We use library like selenium, beautifulsoup to get page source html and get text from it. This may help to extract certain information but it can't extract image links, website links for the required product or information we are extracting eg: while scraping any e-commerce website if along with details like product-title, price etc you want the image and product main page link then preprocessing the html becomes important. Below i have shared an open source repository to get LLM friendly text from webpage which can extract any data including websites and image links.   APIs like Jina Reader API and Firecrawl API can be used to get clean text from any webpage. If you want a complete open-source option and ability to modify the code as per your need(some webs...
Read more

Understanding LSTM

Image
 Understanding LSTM By Guillaume Chevalier — File:The_LSTM_Cell.svg, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=109362147   The Cell state: As RNN finds it difficult to carry previous information for a long input to the final state, so in LSTM we keep a separate state called cell state where previously learned information is available for the model. The mechanism of how this cell state is maintained and the models ability to learn from new inputs is discussed below. Selectively Removing Information from the Cell State: Forget Gate Mechanism At time step ‘t’, we have a previous cell state vector(or matrix)(ct-1) which has encoded features from all inputs before it. We have a previous hidden state vector(ht-1) which encodes the influence of last input to the long term cell state. We have a new input vector(xt) which should make necessary changes to the encoding done so far. Both these vectors are transformed to the same vector space using two(W and U) lea...
Read more