Re-fitting a saved scikit-learn model without some features not used – "ValueError: A given column is not a column of the dataframe"


I’d need to re-fit a scikit-learn pipeline using a smaller dataset, without some features that are actually not used by the model.

(The actual situation is that I’m saving it through joblib and loading it in another file where I need to re-fit is since it contains some custom transformers I made, but adding all features would be a pain since it’s a different kind of model. However this is not important since the same error happens also if I re-fit the model before saving it in the same file where I first trained it).

This is my custom transformer:

class TransformAdoptionFeatures(BaseEstimator, TransformerMixin):
    def __init__(self):

    def fit(self, X, y=None):
        return self

    def transform(self, X):
        adoption_features = X.columns
        feats_munic = [feat for feat in adoption_features if '_munic' in feat]
        feats_adj_neigh = [feat for feat in adoption_features
                           if '_adj' in feat]
        feats_port = [feat for feat in adoption_features if '_port' in feat]

        feats_to_keep_all = feats_munic + feats_adj_neigh + feats_port
        feats_to_keep = [feat for feat in feats_to_keep_all
                         if 'tot_cumul' not in feat]
        return X[feats_to_keep]

And this is my pipeline:

full_pipeline = Pipeline([
    ('transformer', TransformAdoptionFeatures()),
    ('scaler', StandardScaler())

model = Pipeline([
    ("preparation", full_pipeline),
    ("regressor", ml_model)

Where ml_model is whichever scikit-learn machine learning model. Both the full_pipeline and the ml_model are already fitted when saving the model. (In the actual model there is a ColumnTransformer intermediate step that represent the actual full_pipeline, since I need to have different transformers for different columns, but I copied only the important one for brevity).

Issue: I reduced the number of features of the dataset I already used to fit everything, removing some features that are not considered in TransformAdoptionFeatures() (they do not get into the features to keep). Then, I tried to re-fit the model to the new dataset with reduced features and I got this error:

Traceback (most recent call last):

  File "C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\pandas\core\indexes\", line 2889, in get_loc
    return self._engine.get_loc(casted_key)

  File "pandas\_libs\index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc

  File "pandas\_libs\index.pyx", line 97, in pandas._libs.index.IndexEngine.get_loc

  File "pandas\_libs\hashtable_class_helper.pxi", line 1675, in pandas._libs.hashtable.PyObjectHashTable.get_item

  File "pandas\_libs\hashtable_class_helper.pxi", line 1683, in pandas._libs.hashtable.PyObjectHashTable.get_item

KeyError: 'tot_cumul_adoption_pr_y_munic'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\utils\", line 447, in _get_column_indices
    col_idx = all_columns.get_loc(col)

  File "C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\pandas\core\indexes\", line 2891, in get_loc
    raise KeyError(key) from err

KeyError: 'tot_cumul_adoption_pr_y_munic'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "C:\Users\giaco\sbp-abm\municipalities_abm\", line 15, in <module>
    modelSBP = model.SBPAdoption(initial_year=start_year)

  File "C:\Users\giaco\sbp-abm\municipalities_abm\municipalities_abm\", line 103, in __init__
    self._upload_ml_models(ml_clsf_folder, ml_regr_folder)

  File "C:\Users\giaco\sbp-abm\municipalities_abm\municipalities_abm\", line 183, in _upload_ml_models'adoption_in_year', axis=1),

  File "C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\", line 330, in fit
    Xt = self._fit(X, y, **fit_params_steps)

  File "C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\", line 292, in _fit
    X, fitted_transformer = fit_transform_one_cached(

  File "C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\joblib\", line 352, in __call__
    return self.func(*args, **kwargs)

  File "C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\", line 740, in _fit_transform_one
    res = transformer.fit_transform(X, y, **fit_params)

  File "C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\compose\", line 529, in fit_transform

  File "C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\compose\", line 327, in _validate_remainder
    cols.extend(_get_column_indices(X, columns))

  File "C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\utils\", line 454, in _get_column_indices
    raise ValueError(

ValueError: A given column is not a column of the dataframe

I do not understand what does this error is due to, I thought scikit-learn was not storing the name of the columns that I pass.


I found my error and it was actually in the use of the ColumnsTransformer, that is also the only place where the column names enter.

My error was really simple, I just did not update the list of the columns to apply each transformation to removing the names of the features excluded.

Answered By – giacrava

Answer Checked By – Mildred Charles (AngularFixing Admin)

Leave a Reply

Your email address will not be published.