Ultima actualización: 24/May/2025

La validación de modelos es un proceso crucial en el desarrollo de modelos de machine learning. Asegura que el modelo no solo se ajusta bien a los datos de entrenamiento, sino que también generaliza adecuadamente a datos no vistos.

Este proceso permite identificar problemas como el sobreajuste (overfitting), donde el modelo aprende demasiado bien los datos y sus errores en el conjunto de entrenamiento y falla en generalizar a nuevos datos.

📚 Importar librerias

# base libraries for data science
import warnings
import pandas as pd

from deepchecks.tabular import Dataset
from deepchecks.tabular.suites import model_evaluation


from joblib import dump
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.impute import SimpleImputer
from sklearn.metrics import f1_score, precision_score, recall_score, roc_auc_score
from sklearn.model_selection import RandomizedSearchCV, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", category=RuntimeWarning)

La evaluación de modelos (model evaluation), es el proceso de analizar y verificar el rendimiento de un modelo de machine learning. Este proceso es necesario en los proyectos de machine learning en cualquier momento en que se desee evaluar el modelo. Su importancia radica en permitir un análisis exhaustivo del rendimiento del modelo antes de su despliegue, ayudar en la selección y optimización de modelos al permitir comparar su desempeño, y verificar cómo se comporta el modelo con nuevos conjuntos de datos, incluyendo la comparación con datos anteriores. A través de la evaluación del modelo, se pueden identificar problemas cruciales como el sobreajuste (overfitting) al comparar el rendimiento en conjuntos de entrenamiento y prueba, detectar errores sistemáticos, identificar segmentos de datos donde el modelo rinde mal, o verificar si el modelo utiliza bien las características. Esto es fundamental para garantizar la fiabilidad y el rendimiento del modelo antes de ponerlo en producción y para mantener su calidad a lo largo del tiempo.

# Leer datos desde un URL
url_data = "https://www.openml.org/data/get_csv/16826755/phpMYEkMl"
dataset = pd.read_csv(url_data, low_memory=False, na_values="?")

Definicion de tipos de datos

# Features numericas
cols_numeric_float = ["age", "fare"]
cols_numeric_int = ["sibsp", "parch"]
cols_numeric = cols_numeric_float + cols_numeric_int

# Features categoricas
cols_categoric = ["sex", "embarked"]
cols_categoric_ord = ["pclass"]
cols_categorical = cols_categoric + cols_categoric_ord

Categoricas

dataset[cols_categoric] = dataset[cols_categoric].astype("category")
dataset["pclass"] = pd.Categorical(dataset["pclass"], categories=[3, 2, 1], ordered=True)

Numericas

dataset[cols_numeric_float] = dataset[cols_numeric_float].astype("float")
dataset[cols_numeric_int] = dataset[cols_numeric_int].astype("int8")

Variable Target

target = "survived"

dataset[target] = dataset[target].astype("int8")

Si existen duplicados en el dataset, es importante eliminarlos para evitar cualquier sesgo en el conjunto de datos o fuga de datos (data leak) cuando se entrena un modelo de aprendizaje automático.

dataset = dataset.drop_duplicates()

Train / Test split

# split data into features and target

X_features = dataset.drop(target, axis="columns")
Y_target = dataset[target]

# 80% train, 20% test
x_train, x_test, y_train, y_test = train_test_split(
    X_features, Y_target, stratify=Y_target, test_size=0.2, random_state=42
)

👨‍🏭 Feature Engineering

numeric_pipe = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="median")),
    ]
)

categorical_pipe = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("onehot", OneHotEncoder()),
    ]
)

categorical_ord_pipe = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="most_frequent")),
        ("onehot", OrdinalEncoder()),
    ]
)

preprocessor = ColumnTransformer(
    transformers=[
        ("numeric", numeric_pipe, cols_numeric),
        ("categoric", categorical_pipe, cols_categoric),
        ("categoric ordinal", categorical_ord_pipe, cols_categoric_ord),
    ]
)

Create pipeline

data_model_pipeline = Pipeline(
    steps=[("preprocessor", preprocessor), ("model", RandomForestClassifier())]
)

Hyperparameter tunning

Select the best hyperparameters for the models selected in the previous step.

Random Forest

score = "recall"

hyperparameters = {
    "model__max_depth": [4, 5, 7, 9, 10],
    "model__max_features": [2, 3, 4, 5, 6, 7, 8, 9],
    "model__criterion": ["gini", "entropy"],
}


grid_search = RandomizedSearchCV(
    data_model_pipeline,
    hyperparameters,
    cv=5,
    scoring=score,
    n_jobs=8,
)
grid_search.fit(x_train, y_train);

grid_search.best_params_

{'model__max_features': 7,
 'model__max_depth': 4,
 'model__criterion': 'entropy'}

best_data_model_pipeline = grid_search.best_estimator_

Evaluation

y_pred = best_data_model_pipeline.predict(x_test)
recall = recall_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)
auc = roc_auc_score(y_test, y_pred)
print(f"recall: {recall}")
print(f"precision: {precision}")
print(f"f1: {f1}")
print(f"auc: {auc}")

recall: 0.73
precision: 0.8390804597701149
f1: 0.7807486631016043
auc: 0.8217901234567903

Model Validation

train_ds = Dataset(
    pd.concat([x_train, y_train], axis="columns"),
    label=target,
    cat_features=cols_categorical,
    set_index_from_dataframe_index=True,
)
test_ds = Dataset(
    pd.concat([x_test, y_test], axis="columns"),
    label=target,
    cat_features=cols_categorical,
    set_index_from_dataframe_index=True,
)

evaluation_suite = model_evaluation()
suite_result = evaluation_suite.run(train_ds, test_ds, best_data_model_pipeline)
# Note: the result can be saved as html using suite_result.save_as_html()
# or exported to json using suite_result.to_json()
suite_result.show_not_interactive()

deepchecks - WARNING - Cannot use model's built-in feature importance on a Scikit-learn Pipeline, using permutation feature importance calculation instead
deepchecks - INFO - Calculating permutation feature importance. Expected to finish in 9 seconds

Model Evaluation Suite

The suite is composed of various checks such as: Confusion Matrix Report, Boosting Overfit, Calibration Score, etc...
Each check may contain conditions (which will result in pass ✓ / fail ✖ / warning ! / error ⁈) as well as other outputs such as plots or tables.
Suites, checks and conditions can all be modified. Read more about custom suites.

Conditions Summary

Status	Check	Condition	More Info
!	Weak Segments Performance - Train Dataset	The relative performance of weakest segment is greater than 80% of average model performance.	Found a segment with accuracy score of 0.605 in comparison to an average score of 0.815 in sampled data.
!	Weak Segments Performance - Test Dataset	The relative performance of weakest segment is greater than 80% of average model performance.	Found a segment with accuracy score of 0.667 in comparison to an average score of 0.844 in sampled data.
✓	Train Test Performance	Train-Test scores relative degradation is less than 0.1	Found max degradation of -2.17% for metric Precision and class 0.
✓	ROC Report - Train Dataset	AUC score for all the classes is greater than 0.7	All classes passed, minimum AUC found is 0.88 for class 1
✓	ROC Report - Test Dataset	AUC score for all the classes is greater than 0.7	All classes passed, minimum AUC found is 0.89 for class 1
✓	Prediction Drift	Prediction drift score < 0.15	Found model prediction Kolmogorov-Smirnov drift score of 0.08
✓	Simple Model Comparison	Model performance gain over simple model is greater than 10%	All classes passed, average gain for metrics: {'F1': '63.25%'}
✓	Unused Features - Train Dataset	Number of high variance unused features is less or equal to 5	Found 4 high variance unused features
✓	Unused Features - Test Dataset	Number of high variance unused features is less or equal to 5	Found 4 high variance unused features
✓	Model Inference Time - Train Dataset	Average model inference time for one sample is less than 0.001	Found average inference time (seconds): 2.176e-05
✓	Model Inference Time - Test Dataset	Average model inference time for one sample is less than 0.001	Found average inference time (seconds): 6.873e-05

Check With Conditions Output

Weak Segments Performance - Train Dataset

Search for segments with low performance scores. Read More...

Conditions Summary

Status	Condition	More Info
!	The relative performance of weakest segment is greater than 80% of average model performance.	Found a segment with accuracy score of 0.605 in comparison to an average score of 0.815 in sampled data.

Additional Outputs

Showcasing intersections of features with weakest detected segments.
The full list of weak segments can be observed in the check result value.