{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Advanced Drift Detection in Perpetual\n", "\n", "This tutorial demonstrates how to use Perpetual's built-in drift detection and compares it against several state-of-the-art methods found in literature. We will explore how these metrics respond as we gradually increase the amount of drift in the California Housing dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import warnings\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "from perpetual import PerpetualBooster\n", "from scipy import stats\n", "from sklearn.datasets import fetch_california_housing\n", "from sklearn.ensemble import RandomForestClassifier\n", "from sklearn.metrics import mean_squared_error, roc_auc_score\n", "from sklearn.metrics.pairwise import rbf_kernel\n", "from sklearn.model_selection import train_test_split\n", "\n", "warnings.filterwarnings(\"ignore\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Defining Benchmark Metrics\n", "\n", "We implement common drift detection methods for comparison." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def calculate_energy_distance(X_ref, X_curr, sample_size=500):\n", " if len(X_ref) > sample_size:\n", " X_ref = X_ref.sample(sample_size, random_state=42)\n", " if len(X_curr) > sample_size:\n", " X_curr = X_curr.sample(sample_size, random_state=42)\n", "\n", " # Using kernel-based estimate\n", " K_xx = rbf_kernel(X_ref, X_ref)\n", " K_yy = rbf_kernel(X_curr, X_curr)\n", " K_xy = rbf_kernel(X_ref, X_curr)\n", "\n", " return np.mean(K_xx) + np.mean(K_yy) - 2 * np.mean(K_xy)\n", "\n", "\n", "def calculate_mahalanobis_distance(X_ref, X_curr):\n", " # Center and scale data\n", " mean_ref = X_ref.mean(axis=0)\n", " cov_ref = np.cov(X_ref.T)\n", " # Robust covariance estimate if needed, but for benchmark simple cov is fine\n", " inv_cov_ref = np.linalg.pinv(cov_ref)\n", "\n", " mean_curr = X_curr.mean(axis=0)\n", " diff = mean_curr - mean_ref\n", " return np.sqrt(diff.dot(inv_cov_ref).dot(diff))\n", "\n", "\n", "def get_adversarial_auc(X_ref, X_curr):\n", " if len(X_ref) > 1000:\n", " X_ref = X_ref.sample(1000, random_state=42)\n", " if len(X_curr) > 1000:\n", " X_curr = X_curr.sample(1000, random_state=42)\n", " X_adv = pd.concat([X_ref, X_curr])\n", " y_adv = np.array([0] * len(X_ref) + [1] * len(X_curr))\n", " clf = RandomForestClassifier(n_estimators=50, max_depth=5, random_state=42)\n", " clf.fit(X_adv, y_adv)\n", " probs = clf.predict_proba(X_adv)[:, 1]\n", " return roc_auc_score(y_adv, probs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Load Data and Train Model\n", "\n", "We use the California Housing dataset and train a Perpetual model with `save_node_stats=True`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = fetch_california_housing()\n", "\n", "X = pd.DataFrame(data.data, columns=data.feature_names)\n", "y = data.target\n", "\n", "X_train, X_test, y_train, y_test = train_test_split(\n", " X, y, test_size=0.2, random_state=42\n", ")\n", "\n", "model = PerpetualBooster(objective=\"SquaredLoss\", budget=1.0, save_node_stats=True)\n", "model.fit(X_train, y_train)\n", "\n", "y_pred_train = model.predict(X_train)\n", "train_residuals = y_train - y_pred_train\n", "\n", "print(f\"Baseline MSE: {mean_squared_error(y_test, model.predict(X_test)):.4f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Gradual Drift Simulation\n", "\n", "We gradually increase the drift in `MedInc` and record how different metrics respond." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "drift_levels = np.linspace(0, 1.2, 13)\n", "results = []\n", "\n", "for level in drift_levels:\n", " X_curr = X_test.copy()\n", " X_curr[\"MedInc\"] += level # Inject drift\n", "\n", " # Perpetual Unsupervised Metrics\n", " perp_data = model.calculate_drift(X_curr, drift_type=\"data\")\n", " perp_concept = model.calculate_drift(X_curr, drift_type=\"concept\")\n", "\n", " # Multivariate Benchmarks\n", " energy = calculate_energy_distance(X_test, X_curr)\n", " mahalanobis = calculate_mahalanobis_distance(X_test, X_curr)\n", " adv_auc = get_adversarial_auc(X_test, X_curr)\n", "\n", " # Performance Metrics\n", " y_curr_pred = model.predict(X_curr)\n", " mse = mean_squared_error(y_test, y_curr_pred)\n", " curr_residuals = y_test - y_curr_pred\n", " residual_ks = stats.ks_2samp(train_residuals, curr_residuals).statistic\n", "\n", " results.append(\n", " {\n", " \"Level\": level,\n", " \"Perp Data\": perp_data,\n", " \"Perp Concept\": perp_concept,\n", " \"Energy Distance\": energy,\n", " \"Mahalanobis\": mahalanobis,\n", " \"Adversarial AUC\": adv_auc,\n", " \"Residual KS\": residual_ks,\n", " \"MSE\": mse,\n", " }\n", " )\n", "\n", "df_res = pd.DataFrame(results)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_res.head(len(list(drift_levels)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Visualizing Drift Response" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, axes = plt.subplots(2, 1, figsize=(10, 14))\n", "fig.subplots_adjust(hspace=0.3, right=0.75)\n", "\n", "# --- Graph 1: Data Drift Metrics ---\n", "ax1 = axes[0]\n", "ax2 = ax1.twinx()\n", "ax3 = ax1.twinx()\n", "ax4 = ax1.twinx()\n", "\n", "ax3.spines[\"right\"].set_position((\"outward\", 60))\n", "ax4.spines[\"right\"].set_position((\"outward\", 120))\n", "\n", "(p1,) = ax1.plot(\n", " df_res[\"Level\"],\n", " df_res[\"Perp Data\"],\n", " label=\"Perpetual Data Drift\",\n", " marker=\"o\",\n", " color=\"C0\",\n", ")\n", "(p2,) = ax2.plot(\n", " df_res[\"Level\"],\n", " df_res[\"Energy Distance\"],\n", " label=\"Energy Distance\",\n", " marker=\"s\",\n", " color=\"C1\",\n", ")\n", "(p3,) = ax3.plot(\n", " df_res[\"Level\"],\n", " df_res[\"Mahalanobis\"],\n", " label=\"Mahalanobis Distance\",\n", " marker=\"^\",\n", " color=\"C2\",\n", ")\n", "(p4,) = ax4.plot(\n", " df_res[\"Level\"],\n", " df_res[\"Adversarial AUC\"],\n", " label=\"Adversarial AUC\",\n", " marker=\"d\",\n", " color=\"C3\",\n", ")\n", "\n", "ax1.set_xlabel(\"Drift Level (MedInc Shift)\", fontweight=\"bold\")\n", "ax1.set_ylabel(\"Perpetual Data Drift\", color=\"C0\")\n", "ax2.set_ylabel(\"Energy Distance\", color=\"C1\")\n", "ax3.set_ylabel(\"Mahalanobis Distance\", color=\"C2\")\n", "ax4.set_ylabel(\"Adversarial AUC\", color=\"C3\")\n", "\n", "ax1.tick_params(axis=\"y\", colors=\"C0\")\n", "ax2.tick_params(axis=\"y\", colors=\"C1\")\n", "ax3.tick_params(axis=\"y\", colors=\"C2\")\n", "ax4.tick_params(axis=\"y\", colors=\"C3\")\n", "\n", "lines1 = [p1, p2, p3, p4]\n", "ax1.legend(lines1, [line.get_label() for line in lines1], loc=\"upper left\")\n", "ax1.set_title(\"Data Drift Benchmark\", fontsize=14, fontweight=\"bold\")\n", "ax1.grid(True, alpha=0.3)\n", "\n", "# --- Graph 2: Concept Drift & Performance Metrics ---\n", "ax5 = axes[1]\n", "ax6 = ax5.twinx()\n", "ax7 = ax5.twinx()\n", "\n", "ax7.spines[\"right\"].set_position((\"outward\", 60))\n", "\n", "(p5,) = ax5.plot(\n", " df_res[\"Level\"],\n", " df_res[\"Perp Concept\"],\n", " label=\"Perpetual Concept Drift\",\n", " marker=\"o\",\n", " color=\"C0\",\n", ")\n", "(p6,) = ax6.plot(\n", " df_res[\"Level\"],\n", " df_res[\"Residual KS\"],\n", " label=\"Residual KS Stat\",\n", " marker=\"s\",\n", " color=\"C1\",\n", ")\n", "(p7,) = ax7.plot(df_res[\"Level\"], df_res[\"MSE\"], label=\"MSE\", marker=\"^\", color=\"C2\")\n", "\n", "ax5.set_xlabel(\"Drift Level (MedInc Shift)\", fontweight=\"bold\")\n", "ax5.set_ylabel(\"Perpetual Concept Drift\", color=\"C0\")\n", "ax6.set_ylabel(\"Residual KS Stat\", color=\"C1\")\n", "ax7.set_ylabel(\"MSE\", color=\"C2\")\n", "\n", "ax5.tick_params(axis=\"y\", colors=\"C0\")\n", "ax6.tick_params(axis=\"y\", colors=\"C1\")\n", "ax7.tick_params(axis=\"y\", colors=\"C2\")\n", "\n", "lines2 = [p5, p6, p7]\n", "ax5.legend(lines2, [line.get_label() for line in lines2], loc=\"upper left\")\n", "ax5.set_title(\"Concept Drift Benchmark\", fontsize=14, fontweight=\"bold\")\n", "ax5.grid(True, alpha=0.3)\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. Practical Threshold Guidance\n", "\n", "Based on empirical observations, here are the recommended threshold ranges for interpreting these drift scores.\n", "\n", "### Perpetual Unsupervised Scores (Chi-Squared Statistic)\n", "The drift score is an average Chi-squared statistic across model nodes.\n", "\n", "* **< 1.0**: **No Drift.** The current data matches the reference distribution well.\n", "* **1.0 - 2.0**: **Warning.** Slight distributional shift. Monitor model performance closely.\n", "* **> 2.5**: **Significant Drift.** High likelihood of data/concept shift. Retraining or investigative action (e.g., data quality check) is recommended.\n", "\n", "### Benchmark Ranges\n", "* **Energy Distance (MMD)**: Scale-dependent, but values significantly above the 0.0 baseline indicate drift. MMD is a kernel-based metric for distribution distance.\n", "* **Mahalanobis Distance**: Measures standard deviations from the mean; values > 3.0 (3 sigma) are typically considered significant outliers/drift.\n", "* **Adversarial AUC (ROC-AUC)**: Ranges from 0.5 (perfectly indistinguishable) to 1.0 (perfectly separable).\n", " * **0.50 - 0.60**: Low/No drift.\n", " * **0.60 - 0.75**: Moderate drift.\n", " * **> 0.80**: High drift.\n", "\n", "### Performance Diagnostics (Concept Drift)\n", "* **Residual KS Stat**: Measures how much the distribution of prediction errors has changed. A high KS value (> 0.2 in this case) confirms that concept drift is affecting model accuracy.\n", "* **MSE**: A dramatic increase in Mean Squared Error indicates the model is struggling to predict the target under the new shifted distribution.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Conclusion\n", "\n", "In this tutorial, we compared Perpetual's unsupervised drift metrics against state-of-the-art multivariate benchmarks:\n", "\n", "* **Perpetual Data & Concept Drift**: These metrics provide a model-based view of how data shifts affect both the marginal feature distributions and the predictive structure, showing a clear response to gradual drift.\n", "* **Energy Distance**: A robust kernel-based multivariate metric that captures distribution shifts effectively.\n", "* **Mahalanobis Distance**: Useful for detecting changes in the multivariate mean relative to the training feature covariance.\n", "* **Adversarial AUC**: A powerful classifier-based approach that identifies how distinguishable the new data is from the reference data.\n", "\n", "We can see that Perpetual's internal metrics correlate strongly with these established multivariate methods, offering a fast, performant, and unsupervised built-in alternative for drift monitoring during model inference." ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.7" } }, "nbformat": 4, "nbformat_minor": 2 }