Custom Objective Tutorial

Perpetual allows you to define your own objective functions for optimization. This is useful when the standard loss functions (like SquaredLoss or LogLoss) do not fit your specific use case.

In this tutorial, we will demonstrate how to define a custom objective and use it with PerpetualBooster.

1. Imports and Data Preparation

We’ll use the California Housing dataset for this regression example.

import numpy as np
import pandas as pd
from perpetual import PerpetualBooster
from sklearn.datasets import fetch_california_housing
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

# Load data
data = fetch_california_housing()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

2. Defining a Custom Objective

A custom objective in Perpetual is a tuple of three functions:

1. loss(y, pred, weight, group): Per-sample loss.

2. gradient(y, pred, weight, group): Per-sample gradient and hessian. Return (gradient, hessian). If the hessian is constant (e.g. always 1.0 for squared loss), return None instead to enable an optimized code path.

3. initial_value(y, weight, group): Initial prediction before any trees are added (typically the mean or median of y).

Tip: In the Rust API, only loss and gradient are required — initial_value defaults to the weighted mean and default_metric defaults to RMSE. In the Python API all three functions are required in the tuple.

Let’s implement a simple Squared Error objective as an example.

def custom_loss(y, pred, _weight, _group):
    return (y - pred) ** 2


def custom_gradient(y, pred, _weight, _group):
    # Gradient of (y - pred)^2 with respect to pred is 2 * (pred - y)
    # Note: Perpetual handles the scaling, so (pred - y) is sufficient.
    grad = pred - y
    hess = None  # If hess is constant (e.g. 1.0) return None to improve performance
    return grad, hess


def custom_init(y, _weight, _group):
    return np.mean(y)

3. Training and Evaluation

Now we can pass our custom objective to the PerpetualBooster constructor.

# Initialize booster with custom objective
model = PerpetualBooster(objective=(custom_loss, custom_gradient, custom_init))

# Fit the model
model.fit(X_train, y_train)

# Make predictions
preds = model.predict(X_test)

# Evaluate
mse = mean_squared_error(y_test, preds)
print(f"Mean Squared Error with Custom Objective: {mse:.4f}")

4. Comparison with Standard Objective

Let’s compare this with the built-in SquaredLoss objective.

standard_model = PerpetualBooster(objective="SquaredLoss")
standard_model.fit(X_train, y_train)
standard_preds = standard_model.predict(X_test)

standard_mse = mean_squared_error(y_test, standard_preds)
print(f"Mean Squared Error with Standard Objective: {standard_mse:.4f}")

assert np.allclose(preds, standard_preds)
print("Results are identical!")