Gradient Boosting

The Gradient Boosting calculator creates a predictive model from one dependent variable and two or more predictor variables. If the dependent variable is categorical, numiqo runs Gradient Boosting classification. If the dependent variable is metric, numiqo runs Gradient Boosting regression.

To calculate Gradient Boosting online, open Predictive Analytics, select Gradient Boosting, choose the dependent variable, and select the independent variables that should be used as predictors.

What is Gradient Boosting?

Gradient Boosting is an ensemble method that builds many small decision trees in sequence. Each new tree focuses on the current prediction errors of the existing model. The model then adds the new tree with a learning rate, so the final prediction is built gradually instead of from one large tree.

This is the general machine learning method behind TreeNet-style boosted tree models used in predictive analytics software. In this calculator, the feature uses the generic name Gradient Boosting rather than a branded product name.

How the Gradient Boosting calculator works

The calculator trains shallow decision trees on pseudo-residuals. For regression, the pseudo-residual is the difference between the observed value and the current predicted value. For classification, the calculator uses class probabilities and fits trees that improve the score for each category.

1. Initial prediction: Regression starts with the mean of the dependent variable. Classification starts with class proportions.
2. Sequential trees: Each tree is fitted to the current error of the model.
3. Learning rate: The learning rate controls how strongly each new tree changes the existing model.
4. Small trees: The maximum tree depth controls how complex each individual tree can be.
5. Validation split: The calculator reports performance on validation rows that were held out from training when the selected data set is large enough.

Gradient Boosting results

The Gradient Boosting output includes a method table, validation performance, and relative variable importance. Classification models also include a confusion matrix. Regression models report RMSE, MAE, and R-squared.

Validation performance

Validation performance estimates how well the boosted-tree model predicts rows that were not used to fit the trees. Classification models report accuracy. Regression models report RMSE, MAE, and R-squared.

Confusion matrix

The confusion matrix compares the observed category with the predicted category. The rows show the actual categories, and the columns show the predicted categories. The correct percentage for each row shows how often observations from that actual category were classified correctly.

Variable importance

Variable importance summarizes how much each predictor improves the boosted-tree splits. A higher relative importance means that the variable contributed more to reducing the model error across the ensemble. Importance is useful for ranking predictors, but it should not be interpreted as proof of causality.

Gradient Boosting vs. Random Forest

A Random Forest builds many trees independently and combines their predictions. Gradient Boosting builds trees sequentially, where each tree improves the current model. Random Forest is often easier to tune, while Gradient Boosting can achieve strong predictive performance when the number of trees, learning rate, and tree depth are chosen carefully.

When to use Gradient Boosting

Use Gradient Boosting classification when the dependent variable is categorical and the goal is to predict group membership. Use Gradient Boosting regression when the dependent variable is metric and the goal is to predict a numeric value. Common applications include churn prediction, quality risk scoring, demand prediction, process monitoring, and customer segmentation.

Limitations

Rows with missing values in the selected variables are excluded from the calculation. The current implementation uses shallow CART-style trees, squared-error splits on pseudo-residuals, and an internal training/validation split. It does not yet include prediction for new rows, partial dependence plots, or automated hyperparameter search.