Scribd
Upload
48 views2 pages

Learning Rate (Or Eta)

The document discusses key hyperparameters in XGBoost that control aspects like learning rate, number of trees, tree depth, and regularization. It defines each hyperparameter, explains where and when they are used, and provides typical recommended value ranges to prevent overfitting while improving model performance.

Uploaded by

anuragpanda222
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
48 views2 pages

Learning Rate (Or Eta)

The document discusses key hyperparameters in XGBoost that control aspects like learning rate, number of trees, tree depth, and regularization. It defines each hyperparameter, explains where and when they are used, and provides typical recommended value ranges to prevent overfitting while improving model performance.

Uploaded by

anuragpanda222
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

Learning Rate (or eta):

Definition: Controls the step size shrinkage used in updating the weights of the
model during each boosting iteration.
Where to use: Central to controlling the step size during boosting and
preventing overfitting.
When to use: Lower values make the boosting process more conservative and
require more boosting rounds to converge, while higher values may lead to
overfitting.
XGBoost Hyperparameter: learning_rate
Recommended values: Typically in the range [0.01, 0.3].
o
o Number of Estimators (n_estimators):
Definition: Number of boosting rounds or trees to build.
Where to use: Dictates the number of boosting rounds and the overall
complexity of the model.
When to use: Higher values can improve model performance, but increasing the
number of estimators also increases computation time.
XGBoost Hyperparameter: n_estimators
Recommended values: Depends on the size of the dataset and computational
resources, but typically in the range [100, 1000].

• Maximum Depth (max_depth):


o Definition: Maximum depth of a tree in the ensemble.
o Where to use: Controls the depth of individual trees and the complexity of the model.
o When to use: Higher values allow for more complex trees, but too high may lead to
overfitting.
o XGBoost Hyperparameter: max_depth
o Recommended values: Typically in the range [3, 10].
• Minimum Child Weight (min_child_weight):
o Definition: Minimum sum of instance weight required in a child node. It helps prevent
overfitting by controlling the minimum size of child nodes.
o Where to use: Ensures that each leaf node has a minimum number of instances, thus
reducing the complexity of the model.
o When to use: Higher values make the algorithm more conservative and reduce the risk
of overfitting.
o XGBoost Hyperparameter: min_child_weight
o Recommended values: Typically in the range [1, 10].
• Subsample:
o Definition: Fraction of observations to be randomly sampled for each tree. It introduces
randomness and reduces overfitting.
o Where to use: Controls the randomness of the data sampling process for each tree.
o When to use: Lower values make the model more robust to noise but may lead to
underfitting.
o XGBoost Hyperparameter: subsample
o Recommended values: Typically in the range [0.5, 1.0].
• Colsample bytree:
o Definition: Fraction of features to be randomly sampled for each tree. It introduces
randomness and reduces overfitting.
o Where to use: Controls the randomness of feature selection for each tree.
o When to use: Lower values reduce overfitting by introducing more randomness in
feature selection.
o XGBoost Hyperparameter: colsample_bytree
o Recommended values: Typically in the range [0.5, 1.0].
• Gamma:
o Definition: Minimum loss reduction required to make a further partition on a leaf node.
It acts as regularization by controlling the complexity of trees.
o Where to use: Helps prevent overfitting by penalizing overly complex trees.
o When to use: Higher values make the algorithm more conservative.
o XGBoost Hyperparameter: gamma
o Recommended values: Typically in the range [0, 0.2].
• Regularization Parameters (reg_alpha and reg_lambda):
o Definition: L1 and L2 regularization terms applied to the weights. They help prevent
overfitting by penalizing large parameter values.
o Where to use: Controls the amount of regularization applied to the model.
o When to use: Increase values to increase regularization and reduce overfitting.
o XGBoost Hyperparameters: reg_alpha, reg_lambda
o Recommended values: Typically in the range [0, 0.5].

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy