#Problems with my first ML project

11 messages · Page 1 of 1 (latest)

jovial rose
#

I'm trying to build a churn prediction model, but I'm having some difficulties with the results. I've performed EDA to select my features and I'm using an XGBoostClassifier for the model.

However, I'm getting poor metrics. My model has very low recall and is failing to predict churn. It's generating too many false positives, and I don't know what else to do.

For some context, my dataset has a churn rate of approximately 16.5%. I have already tried using SMOTE and setting the scale_pos_weight parameter, but neither of these methods improved the results.

Any suggestions would be greatly appreciated.

carmine vapor
#

Why your XGBoost churn model is underperforming, especially in terms of recall and false positives?

You’re working with:
Class imbalance (~16.5% churners)
XGBoostClassifier
Tried SMOTE and scale_pos_weight

Check if your metrics are misleading due to imbalance. Use a confusion matrix and classification report (sklearn). Prioritize recall and precision, not accuracy. If you're not already, use Stratified K-Fold or Stratified train/test split to maintain class distribution.

You said SMOTE and scale_pos_weight didn’t help — that’s not uncommon. Here's why and what to try: Better strategy than SMOTE: Try under-sampling the majority class instead of oversampling. SMOTE often generates noisy synthetic churn cases. Or try SMOTE + Tomek Links or SMOTE + Edited Nearest Neighbors (ENN). Better use of scale_pos_weight:

Did you try setting it via: Python

scale_pos_weight = (number of negative samples) / (number of positive samples)

In your case, maybe something like scale_pos_weight=5? Try tuning this — it’s sensitive.

jovial rose
carmine vapor
#

Hyperparameter Tuning (very important)

A poorly tuned XGBoost model often:

Defaults to being overly conservative (low recall)

Struggles with rare class detection

Start with:

param_grid = {
    'max_depth': [3, 5, 7],
    'learning_rate': [0.01, 0.05, 0.1],
    'n_estimators': [100, 300, 500],
    'gamma': [0, 1, 5],
    'subsample': [0.6, 0.8, 1.0],
    'colsample_bytree': [0.6, 0.8, 1.0],
    'scale_pos_weight': [1, 3, 5, 7]
}

Use RandomizedSearchCV or Optuna for tuning (grid search is slow).

#

EDA might have selected features statistically, but churn modeling benefits from:

Behavioral aggregates (e.g., total logins last 30 days, usage patterns)

Recent vs. historic behavior deltas (e.g., drop in activity, payments)

Derived features like interaction rates, decay rates, etc.

Also: test removing unimportant features — XGBoost is tree-based, but junk features still introduce noise.

Use:

from xgboost import plot_importance
plot_importance(model, max_num_features=15)
jovial rose
jovial rose
carmine vapor
#

Nice sounds like a good plan I hope so too.

#

Try something like above if that does not help look to the feature engineering likely.

jovial rose