🔒 Regression - predictions not being added to columns

[thin orchid]

Any idea as to why some are correctly being filled with the predictive values and the rest are left at 0.0?

# Define features and target
columns_to_drop = ['tempmax', 'tempmin']
features = sub_beijing.drop(columns=[col for col in columns_to_drop if col in sub_beijing.columns])
target_max = sub_beijing['tempmax']
target_min = sub_beijing['tempmin']# Split the data into training and testing sets
X_train, X_test, y_train_max, y_test_max = train_test_split(features, target_max, test_size=0.2, random_state=42)
_, _, y_train_min, y_test_min = train_test_split(features, target_min, test_size=0.2, random_state=42)

# Train the model for tempmax
ridge_max = Ridge()
ridge_max.fit(X_train, y_train_max)

# Train the model for tempmin
ridge_min = Ridge()
ridge_min.fit(X_train, y_train_min)

# Predict tempmax and tempmin for the test set
y_pred_max = ridge_max.predict(X_test)
y_pred_min = ridge_min.predict(X_test)

# Ensure columns exist and set correct dtype to avoid FutureWarning
if 'predicted_tempmax' not in sub_beijing.columns:
    sub_beijing['predicted_tempmax'] = 0.0
if 'predicted_tempmin' not in sub_beijing.columns:
    sub_beijing['predicted_tempmin'] = 0.0

# Add predictions to the original dataframe using .loc
sub_beijing.loc[X_test.index, 'predicted_tempmax'] = y_pred_max
sub_beijing.loc[X_test.index, 'predicted_tempmin'] = y_pred_min

# Optional: Calculate and print the mean squared error for evaluation
mse_max = mean_squared_error(y_test_max, y_pred_max)
mse_min = mean_squared_error(y_test_min, y_pred_min)

print(f'Mean Squared Error for tempmax: {mse_max}')
print(f'Mean Squared Error for tempmin: {mse_min}')

# Display the dataframe with predictions
print(sub_beijing[['tempmax', 'predicted_tempmax', 'tempmin', 'predicted_tempmin']].head(60))

[fallow junco]

@thin orchid

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[cyan iron]

When you're modifying the original df with .loc

Aren't you indexing the same thing twice

Oh nvm

Different columns

weird, the code seems fine. Are you sure your models are not just outputting 0 for many inputs?

[thin orchid]

The code above this is

from sklearn.model_selection import train_test_split
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error

# Ensure sub_beijing is not a view
sub_beijing = sub_beijing.copy()

# Ensure columns exist and set correct dtype to avoid FutureWarning
if 'predicted_tempmax' not in sub_beijing.columns:
    sub_beijing['predicted_tempmax'] = 0.0
if 'predicted_tempmin' not in sub_beijing.columns:
    sub_beijing['predicted_tempmin'] = 0.0

So it's just leaving them Null.
I asked ChatGPT: "The issue arises because we are using two separate train-test splits for y_train_max and y_train_min, causing their test indices to be different. This discrepancy leads to adding predictions to mismatched indices, leaving some rows empty."

So I'm gonna try predicting the tempmax in one model and run another one for tempmin. Hopefully that fixes it.
I appreciate you taking a look at it. I'll post the update when done.

[fallow junco]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.