🔒 Help with filtering dataset, random forest prediction

[lean olive]

Hello im taking my first steps in DS, and here I have an exercise I cannot continue.

The provided starter Colab removes all non-numeric columns from both datasets. it is intended by the exercise itself. Nothing should be done with other columns (yet). The rule is that I can only make filters. Nothing else can be changed. The ultimate goal is to keep only sale prices for CABA. I also cannot touch the model or its parameters. The objetive is to predict prices with all those filters.

Things I do not understand yet:
Training and learning dataset should have the same filters?
I'm having trouble imputing missing values, and I feel that's the key for a good prediction.

Here is the most important db info:

Data columns (total 17 columns):
 #   Column            Non-Null Count    Dtype  
---  ------            --------------    -----  
 2   lat               422620 non-null   float64
 3   lon               422604 non-null   float64
 5   publisher_id      663922 non-null   object 
 7   location_0        982787 non-null   object 
 8   location_1        982769 non-null   object 
 12  operation_type    982487 non-null   object 
 13  property_type     982784 non-null   object 
 14  source            1283423 non-null  object 
 15  price             984477 non-null   float64 ```

Here are the important columns info: 

```df_ent["currency_type"].unique()
array(['dolares', 'Otro', 'pesos'], dtype=object)```

```df_ent["location_1"].unique()
array(['Ciudad Autónoma de Buenos Aires', 'Capital Federal', 'CABA'], dtype=object)```

Here is the model: 
```RandomForestRegressor(max_depth=5, n_estimators=50, n_jobs=-1, random_state=42)```

Filters for training and learning are in the images.
The `df_ap = df_ap.select_dtypes('number')` it is intended by the exercise itself.

You can also find in the images an histogram with the not so good final predictions

Thank you in advance!

[olive temple]

@lean olive

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[harsh monolith]

can you explain what you think "training and learning dataset" means?

also, can you tell me which features are numeric, and why it would be an issue to use only those features?

[lean olive]

From my understanding,

traininig means the part of the dataset that im going to clean first and the use it for training the model.

The learning dataset is what the algorithm is going to use in order to predict the prices.

[harsh monolith]

that's not quite right.
"learning dataset" isn't a thing. there's a training set, and a test set. though the training set is a "learning dataset"

[lean olive]

As numeric features I have: id, lat, lon, price, publisher id

In lat, lon I have like 75% of missing values. I have only 422620 values out of 1283423.

But it will be an issue because I wont filter by the categorical features that I want: CABA, selling prices... The goal is not to predict the prices from every city, just that one in particular

df_ent["location_1"].unique()
array(['Ciudad Autónoma de Buenos Aires', 'Capital Federal', 'CABA'], dtype=object)

I've already cleaned it but it had other cities

df_ent["currency_type"].unique()
array(['dolares', 'Otro', 'pesos'], dtype=object)

I have different currencies, and I want to all be the same

Maybe I'm mistaking myself and i'm not quite understanding how filtering and imputating missing values is

[harsh monolith]

do you see the problem with treating publisher ID as a number?

[lean olive]

Yes, its making my data go all over the place, do I have to delete it from the training set?

What about imputating missing values in lat and lon?
I've used the median, but I end up with like 80% of my predicted data with just one number... that makes sense since im telling the model "ok all of these houses are in the exact same place"

[harsh monolith]

unique IDs don't mean anything mathematically (it would, for example, make no sense to add two of them). and they also don't tell you anything about what they refer to. they're just unique identifiers.

[harsh monolith]

you also have to be careful about latitude and longitude. two cities can have the same latitude, but be in Africa and Europe.

[lean olive]

So, a good practice will be to delete them from the training set?

[harsh monolith]

you shouldn't train the model on unique IDs, no

[lean olive]

Got it, thank you

[lean olive]

Exactly, but I'm just filtering for one city in particular at the beginning

yet I still have missing values... so every long lat is relatively similar at the beginning. Then when I fill missing values with the median the model is not predicting right

This is what I'm trying to say, taht im doing:

df_ent['lat'] = df_ent['lat'].fillna(df_ent['lat'].median())
df_ent['lon'] = df_ent['lon'].fillna(df_ent['lon'].median())
df_ent = df_ent.fillna(df_ent.median(numeric_only=True))

[harsh monolith]

you have to use different imputation methods for different kinds of data. can you see why imputing lat and long with the overall median is a bad idea?

[lean olive]

Yes, because every single value will be the median (over 70% of my data will be the median)... That's the reason why my prediction has a big spike in the middle

What other ways of imputing lat and long can I look for?

[harsh monolith]

lat and long should almost always be used together, since they're basically coordinates in a 2d grid. so any time a lat or long is missing, try to figure out which other data point is probably closest to that one

[lean olive]

So using another variable to predict a (lat, long) data point? or to use something like nearest neighbors imputation?

[harsh monolith]

yes--you said sort of the same thing two different ways

[lean olive]

Ok I will try that. Thank you very much for the help! I appreciate it

[olive temple]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.