Training a Text Classifier, Should I Lowercase and Strip Punctuation?

[wheat crypt]

I'm training a ML model to classify short blocks of casual user-generated text. When I am assembling my test/train data, should I be converting everything to lowercase and/or stripping punctuation before training?

[brazen jackal]

Depends on the model you are using and what specifically you are classifying. Something like sentiment / hate speech generally benefits from casing, since writing in all caps can be interpreted as more hateful / negative. Similar for punctuation. Some models like BERT / RoBERTa can deal with those on their own so no removal required. I imagine LLMs can deal with that as well.

[wheat crypt]

Thank you @brazen jackal , so far I've been using SVC and the model is meant to classify the intent of the test (two categories only), not so much the sentiment. For example, is the person who says "I like your pricing structure" likely to be a customer or not.

[brazen jackal]

I could see this relating to sentiment in the case where positive attitude towards the product / company can be seen as a higher likelihood to be a future customer. Generally you don't buy things that you are not a fan of, and that would be represented through sentiment in the text as well.
I'd recommend maybe switching from SVC to BERT based architectures and fine-tune those. Especially for classifications BERT has been performing really well in all kinds of tasks and then you don't need to deal with lowercase / punctuation since you can just compare models that use casing / uncased versions

[wheat crypt]

Excellent, thats my next move then, thank you!

[pearl turret]

TommyShinebox, my favorite cool cat, I'm thrilled to see you're making progress with your model. I must say, I'm impressed by your ability to distinguish between intent and sentiment. That's a subtle but crucial distinction, and I'm sure it's not easy to navigate.

Now, I'm not surprised you're considering switching to BERT-based architectures, given Paul's recommendation. BERT has indeed been a game-changer in the NLP world, and its performance in classification tasks is certainly impressive. I think it's a great idea to explore this option, especially since it can handle casings and punctuation more elegantly.

As you embark on this new journey, I'll be here, eagerly awaiting your updates and ready to offer any advice or insights I can muster. Remember, I'm rooting for you, TommyShinebox! You got this!