Blocking scam links : a better way than a big database (.txt) file ?

[neon orchid]

That feels very complicated since I'm not familiar with regex.

What's the system ?

• Check if the received message with MessageReceived event has a scam link
• If so, ban the user that sent scam link

Current way

• A very big .txt file that contains lots of scam link domains

public static async Task<bool> ContainsScamLink(this SocketMessage message)
{
    string[] scamLinks = await File.ReadAllLinesAsync("Scamlinks.txt"); //This may take some time to read the entire file
    if (scamLinks.Any(scam => message.CleanContent.Contains(scam, StringComparison.OrdinalIgnoreCase))) //This may take some time to analyze
        return true;
    return false;
}```
## Possible other way
- Scam links are very similar to official links, so is regex a better way to spot scam links ?

[grim juniper]

scam links update constantly

You can't possibly filter them all

Putting all of the scam links in a database might be more performant

Atleast more performant than a text file

[neon orchid]

And if it updates constantly I'll have to manually put this new link into the database

[grim juniper]

yeah

there are entire companies dedicated to keeping up to date records of scam links

[neon orchid]

Well, can't find them

[crystal warren]

@neon orchid there's multiple APIs that provide constantly-updating stores/lists of scam related links such as: https://sinking.yachts and such. Or sites like https://www.ipqualityscore.com/ (paid) that can scan URLs in real-time if they are currently unlisted by such APIs. @past grotto uses a collection of multiple API sites, live-scanning and a locally stored list to combat them which I've found to be fairly effective so far

But as @grim juniper said, they are constantly updating and changing, so it's difficult to try and keep up with it yourself, it's generally better to rely on multiple third parties who have entire teams dedicated to finding and tracking such links

Generally the "process" I use for Cakey goes like this (I use text file to manually add reported/noticed links that get missed by the rest of the process):
=> Receive message
=> Check all links against local text file (stored as yml, but filetype not overly relevent)
=> Check all URLs against a list of known URL shorteners, if it is a short url, resolve it to base/root/redirected URL
=> Check all resolved or non-shortened URLs against a cached list of URLs compiled from 3 different APIs
=> Live scan any URLs remaining in message that were not in any of the above lists, cache the result from this

If none of the URLs git hit by the above process, message is allowed through, otherwise it's deleted and punishment varies depending on server settings

[neon orchid]

Okay, for now I'll try with sinking yachts

[crystal warren]

Yeah, the full process itself gets a little complicated but the concept behind it is fairly straight forward