A machine learning approach to dealing with cheaters

[silver night]

The title does sound crazy, and it probably is. It has been a long dilemma between myself and I to attempt implementing such a thing. I am open to feedback and constructive discussion regarding this approach. You will find attached below all the important details about how I went on doing this.
(PDF document removed from this message due to updates, scroll down for the later versions and the second version of unsupervised ML).

[jolly carbon]

Nice one, but did you consider other metrics/inputs (which ones, and why did you not consider them), even those that are not specific to your gamemode, but that could reveal a cheater?

And how did you actually capture the data for your analysis? Client only/Server only/Both? Did you see positives/negatives in your chosen approach? What if cheaters bypass your data capture, or even poison their own data to drop their own 'cheating ratio'

[silver night]

The reason behind choosing the reactionTime, yaw, pitch and weapon as features is because those are the only reliable metrics I could capture from players. The data was captured in both server and client. There are certain thresholds of those metrics that indicate that a kill is suspicious or not, and by measuring the aim deltas frame by frame then averaging them I get to actually analyse the player's crosshair movement vertically and horizontally based on the agreed-upon standard in the gaming industry. Those averaged values indicate whether an aim is actually extremely snappy or not, and if so in which direction. Obviously, there are patterns behind it hence why the ML approach, sometimes the line between a suspicious kill and a normal one is extremely thin and blurry (see fig 1).
There were most definitely some false positives due to how some weapons behave in-game, however not very impactful to poison the dataset. One kill detected as suspicious is not enough to determine whether a person is actually cheating or not, however combining more measurements such as the time window, kill streaks with the model can definitely tell you who is cheating and who is not because a cheater will eventually have more suspicious kills than non-suspicious ones (the case for my server and today's analysis). I don't think (and at least hope not) that cheaters will be able to bypass the delta time calculations unless they play the game without rendering graphics.

When it comes to treating cases that do not happen to belong to my gamemode, it is not possible for me to gather such data unless someone else wants to collaborate, then we can have more models that deal with different kinds of cheats.

[jolly carbon]

I hope you are aware of edge cases like this: https://www.youtube.com/watch?v=tU3e8TNtTzw

Do you actually consider kills being the determining factor, or the damage given

[silver night]

I am very well of such possibilities, hence why on a first glance the algorithm I chose might look overkill for such a purpose but it can handle edge cases very well assuming that you supply it with enough data to do so, maybe if there's a way to include more metrics directly from the engine or MTA itself it would make the task at hand much easier

[silver night]

as of now I consider the kill because the damage given won't serve me as much, its weighing factor was almost 0

[jolly carbon]

That's why I asked about other inputs/metrics, because you might not have the data/no way to capture it

[silver night]

given its insignificance it was better taken off than kept

[silver night]

Indeed, however I worked with what I was given. This could be the start of something totally new in MTA

With more data and metrics it could turn into something very powerful

[jolly carbon]

So if somebody were to team up with a cheater, the cheater could simply damage others down to a threshold, but never finish, and let the second guy without cheats finish the kill

Its probably not going to be the case most of the time, but its a possibility to get around the detection

[silver night]

The determining factor here is not who started the kill, but how they started it. If let's say the other person reacted within the norms of movement and reaction time, his kill will not be flagged as suspicious

If however the aim happened to be extremely snappy and accurate (a yaw avg of > 10 or < 0.01) then it will be flagged as suspicious

[jolly carbon]

Did you also try to approach the problem from a cheater's side by creating your own aim cheat?

[silver night]

I did simulate a cheat in these different scenarios using Lua, yes

I also allowed more cheaters in the server that I could capture screenshots of using cheat menus

it led to gathering more authentic data

[jolly carbon]

Did you also capture their gameplay/inputs for replay?

[silver night]

No, however I did remember that a server had a similar feature before so the next step in order to improve this further would to capture their inputs and provide a replay functionality in-game

[jolly carbon]

With a game capture you could keep retesting your model easily

[silver night]

Yes

In any case this is a start, we could take it more steps further if enough interest is given to it

more metrics and more data will simply mean better detection and protection

and also with ensemble learning techniques, it'll be possible to generalize predictions from multiple models at once

[jolly carbon]

The problem with models like these that its hard to apply on the "global" scale across all servers

[keen quail]

In MTA specifically can it be made to track mouse movements for example the snapping that happens during aimbot scenarios. Whether its through capturing or having it be present in the live environment

[jolly carbon]

So it's not really a good solution for MTA AC

[silver night]

With its very oriented nature, I don't think it can generalize on a global scale unless there are global metrics and more data to train it on

and that data doesn't have to generalize only with aim hacks

it could go further beyond to anything else

A model is only as good as the data its trained on, if there's proper data for whatever it may be it can be generalized through multiple pipelines and eventually would result in something that would improve the qol of every user

[silver night]

I forgot to include this in the document. This plot concretely shows the accuracy of the model's predictions when it's given new data it has never seen before.

[silver night]

Here's the updated document:

[median seal]

I like this idea, just one question im curious about, since you go with supervised based approach, how you got these ground truth data for whatever a sample is suspicious or non-suspicious? by marked manually with human admin?

[silver night]

I've always had intuition about who's cheating and who isn't through admin panel screenshots. Although, I was thinking of writing another document explaining it the unsupervised way

I'll try to find some time to explain through clustering, it'll explain more about how this data was conceived

[silver night]

I'll be posting about the unsupervised approach very soon while diving into the details of how K-means clustering works. I think it will be much more interesting :D

[silver night]

As promised, this is the document explaining the unsupervised approach.

I am looking forward to hearing your feedbacks.