I tested the difference between Windows and Linux, and Windows seems to be more stable and smooth when playing videos, while on Linux there's a huge increase in CPU usage and my computer fans go absolutely bonkers over what seemingly is a small task.
Bare in mind, I do watch videos at 2x speed which may also be a contribution, but even under Windows I don't have a problem.
Issue occurs with Chromium and Firefox, except with Firefox, the process Isolated Web Co seems to require a huge amount of CPU when playing videos.