I noticed that after about half an hour of watching I'd start to buffer a lot - after investigating it turned out the server was being thermally throttled and the CPU was at 100%, because the default transcode config was set to "Software (no GPU)". I changed it to "Video Acceleration API (VA-API) (Best Effort)", which I think is right for my N100 docker setup on the vaapi image? "Intel QuickSync (Best Effort)" was also an option.
Anyways - I had my friend who's running the Windows build on a system with a dedicated Nvidia GPU check, and their default transcode config was also set to "Software (no GPU)". Is that the default value on purpose? We are both on 0.19.3. They changed to "nvidia (cuda)".