My Series X controller constantly drains batteries, even if not on.

[lime island]

Is that fixable in any way or do I just have to take the batteries out every time I turn it off?