Update: I managed to make a working solution for this, but there is still one thing that I don't understand here.
I'm using a local euler angle offset to add mouse rotation to my camera. The X offset is being applied properly, but my Y offset is being treated as a continuous rotation, rotating the camera further and further each frame. My workaround for this is to set the offset to 0 after each frame, but I feel like there should be a more proper way of doing this, a solution where the X offset and Y offset would both be treated as a static offset, instead of X being treated like a static offset and Y as a continuous rotation...