I've seen this happen on a few different maps, but it get it most consistently on Hirschfelden.
Game runs fine, perfectly smooth normally, and in binocular view too. But when putting the binos away, I get a huge drop in frames.
It also seems to be better/worse depending on which direction I look? For example in this clip, the fps drop is fairly minor when looking North, but worse when looking South.
I would assume it's related to the amount of terrain/textures being rendered, but in this example looking south is towards the edge of the map so... I'm not sure that theory works.