To preface, I'm completely aware this a non issue and fits with design language of the game. I can see that developer took great care of fitting historical design into the game.
Prime example is the granary, a great representation what medieval granaries ought to have looked liked including mushroom-shaped staddle stones.
That said, their whole point is to stop rats getting into the food store which is rendered useless with the addition on the side. (I do realise there are ladders up to the floor as well but those were used only during entering the building and then moved)
It's utterly tiny thing but annoys the heck out of me. I love architecture and design and that's exactly why I love this game.