Often I find myself in situations where I besiege an enemy city and their army just leaves, only to come back in greater numbers.
When this happens and I'm the defender, I do admit it's very satisfying to sneak out of my city, behead the enemy general and go back inside my walls... but it shouldn't be a thing by default.
Solution:
Either make a "backdoor" building that allows armies to escape the city while under siege or make it some sort of special unit trait.
By default, besieged units are besieged, surrounded by enemies and shouldn't be able to just prowl around their territory. Not everyone is Helm Hammerhand, but we do love Helm Hammerhand.