Everything was chill in my 2nd base (please see location) while I was manually mining iron ore south of the map. When I teleported, I was suddenly surrounded and attacked by 8 T3 brigands. My villagers and soldiers went into panic mode and everything went from 0 to 100 real quick. 1/3 of my men died and I was stunned with what happened.
As per my observation, bandit/brigand patrols are already constantly patrolling the map. The difference is that, they only get rendered once the player is close/at a certain distance in order to save CPU-GPU processing power.
I loaded my save again, waited for a few minutes, then proceeded to use the Travel Post. They were then at a far away distance from the village, which supports the statement in my 2nd paragraph.
I did a quick search for previous reports here in the 'bugs-n-crashes' channel, and yes walls will be useless if this issue isn't resolved.