Get rid of overlapping "area of effect" structures in your base.
I had 15 followers. I had 3 farm level 2 completely decked out with 6 scarecrows, 6 propaganda speakers, 6 windmills, 3 devotion statues with the big one to collect from them, and 12 signpost decorations for different seeds. My game was lagging so bad I was getting the lingering text box bug every time I played. Even in every crusade. Frequent freezes and crashes, and the text bug/crash even prevented me from getting one of my talisman piece rewards from sacrificing a follower to Midas. (Even though it counted as a sacrifice and now I can't get my last fleece for %100)
ANYWAY: My spouse is a game developer and mentioned that the unity game engine has a hard time dealing with overlapping things. ALL those structures have a blue "area of effect" around them and they all were over top of each other in a huge mess.
I trimmed the fat and now successfully run 5 lv.2 farms with 20 followers and no lag. Each farm has a level 1 windmill and level 2 scarecrow. That's it. Lag is gone. Game runs great.
Oh I also got rid of my decorative flooring but I'm not confident that helped much or at all. Mostly the area of effect thing.
PS5 btw.