here is my thoughts about why smaller team will be kinda painful, but take it with a grain of salt.
based on my solo/squad queue experience with solo being more prevalent, usually there are 2 Windguards in the team in a squad play to heal everyone and give Rage to the better players to score a decisive wipe in Carnage/Survival/Gates of Fire.
this is manageable in a 6v6 because you have 4 other players in the team that can pick up on Windguard's lack of firepower (since Blast wall damage is nerfed again) and deal with the very powerful Rage that the Windguard can give to other players.
if the Carnage game downsizes to 4v4, it will most likely fall into Survival strat where a team would pick 2 Windguard and 2 other damage dealer, because that's one of the strongest combination you can have. at this point having more than 5 classes will be oof because people will pick 2 Windguards and that leaves only 2 more empty spot to fill in.
turning the game into 4v4 (Carnage especially) will just be Survival without having to revive souls.
another thing that is worrying if you downsize the game to 4v4 is that there will be too many power ups in the game.
there are players out there that are hella good at the game and they will become unkillable if you let them cycle all the shield, health, and stamina power ups.
in 6v6 this isn't too much of a problem because there are other people needing it, but in 4v4 you will be able to cycle the power ups even more.
this is a very big issue in Spoils of War where a Marauder can just keep cycling shields, health, and stamina pick up and defend themselves with Gust while farming gold.
yes Gust recharge has been nerfed but it still doesn't stop very experienced player from being almost invulnerable.
I don't play enough Spoils of War to comment on it even further than that, but Gates of Fire will also be janky with how the spawn works like Aastrid mentioned.