I always find myself having to go into the settings so I can hear players when spectating, but I love playing against others without the "hear other players" box checked.
Maybe just make it default that you can hear players in spectator mode no matter what, and keep the silence for when you're playing; if it's unchecked.
Thanks!