You enter a que, 4 people, 1 person leaves, you que again! another 4 people, one of them you've already seen the name of! they leave again!
This continues on for whatever reason that person thinks dodging a que is fun for. So I'd say, put a 1 minute cooldown on them re-queing if they specifically "leave" the Dungeon Vote screen so that at least for the next 60 seconds, you wont be qued up with them anymore.
If people Leave for other reasons, chances are they'll need a 1 minute to check something or re-do gems or open the door or something so it wouldn't punish the people who do it for a good reason.
Unless I'm not seeing something... I'd love to be shown why people do this, for good reasons?