Apparently even when glass cards are debuffed, destroying them still ramps Glass Joker.
One of my runs was finishing off with Verdant Leaf and i used Hanged Man on two debuffed glass cards, expecting it to not increment Glass Joker. Usually when you destroy cards to ramp jokers (e.g. destroying face cards for Canio) it doesn't work, but for some reason it did.
Asking around the community, people seem to find this weird as well.