Going in the escape after beating Agahnim

[crystal spire]

Hi guys, during my run, I didn't clear the escape chest before beating Agahnim. I don't have the Pearl or the Mirror. Can I go back into the escape area to clear it without these items? Because now, i can't enter in the castle without teleport to dark world 😦
(for context, i'm in a multiworld session, so the logic can be tricky)

[tall sorrel]

you can enter the castle courtyard though the secret passage under the bush that has Uncle

you can also fall through a grave near sanctuary to the back section of the sewers if you have glove