If it's calcium WITHOUT d3, you can leave it in. He won't overdose on it, it's only with D3 it begins to be a concern (if I recall correctly excess calcium is just excreted in the waste. excess D3 is different because it's a fat soluble vitamin)
The calcium in the tank is for self regulation, so they should lose interest if they don't need more calcium. because he is new maybe Soot is curious and investigating (they poke things with their tongues, after all). If you're just finding powder mess everywhere, soot might just be walking through it, many of our geckos make a calcium mess that way