When pieces in a village get destroyed, they should leave behind a layout (The blue holograms used when building builds) for easier repair. You could directly interact with holograms, and it'll automatically open the building menu and select the missing piece.
And another idea that i would love, would be for villagers to pick up the dropped resources if they're still nearby, and fix the village on their own. But that might be hard to implement.