iirc GridContainer and other similar control nodes will constantly take control of ALL formatting options for any child control nodes, scale included.
#TextureButton.rect_scale keeps resetting to 1,1
pulsar island
So your best bet is to either find another way to keep each texture at a 128x128 size or to deal with nesting more child nodes
Maybe you can set the minimum size of the control nodes to be 128x128? (i forget if that's a new thing in godot 4) It sounds to me like the control nodes are getting resized to 0x0, and by extension aren't being moved around correctly.