Algorithm that "smooths" grid mesh?

[golden light]

Hi! I am making a golf game ⛳, and I have this 3D map that I am generating from an image. Each pixel's color represents a type of terrain (fx. green for fairway, brown for bunker). I lay all the vertices out in a grid formation, and assign materials based on the pixel color in the generator image. The height of the vertices also depend on the pixel color with different perlin noise functions for each type.

As seen in the image, the grid is clearly visible. I was wondering if there's an algorithm that can help me make it look smooth. I don't know if perfectly smooth is achievable, but I would at least like to get rid of the hard edges / pixelated look, by "rotating" some triangle pairs 90 degrees and setting each triangle's material rather than the entire squares that make up the grid. See the second image

The more crazy (but better) solution will require moving the vertices away from the grid structure, but I'm not sure that this is feasible. See the third image

Any help coming up with a solution is greatly appreciated 🙏

Wait isn't this basically marching squares?

[shrewd pivot]

Basically

You can also try to do some other forms of processing but I'm guessing that's not as easy to get good results with

[golden light]

So what if I draw one mesh for each material type. I'm thinking this might give me a performance boost? Also, it should work nicely alongside the marching squares

And I'm guessing I'll have to make each pixel in the generator iamge correspond to a vertex instead of a square. This is probably the key 💡