I went AFK for a short while with my cursor accidentally at the edge of the screen. When I came back, there were floating point precision errors in the graphics and the memory usage was extreme!
The grid I'm at in this screenshot is ~#2000 out from the rift. Theory from a computer science student is that once a chunk is in memory, its never removed.
Issue resolves after restarting the game.