Thematically, it is bizarre that the Incan leader is anti-synergistic with the Inca civilization. Both are pretty low tier, and they are worse together than other combinations.
As I see it, there's a pretty quick fix and a much needed buff that you could do. What if Pachacuti counted unique improvements as if they were mountains? This would allow the terrace farm to act as a mountain. It would also give him a general buff to culture gain, which would be a nice edge to have. It would allow terrace farms to not count against Pachacuti's ability.
Alternatively, perhaps buildings should get a food adjacency for both rough terrain and mountains, and the specialist ability applies to buildings adjacent to either rough terrain or mountains. But that would be a much greater buff.