Also tried tinkering around with all the shadow options in the game, tried changing from VULKAN to DX12 but nothing seems to help.
Never had this problem anywhere else in the game, except now :/
Also, here are my specs:
Ryzen 7 3800x
Sapphire RX590 Nitro+ (Latest Driver)
32GB RAM @ 3000MHz
Installed on a SSD