So my friends and I were running in to this exact same issue initially. FW rules were set up, my (the server host) ports were forwarded. We could all join the server, but the only way we could run a raid is if I "hosted" the raid. We would run in to the same thing if anyone else hosted a raid. Our fix was multifaceted. For myself and two of my friends, Enabling Native Sockets and Use UPnP allowed all of us to connect.
However, one of us were still having the issue. The fix for him was to manually port forward 25565. He has a weird managed ISP router, that has UPnP disabled, and can't be turned on. Maybe this well help someone. It's been smooth sailing since!
FYI: Not using a VPN. I use a reverse proxy set up and my friends are connecting using a domain name and not a direct IP address (I host a lot of other servers for other games). But our issue was the same when troubleshooting with a direct IP.