Well, this is a bummer.
I ordered HA Voice Preview, it just arrived and I tried to link it to my HA instance. But it seems that it can never work if you have proper and correct TLS certificate, the only way to use voice assistant is to keep everything flowing unencrypted on your network.
And the issue regarding that is depressing to read, seems like no encryption is the recommendation from devs: https://github.com/esphome/home-assistant-voice-pe/issues/315
Now I need to return the device. I feel this should be pinned somewhere, so folks who care about security and want fully local and secure option know to avoid it...