My private groups solved this by using Matrix for text chat and Mumble for voice. It has push-to-talk and outstanding sound quality. Hosted Mumble servers are cheap, and self-hosting is pretty easy.

When Element Call (MatrixRTC) eventually leaves beta, we might switch to that, but it’s hard to beat Mumble for audio.