It doesn’t really matter that much imo. Virtualization is done by kvm or xen, both distro independent. Qemu, libvirt and anything else you might need will only differ version, which may or may not matter to you (probably not). Maybe you can gain some performance by building from source, whereas something like Gentoo might come out ahead… Then again, you can build from source on any distro so…