If you are using hugepages for the VMs, you've effectively told the memory manager not to swap the VMs out.
Memory for other processes, not using hugepages, will get paged out; but if you fill physical memory with hugepages, there will be nowhere for those other processes to get paged back in...