Asus x99-E WS motherboard.
I think I narrowed it down a little bit more. When running two win10 VM's at the same time, the VM with the GPU in the third PCIe slot got the problems. This was the case with either the 1070 or 1060.
What is weird is that it didn't matter if both VM's were running or not, as long as the GPU was connected to the MB, the issue occurred.
I changed the machine type to Q35 and now it works well.