Hi Richard,

Benchmarking virtual environments can be quite tricky. For example, benchmarking one VM could give you excellent results while benchmarking all the VMs at the same time could give you poor results as they share resources. If the assigned resources to the VM are dynamic, it could even change over time.

If you want to get representable results you need to understand how the Virtual machines are spread over the physical machines and test the worst-case scenario (spinning up the max amount of VMs on the physical infrastructure and benchmarking them all at once). This can be quite an undertaking depending on the HW on which you run your virtualized environment.

We have seen a lot of issues with virtualized environments. To give you some heads-up on issues we have encountered:

• The NAS controller was unable to follow with the load (while on disks there was still a lot of margin).
• If you are sharing a disk with multiple VM's make sure you are monitoring it correctly. The machine might show you are only using 10% of the disk, while you have actually 10 VMs using 10% of your disk. Your disk queue length will be a better indicator to warn you if are saturating on disk.
• Time is very important for DB and DM servers as this can otherwise mess up your synchronization. Ensure all VMs are at the exact same time by using the same NTP server on all of them. The time is calculated based on CPU cycles and we have seen VMs drifting minutes away from each other and then jumping back after 5 min back to the correct time (going back in time or jumping to the future).
• When virtualizing your HW you might lose a lot of performance on them (depending on how well they cope with virtualization). For example, your CPU might disable some of the cache it has to allow you to virtualize the CPU, this will lead to a big reduction in performance. This is something you will need to discover by benchmarking your servers.
• If you have a large virtualized environment, make sure to spread your VM snapshots over time as otherwise they might saturate your network at a given time and lead to dropped packages in your switches.