Part one of this discussion on hidden virtualization costs went over the power and heat costs and the management concerns of virtualization. Now, in part two, we will turn our attention to networking issues and the problem of virtual machine sprawl. Part three will discuss licensing, performance and storage.
Virtualized servers have unique networking concerns. In a virtual network, you will see a larger than average number of IP addresses, more than usual subnets and VLANs, high packets per second, greater susceptibility to switch port problems, greater need to plan machine per host density and MAC address management issues.
Depending on the density of virtual machines achieved per physical host, you will quickly realize that you need a networking plan. From the outside looking in, 100 physical servers might not seem like a big deal, but if you average a density of six virtual machines per physical server, you are actually looking at 600 virtual machines and 100 physical servers, which translates to 700 resources that require network configuration and IP addresses.
From the networking perspective, it is important to think of virtual machines as being no different than physical servers. It is extremely important to plan out the configuration of the network and its subnetting based on the total number of virtual machines and physical servers. Otherwise, you are going to be in for a rude awakening.
As the physical host servers and the virtual machines are scaled out, the number of subnets and VLANs will grow proportionately, so be aware that there will be an increased number of subnets and VLANs that will need to be managed.
It is also worth noting that if you are setting up access-lists (ACLs) between VLANs (an example would be a Cisco 6509 with a multi interface Firewall Service Module or FWSM), you should be ready to manage a fairly large number of object groups and a relatively large ACL and firewall config in general.
One obvious problem that will need to be addressed is that a single physical interface running multiple virtual machines can introduce very high packet per second (PPS) throughput on the network adapter and the physical switch port. Even gigabit switches can have their switch port buffers overloaded due to very high PPS network traffic, which manifests as dropped packets and retransmitted packets on the switch port.
One of the only ways to get around this without affecting your virtual machine's network performance is to buy higher-end Gigabit switches that have robust hardware buffers that can keep up with high bandwidth and high PPS throughput.
Another problem is that a single physical switch port could go down or may have a serious issue that could in turn impact all virtual machines that are bound to the physical network adapter that is connected to this particular switch port.
Along the same lines, when troubleshooting poor network performance on a virtual machine, you have to consider other factors, such as the other virtual machines that are running on the same physical network interface could be causing buffering issues on the physical interface itself. These and other factors make network monitoring more challenging in a virtual environment as compared to a non-virtualized environment.
Virtual machine per physical host density should always be a top consideration when it comes to planning and designing your virtual network. As the virtualized environment sprawls, you should expect to manage a very large number of IP addresses, hence a very large number of MAC addresses, subnet addresses and VLANs.
The consequences are along the same lines as what might be faced in a large enterprise network containing many servers; the core network layer must be robust, stable and fast, such as in a Cisco 6509E chassis with. Expect to burn through a large number of IP addresses very quickly; consider using smaller subnets.
Also, in order to keep the broadcast traffic at a manageable level, consider breaking up the subnets into /23 networks to allow for around 500 virtual machines per subnet and still having acceptable network performance. Depending on how inter-VLAN traffic is managed and if ACLs are applied to each VLAN, consider using a robust Firewall Services Module (FWSM) solution as a large number of interfaces and a high throughput will be needed. With a large number of IP addresses comes a hefty ARP table, which once again translates into spending the money up front on the networking gear that can do the job correctly.
One final thing to mention are the issues with MAC address management in virtualized environments. You don't have to worry about this normally, since in a physical server the MAC address is burned into the physical network adapter, but this is not so with virtual servers.
A virtual network adapter gets either a user-statically assigned or a dynamically assigned MAC address. Either way, you are looking at a large number of MAC addresses in your network, and more importantly, multiple MAC addresses per single physical network interface as well as each physical network switch port. With an increase in the number of switches and having multiple MAC addresses per switch port, tracing down a rogue MAC address can become quite a taxing exercise.
In some cases, you may find you're trading physical server sprawl for virtual machine sprawl.
As an example, a company may have had 100 servers being managed in their data center facility six months ago. After server consolidation, the company might reduce that physical server footprint to 50 servers. Six months later, the same organization might find that it is not only managing the same 50 physical servers, but it is also managing eight virtual machines on each physical host server for a total of 450 managed machines. That's a 450% increase in managed machines, which quickly negates the 50% reduction only six months earlier!
One reason behind this explosion in the number of machines being managed is that virtual machines are so easy to create. In a traditional IT department, when a business unit requires a new server, an IT administrator must go through a series of steps in order to see to that need. In the physical world, a number of controls and processes are put in place, and the first step is physical procurement. Once a server specification is defined and the order placed, it takes time for the server to ship and arrive.
Then, the next step of physically handling the server takes place. The physical server is unboxed, racked and stacked, cabled and configured. These steps are almost always controlled by the IT department, simply because physical access is needed to actually touch the equipment.
Virtualization is much simpler. Physical access to the server is not typically required to create a virtual machine. And because standing up a new virtual machine can be done by basically just copying a single file from one location to another, almost anyone in the organization can do it. Unchecked and uncontrolled use of virtualization can lead to virtual machine sprawl, where virtual machines become unmanaged and uncontrolled as they sporadically appear throughout the IT infrastructure.
Click here to proceed to part three, which will discuss licensing, performance and storage.
About the authors: David Marshall is a senior member of the reference architect team at Surgient, Inc., and he specializes in server virtualization, virtualization applications and Windows administration. He also runs the InfoWorld Virtualization Report, as well as the virtualization news blog, VMBlog.com. David is also a co-author of Advanced Server Virtualization: VMware and Microsoft Platforms in the Virtual Data Center, a book that details years of hands on experience using and implementing server virtualization solutions.
Dan Knezevic is a senior network engineer and a team lead for the data center operations team at Surgient Inc, providing expertise in the data center network and server infrastructure as well as virtualization platforms. He also specializes in network security and enterprise storage solutions. He brings six years of virtualization integration experience in the data center environment.