The calculation seems simple: On a KVM host that has 16 GB, you should be able to run a maximum of 32 VMs with...
512 MB RAM each. However, by applying some clever tricks, you can optimize memory use and get more out of your physical servers.
To start with, you need to remember that from the hypervisor perspective, a KVM VM is just a Linux process. That means it follows generic rules about virtual and resident memory, just as any other Linux process. When it starts, a KVM VM sets pointers to virtual memory addresses. Virtual RAM is not real memory though, it's just an address space that allows processes to set pointers. Basically, the exact amount of virtual RAM a process is using doesn't really matter.
When a VM is active, resident memory is allocated. This is memory that is actively used, and the amount of resident memory that is claimed by a process is equal to a virtual machine's real memory usage. The interesting thing to observe is that there is no one-to-one relationship between the amount of memory that a VM is using (figure 1) and the VM's allocated resident memory (figure 2).
As you can see, our test VM has 238 MB of RAM marked as used, but the amount of resident memory reserved on the host OS is much higher. However, this doesn't matter. What does matter is that the amount of resident memory a VM uses is lower than the amount of RAM allocated to the VM.
Using memory swap
There's another aspect that plays a role in the number of VMs a server can host, and that is the usage of disk swap space. The Linux kernel efficiently tracks whether allocated memory is actively being used. You can observe this from the /proc/meminfo file, from which the essential lines are copied below:
Active: 2097832 kB
Inactive: 1136816 kB
Active(anon): 1537524 kB
Inactive(anon): 101456 kB
Active(file): 560308 kB
Inactive(file): 1035360 kB
Active memory is memory that should be available as RAM, inactive memory is memory that hasn't been used recently and is available to be swapped out. As an administrator, you should observe statistics about active versus inactive memory over an extended period. Make sure that the amount of swap space on your server is at least equal to the maximum amount of inactive memory that you have observed, because memory swap can and will be used by your VMs.
When using memory swap, you should also make sure that active memory isn't swapped in and out, because that will result in reduced performance. Active memory should be in RAM and not in disk swap space. You can see whether active memory is being swapped in and out by using the vmstat command. For example, run the vmstat 2 10 command to display a system usage summary in 10 loops with a 2 seconds interval. Make sure you observe the SI and SO (swap in/swap out) statistics. It's a bad sign if you see a high number for these statistics.
OOM and memory swap
You should also monitor your servers to make sure that no out of memory (OOM) situation arises. OOM can happen if the total amount of RAM allocated to all VMs is higher than the total amount of RAM plus available swap space. Initially you may succeed in running more VMs than the amount of RAM plus swap space, but when the VMs become more active, they demand more RAM, which can easily lead to an OOM situation.
In an OOM situation, random processes will be terminated, which means that VMs can spontaneously fail. It is not that difficult to avoid OOM; just make sure that the total amount of virtual RAM assigned to your VMs is less than the total amount of RAM plus swap space.
Share memory with kernel samepage merging
Another way to optimize KVM memory use is by running kernel samepage merging (KSM) The KSM daemon monitors for kernel shared memory and tries to observe use of shared memory within VMs. This occurs, for example, when multiple servers are running the same operating system. Instead of allocating the same operating system in memory, KSM ensures that it is loaded once and shared among VMs.
Even though KSM can lead to an effective reduction of the amount of RAM required to run VMs, it can lead to higher CPU loads. Therefore monitor the process to ensure it isn't causing performance problems elsewhere on your server.
Because the Linux kernel efficiently differentiates between active and inactive memory, swap can efficiently be used to optimize memory needs for KVM virtual machines. Make sure though that the total amount of RAM allocated to all of your VMs fits in the total amount of RAM and swap available on the host server. KSM may help in minimizing the memory needs as well, but you should observe that it's not causing any performance problems. By using these standard Linux kernel solutions to optimize the use of memory by KVM virtual machines, you'll be capable of running more VMs than the physical amount of RAM really allows you. But by all means, make sure that you have enough swap space available.