Container-based virtualization, called Linux containers (LXC), can be a very efficient alternative to a conventional hypervisor. However, if not set up correctly, virtual containers can consume resources and starve other processes, potentially hurting performance and eliminating the advantage. To prevent this problem, administrators can use cgroups to impose resource limits on processes and ensure their environment runs efficiently.
What are cgroups?
A cgroup is a technique that allows administrators to create groups of resources and limit the availability of those resources for certain processes. Without cgroups, a virtual container can claim all available memory and starve other processes. With cgroups, the administrator can limit the resources a specific container can use. These limits are created on the host operating system and apply to all virtual containers within a group.
Starting a container in LXC
After creating an
To adjust those resources, an administrator can define limits in the cgroup configuration files. These configuration files are available on mount points in the file system of the host operating system. On SUSE Linux Enterprise Server 11.3, for instance, they are located in /sys/fs/cgroup. Every resource has its own subdirectory in the structure. You can modify these files to set resource limits for virtual containers.
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,noexec,nosuid,nodev,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,noexec,nosuid,nodev,cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,noexec,nosuid,nodev,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,noexec,nosuid,nodev,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,noexec,nosuid,nodev,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,noexec,nosuid,nodev,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,noexec,nosuid,nodev,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,noexec,nosuid,nodev,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup
Let's take a look at how to adjust these limits. Memory access is controlled through /sys/fs/cgroup/memory. When navigating to that directory, you will find a subdirectory with the name lxc, under which a subdirectory is created for every container. Two control files are of particular interest: memory.max_usage_in_bytes and memory.usage_in_bytes.
lin:/sys/fs/cgroup/memory/lxc/sles-lxc1 # ls
lin:/sys/fs/cgroup/memory/lxc/sles-lxc1 # cat memory.max_usage_in_bytes
lin:/sys/fs/cgroup/memory/lxc/sles-lxc1 # cat memory.usage_in_bytes
In the listing above you can see that the max_usage_in_bytes is currently set to about 15 MB,
and the current usage in bytes is about 14 MB. If you want to increase the memory
available to this container, you can echo a new value to the file. If for instance, you want 30
MB of RAM available to this container, use the command
echo 30605312 >
When you create another container, it will also get a configuration file. This file is typically
in /etc/lxc, in a subdirectory that has the same name as the container. The complete
configuration of the container is stored in this file, including its cgroup settings. An
administrator can limit specific resources for a virtual container from within this directory. If,
for instance, you want to make the memory limitation a permanent part of the configuration file,
you would add the line
lxc.cgroup.memory.max_usage_in_bytes with its value to the end
of the configuration file. The next time you start a virtual container, it will be created with a
maximum memory limit that you specify in the configuration file.
Apart from limiting the amount of resources available to specific containers, cgroups can also limit a container's access to specific devices. You can see an example of this in the container configuration file, where it will display device access properties.
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 254:0 rwm
An administrator can easily add to these lines to define which devices a virtual container can access to create a more secure environment.
This was first published in February 2014