Container-based virtualization, called Linux containers (LXC), can be a very efficient alternative to a conventional hypervisor. However, if not set up correctly, virtual containers can consume resources and starve other processes, potentially hurting performance and eliminating the advantage. To prevent this problem, administrators can use cgroups to impose resource limits on processes and ensure their environment runs efficiently.
What are cgroups?
A cgroup is a technique that allows administrators to create groups of resources and limit the availability of those resources for certain processes. Without cgroups, a virtual container can claim all available memory and starve other processes. With cgroups, the administrator can limit the resources a specific container can use. These limits are created on the host operating system and apply to all virtual containers within a group.
Starting a container in LXC
After creating an LXC virtual container, the system copies the container's files to a chroot environment where they form the container. Next, the container starts on top of the host operating system, which controls the container's resource use. When you start a new container, Linux creates a cgroup for that container, which allows you to limit the resources that are available to the container.
To adjust those resources, an administrator can define limits in the cgroup configuration files. These configuration files are available on mount points in the file system of the host operating system. On SUSE Linux Enterprise Server 11.3, for instance, they are located in /sys/fs/cgroup. Every resource has its own subdirectory in the structure. You can modify these files to set resource limits for virtual containers.
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,noexec,nosuid,nodev,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,noexec,nosuid,nodev,cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,noexec,nosuid,nodev,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,noexec,nosuid,nodev,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,noexec,nosuid,nodev,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,noexec,nosuid,nodev,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,noexec,nosuid,nodev,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,noexec,nosuid,nodev,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,noexec,nosuid,nodev,perf_event)
Let's take a look at how to adjust these limits. Memory access is controlled through /sys/fs/cgroup/memory. When navigating to that directory, you will find a subdirectory with the name lxc, under which a subdirectory is created for every container. Two control files are of particular interest: memory.max_usage_in_bytes and memory.usage_in_bytes.
lin:/sys/fs/cgroup/memory/lxc/sles-lxc1 # ls
cgroup.clone_children memory.max_usage_in_bytes memory.swappiness
cgroup.event_control memory.move_charge_at_immigrate memory.usage_in_bytes
cgroup.procs memory.numa_stat memory.use_hierarchy
memory.failcnt memory.oom_control notify_on_release
memory.force_empty memory.soft_limit_in_bytes tasks
lin:/sys/fs/cgroup/memory/lxc/sles-lxc1 # cat memory.max_usage_in_bytes
lin:/sys/fs/cgroup/memory/lxc/sles-lxc1 # cat memory.usage_in_bytes
In the listing above you can see that the max_usage_in_bytes is currently set to about 15 MB, and the current usage in bytes is about 14 MB. If you want to increase the memory available to this container, you can echo a new value to the file. If for instance, you want 30 MB of RAM available to this container, use the command
echo 30605312 > memory.max_usage_in_bytes.
When you create another container, it will also get a configuration file. This file is typically in /etc/lxc, in a subdirectory that has the same name as the container. The complete configuration of the container is stored in this file, including its cgroup settings. An administrator can limit specific resources for a virtual container from within this directory. If, for instance, you want to make the memory limitation a permanent part of the configuration file, you would add the line
lxc.cgroup.memory.max_usage_in_bytes with its value to the end of the configuration file. The next time you start a virtual container, it will be created with a maximum memory limit that you specify in the configuration file.
Apart from limiting the amount of resources available to specific containers, cgroups can also limit a container's access to specific devices. You can see an example of this in the container configuration file, where it will display device access properties.
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 254:0 rwm
An administrator can easily add to these lines to define which devices a virtual container can access to create a more secure environment.