Creating Linux container compute constrictions with cgroups

Use cgroups to limit the resources a virtual container can consume to prevent oversubscription and contention.

Container-based virtualization, called Linux containers (LXC), can be a very efficient alternative to a conventional...

hypervisor. However, if not set up correctly, virtual containers can consume resources and starve other processes, potentially hurting performance and eliminating the advantage. To prevent this problem, administrators can use cgroups to impose resource limits on processes and ensure their environment runs efficiently.

What are cgroups?

A cgroup is a technique that allows administrators to create groups of resources and limit the availability of those resources for certain processes. Without cgroups, a virtual container can claim all available memory and starve other processes. With cgroups, the administrator can limit the resources a specific container can use. These limits are created on the host operating system and apply to all virtual containers within a group.

Starting a container in LXC

After creating an LXC virtual container, the system copies the container's files to a chroot environment where they form the container. Next, the container starts on top of the host operating system, which controls the container's resource use. When you start a new container, Linux creates a cgroup for that container, which allows you to limit the resources that are available to the container.

To adjust those resources, an administrator can define limits in the cgroup configuration files. These configuration files are available on mount points in the file system of the host operating system. On SUSE Linux Enterprise Server 11.3, for instance, they are located in  /sys/fs/cgroup. Every resource has its own subdirectory in the structure. You can modify these files to set resource limits for virtual containers.

cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,noexec,nosuid,nodev,cpuset)

cgroup on /sys/fs/cgroup/cpu type cgroup (rw,noexec,nosuid,nodev,cpu)

cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,noexec,nosuid,nodev,cpuacct)

cgroup on /sys/fs/cgroup/memory type cgroup (rw,noexec,nosuid,nodev,memory)

cgroup on /sys/fs/cgroup/devices type cgroup (rw,noexec,nosuid,nodev,devices)

cgroup on /sys/fs/cgroup/freezer type cgroup (rw,noexec,nosuid,nodev,freezer)

cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,noexec,nosuid,nodev,net_cls)

cgroup on /sys/fs/cgroup/blkio type cgroup (rw,noexec,nosuid,nodev,blkio)

cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,noexec,nosuid,nodev,perf_event)

Let's take a look at how to adjust these limits. Memory access is controlled through /sys/fs/cgroup/memory. When navigating to that directory, you will find a subdirectory with the name lxc, under which a subdirectory is created for every container. Two control files are of particular interest: memory.max_usage_in_bytes and memory.usage_in_bytes.

lin:/sys/fs/cgroup/memory/lxc/sles-lxc1 # ls  

cgroup.clone_children memory.max_usage_in_bytes    memory.swappiness

cgroup.event_control  memory.move_charge_at_immigrate memory.usage_in_bytes

cgroup.procs      memory.numa_stat         memory.use_hierarchy

memory.failcnt     memory.oom_control        notify_on_release

memory.force_empty   memory.soft_limit_in_bytes    tasks

memory.limit_in_bytes memory.stat

lin:/sys/fs/cgroup/memory/lxc/sles-lxc1 # cat memory.max_usage_in_bytes


lin:/sys/fs/cgroup/memory/lxc/sles-lxc1 # cat memory.usage_in_bytes


In the listing above you can see that the max_usage_in_bytes is currently set to about 15 MB, and the current usage in bytes is about 14 MB. If you want to increase the memory available to this container, you can echo a new value to the file. If for instance, you want 30 MB of RAM available to this container, use the command echo 30605312 > memory.max_usage_in_bytes.

When you create another container, it will also get a configuration file. This file is typically in /etc/lxc, in a subdirectory that has the same name as the container. The complete configuration of the container is stored in this file, including its cgroup settings. An administrator can limit specific resources for a virtual container from within this directory. If, for instance, you want to make the memory limitation a permanent part of the configuration file, you would add the line lxc.cgroup.memory.max_usage_in_bytes with its value to the end of the configuration file. The next time you start a virtual container, it will be created with a maximum memory limit that you specify in the configuration file.

Apart from limiting the amount of resources available to specific containers, cgroups can also limit a container's access to specific devices. You can see an example of this in the container configuration file, where it will display device access properties.

lxc.cgroup.devices.allow = c 5:0 rwm

lxc.cgroup.devices.allow = c 4:0 rwm

lxc.cgroup.devices.allow = c 4:1 rwm

# /dev/{,u}random

lxc.cgroup.devices.allow = c 1:9 rwm

lxc.cgroup.devices.allow = c 1:8 rwm

lxc.cgroup.devices.allow = c 136:* rwm

lxc.cgroup.devices.allow = c 5:2 rwm

# rtc

lxc.cgroup.devices.allow = c 254:0 rwm

An administrator can easily add to these lines to define which devices a virtual container can access to create a more secure environment.

This was first published in February 2014

Dig Deeper on Open source virtualization



Find more PRO+ content and other member only offers, here.



Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: