Under cgroups v1, each controller may be mounted against a
separate cgroup filesystem that provides its own hierarchical
organization of the processes on the system. It is also possible
to comount multiple (or even all) cgroups v1 controllers against
the same cgroup filesystem, meaning that the comounted
controllers manage the same hierarchical organization of
processes.
For each mounted hierarchy, the directory tree mirrors the
control group hierarchy. Each control group is represented by a
directory, with each of its child control cgroups represented as
a child directory. For instance, /user/joe/1.session represents
control group 1.session, which is a child of cgroup joe, which is
a child of /user. Under each cgroup directory is a set of files
which can be read or written to, reflecting resource limits and a
few general cgroup properties.
Tasks (threads) versus processes
In cgroups v1, a distinction is drawn between processes and
tasks. In this view, a process can consist of multiple tasks
(more commonly called threads, from a user-space perspective, and
called such in the remainder of this man page). In cgroups v1,
it is possible to independently manipulate the cgroup memberships
of the threads in a process.
The cgroups v1 ability to split threads across different cgroups
caused problems in some cases. For example, it made no sense for
the memory controller, since all of the threads of a process
share a single address space. Because of these problems, the
ability to independently manipulate the cgroup memberships of the
threads in a process was removed in the initial cgroups v2
implementation, and subsequently restored in a more limited form
(see the discussion of "thread mode" below).
Mounting v1 controllers
The use of cgroups requires a kernel built with the CONFIG_CGROUP
option. In addition, each of the v1 controllers has an
associated configuration option that must be set in order to
employ that controller.
In order to use a v1 controller, it must be mounted against a
cgroup filesystem. The usual place for such mounts is under a
tmpfs(5) filesystem mounted at /sys/fs/cgroup. Thus, one might
mount the cpu controller as follows:
mount -t cgroup -o cpu none /sys/fs/cgroup/cpu
It is possible to comount multiple controllers against the same
hierarchy. For example, here the cpu and cpuacct controllers are
comounted against a single hierarchy:
mount -t cgroup -o cpu,cpuacct none /sys/fs/cgroup/cpu,cpuacct
Comounting controllers has the effect that a process is in the
same cgroup for all of the comounted controllers. Separately
mounting controllers allows a process to be in cgroup /foo1 for
one controller while being in /foo2/foo3 for another.
It is possible to comount all v1 controllers against the same
hierarchy:
mount -t cgroup -o all cgroup /sys/fs/cgroup
(One can achieve the same result by omitting -o all, since it is
the default if no controllers are explicitly specified.)
It is not possible to mount the same controller against multiple
cgroup hierarchies. For example, it is not possible to mount
both the cpu and cpuacct controllers against one hierarchy, and
to mount the cpu controller alone against another hierarchy. It
is possible to create multiple mount with exactly the same set of
comounted controllers. However, in this case all that results is
multiple mount points providing a view of the same hierarchy.
Note that on many systems, the v1 controllers are automatically
mounted under /sys/fs/cgroup; in particular, systemd(1)
automatically creates such mounts.
Unmounting v1 controllers
A mounted cgroup filesystem can be unmounted using the umount(8)
command, as in the following example:
umount /sys/fs/cgroup/pids
But note well: a cgroup filesystem is unmounted only if it is not
busy, that is, it has no child cgroups. If this is not the case,
then the only effect of the umount(8) is to make the mount
invisible. Thus, to ensure that the mount is really removed, one
must first remove all child cgroups, which in turn can be done
only after all member processes have been moved from those
cgroups to the root cgroup.
Cgroups version 1 controllers
Each of the cgroups version 1 controllers is governed by a kernel
configuration option (listed below). Additionally, the
availability of the cgroups feature is governed by the
CONFIG_CGROUPS kernel configuration option.
cpu (since Linux 2.6.24; CONFIG_CGROUP_SCHED)
Cgroups can be guaranteed a minimum number of "CPU shares"
when a system is busy. This does not limit a cgroup's CPU
usage if the CPUs are not busy. For further information,
see Documentation/scheduler/sched-design-CFS.rst (or
Documentation/scheduler/sched-design-CFS.txt in Linux 5.2
and earlier).
In Linux 3.2, this controller was extended to provide CPU
"bandwidth" control. If the kernel is configured with
CONFIG_CFS_BANDWIDTH, then within each scheduling period
(defined via a file in the cgroup directory), it is
possible to define an upper limit on the CPU time
allocated to the processes in a cgroup. This upper limit
applies even if there is no other competition for the CPU.
Further information can be found in the kernel source file
Documentation/scheduler/sched-bwc.rst (or
Documentation/scheduler/sched-bwc.txt in Linux 5.2 and
earlier).
cpuacct (since Linux 2.6.24; CONFIG_CGROUP_CPUACCT)
This provides accounting for CPU usage by groups of
processes.
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/cpuacct.rst (or
Documentation/cgroup-v1/cpuacct.txt in Linux 5.2 and
earlier).
cpuset (since Linux 2.6.24; CONFIG_CPUSETS)
This cgroup can be used to bind the processes in a cgroup
to a specified set of CPUs and NUMA nodes.
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/cpusets.rst (or
Documentation/cgroup-v1/cpusets.txt in Linux 5.2 and
earlier).
memory (since Linux 2.6.25; CONFIG_MEMCG)
The memory controller supports reporting and limiting of
process memory, kernel memory, and swap used by cgroups.
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/memory.rst (or
Documentation/cgroup-v1/memory.txt in Linux 5.2 and
earlier).
devices (since Linux 2.6.26; CONFIG_CGROUP_DEVICE)
This supports controlling which processes may create
(mknod) devices as well as open them for reading or
writing. The policies may be specified as allow-lists and
deny-lists. Hierarchy is enforced, so new rules must not
violate existing rules for the target or ancestor cgroups.
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/devices.rst (or
Documentation/cgroup-v1/devices.txt in Linux 5.2 and
earlier).
freezer (since Linux 2.6.28; CONFIG_CGROUP_FREEZER)
The freezer cgroup can suspend and restore (resume) all
processes in a cgroup. Freezing a cgroup /A also causes
its children, for example, processes in /A/B, to be
frozen.
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/freezer-subsystem.rst
(or Documentation/cgroup-v1/freezer-subsystem.txt in Linux
5.2 and earlier).
net_cls (since Linux 2.6.29; CONFIG_CGROUP_NET_CLASSID)
This places a classid, specified for the cgroup, on
network packets created by a cgroup. These classids can
then be used in firewall rules, as well as used to shape
traffic using tc(8). This applies only to packets leaving
the cgroup, not to traffic arriving at the cgroup.
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/net_cls.rst (or
Documentation/cgroup-v1/net_cls.txt in Linux 5.2 and
earlier).
blkio (since Linux 2.6.33; CONFIG_BLK_CGROUP)
The blkio cgroup controls and limits access to specified
block devices by applying IO control in the form of
throttling and upper limits against leaf nodes and
intermediate nodes in the storage hierarchy.
Two policies are available. The first is a proportional-
weight time-based division of disk implemented with CFQ.
This is in effect for leaf nodes using CFQ. The second is
a throttling policy which specifies upper I/O rate limits
on a device.
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/blkio-controller.rst
(or Documentation/cgroup-v1/blkio-controller.txt in Linux
5.2 and earlier).
perf_event (since Linux 2.6.39; CONFIG_CGROUP_PERF)
This controller allows perf monitoring of the set of
processes grouped in a cgroup.
Further information can be found in the kernel source
files
net_prio (since Linux 3.3; CONFIG_CGROUP_NET_PRIO)
This allows priorities to be specified, per network
interface, for cgroups.
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/net_prio.rst (or
Documentation/cgroup-v1/net_prio.txt in Linux 5.2 and
earlier).
hugetlb (since Linux 3.5; CONFIG_CGROUP_HUGETLB)
This supports limiting the use of huge pages by cgroups.
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/hugetlb.rst (or
Documentation/cgroup-v1/hugetlb.txt in Linux 5.2 and
earlier).
pids (since Linux 4.3; CONFIG_CGROUP_PIDS)
This controller permits limiting the number of process
that may be created in a cgroup (and its descendants).
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/pids.rst (or
Documentation/cgroup-v1/pids.txt in Linux 5.2 and
earlier).
rdma (since Linux 4.11; CONFIG_CGROUP_RDMA)
The RDMA controller permits limiting the use of RDMA/IB-
specific resources per cgroup.
Further information can be found in the kernel source file
Documentation/admin-guide/cgroup-v1/rdma.rst (or
Documentation/cgroup-v1/rdma.txt in Linux 5.2 and
earlier).
Creating cgroups and moving processes
A cgroup filesystem initially contains a single root cgroup, '/',
which all processes belong to. A new cgroup is created by
creating a directory in the cgroup filesystem:
mkdir /sys/fs/cgroup/cpu/cg1
This creates a new empty cgroup.
A process may be moved to this cgroup by writing its PID into the
cgroup's cgroup.procs file:
echo $$ > /sys/fs/cgroup/cpu/cg1/cgroup.procs
Only one PID at a time should be written to this file.
Writing the value 0 to a cgroup.procs file causes the writing
process to be moved to the corresponding cgroup.
When writing a PID into the cgroup.procs, all threads in the
process are moved into the new cgroup at once.
Within a hierarchy, a process can be a member of exactly one
cgroup. Writing a process's PID to a cgroup.procs file
automatically removes it from the cgroup of which it was
previously a member.
The cgroup.procs file can be read to obtain a list of the
processes that are members of a cgroup. The returned list of
PIDs is not guaranteed to be in order. Nor is it guaranteed to
be free of duplicates. (For example, a PID may be recycled while
reading from the list.)
In cgroups v1, an individual thread can be moved to another
cgroup by writing its thread ID (i.e., the kernel thread ID
returned by clone(2) and gettid(2)) to the tasks file in a cgroup
directory. This file can be read to discover the set of threads
that are members of the cgroup.
Removing cgroups
To remove a cgroup, it must first have no child cgroups and
contain no (nonzombie) processes. So long as that is the case,
one can simply remove the corresponding directory pathname. Note
that files in a cgroup directory cannot and need not be removed.
Cgroups v1 release notification
Two files can be used to determine whether the kernel provides
notifications when a cgroup becomes empty. A cgroup is
considered to be empty when it contains no child cgroups and no
member processes.
A special file in the root directory of each cgroup hierarchy,
release_agent, can be used to register the pathname of a program
that may be invoked when a cgroup in the hierarchy becomes empty.
The pathname of the newly empty cgroup (relative to the cgroup
mount point) is provided as the sole command-line argument when
the release_agent program is invoked. The release_agent program
might remove the cgroup directory, or perhaps repopulate it with
a process.
The default value of the release_agent file is empty, meaning
that no release agent is invoked.
The content of the release_agent file can also be specified via a
mount option when the cgroup filesystem is mounted:
mount -o release_agent=pathname ...
Whether or not the release_agent program is invoked when a
particular cgroup becomes empty is determined by the value in the
notify_on_release file in the corresponding cgroup directory. If
this file contains the value 0, then the release_agent program is
not invoked. If it contains the value 1, the release_agent
program is invoked. The default value for this file in the root
cgroup is 0. At the time when a new cgroup is created, the value
in this file is inherited from the corresponding file in the
parent cgroup.
Cgroup v1 named hierarchies
In cgroups v1, it is possible to mount a cgroup hierarchy that
has no attached controllers:
mount -t cgroup -o none,name=somename none /some/mount/point
Multiple instances of such hierarchies can be mounted; each
hierarchy must have a unique name. The only purpose of such
hierarchies is to track processes. (See the discussion of
release notification below.) An example of this is the
name=systemd cgroup hierarchy that is used by systemd(1) to track
services and user sessions.
Since Linux 5.0, the cgroup_no_v1 kernel boot option (described
below) can be used to disable cgroup v1 named hierarchies, by
specifying cgroup_no_v1=named.
