crun: cannot set memory swappiness with cgroupv2: OCI runtime error

2024-05-10

When I was working on deploying Rust on arm64 I encountered the following error when trying to restart Podman container via Github Actions Webhook and Portainer:

{"message":"Error updating service","details":"start container error: Error response from daemon: crun: cannot set memory swappiness with cgroupv2: OCI runtime error"}

I googled and tried all solutions I met in the Internet. Unfortunately, none of them worked for me. Finally, I found the solution by myself. The error was caused by the runtime of the container. The default runtime of the container was crun. I think, this is the default runtime for Podman. And it does not seem to work well with cgroupsv2. I tried a few ways of downgrading the host machine to cgroupv1, but nothing worked. Eventually, I had to change the runtime to runc (that must be the default runtime for Docker, by the way) and the error was gone.

dnf install runc
runc --version

Now you can change the runtime of the container to runc via Portainer or Podman CLI: Container -> Runtime & Resources -> Runtime -> runc

You can check the runtime of the container via Podman CLI:

podman ps
podman inspect 869289ae4c2b |grep Runtime

The error was gone and the container was running without any problem.

Note on Loginctl#

You might experience also this error: runc create failed: unable to start container process: unable to apply cgroup configuration: unable to start unit

For fixing it, I used this:

loginctl enable-linger 1000

Understanding Systemd and User Sessions#

Systemd manages user sessions through "lingering." Normally, when a user logs out, their processes are terminated, and the session ends. This behavior ensures that resources are freed and not left consuming system resources unnecessarily.

What Does loginctl enable-linger Do?#

The loginctl enable-linger command alters this behavior for a specified user. When lingering is enabled for a user, their processes are allowed to continue running in the background even after they have logged out. Here’s what it involves:

  • Persistent User Processes: Enabling lingering allows user processes to continue running after the user has logged out. This is particularly useful for long-running tasks or services that should not be interrupted by the user’s logout.
  • Implications for Docker and Containers: For Docker, especially when it is running tasks as a non-root user or when containers are set to auto-start under a specific user, enabling lingering ensures that these processes can start or keep running without an active user session. This is crucial for Docker operations initiated by user-level systemd services or background jobs.

the initial issue with the container mentioned problems likely related to the management of cgroups and possibly session-related limitations on process lifetimes. The error suggested that Docker or the container runtime was trying to initiate operations that required persistent processes, but was unable to maintain them due to the default session handling behavior.

By enabling lingering for the user (user with ID 1000):

  • Persistent Sessions for Docker: This ensured that any Docker processes started by this user could continue running in the background independently of the user’s login state.
  • Resource Allocation and Management: It allowed Docker to manage resources such as cgroups without interruptions or limitations imposed by the session ending upon user logout.

Conclusion#

If you encounter the error crun: cannot set memory swappiness with cgroupv2: OCI runtime error when trying to restart a container, you might need to change the runtime of the container to runc to solve the problem.

Backlinks: