Docker out of disk space on aufs although df shows plenty

Symptoms: cannot start any containers

$ docker run continuumio/miniconda3
docker: Error response from daemon: mkdir /var/run/docker/libcontainerd/a40677f8a1bbaf43d0bd3c1935c4ee69330ac1b0c6bb577ebf66f78edaae2d48: no space left on device.
$ sudo df -h /var/run/docker/libcontainerd
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           201M  748K  200M   1% /run

I have removed exited containers and dangling or old images. Some info (I’m running Ubuntu on AWS with 2 GB of RAM):

  • Docker with nodeBB redis is not configured properly?
  • Symfony 3 and Docker (nginx, php7.1-fpm mysql8) Performances low on Windows
  • Use docker-compose with docker swarm
  • Changing the configuration of zookeeper docker image
  • How can I set an overlay network within docker compose
  • Docker nmap one container from inside another?
  • $ docker info
    Containers: 9
     Running: 7
     Paused: 0
     Stopped: 2
    Images: 278
    Server Version: 1.12.3
    Storage Driver: aufs
     Root Dir: /home/ubuntu/docker/volumes/docker_images/var/aufs
     Backing Filesystem: extfs
     Dirs: 336
     Dirperm1 Supported: false
    Logging Driver: json-file
    Cgroup Driver: cgroupfs
    Plugins:
     Volume: local
     Network: bridge overlay null host
    Swarm: inactive
    Runtimes: runc
    Default Runtime: runc
    Security Options: apparmor
    Kernel Version: 3.13.0-48-generic
    Operating System: Ubuntu 14.04.5 LTS
    OSType: linux
    Architecture: x86_64
    CPUs: 1
    Total Memory: 1.954 GiB
    Name: ip-192-168-1-205
    ID: 2CFI:6RYE:PRVQ:ZOSP:4DNU:O727:IQ57:BL4H:TGMD:L3KY:HD7G:VAUJ
    Docker Root Dir: /home/ubuntu/docker/volumes/docker_images/var/aufs
    Debug Mode (client): false
    Debug Mode (server): false
    Registry: https://index.docker.io/v1/
    WARNING: No swap limit support
    $ uname -a
    Linux ip-192-168-1-205 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
    

    My /var/lib/docker is a symlink to /home/ubuntu/docker/volumes/docker_images/var/

    $ df -h
    Filesystem      Size  Used Avail Use% Mounted on
    udev            996M   12K  996M   1% /dev
    tmpfs           201M  748K  200M   1% /run
    /dev/xvda1      7.8G  2.2G  5.2G  30% /
    none            4.0K     0  4.0K   0% /sys/fs/cgroup
    none            5.0M     0  5.0M   0% /run/lock
    none           1001M  592K 1000M   1% /run/shm
    none            100M     0  100M   0% /run/user
    /dev/xvdf        16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images
    /dev/xvdg        99G  3.3G   91G   4% /home/ubuntu/docker/volumes/hub
    none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/e693c6c76faf6bcc61b262cda110985f491f3bf8dcb3557d0bce857027759fc8
    none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/9915d2a8761471958ba4268a15cca6149ddedfdfa9007cfc718536cd4189e9ae
    none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/1ff065d23fd79042d62697c678f980eb5e7bb08493422e905835b064dad96e66
    none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/e94aed0d3162e222de6c96a4e897d70e82f1fd24dbc8a57aa4a5c4ddc15fc827
    none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/9aa83823063aab902fa82515d7cb5c6ee3d3e64a789be23367d70d445a319fb6
    none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/a800385412c324efd6bdd7df48aee0c5db7f69a8d333b60b9ae96839b05339d2
    none             16G  7.5G  7.4G  51% /home/ubuntu/docker/volumes/docker_images/var/aufs/mnt/e109c3c4aa7acefcc69f2436275a9d61bdf8b1b968e150d9063bbbd276e5cf3c
    

    Any ideas how to resolve this? My build system is stalled :/ I haven’t tried restarting docker daemon as it requires some manual effort to restart every container.

    Update:

    $ df -i
    Filesystem      Inodes  IUsed   IFree IUse% Mounted on
    udev            254821    398  254423    1% /dev
    tmpfs           256060 256060       0  100% /run
    /dev/xvda1      524288  86504  437784   17% /
    none            256060     11  256049    1% /sys/fs/cgroup
    none            256060      1  256059    1% /run/lock
    none            256060     25  256035    1% /run/shm
    none            256060      2  256058    1% /run/user
    /dev/xvdf      1048576 150320  898256   15% /home/ubuntu/docker/volumes/docker_images
    /dev/xvdg      6553600   3207 6550393    1% /home/ubuntu/docker/volumes/hub
    

    tmpfs 100% used! Next I’ll need to figure out what can I delete.

    Update 2: I was hit by https://github.com/docker/docker/issues/22513

    My solution (run as root):

    # ifconfig | grep veth | cut -d' ' -f1 | xargs echo
    veth087c8e6 veth1fb6674 veth5b8c14c veth801416c veth8a51820
    
    # find /run/network -name 'ifstate.*' | egrep -w -v -e `ifconfig | grep veth | cut -d' ' -f1 | xargs echo | tr ' ' '|'` | xargs -n1 rm
    

  • boot2docker resulting in “Cannot connect to the Docker daemon. Is 'docker -d' running on this host?”
  • When to use auto mapped Docker data volume
  • Cannot connect from MySQL Workbench to dockerized MySQL server
  • [pacemaker]Docker compose autostart
  • Trying to use webpack-dev-server in container, getting errors that are not there otherwise
  • Strategy for versioning Docker images linked to other contains via docker compose
  • One Solution collect form web for “Docker out of disk space on aufs although df shows plenty”

    As I already described in the question (I’m the OP) the issue was running out of inodes due to kernel bug as described at https://github.com/docker/docker/issues/22513

    My solution was to delete those ifstate files which didn’t correspond to any of the interfaces listed by ifconfig:

    # ifconfig | grep veth | cut -d' ' -f1 | xargs echo
    veth087c8e6 veth1fb6674 veth5b8c14c veth801416c veth8a51820
    
    # find /run/network -name 'ifstate.*' | egrep -w -v -e `ifconfig | grep veth | cut -d' ' -f1 | xargs echo | tr ' ' '|'` | xargs -n1 rm
    
    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.