Docker /var/lib/docker/devicemapper/devicemapper using 2.7GB

I am a developer in codiva online ide. Every time the user runs a java program, we start a new container. We use devicemapper storage driver because that is the only way we were able to set an upper limit on the container size.

After a few months of usage, we are seeing that the disk usage is increasing, and /var/lib/docker/devicemapper/devicemapper takes around 2.7GB.

  • Docker: Staging multiple containers for one application
  • How to monitor Nodejs app running on Kubernetes
  • Download key with `gpg --recv-key` and simultaneously check fingerprint in a script
  • Can you please suggest on best ways to cleanup docker containers
  • Managing resources (database, elasticsearch, redis, etc) for tests using Docker and Jenkins
  • How to change docker installation directory in centos 7 with docker 1.13 version
  • I have deleted all the docker containers that are not running, and any dangling volumes, unfortunately, the disk usage for /var/lib/docker/devicemapper/devicemapper still show up as 2.7GB.

    What is the best way to reclaim this space? We don’t need any persistent storage, we are okay with deleting all the data, because any data that needs to be stored, we mount the host directory directly, and those files are managed outside of docker.

    sudo docker ps -a
    CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
    

    There are no running or even stopped containers.
    The docker info results here.

    $ sudo docker info
    Containers: 1
    Images: 14
    Server Version: 1.9.1
    Storage Driver: devicemapper
     Pool Name: docker-202:1-143417-pool
     Pool Blocksize: 65.54 kB
     Base Device Size: 107.4 GB
     Backing Filesystem: ext4
     Data file: /dev/loop0
     Metadata file: /dev/loop1
     Data Space Used: 2.591 GB
     Data Space Total: 107.4 GB
     Data Space Available: 1.469 GB
     Metadata Space Used: 2.531 MB
     Metadata Space Total: 2.147 GB
     Metadata Space Available: 1.469 GB
     Udev Sync Supported: true
     Deferred Removal Enabled: false
     Deferred Deletion Enabled: false
     Deferred Deleted Device Count: 0
     Data loop file: /var/lib/docker/devicemapper/devicemapper/data
     Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
     Library Version: 1.02.77 (2012-10-15)
    Execution Driver: native-0.2
    Logging Driver: json-file
    Kernel Version: 3.13.0-76-generic
    Operating System: Ubuntu 14.04.3 LTS
    CPUs: 1
    Total Memory: 992.5 MiB
    Name: ip-172-31-26-50
    ID: NIQ5:2NDW:SB77:ZNZU:UO6G:EYLB:JIW7:SQLL:QLWY:LNRM:SU6P:ZS7X
    WARNING: No swap limit support
    

    There are no dangling or orphaned volumes.

    sudo docker volume ls
    DRIVER              VOLUME NAME
    

    The only image is java:8

    sudo docker images
    REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
    java                8                   289bdffc37d7        3 weeks ago         669.2 MB
    

    I had already followed the instructions is https://lebkowski.name/docker-volumes/ and still it is using unnecessary extra space.

    Edit1:
    I have added the results of docker images -a. This shows a large number unnamed images. The size seems suspicious, as the sum of all these images goes upto 4.5GB, but my entire /var/lib/docker is only around 2.7GB.

    $ sudo docker images -a
    REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
    java                8                   289bdffc37d7        3 weeks ago         669.2 MB
    <none>              <none>              5d7cc49c4372        3 weeks ago         668.8 MB
    <none>              <none>              9e756f3b105a        3 weeks ago         310.9 MB
    <none>              <none>              7615ebf33bb7        3 weeks ago         310.9 MB
    <none>              <none>              ac079291ad51        3 weeks ago         310.9 MB
    <none>              <none>              cd02ed4609aa        3 weeks ago         310.9 MB
    <none>              <none>              07bbd1c2007f        3 weeks ago         310.9 MB
    <none>              <none>              6331ed9a346a        3 weeks ago         310.9 MB
    <none>              <none>              b647ad8e0a08        3 weeks ago         310.9 MB
    <none>              <none>              f23012d4423e        3 weeks ago         310.9 MB
    <none>              <none>              0d3e866c82f3        3 weeks ago         301 MB
    <none>              <none>              c28cbef85c39        3 weeks ago         169.7 MB
    <none>              <none>              3f0d3d140ce1        3 weeks ago         125.1 MB
    <none>              <none>              17bd2058e0c6        3 weeks ago         125.1 MB
    

    On trying to one of them I get the error.
    $ sudo docker rmi 5d7cc49c4372
    Error response from daemon: conflict: unable to delete 5d7cc49c4372 (cannot be forced) – image has dependent child images
    Error: failed to remove images: [5d7cc49c4372]

    On reading about it, is seems that the only required image java:8 depends on these child images.

    $ sudo docker history 289bdffc37d7
    IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
    289bdffc37d7        3 weeks ago         /bin/sh -c /var/lib/dpkg/info/ca-certificates   418.2 kB            
    5d7cc49c4372        3 weeks ago         /bin/sh -c set -x  && apt-get update  && apt-   357.9 MB            
    7615ebf33bb7        3 weeks ago         /bin/sh -c #(nop) ENV CA_CERTIFICATES_JAVA_VE   0 B                 
    9e756f3b105a        3 weeks ago         /bin/sh -c #(nop) ENV JAVA_DEBIAN_VERSION=8u9   0 B                 
    ac079291ad51        3 weeks ago         /bin/sh -c #(nop) ENV JAVA_VERSION=8u91         0 B                 
    07bbd1c2007f        3 weeks ago         /bin/sh -c #(nop) ENV JAVA_HOME=/usr/lib/jvm/   0 B                 
    cd02ed4609aa        3 weeks ago         /bin/sh -c {   echo '#!/bin/sh';   echo 'set    87 B                
    b647ad8e0a08        3 weeks ago         /bin/sh -c #(nop) ENV LANG=C.UTF-8              0 B                 
    6331ed9a346a        3 weeks ago         /bin/sh -c echo 'deb http://httpredir.debian.   61 B                
    f23012d4423e        3 weeks ago         /bin/sh -c apt-get update && apt-get install    9.93 MB             
    0d3e866c82f3        3 weeks ago         /bin/sh -c apt-get update && apt-get install    131.2 MB            
    c28cbef85c39        3 weeks ago         /bin/sh -c apt-get update && apt-get install    44.67 MB            
    3f0d3d140ce1        3 weeks ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B                 
    17bd2058e0c6        3 weeks ago         /bin/sh -c #(nop) ADD file:76679eeb94129df23c   125.1 MB 
    

    That means, the untagged images cannot be removed, and I think by design the total size used should still be ~670MB since docker stores only the diffs for dependent images.

    Please let me know how to reduce the disk usage of devicemapper.

  • Dockerfile strategies for Git
  • Using private svn+bower doesn't work in dockerfile
  • Deploy one image on multi-container docker Elastic BeansTalk Service
  • docker custom network connexion refused between
  • docker push to dockerhub private repo sometimes pushes existing layer again (ubuntu base layer)
  • Docker: mounting volume and run node apps
  • 2 Solutions collect form web for “Docker /var/lib/docker/devicemapper/devicemapper using 2.7GB”

    You have to remove the volumes associated with the container!

    To achieve this, use this command:

    docker rm -v $(docker ps -a -q -f status=exited)
    

    This appears to be a rather old bug with devicemapper, especially if you are using ext3. If you are using ext3, then upgrading to ext4 may solve your issues. The main symptom of the bug is that devicemapper doesn’t free up deleted images and volumes.

    Short of that, you can completely purge /var/lib/docker and do a reinstall since you say you’re ok with data loss.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.