How can I inspect the file system of a failed `docker build`?
I’m trying to build a new Docker image for our development process, using
cpanm to install a bunch of Perl modules as a base image for various projects.
While developing the Dockerfile,
cpanm returns a failure code because some of the modules did not install cleanly.
I’m fairly sure I need to get
apt to install some more things.
My question is, where can I find the
/.cpanm/work directory quoted in the output, in order to inspect the logs? In the general case, how can I inspect the file system of a failed
docker build command?
Morning edit After biting the bullet and running a
find I discovered
Is this reliable, or am I better off building a “bare” container and running stuff manually until I have all the things I need?
4 Solutions collect form web for “How can I inspect the file system of a failed `docker build`?”
Everytime docker successfully executes a
RUN command from a Dockerfile, a new layer in the image filesystem is committed. Conveniently you can use those layers ids as images to start a new container.
Take the following Dockerfile:
FROM busybox RUN echo 'foo' > /tmp/foo.txt RUN echo 'bar' >> /tmp/foo.txt
and build it:
$ docker build -t so-2622957 . Sending build context to Docker daemon 47.62 kB Step 1/3 : FROM busybox ---> 00f017a8c2a6 Step 2/3 : RUN echo 'foo' > /tmp/foo.txt ---> Running in 4dbd01ebf27f ---> 044e1532c690 Removing intermediate container 4dbd01ebf27f Step 3/3 : RUN echo 'bar' >> /tmp/foo.txt ---> Running in 74d81cb9d2b1 ---> 5bd8172529c1 Removing intermediate container 74d81cb9d2b1 Successfully built 5bd8172529c1
You can now start a new container from
$ docker run --rm 00f017a8c2a6 cat /tmp/foo.txt cat: /tmp/foo.txt: No such file or directory $ docker run --rm 044e1532c690 cat /tmp/foo.txt foo $ docker run --rm 5bd8172529c1 cat /tmp/foo.txt foo bar
of course you might want to start a shell to explore the filesystem and try out commands:
$ docker run --rm -it 044e1532c690 sh / # ls -l /tmp total 4 -rw-r--r-- 1 root root 4 Mar 9 19:09 foo.txt / # cat /tmp/foo.txt foo
When one of the Dockerfile command fails, what you need to do is to look for the id of the preceding layer and run a shell in a container created from that id:
docker run --rm -it <id_last_working_layer> bash -il
Once in the container:
- try the command that failed, and reproduce the issue
- then fix the command and test it
- finally update your Dockerfile with the fixed command
If you really need to experiment in the actual layer that failed instead of working from the last working layer, see Drew’s answer below.
The top answer works in the case that you want to examine the state immediately prior to the failed command.
However, the question asks how to examine the state of the failed container itself. In my situation, the failed command is a build that takes several hours, so rewinding prior to the failed command and running it again takes a long time and is not very helpful.
The solution here is to find the container that failed:
$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6934ada98de6 42e0228751b3 "/bin/sh -c './utils/" 24 minutes ago Exited (1) About a minute ago sleepy_bell
Commit it to an image:
$ docker commit 6934ada98de6 sha256:7015687976a478e0e94b60fa496d319cdf4ec847bcd612aecf869a72336e6b83
And then run the image [if necessary, running bash]:
$ docker run -it 7015687976a4 [bash -il]
Now you are actually looking at the state of the build at the time that it failed, instead of at the time before running the command that caused the failure.
What I would do is comment out the Dockerfile below and including the offending line. Then you can run the container and run the docker commands by hand, and look at the logs in the usual way. E.g. if the Dockerfile is
RUN foo RUN bar RUN baz
and it’s dying at bar I would do
RUN foo # RUN bar # RUN baz
$ docker build -t foo . $ docker run -it foo bash container# bar ...grep logs...
Docker caches the entire filesystem state after each successful
- to examine the latest state before your failing
RUNcommand, comment it out in the Dockerfile (as well as any and all subsequent
RUNcommands), then run
- to examine the state after the failing
RUNcommand, simply add
|| trueto it to force it to succeed; then proceed like above (keep any and all subsequent
RUNcommands commented out, run
Tada, no need to mess with Docker internals or layer IDs, and as a bonus Docker automatically minimizes the amount of work that needs to be re-done.