com.spotify.docker.client.DockerRequestException: Request error: DELETE unix://localhost:80/v1.12/containers/…: 409

I am working on a Java application which deploys web artifacts in Apache Tomcat Docker Containers with the use of Google Kubernetes. I am using https://github.com/spotify/docker-client in order to carry out Docker Image and Container handling activities and https://github.com/fabric8io/fabric8/tree/master/components/kubernetes-api for Kubernetes related functionalities.

In this application, I have added a functionality which enables the user to remove a web artifact which the user deploys.

  • Dockerfile created for JBoss with MySQL image
  • Docker ignores ENTRYPOINT
  • How can you limit which users have Docker exec access in a shared environment?
  • minikube : not able to connect a locally deployed nginx service
  • An error occurred trying to connect - docker when not using sudo
  • Compiling and running in different containers
  • When removing I,

    1. delete the Kubernetes replication controller which I use to generate the desired number of pod replicas

    2. separately delete off the replica pods (as pods are not deleted automatically when the replication controller is deleted in the corresponding method in the Java API)

    3. delete off the corresponding Service created

    4. delete off the Docker Containers corresponding to the pods deleted off

    5. finally, remove the Docker Image used for the deployment

    Following code shows the removal functionality implemented:

    public boolean remove(String tenant, String appName) throws WebArtifactHandlerException {
            String componentName = generateKubernetesComponentName(tenant, appName);
            final int singleImageIndex = 0;
            try {
                if (replicationControllerHandler.getReplicationController(componentName) != null) {
                    String dockerImage = replicationControllerHandler.getReplicationController(componentName).getSpec()
                            .getTemplate().getSpec().getContainers().get(singleImageIndex).getImage();
                    List<String> containerIds = containerHandler.getRunningContainerIdsByImage(dockerImage);
                    replicationControllerHandler.deleteReplicationController(componentName);
                    podHandler.deleteReplicaPods(tenant, appName);
                    serviceHandler.deleteService(componentName);
                    Thread.sleep(OPERATION_DELAY_IN_MILLISECONDS);
                    containerHandler.deleteContainers(containerIds);
                    imageBuilder.removeImage(tenant, appName, getDockerImageVersion(dockerImage));
                    return true;
                } else {
                    return false;
                }
            } catch (Exception exception) {
                String message = String.format("Failed to remove web artifact[artifact]: %s",
                        generateKubernetesComponentName(tenant, appName));
                LOG.error(message, exception);
                throw new WebArtifactHandlerException(message, exception);
            }
        } 
    

    Implementation of the Docker Container deletion functionality is as follows:

    public void deleteContainers(List<String> containerIds) throws WebArtifactHandlerException {
            try {
                for (String containerId : containerIds) {
                    dockerClient.removeContainer(containerId);
                    Thread.sleep(OPERATION_DELAY_IN_MILLISECONDS);
                }
            } catch (Exception exception) {
                String message = "Could not delete the Docker Containers.";
                LOG.error(message, exception);
                throw new WebArtifactHandlerException(message, exception);
            }
        }
    

    In the above case although the execution of the desired function takes place without any sort of issue, at certain instances I tend to get the following exception.

    Sep 11, 2015 3:57:28 PM org.apache.poc.webartifact.WebArtifactHandler remove
    SEVERE: Failed to remove web artifact[artifact]: app-wso2-com
    org.apache.poc.miscellaneous.exceptions.WebArtifactHandlerException: Could not delete the Docker Containers.
        at org.apache.poc.docker.JavaWebArtifactContainerHandler.deleteContainers(JavaWebArtifactContainerHandler.java:80)
        at org.apache.poc.webartifact.WebArtifactHandler.remove(WebArtifactHandler.java:206)
        at org.apache.poc.Executor.process(Executor.java:222)
        at org.apache.poc.Executor.main(Executor.java:46)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
    Caused by: com.spotify.docker.client.DockerRequestException: Request error: DELETE unix://localhost:80/v1.12/containers/af05916d2bddf73dcf8bf41c6ea7f5f3b859c90b97447a8248ffa7b5b3968691: 409
        at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1061)
        at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1021)
        at com.spotify.docker.client.DefaultDockerClient.removeContainer(DefaultDockerClient.java:544)
        at com.spotify.docker.client.DefaultDockerClient.removeContainer(DefaultDockerClient.java:535)
        at org.wso2.carbon6.poc.docker.JavaWebArtifactContainerHandler.deleteContainers(JavaWebArtifactContainerHandler.java:74)
        ... 8 more
    Caused by: com.spotify.docker.client.shaded.javax.ws.rs.ClientErrorException: HTTP 409 Conflict
        at org.glassfish.jersey.client.JerseyInvocation.createExceptionForFamily(JerseyInvocation.java:991)
        at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:975)
        at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:795)
        at org.glassfish.jersey.client.JerseyInvocation.access$500(JerseyInvocation.java:91)
        at org.glassfish.jersey.client.JerseyInvocation$5.completed(JerseyInvocation.java:756)
        at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:189)
        at org.glassfish.jersey.client.ClientRuntime.access$300(ClientRuntime.java:74)
        at org.glassfish.jersey.client.ClientRuntime$1.run(ClientRuntime.java:171)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:320)
        at org.glassfish.jersey.client.ClientRuntime$2.run(ClientRuntime.java:201)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    

    I searched a large number of sources for any help for this but still I wasn’t able to avoid it in all instances, I execute this functionality.

    At the beginning I tended to get this issue more often than now, but allowing the executing thread to sleep at the end of deleting each Docker Container and before deleting any Docker Containers, gradually reduced the number of instances I am getting this issue.

    Is sleeping the thread the ultimate solution for this issue or is there any other reason for this issue to pop and a solution that can help me to avoid this exception? Any help is greatly appreciated.

  • Can't set shell variable in Dockerfile
  • docker container naming pattern
  • Unable to locate package while building Docker image
  • HTTPS request from Docker container to LAN IP fails, but curl works
  • wget (or any connection) inside docker can't connect to servers own ip address
  • Docker attach delay
  • One Solution collect form web for “com.spotify.docker.client.DockerRequestException: Request error: DELETE unix://localhost:80/v1.12/containers/…: 409”

    Unfortunately I’m not familiar with the Java client library.

    My suggestion would be to try using the regular command-line client (kubectl). If that works, then you know the problem is in the Java client library or your usage of it. If using the command line client doesn’t work, then there will be more people who can help you (since a lot more people are familiar with the command-line client than with the Java client library).

    In other words
    % kubectl delete pods … # –cascade=true by default
    % kubectl delete services …

    I’m curious why you need step (4) and (5). Step (4) should happen automatically when you delete the pod, and step (5) should happen automatically in the background.

    If the two lines of “kubectl delete” work, then the problem is with the Java client library or your usage of it. As a starting point I would suggest remove calling deleteContainers() and removeImage() from your Java code and see if that helps. I think those steps are unnecessary.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.