Gitlab docker registry pull failed EOF

1. Introduction

I’ve got Gitlab setup using a docker image https://github.com/sameersbn/docker-gitlab (sameersbn/docker-gitlab) and want to setup gitlab private registry.

1.1 Problem

All of this is behind reverse proxy load balancer using nginx. I am able to push into registry, but pull has some problem I am not able to resolve.

  • [pacemaker]Docker compose autostart
  • Docker image seemingly having missing layers
  • How to run HHVM in Docker as a daemon?
  • Which capabilities can I drop in a Docker Nginx container?
  • How to manage multiple backend stacks for development?
  • How should a container running in network mode bridge communicate with another container running in host mode?
  • The output tells me retrying in 5sec and then increases over time, but after about minute it outputs Download complete on some layers. When all layers outputs Download complete. Pull results then like Unexpected EOF.

    Infrastructure goes like this:


    Proxy load balancer -----> gitlab registry
    |
    |
    Another proxy load balancer -----> Gitlab

    2. Terminal outputs

    Docker login and push works fine. The problem is with pull.

    2.1 Docker pull

    2.1.1 expected result

    root@server:~# docker pull registry.dalten.com/docker/php5.6-cli/image
    Using default tag: latest
    latest: Pulling from docker/php5.6-cli/image
    10a267c67f42: Already exists 
    370377701f89: Pulling fs layer 
    455c73a122bc: Pulling fs layer 
    2cdcd82f5338: Pulling fs layer 
    3ac634f20449: Waiting 
    a19eeb6546ae: Waiting 
    4ca536f5a464: Waiting 
    4788559f39db: Waiting 
    cc1917e72a80: Waiting 
    42feb0ca362a: Pull complete 
    f74f29b91bbc: Pull complete 
    d104213bfd22: Pull complete 
    6db9d2d35d48: Pull complete 
    806b8ac2f0b7: Pull complete 
    661bb76178ac: Pull complete 
    3727be25d45f: Pull complete 
    04fed213644a: Pull complete 
    a78f18e044f7: Pull complete 
    849a2f576c66: Pull complete 
    259300decf26: Pull complete 
    0d9f42b121be: Pull complete 
    2483d273017a: Pull complete 
    3dfaf57e1d54: Pull complete 
    92fbf8438206: Pull complete 
    7d5100ec575c: Pull complete 
    52b1dff7320d: Pull complete 
    500fa7f84392: Pull complete 
    aff937c2773b: Pull complete 
    92f8bc1201f3: Pull complete 
    Digest: sha256:d3d06d2cebd333af21565639503ddded64db47d5641ee9ff93167200b5fcad7a
    Status: Downloaded newer image for registry.dalten.com/docker/php5.6-cli/image:latest
    

    2.1.2 Actual behavior

    docker pull registry.dalten.com/docker/php5.6-cli/image
    Using default tag: latest
    latest: Pulling from docker/php5.6-cli/image
    10a267c67f42: Already exists 
    370377701f89: Pulling fs layer 
    455c73a122bc: Pulling fs layer 
    2cdcd82f5338: Pulling fs layer 
    3ac634f20449: Waiting 
    a19eeb6546ae: Waiting 
    a19eeb6546ae: Downloading 4.398 MB/4.398 MB
    4788559f39db: Downloading 102.6 MB/102.6 MB
    cc1917e72a80: Download complete 
    42feb0ca362a: Download complete 
    f74f29b91bbc: Download complete 
    d104213bfd22: Download complete 
    6db9d2d35d48: Download complete 
    806b8ac2f0b7: Download complete 
    661bb76178ac: Download complete 
    3727be25d45f: Download complete 
    04fed213644a: Download complete 
    a78f18e044f7: Download complete 
    849a2f576c66: Download complete 
    259300decf26: Download complete 
    0d9f42b121be: Download complete 
    2483d273017a: Downloading 24.12 MB/24.12 MB
    3dfaf57e1d54: Download complete 
    92fbf8438206: Download complete 
    7d5100ec575c: Download complete 
    52b1dff7320d: Download complete 
    500fa7f84392: Download complete 
    aff937c2773b: Download complete 
    92f8bc1201f3: Download complete 
    unexpected EOF
    

    2.3 Docker login

    sudo docker login registry.dalten.com
    Username (xxxxxx): 
    Password: 
    Login Succeeded
    

    2.4 Docker push

    sudo docker push registry.dalten.com/docker/php5.6-cli/image
    The push refers to a repository [registry.dalten.com/docker/php5.6-cli/image]
    d8b353eb3025: Pushed 
    f2e85bc0b7b1: Pushed 
    fc9e1e5e38f7: Pushed 
    fe9a3f9c4559: Pushed 
    6a8bf8c8edbd: Pushed 
    latest: digest: sha256:ea1d854d38be82f54d39efe2c67000bed1b03348bcc2f3dc094f260855dff368 size: 1357
    

    3. Configurations

    Configurations contains docker-compose.yml files which I use for running services. Gitlab is running fine. Every configuration works. The only problem is with registry.

    Another thing is that registry works while in insecured state. The problem must be hidden somewhere inside load balancer (nginx) configuration. (Some header has not been sent, limit exceeded, or so)

    3.1 Gitlab Docker-compose

    Note: I cutted compose of irrelevant information

    version: '2'
    services:
      registry:
        image: registry:latest
        environment:
          REGISTRY_AUTH_TOKEN_ISSUER: gitlab-issuer
          REGISTRY_AUTH_TOKEN_REALM: https://git.dalten.cz/jwt/auth
          REGISTRY_AUTH_TOKEN_SERVICE: container_registry
          REGISTRY_LOG_LEVEL: debug
          REGISTRY_STORAGE_DELETE_ENABLED: 'true'
          REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certs/fullchain.pem
          REGISTRY_HTTP_SECRET: secret
        volumes:
        - /mnt/storage/srv/gitlab1/data/shared/registry:/registry
        - /mnt/storage/srv/gitlab1/certs:/certs
        ports:
        - 5000:5000/tcp
      gitlab:
        image: sameersbn/gitlab:latest
        hostname: git.dalten.cz
        environment:
          GITLAB_REGISTRY_ENABLED: 'true'
          GITLAB_REGISTRY_HOST: registry.dalten.com
          GITLAB_REGISTRY_API_URL: http://registry:5000
          GITLAB_REGISTRY_ISSUER: gitlab-issuer
          GITLAB_REGISTRY_CERT_PATH: /certs/fullchain.pem
          GITLAB_REGISTRY_KEY_PATH: /certs/privkey.pem
        volumes:
        - /mnt/storage/srv/gitlab1/certs:/certs
        - /mnt/storage/srv/gitlab1/data:/home/git/data/
        ports:
        - 8643:80/tcp
        - 1022:22/tcp
    

    3.2 Nginx registry docker-compose

    version: '2'
    services:
      nginx:
        image: sameersbn/nginx:1.10.3
        volumes:
        - /srv/nginx/sites-enabled:/etc/nginx/sites-enabled
        - /mnt/storage/lb/letsencrypt:/etc/nginx/ssl
        ports:
        - 80:80/tcp
        - 443:443/tcp
    

    3.2.1 Nginx Virtual host file

    I use Nginx for my reverse proxy configuration (Load Balancer). There are stored all certificates for https traffic.

        # For versions of Nginx > 1.3.9 that include chunked transfer encoding
    # support Replace with appropriate values where necessary
    upstream docker-registry {
     server 10.126.0.242:5000;
    }
    
    server {
      listen 80;
      server_name registry.dalten.com
      return 301 https://$http_host:$request_uri;
      access_log /var/log/nginx/access.log;
      error_log  /var/log/nginx/error.log;
    }
    
    server {
     listen 443 default_server;
     server_name registry.dalten.com
    
     add_header Docker-Distribution-Api-Version registry/2.0 always;
    
     ssl on;
     ssl_certificate /etc/nginx/ssl/live/registry.dalten.com/fullchain.pem;
     ssl_certificate_key /etc/nginx/ssl/live/registry.dalten.com/privkey.pem;
    
      ssl_session_timeout 5m;
    
     client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image
    uploads
     # required to avoid HTTP 411: see Issue #1486
     # (https://github.com/dotcloud/docker/issues/1486)
     chunked_transfer_encoding on;
     proxy_set_header X-Original-URI $request_uri;
     proxy_set_header Docker-Distribution-Api-Version registry/2.0;
    
    access_log /var/log/nginx/access.log;
     error_log /var/log/nginx/error.log;
    
    location /
    {
         # let Nginx know about our auth file
         proxy_pass http://docker-registry;
         proxy_set_header Host $host; # required for docker client's sake
         proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $scheme;
     }
     location /_ping {
         proxy_pass http://docker-registry;
     }
     location /v1/_ping {
         proxy_pass http://docker-registry;
     }
     location /v2/ {
         # To add basic authentication to v2 use auth_basic setting plus
         # add_header
         add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;
         proxy_pass http://docker-registry;
         proxy_set_header Host $http_host; # required for docker client's sake
         proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $scheme;
         proxy_read_timeout 900;
     }
    }
    

    3. Logs

    If you require any more logs feel free to ask!

    3.1 Docker pull

    4. Thank you for all your help!

  • Docker Exec Cronjobs Filling Ec2 instance disk space, why?
  • Am I using flyway wrong?
  • How to access xdebug session on host machine for unit tests that run inside docker container?
  • Docker not running on OS/X
  • Running Cloudant as docker container with docker compose
  • How do I use map[string]struct{}
  • One Solution collect form web for “Gitlab docker registry pull failed EOF”

    Problem solved by adding proxy_max_temp_file_size 0; inside nginx virtual host configuration!

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.