SSL certificate verification fails inside docker container on specific server

I’m running into a strange problem with certificates that I can’t figure out how to debug. When I run wget inside of a docker container on one specific server it cannot verify certificates. The same wget works fine on the server machine itself (outside docker) and it works inside that same docker container on different servers.

Here’s the setup for the docker container:

  • How can I get a Docker image's label if the label name has a “.” in it?
  • Docker - Not able to telnet Linked Container
  • making requests to localhost from inside docker container
  • Dockerfile error with AWS Elastic Beanstalk; works otherwise, are there differences?
  • Setting up a remote private Docker registry
  • Can SSH in but not out of docker container: network unreachable
  • docker run --rm -ti debian:jessie bash
    apt-get update
    apt-get install wget
    wget https://google.com
    

    The response is:

    converted 'https://google.com' (ANSI_X3.4-1968) -> 'https://google.com' (UTF-8)
    --2016-06-22 14:22:02--  https://google.com/
    Resolving google.com (google.com)... 216.58.217.142, 2607:f8b0:4004:807::200e
    Connecting to google.com (google.com)|216.58.217.142|:443... connected.
    ERROR: The certificate of 'google.com' is not trusted.
    ERROR: The certificate of 'google.com' hasn't got a known issuer.
    The certificate's owner does not match hostname 'google.com'
    

    Since this same process works on other servers, it seems like the problem could only be some certificate problem on that server itself. But I must be confused: why should the certificates on the server itself have anything to do with what’s happening inside of the docker container?

    I would really appreciate any insight into this, in particular any debugging steps I can take to understand the problem better.

  • AWS Cloudwatch logs with Docker Container - NoCredentialProviders: no valid providers in chain
  • Create new container with interactive shell
  • Docker DNS Error
  • How to know the reason why a docker container exits?
  • Handling software updates in Docker images
  • Docker inspect with unix:///var/run/docker.sock (in Java or Scala)
  • 3 Solutions collect form web for “SSL certificate verification fails inside docker container on specific server”

    Docker uses iptables.

    If you have iptable rules set up it’s possible to direct EVERY https request to your own running server.

    If you are, for example, running jenkins locally and using iptables to redirect 443 to default 8080 port than all your container traffic to port 443 ports will be redirected to that local jenkins server which will be unable to verify your certificate.
    We ran into this problem when using Jenkins to build our docker images. our jenkins used iptables to get around running jenkins as root.

    It seems that the certificates are out of date inside the jessie image.

    try apt-get install ca-certificates before the wget

    This worked fine for me, though to be safe, make sure your “ca-certificates” package is up to date. Most likely, you have some kind of security device on the network that is inspecting the traffic, and to do so, decrypting and encrypting with it’s own certificate. Here’s the certificate I get from my own testing:

    bash$ openssl s_client -showcerts -connect www.google.com:443
    CONNECTED(00000003)
    depth=3 C = US, O = Equifax, OU = Equifax Secure Certificate Authority
    verify return:1
    depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
    verify return:1
    depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
    verify return:1
    depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = www.google.com
    verify return:1
    ---
    Certificate chain
     0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
       i:/C=US/O=Google Inc/CN=Google Internet Authority G2
    -----BEGIN CERTIFICATE-----
    MIIEgDCCA2igAwIBAgIIdnEF+1C/AZowDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE
    BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
    cm5ldCBBdXRob3JpdHkgRzIwHhcNMTYwNjA4MTIzNzI5WhcNMTYwODMxMTIzMDAw
    WjBoMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwN
    TW91bnRhaW4gVmlldzETMBEGA1UECgwKR29vZ2xlIEluYzEXMBUGA1UEAwwOd3d3
    Lmdvb2dsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCxYx0X
    auhoQ4cobo6J3UMUNCbzKmJ/XSzDB5RLtjvbvtDfCMHm8hO91vvlcKRRrwqdYpiw
    2zUcPDyjwOrZZsJlQglQw/rRpbfQQ6aKsKQWiT3sAIz5joXXi/622YhhGAAdyGGy
    /tzsQkW0IqWAIFLFBbHWMvgDmvMwacps34B80U+p5Iq2xx2xHegl0RMb4HfSEpW/
    H4A0MKvYR6uZZEEr39E+4R6IY9HSv1ZDq8csspyWjXpaIxd6ZD6+lGwvgyszQtNa
    0aEP9+tNhF7jPRD5TedfvyOz81dUCvE0O2E+nfripG2gNBm4r5N6XWUH/lvoopaR
    00eE2fKpk/fvZgZXAgMBAAGjggFLMIIBRzAdBgNVHSUEFjAUBggrBgEFBQcDAQYI
    KwYBBQUHAwIwGQYDVR0RBBIwEIIOd3d3Lmdvb2dsZS5jb20waAYIKwYBBQUHAQEE
    XDBaMCsGCCsGAQUFBzAChh9odHRwOi8vcGtpLmdvb2dsZS5jb20vR0lBRzIuY3J0
    MCsGCCsGAQUFBzABhh9odHRwOi8vY2xpZW50czEuZ29vZ2xlLmNvbS9vY3NwMB0G
    A1UdDgQWBBStL+4j1/n+vGwj3sL861LWCYDUGTAMBgNVHRMBAf8EAjAAMB8GA1Ud
    IwQYMBaAFErdBhYbvPZotXb1gba7Yhq6WoEvMCEGA1UdIAQaMBgwDAYKKwYBBAHW
    eQIFATAIBgZngQwBAgIwMAYDVR0fBCkwJzAloCOgIYYfaHR0cDovL3BraS5nb29n
    bGUuY29tL0dJQUcyLmNybDANBgkqhkiG9w0BAQsFAAOCAQEAK1BpCyAbID8gpwWI
    bQReJv81H/qvYvaaOFa7PLnqHhaAZmzjV1tkCsVB60IgsBDoNuPdtJ5klpxV+njs
    VhDnxneaHL21zwUCuZvNVyYL9VCSYGWV1iNe6PtYYtbWt7of6bEiwZsSuPWaRuRp
    YcvJH+mpv/EIDw1shU5UK3FpvnSHEH2jrs2psnC4BYSovT3pH2nxTCpiLiya1UNn
    +qtqiCJyDKhEV6f1j+Awg/Fr+tVCZLjHcSmGqL3DNcHbCUalXrq4EwVK3Pg8lghU
    Dm3e0J3EcjgMacIg+RP+2pOM5GvIwQ6BrKbmcnTjqsjcG1tQEV0ZSb6hx2bGYhc2
    3TG+rQ==
    -----END CERTIFICATE-----
     1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
       i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
    -----BEGIN CERTIFICATE-----
    MIID8DCCAtigAwIBAgIDAjqDMA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNVBAYTAlVT
    MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
    YWwgQ0EwHhcNMTMwNDA1MTUxNTU2WhcNMTYxMjMxMjM1OTU5WjBJMQswCQYDVQQG
    EwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzElMCMGA1UEAxMcR29vZ2xlIEludGVy
    bmV0IEF1dGhvcml0eSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
    AJwqBHdc2FCROgajguDYUEi8iT/xGXAaiEZ+4I/F8YnOIe5a/mENtzJEiaB0C1NP
    VaTOgmKV7utZX8bhBYASxF6UP7xbSDj0U/ck5vuR6RXEz/RTDfRK/J9U3n2+oGtv
    h8DQUB8oMANA2ghzUWx//zo8pzcGjr1LEQTrfSTe5vn8MXH7lNVg8y5Kr0LSy+rE
    ahqyzFPdFUuLH8gZYR/Nnag+YyuENWllhMgZxUYi+FOVvuOAShDGKuy6lyARxzmZ
    EASg8GF6lSWMTlJ14rbtCMoU/M4iarNOz0YDl5cDfsCx3nuvRTPPuj5xt970JSXC
    DTWJnZ37DhF5iR43xa+OcmkCAwEAAaOB5zCB5DAfBgNVHSMEGDAWgBTAephojYn7
    qwVkDBF9qn1luMrMTjAdBgNVHQ4EFgQUSt0GFhu89mi1dvWBtrtiGrpagS8wDgYD
    VR0PAQH/BAQDAgEGMC4GCCsGAQUFBwEBBCIwIDAeBggrBgEFBQcwAYYSaHR0cDov
    L2cuc3ltY2QuY29tMBIGA1UdEwEB/wQIMAYBAf8CAQAwNQYDVR0fBC4wLDAqoCig
    JoYkaHR0cDovL2cuc3ltY2IuY29tL2NybHMvZ3RnbG9iYWwuY3JsMBcGA1UdIAQQ
    MA4wDAYKKwYBBAHWeQIFATANBgkqhkiG9w0BAQsFAAOCAQEAqvqpIM1qZ4PtXtR+
    3h3Ef+AlBgDFJPupyC1tft6dgmUsgWM0Zj7pUsIItMsv91+ZOmqcUHqFBYx90SpI
    hNMJbHzCzTWf84LuUt5oX+QAihcglvcpjZpNy6jehsgNb1aHA30DP9z6eX0hGfnI
    Oi9RdozHQZJxjyXON/hKTAAj78Q1EK7gI4BzfE00LshukNYQHpmEcxpw8u1VDu4X
    Bupn7jLrLN1nBz/2i8Jw3lsA5rsb0zYaImxssDVCbJAJPZPpZAkiDoUGn8JzIdPm
    X4DkjYUiOnMDsWCOrmji9D6X52ASCWg23jrW4kOVWzeBkoEfu43XrVJkFleW2V40
    fsg12A==
    -----END CERTIFICATE-----
     2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
       i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
    -----BEGIN CERTIFICATE-----
    MIIDfTCCAuagAwIBAgIDErvmMA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT
    MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0
    aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDIwNTIxMDQwMDAwWhcNMTgwODIxMDQwMDAw
    WjBCMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEbMBkGA1UE
    AxMSR2VvVHJ1c3QgR2xvYmFsIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
    CgKCAQEA2swYYzD99BcjGlZ+W988bDjkcbd4kdS8odhM+KhDtgPpTSEHCIjaWC9m
    OSm9BXiLnTjoBbdqfnGk5sRgprDvgOSJKA+eJdbtg/OtppHHmMlCGDUUna2YRpIu
    T8rxh0PBFpVXLVDviS2Aelet8u5fa9IAjbkU+BQVNdnARqN7csiRv8lVK83Qlz6c
    JmTM386DGXHKTubU1XupGc1V3sjs0l44U+VcT4wt/lAjNvxm5suOpDkZALeVAjmR
    Cw7+OC7RHQWa9k0+bw8HHa8sHo9gOeL6NlMTOdReJivbPagUvTLrGAMoUgRx5asz
    PeE4uwc2hGKceeoWMPRfwCvocWvk+QIDAQABo4HwMIHtMB8GA1UdIwQYMBaAFEjm
    aPkr0rKV10fYIyAQTzOYkJ/UMB0GA1UdDgQWBBTAephojYn7qwVkDBF9qn1luMrM
    TjAPBgNVHRMBAf8EBTADAQH/MA4GA1UdDwEB/wQEAwIBBjA6BgNVHR8EMzAxMC+g
    LaArhilodHRwOi8vY3JsLmdlb3RydXN0LmNvbS9jcmxzL3NlY3VyZWNhLmNybDBO
    BgNVHSAERzBFMEMGBFUdIAAwOzA5BggrBgEFBQcCARYtaHR0cHM6Ly93d3cuZ2Vv
    dHJ1c3QuY29tL3Jlc291cmNlcy9yZXBvc2l0b3J5MA0GCSqGSIb3DQEBBQUAA4GB
    AHbhEm5OSxYShjAGsoEIz/AIx8dxfmbuwu3UOx//8PDITtZDOLC5MH0Y0FWDomrL
    NhGc6Ehmo21/uBPUR/6LWlxz/K7ZGzIZOKuXNBSqltLroxwUCEm2u+WR74M26x1W
    b8ravHNjkOR/ez4iyz0H7V84dJzjA1BOoa+Y7mHyhD8S
    -----END CERTIFICATE-----
    ---
    Server certificate
    subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
    issuer=/C=US/O=Google Inc/CN=Google Internet Authority G2
    ---
    No client certificate CA names sent
    ---
    SSL handshake has read 3727 bytes and written 415 bytes
    ---
    New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
    Server public key is 2048 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    SSL-Session:
        Protocol  : TLSv1.2
        Cipher    : ECDHE-RSA-AES128-GCM-SHA256
        Session-ID: 09AF6D01D3E3059EA0E4543E880035C34D74CEFCBB9D20F34F8CC1789D2485B2
        Session-ID-ctx: 
        Master-Key: 575CCE0D8562480D591DE3983B2B6709D1FF5F0FCF219FFF66C30B90A5A906E5A8BD6688DED22EDFE6F7DC9702915E5B
        Key-Arg   : None
        PSK identity: None
        PSK identity hint: None
        SRP username: None
        TLS session ticket lifetime hint: 100800 (seconds)
        TLS session ticket:
        0000 - 3e 73 9d 09 9a 16 a9 a2-70 64 76 b4 16 b1 ca d0   >s......pdv.....
        0010 - 70 37 62 e2 d3 e6 ac b3-31 31 4d 4b 1c 9b 2b 6c   p7b.....11MK..+l
        0020 - cc 1c 0d 3d ae dc ce c2-d4 36 41 4c 04 54 f0 e3   ...=.....6AL.T..
        0030 - 15 03 04 b5 32 0d 8b c0-5b c0 d6 03 8d df d8 bf   ....2...[.......
        0040 - 74 7c ae ac da 3b 1a 8d-d7 56 3d 3a ee dd 69 d3   t|...;...V=:..i.
        0050 - fb 2d 34 4a c4 51 0c e6-39 18 20 f1 cc 5d ab 66   .-4J.Q..9. ..].f
        0060 - 9f f9 47 6f b4 09 6f 4f-42 6c 72 42 fd 92 a3 3b   ..Go..oOBlrB...;
        0070 - 95 3d a1 14 e5 33 b8 b4-8a de 0f f4 4b b6 08 2b   .=...3......K..+
        0080 - bb f6 18 3c 51 90 c8 ce-8c 9d 84 37 de be 07 72   ...<Q......7...r
        0090 - 5d 5a fa 6a 28 70 95 29-28 5e 0d 26 0f 59 c7 d2   ]Z.j(p.)(^.&.Y..
        00a0 - b5 86 1e 99                                       ....
    
        Start Time: 1466605956
        Timeout   : 300 (sec)
        Verify return code: 0 (ok)
    ---
    

    To make this work on your own network, you’ll need to add the CA from your local security appliance into your container:

    sudo cp ca.pem /usr/local/share/ca-certificates/my-ca.crt
    sudo update-ca-certificates
    
    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.