Celery and Flask in same docker-compose

I’m trying to use docker-compose to spawn my Flask/Celery/Redis services.

Here’s my docker-compose.yml:

  • How do I remove old service images after an update?
  • How do you trace docker images brought in by a pull?
  • local files missing from docker-machine container
  • using ansible for provisioning docker containers
  • How can I understand the role of “/bin/true” command in “docker run …” command?
  • Generated files on host from docker
  • flask:
        build: .
        command: "python3 app.py"
        ports:
            - '5000:5000'
        links:
            - redis
        volumes:
            - .:/usr/src/app:ro
    
    celery:
        build: .
        command: "celery -A app.celery worker --loglevel=info"
        volumes:
            - .:/usr/src/app:ro
    
    redis:
        image: redis
        ports:
            - '6379:6379'
    

    When I run this docker-compose, both Flask and Redis start fine and function as expected. Regarding Celery, Docker reports: flaskcelery_celery_1 exited with code 1, with no other info.

    If I run my three services without Docker, and start Celery with celery -A app.celery worker --loglevel=info, my app functions just fine.

    Some more info, if necessary:

    Dockerfile: (this image installs requirements.txt on build as well)

    FROM python:3.5-onbuild
    EXPOSE 5000
    

    requirements.txt:

    flask==0.11.1
    celery==3.1.23
    

    docker-compose up output:

    Starting flaskcelery_celery_1
    Starting flaskcelery_redis_1
    Starting flaskcelery_flask_1
    Attaching to flaskcelery_celery_1, flaskcelery_redis_1, flaskcelery_flask_1
    redis_1  |                 _._                                                  
    redis_1  |            _.-``__ ''-._                                             
    redis_1  |       _.-``    `.  `_.  ''-._           Redis 3.2.3 (00000000/0) 64 bit
    redis_1  |   .-`` .-```.  ```\/    _.,_ ''-._                                   
    redis_1  |  (    '      ,       .-`  | `,    )     Running in standalone mode
    redis_1  |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
    redis_1  |  |    `-._   `._    /     _.-'    |     PID: 1
    redis_1  |   `-._    `-._  `-./  _.-'    _.-'                                   
    redis_1  |  |`-._`-._    `-.__.-'    _.-'_.-'|                                  
    redis_1  |  |    `-._`-._        _.-'_.-'    |           http://redis.io        
    redis_1  |   `-._    `-._`-.__.-'_.-'    _.-'                                   
    redis_1  |  |`-._`-._    `-.__.-'    _.-'_.-'|                                  
    redis_1  |  |    `-._`-._        _.-'_.-'    |                                  
    redis_1  |   `-._    `-._`-.__.-'_.-'    _.-'                                   
    redis_1  |       `-._    `-.__.-'    _.-'                                       
    redis_1  |           `-._        _.-'                                           
    redis_1  |               `-.__.-'                                               
    redis_1  | 
    redis_1  | 1:M 23 Aug 10:23:08.409 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
    redis_1  | 1:M 23 Aug 10:23:08.409 # Server started, Redis version 3.2.3
    redis_1  | 1:M 23 Aug 10:23:08.409 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
    redis_1  | 1:M 23 Aug 10:23:08.409 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
    redis_1  | 1:M 23 Aug 10:23:08.409 * The server is now ready to accept connections on port 6379
    flaskcelery_celery_1 exited with code 1
    flask_1  |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
    flask_1  |  * Restarting with stat
    flask_1  |  * Debugger is active!
    flask_1  |  * Debugger pin code: 196-119-737
    

  • How do you know that public Docker images are secure?
  • IntelliJ: use docker jvm or docker maven?
  • using gcloud docker client on coreos
  • Docker Compose circular container linking
  • How to import a CSV inside a Docker container with Java 8?
  • When I use PyCharm and odoo, [Errno 2] No such file or directory Process finished with exit code 2
  • One Solution collect form web for “Celery and Flask in same docker-compose”

    Solved my problem. I eventually figured out I could get a command prompt on the Docker image:

    docker build -t <image name> .
    docker run -it <image name> /bin/bash
    

    Then trying to run celery within the container revealed the problem:

    root@4a6edc5d7372:/usr/src/app# celery -A app.celery worker --loglevel=info
    Running a worker with superuser privileges when the
    worker accepts messages serialized with pickle is a very bad idea!
    
    If you really want to continue then you have to set the C_FORCE_ROOT
    environment variable (but please think about this before you do).
    
    User information: uid=0 euid=0 gid=0 egid=0
    

    Docker usually runs as root, and Celery doesn’t like running as root for security reasons (I believe you can get code execution with pickle deserialization). The safer solution was to set the celery container to run as nobody. Working docker-compose.yml:

    flask:
        build: .
        command: "python3 app.py"
        ports:
            - '5000:5000'
        links:
            - redis
            - celery
        volumes:
            - .:/usr/src/app:ro
    
    celery:
        build: .
        command: "celery -A app.celery worker --loglevel=info"
        user: nobody
        links:
            - redis
        volumes:
            - .:/usr/src/app:ro
    
    redis:
        image: redis
        ports:
            - '6379:6379'
    
    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.