Containers are being spawned however the issued command fails when spawning multiple number of docker containers

We have a custom chrome extension that performs a certain set of web related tasks. The tasks need to be exclusive and hence we went forward with the concept of running each task on an individual docker container running chrome

We currently have 4 ec2 instances (c4.2xlarge) running ubuntu and then chrome inside a docker container.

  • Cucumber on Travis CI container
  • Nginx load balancer upstream SSL and Docker
  • How to configure CouchDB authentication in Docker?
  • netcat for MySQL connection forwarding
  • war-file as a part of container or shared folder when running docker
  • Connecting to Dockerized MySQL from remote client
  • We also have one more smaller ec2 instance that reads tasks from mongodb and allocates them to the servers filling them one after the other. So if a task is added it is assigned to one of the instances which then picks it up creates a container performs the task and then destroys the container. Each of these tasks take roughly 2 to 6 minutes to complete. We are using dockerode to dynamically create and destroy containers on the fly within nodejs.

    We performed a lot of benchmarks and come to the conclusion that each server can run a max of 55 tasks, so we have set a threshold of maximum 45 tasks per server at a given time to be on the safer side.

    This is part of the code that runs on all the 4 instances

    function start_polling_mongo() {
            server_id: instance_id,
            task_status: 1
        }, function(err, all_tasks) {
            //update the tasks
                _id: {
                    "$in": all_tasks
            }, {
                task_status: 2
            }, {
                multi: true
            }, function(err) {
                //perform the tasks
                all_tasks.forEach(function(task) {
                    setTimeout(function() {
                    }, 1000);
        setTimeout(start_polling_mongo, 5000);
    function run_task_in_container(task_id) {
        console.log('Running Task id : ' + task_id);
        var commands = '';
        commands += 'Xvfb :99 & ';
        commands += 'export DISPLAY=:99 & ';
        commands += 'x11vnc -rfbport ' + task_id + ' -display :99 -bg -q & ';
        commands += 'google-chrome --no-first-run --load-extension=/home/chrome_extension' + task_id;
        var exposed_ports = {};
        exposed_ports[task_id + '/tcp'] = {};
        var port_bindings = {};
        port_bindings[task_id + '/tcp'] = [{
            HostPort: task_id.toString()
            Image: 'docker-chrome-final',
            ExposedPorts: exposed_ports,
            Cmd: ['bash', '-c', commands],
            Labels: {
                "task_id": task_id.toString()
        }, function(err, container) {
            if (err) {
                console.log('error1 :');
                Privileged: true,
                PortBindings: port_bindings
            }, function(err, data) {
                if (err) {
                    console.log('error2 :');

    The problem is that ever batch of 150 to 180 tasks there are random containers that are created and started however chrome is not running on the same (usually 2-3 containers on every server). I confirmed that by successfully VNC’ing into the server to be presented with a blank screen (XVFB).

    Any input is highly appreciated!

  • How to make mysql and php work together and without show the database password
  • Docker container can't be reached from another PC
  • Pass docker-compose environment to symfony configuration
  • Effect of a driver in the host OS to a Docker process
  • “debconf: delaying package configuration, since apt-utils is not installed” :> google notebooks bash error
  • Docker: Do I need to rebuild the container from scratch whenever I need to add software?
  • Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.