Docker SIGTERM not being delivered to node.js/coffee app when started with flags

I’ve set up listeners in my application to catch SIGTERM, SIGINT and SIGUSR2:

# kill
process.on 'SIGTERM', ->
    killExecutors 'SIGTERM'

# ctrl + c
process.on 'SIGINT', ->
    killExecutors 'SIGINT'

# nodemon signal
process.on 'SIGUSR2', ->
    killExecutors 'SIGUSR2'

It works as expected. When I run it inside a docker instance:

  • How to assign static public IP to docker container
  • How to persist changes made by maven dependency:go-offline in docker image
  • How to link my custom domain to a bluemix container
  • Error: unsupported type in amqp table: undefined
  • Running Portainer in a Docker Container with Apache 2.4 mod_proxy and basic auth
  • Binaries inside Docker container images not being started when not explicitly called
  • FROM node:4.4.7
    
    MAINTAINER Newborns <newborns@versul.com.br>
    
    COPY . /src
    
    EXPOSE 7733
    
    WORKDIR /src
    RUN npm install
    
    CMD ["./node_modules/.bin/coffee", "feeder.coffee"]
    

    Everything works fine, too. BUT, when I add a node flag to the execution

    FROM node:4.4.7
    
    MAINTAINER Newborns <newborns@versul.com.br>
    
    COPY . /src
    
    EXPOSE 7733
    
    WORKDIR /src
    RUN npm install
    
    CMD ["./node_modules/.bin/coffee", "--nodejs", "--max_old_space_size=384", "feeder.coffee"]
    

    it stops catching the signals. I’ve tried to change de CMD exec form to

    CMD ./node_modules/.bin/coffee --nodejs --max_old_space_size=384 feeder.coffee
    

    but still doesn’t work. What changes between the execution with and without flags?

    EDIT:

    Actually, what happens is that docker starts one process when no flags are passed

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root         1  4.2  1.0 960940 85424 ?        Ssl  20:21   0:01 node ./node_modules/.bin/coffee feeder.coffee
    root        16  0.1  0.0  20220  2884 ?        Ss   20:22   0:00 bash
    root        20  0.0  0.0  17500  2064 ?        R+   20:22   0:00 ps -aux
    

    and two processes when flags are passed

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root         1  0.1  0.3 707704 25272 ?        Ssl  20:17   0:00 node ./node_modules/.bin/coffee --nodejs --max_old_space_size=384 feeder.coffee
    root        10  1.7  1.1 965900 90068 ?        Sl   20:17   0:01 /usr/local/bin/node --max_old_space_size=384 /src/node_modules/.bin/coffee feeder.coffee
    

    question is: why?

  • Flow of registration between kafka and zookeeper
  • How to get the dockerTag label via the Artifactory API?
  • How to change dockerd parameters with systemd? [duplicate]
  • Docker fig multiple server deployment
  • Invalid ELF header xgboost (using a pkl in a Docker container)
  • Access application of one container from another with link/port forward
  • One Solution collect form web for “Docker SIGTERM not being delivered to node.js/coffee app when started with flags”

    TL;DR Use a Javascript loader file instead of the coffee executable when you need to use extended node options to avoid the technicalities of signals with forked processes under Docker.

    require('coffee-script').register();
    require('./whatever.coffee').run();
    

    Then

    node --max_old_space_size=384 app.js
    

    Now, onto the technicalities…

    Docker and signals

    The initial process in a container is PID 1 in the containers namespace. PID 1 (or the init process) is treated as a special case by the kernel with regards to signal handling.

    1. If the init process does not install a signal handler, that signal won’t be sent to it.
    2. Signals do not propagate automatically from an init process, the process must manage this.

    So a docker process is expected to handle signals itself.

    Coffeescripts --nodejs option

    As you have noted, coffee will fork a child node process when it has the --nodejs option to be able to pass the extra options on.

    This initially presents some odd behaviour outside of docker with signal handling (on osx at least). A SIGINT or SIGTERM will be forwarded onto the child but also kill the parent coffee process immediately, no matter how you handle the signal in your code (which is running in the child).

    A quick example

    process.on 'SIGTERM', -> console.log 'SIGTERM'
    process.on 'SIGINT', -> console.log 'SIGINT'
    
    cb = -> console.log "test"
    setTimeout cb, 5000
    

    When you run this and ctrlc, the signal is forwarded on to the child process and handled. The parent process closes immediately though and returns to the shell.

    $ coffee --nodejs --max_old_space_size=384 block_signal_coffee.coffee 
    ^C
    SIGINT
    $ <5ish second pause> test
    

    Then the child process with your code continues running in the background for 5 seconds and eventually outputs test.

    Docker and coffee --nodejs

    The main problem is the parent coffee process does not handle any signals in code, so the signals don’t arrive and are not forwarded onto the child. This probably requires a change to coffeescript’s launcher code to fix.

    The signal quirk coffee --nodejs presents outside of Docker could also be bad if it happened under Docker. If the main container process (the parent of the fork) exits before your signal handlers have a chance to complete in the child, the container will close around them. This scenario is unlikely to happen if the above problem is fixed by just forwarding signals onto the child.

    An alternative to using the suggested javascript loader or fixing coffee scripts loader, would be to use an actual init process, like runit or supervisor but that adds another layer of complexity in between docker and your service.

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.