ZMQ crashes “randomly” in aiohttp web service

We have a aiohttp based web services which uses ZMQ to send jobs to workers and waits for the result. We are of course using the ZMQ eventloop, so we can wait for ZMQ sockets. “Sometimes” the process crashes and we get this stack trace:

...
await socket.send(z, flags=flags)
File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/future.py", line 165, in send
kwargs=dict(flags=flags, copy=copy, track=track),
File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/future.py", line 276, in _add_send_event
timeout_ms = self._shadow_sock.sndtimeo
File "/usr/local/lib/python3.5/dist-packages/zmq/sugar/attrsettr.py", line 45, in _getattr_
return self._get_attr_opt(upper_key, opt)
File "/usr/local/lib/python3.5/dist-packages/zmq/sugar/attrsettr.py", line 49, in _get_attr_opt
return self.get(opt)
File "zmq/backend/cython/socket.pyx", line 449, in zmq.backend.cython.socket.Socket.get (zmq/backend/cython/socket.c:4920)
File "zmq/backend/cython/socket.pyx", line 221, in zmq.backend.cython.socket._getsockopt (zmq/backend/cython/socket.c:2860)

“Sometimes” means, that the code works fine, if I just run it on my test machine. We encountered the problem in some rare cases when using docker containers, but were never able to reproduce it in an reliable way. Since we moved our containers into a Kubernetes cluster, it occurs much more often. Does anybody know, what could be the source of the above stack trace?

  • Error using mount command within Dockerfile
  • View Tensorboard via Chorme on Windows
  • docker compose, swarm and ip address inside dockerfile
  • How to share localhost between two different Docker containers?
  • docker-compose postgresql implementation in python app
  • create a web2py docker image and access it through browser
  • npm command not available in Docker container
  • How to configure docker to use /opt instead of /var
  • AWS EC2 Container Service Container Id, what does this mean?
  • Docker port mapping not working
  • Docker Weave and WeaveDNS issues
  • mariadb docker container on OS X: can't create database directory
  • One Solution collect form web for “ZMQ crashes “randomly” in aiohttp web service”

    aiohttp is not intended to be used with vanilla pyzmq.
    Use aiozmq loopless streams instead.

    See also https://github.com/zeromq/pyzmq/issues/894 and https://github.com/aio-libs/aiozmq/blob/master/README.rst

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.