ZMQ crashes “randomly” in aiohttp web service

We have a aiohttp based web services which uses ZMQ to send jobs to workers and waits for the result. We are of course using the ZMQ eventloop, so we can wait for ZMQ sockets. “Sometimes” the process crashes and we get this stack trace:

...
await socket.send(z, flags=flags)
File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/future.py", line 165, in send
kwargs=dict(flags=flags, copy=copy, track=track),
File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/future.py", line 276, in _add_send_event
timeout_ms = self._shadow_sock.sndtimeo
File "/usr/local/lib/python3.5/dist-packages/zmq/sugar/attrsettr.py", line 45, in _getattr_
return self._get_attr_opt(upper_key, opt)
File "/usr/local/lib/python3.5/dist-packages/zmq/sugar/attrsettr.py", line 49, in _get_attr_opt
return self.get(opt)
File "zmq/backend/cython/socket.pyx", line 449, in zmq.backend.cython.socket.Socket.get (zmq/backend/cython/socket.c:4920)
File "zmq/backend/cython/socket.pyx", line 221, in zmq.backend.cython.socket._getsockopt (zmq/backend/cython/socket.c:2860)

“Sometimes” means, that the code works fine, if I just run it on my test machine. We encountered the problem in some rare cases when using docker containers, but were never able to reproduce it in an reliable way. Since we moved our containers into a Kubernetes cluster, it occurs much more often. Does anybody know, what could be the source of the above stack trace?

  • docker: net: no such interface
  • Jenkins plugin for docker
  • Licensing applications inside a Docker container
  • Docker host & no space left on device
  • How can a bash script restart a process on non-0 exit while sending signals to child
  • Trouble setting up LDAPS for login in Sonarqube Docker container
  • Running SonarQube with Docker in CI/CD pipeline
  • Is there a clever way to do this with Docker (maybe via save/load/diff)?
  • Multiple instances of an application
  • Docker failing to see updated fixtures CSV in rspec test directory
  • make new docker container after it has been deleted
  • How to create flexible onbuild docker image that uses env vars?
  • One Solution collect form web for “ZMQ crashes “randomly” in aiohttp web service”

    aiohttp is not intended to be used with vanilla pyzmq.
    Use aiozmq loopless streams instead.

    See also https://github.com/zeromq/pyzmq/issues/894 and https://github.com/aio-libs/aiozmq/blob/master/README.rst

    Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.