ZMQ crashes “randomly” in aiohttp web service
We have a
aiohttp based web services which uses
ZMQ to send jobs to workers and waits for the result. We are of course using the ZMQ eventloop, so we can wait for ZMQ sockets. “Sometimes” the process crashes and we get this stack trace:
... await socket.send(z, flags=flags) File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/future.py", line 165, in send kwargs=dict(flags=flags, copy=copy, track=track), File "/usr/local/lib/python3.5/dist-packages/zmq/eventloop/future.py", line 276, in _add_send_event timeout_ms = self._shadow_sock.sndtimeo File "/usr/local/lib/python3.5/dist-packages/zmq/sugar/attrsettr.py", line 45, in _getattr_ return self._get_attr_opt(upper_key, opt) File "/usr/local/lib/python3.5/dist-packages/zmq/sugar/attrsettr.py", line 49, in _get_attr_opt return self.get(opt) File "zmq/backend/cython/socket.pyx", line 449, in zmq.backend.cython.socket.Socket.get (zmq/backend/cython/socket.c:4920) File "zmq/backend/cython/socket.pyx", line 221, in zmq.backend.cython.socket._getsockopt (zmq/backend/cython/socket.c:2860)
“Sometimes” means, that the code works fine, if I just run it on my test machine. We encountered the problem in some rare cases when using docker containers, but were never able to reproduce it in an reliable way. Since we moved our containers into a Kubernetes cluster, it occurs much more often. Does anybody know, what could be the source of the above stack trace?
One Solution collect form web for “ZMQ crashes “randomly” in aiohttp web service”
aiohttp is not intended to be used with vanilla pyzmq.
Use aiozmq loopless streams instead.