More details of ØMQ and pyzmq

ØMQ is a fast, lightweight messaging library, and pyzmq is the Python bindings.

ØMQ in 100 words (From the excellent Guide):

ØMQ (ZeroMQ, 0MQ, zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry whole messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fanout, pub-sub, task distribution, and request-reply. It’s fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ØMQ is from iMatix and is LGPL open source.

ØMQ works with a variety of socket types, each of which specifies a different send/recv pattern:

REQ : send/recv/send (counterpart: REP)

REP : recv/send/recv (counterpart: REQ)

XREQ: LRU load-balanced requests (counterpart: XREP)

XREP: IDENTITY-routed replies (counterpart: XREQ)

PUB : send-only, to all peers (counterpart: SUB)

SUB : recv-only, from all peers (counterpart: PUB)

PUSH: send-only, load-balanced across peers (counterpart: PULL)

PULL: recv-only, fair-queued among peers (counterpart: PUSH)

PAIR: symmetric exclusive-pair

XREQ is also called a DEALER, and XREP is called ROUTER, which are new, more descriptive names.

IPython’s connection diagram

IPython uses MonitoredQueue devices, as schedulers that relay messages between engines and clients, but also send every message to the Hub, which uses the information to track the state of the cluster.

Creating some Sockets

With ØMQ, you always create a context first. Typically, you have just one Context per process, and can use the classmethod zmq.Context.instance() to create it. Then you use the Context’s socket() method to create sockets of various types.

import zmq

ctx = zmq.Context.instance()

a = ctx.socket(zmq.PAIR)
b = ctx.socket(zmq.PIAR)

p = a.bind_to_random_port('tcp://127.0.0.1')
b.connect('tcp://127.0.0.1:%i'%p)

a.send('hi')
print b.recv()
# 'hi'

non-copying sends

pyzmq supports non-copying sends of objects that provide the buffer interface, which includes numpy arrays.

A = numpy.ones(16)
a.send(A, copy=False)

And on the other side, when you do a non-copying recv, you get a Message object, which also provides the buffer interface:

msg = b.recv(copy=False)
B = numpy.frombuffer(msg)

Note

noncopying in code directory

Exercise

Can you write a pair of function for sending and receiving numpy arrays without copying?

noncopying solution

Using pyzmq for inter-engine communication

Can we design some inter-engine communication with pyzmq?

What sorts of send/recv patterns might be useful?

How do we determine who connects to whom?