In [14]:
%pxresult
IPython has a few magics for working with your engines.
This assumes you have started an IPython cluster, either with the notebook interface,
or the ipcluster/controller/engine commands.
%pylab inline
from IPython import parallel
rc = parallel.Client()
dv = rc[:]
rc.ids
Creating a Client registers the parallel magics %px, %%px, %pxresult, pxconfig, and %autopx.
These magics are initially associated with a DirectView always associated with all currently registered engines.
Now we can execute code remotely with %px:
%px a=5
%px print a
%px a
with dv.sync_imports():
import sys
stderr comes to stderr
%px print >> sys.stderr, "ERROR"
You don't have to wait for results. The %pxconfig magic lets you change the default blocking/targets for the %px magics:
%pxconfig --noblock
%px import time
%px time.sleep(3*rank)
%px time.time()
But you will notice that this didn't output the result of the last command.
For this, we have %pxresult, which displays the output of the latest request:
%pxresult
Remember, an IPython engine is IPython, so you can do magics remotely as well!
%pxconfig --block
%px %pylab inline
%%px can also be used as a cell magic, for submitting whole blocks.
This one acceps --block and --noblock flags to specify
the blocking behavior, though the default is unchanged.
dv.scatter('rank', dv.targets, flatten=True)
%%px --noblock
x = linspace(0,pi,1000)
plt.plot(x,sin((rank+1)*x))
plt.title("Plot %i" % rank)
%pxresult
It also lets you choose some amount of the grouping of the outputs with --group-outputs:
The choices are:
engine - all of an engine's output is collected togethertype - where stdout of each engine is grouped, etc. (the default)order - same as type, but individual displaypub outputs are interleaved.
That is, it will output the first plot from each engine, then the second from each,
etc.%%px --group-outputs=engine
x = linspace(0,pi,1000)
plt.plot(x,sin((rank+1)*x))
plt.title("Plot %i" % rank)
When you specify 'order', then individual display outputs (e.g. plots) will be interleaved.
%pxresult takes the same output-ordering arguments as %%px,
so you can view the previous result in a variety of different ways with a few sequential calls to %pxresult:
%%px --noblock
import os
from IPython.display import display, Math
display(Math("sin(x^%i)" % (rank+1)))
x = linspace(0,pi,1000)
plt.plot(x,sin(x**(rank+1)))
plt.title("Plot %i" % rank)
%pxresult --group-outputs=order
When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):
def generate_output():
"""function for testing output
publishes two outputs of each type, and returns something
"""
import sys,os
from IPython.core.display import display, HTML, Math
print "stdout"
print >> sys.stderr, "stderr"
display(HTML("<a href='http://ipython.org'>Visit IPython</a>"))
return os.getpid()
dv['generate_output'] = generate_output
You can also have more than one set of parallel magics registered at a time.
The View.activate() method takes a suffix argument, which is added to 'px'.
e0 = rc[-1]
e0.block = True
e0.activate('0')
%px0 generate_output()
%px generate_output()
As mentioned above, we can redisplay those same results with various grouping:
%pxresult --group-outputs engine
When you raise exceptions with the parallel exception, the CompositeError raised locally will display your remote traceback.
%%px
from numpy.random import random
A = random((100,100,'invalid shape'))
Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic.
%%px
%%timeit
from numpy.random import random
from numpy.linalg import norm
A = random((100*(rank+1),100*(rank+1)))
norm(A,2)
And you can run line magics remotely as well
%%px
import os
from numpy.random import random
from numpy.linalg import norm
A = random((200*(rank+1),200*(rank+1)))
pid = os.getpid()
!echo 'hello from $pid'
%timeit norm(A,2)
Parallel magics, line magics, display protocol, and cell magics all come together for parallel remote plots in R:
X = np.linspace(0, 3*np.pi, 32)
Y = np.sin(X)
dv.scatter('X', X)
dv.scatter('Y', Y)
plot(X,Y, '-o')
%px %load_ext rmagic
%px %Rpush X Y
%%px
%%R -i X,Y -o XYcoef
XYlm = lm(Y~X)
XYcoef = coef(XYlm)
print(summary(XYlm))
par(mfrow=c(2,2))
plot(XYlm)
And the same for parallel remote Cython
%px %load_ext cythonmagic
%%px
%%cython
cimport cython
from libc.math cimport exp, sqrt, pow, log, erf
@cython.cdivision(True)
cdef double std_norm_cdf(double x) nogil:
return 0.5*(1+erf(x/sqrt(2.0)))
@cython.cdivision(True)
def black_scholes(double s, double k, double t, double v,
double rf, double div, double cp):
"""Price an option using the Black-Scholes model.
s : initial stock price
k : strike price
t : expiration time
v : volatility
rf : risk-free rate
div : dividend
cp : +1/-1 for call/put
"""
cdef double d1, d2, optprice
with nogil:
d1 = (log(s/k)+(rf-div+0.5*pow(v,2))*t)/(v*sqrt(t))
d2 = d1 - v*sqrt(t)
optprice = cp*s*exp(-div*t)*std_norm_cdf(cp*d1) - \
cp*k*exp(-rf*t)*std_norm_cdf(cp*d2)
return optprice
dv.scatter('call', [50., 100., 150., 200.], flatten=True)
%px %timeit black_scholes(call, 100.0, 1.0, 0.3, 0.03, 0.0, -1)
%px black_scholes(call, 100.0, 1.0, 0.3, 0.03, 0.0, -1)
Last, but not least, opening a QtConsole onto an engine
%px0 %qtconsole