Using IPython for parallel computing

http://minrk.github.com/scipy-tutorial-2011

• Brian Granger, Cal Poly Physics
• Fernando Perez, UC Berkeley Neuroscience
• Min Ragan-Kelley, UC Berkeley Applied Science & Technology

• Installation and dependencies
• Overview and getting started
  • Introduction
  • Architecture overview
  • Getting Started
  • Starting the IPython controller and engines
• Performance
  • Latency
  • Throughput
  • Map
• Starting the IPython controller and engines
  • General considerations
  • Using ipcluster
  • Configuring an IPython cluster
  • Using the ipcontroller and ipengine commands
• Basics of remote execution
  • Creating a Client instance
  • Remote Execution with apply()
  • Moving Python objects around
  • Other things to look at
  • Moving on
• IPython’s Direct interface
  • Creating a DirectView instance
  • Quick and easy parallelism
  • Calling Python functions
  • Moving Python objects around
  • Exercise: Matrix Multiply
  • Other things to look at
• Multiplexing Exercises
  • Monte Carlo π
  • Remote Iterators
• The IPython task interface
  • Creating a LoadBalancedView instance
  • Quick and easy parallelism
  • Dependencies
  • Retries and Resubmit
  • Schedulers
  • More details
• DAG Dependencies
  • Why are DAGs good for task dependencies?
  • A Sample DAG
• Parallel examples
  • 150 million digits of pi
  • Parallel options pricing
  • Exercise
• Solution
• More details of ØMQ and pyzmq
  • Creating some Sockets
  • non-copying sends
  • Exercise
  • Using pyzmq for inter-engine communication
• Connection Diagrams of The IPython ZMQ Cluster
  • All Connections
• Using MPI with IPython
  • Additional installation requirements
  • Starting the engines with MPI enabled
  • Actually using MPI
• IPython’s Task Database
  • Example Queries
• Transitioning from IPython.kernel to IPython.parallel
  • Creating a Client
  • Apply
  • MultiEngine to DirectView
  • Task to LoadBalancedView
• Security details of IPython
  • Process and network topology
  • Application logic
  • Secure network connections
  • Specific security vulnerabilities
  • Other security measures
  • Summary
• Details of Parallel Computing with IPython
  • Caveats