Parallel programming in Python using the multiprocessing module
Spark gives you tools to distribute and map threads across processors
and disks, but very often it is necessary to have more control over
your parallel tasks. In this class, we will use some alternatives to
program in parallel in python and take advantage of your multicore
machine.
Go
to this
site and follow the alternative scripts. Time all of them and
check which one(s) work better in your machine.
Note: When using the method pool.apply_async the function to be
invoked needs to be defined in a separate file and be imported to the
program, otherwise your code will not work.