Getting started with Multiprocessing in Python

Post to Twitter

Multiprocessing in Python is pretty easy given the amount of work Python is doing for you behind the scenes. Today I’ll show you a few example to get started with Python 2.7.x multiprocessing.


Let’s start off with a simple example:

import multiprocessing

print multiprocessing.cpu_count()

Running the above code will print out the amount of processors on the current system. On my Mac Mini this value is: 4.

Let’s actually build a small example that fires off one process. Keep in mind this is an entirely new process, not a thread.

import uuid
import multiprocessing

def my_function():
    print 'My Unique Id: {0}'.format(uuid.uuid1())

if __name__ == '__main__':
    process = multiprocessing.Process(target=my_function)
    process.start()

Using multiprocessing.Process you can pass in the function you want to call, in this case its my_function. A new (separate) process will be kicked off and you should see output like this:

My Unique Id: a51d2c7d-8799-11b2-cf1f-a8206617d2d9

To fire off two processes you could do this:

import uuid
import multiprocessing

def my_function():
    print 'My Unique Id: {0}'.format(uuid.uuid1())

if __name__ == '__main__':
    for x in range(2):
        process = multiprocessing.Process(target=my_function)
        process.start()

Want to fire off 10 processes? Easy enough:

import uuid
import multiprocessing

def my_function():
    print 'My Unique Id: {0}'.format(uuid.uuid1())

if __name__ == '__main__':
    for x in range(10):
        process = multiprocessing.Process(target=my_function)
        process.start()

What happens if you need to pass an argument to the target method? Its pretty easy actually but keep in mind the argument has to be serializable via Pickle. In the example below I’ll pass a worker id to the function.

import uuid
import multiprocessing

def my_function(worker_id):
    print 'My Worker Id is: {0}'.format(worker_id)

if __name__ == '__main__':
    for x in range(2):
        process = multiprocessing.Process(target=my_function, args=(uuid.uuid1(),))
        process.start()

Take a look now at this example:

import time
import multiprocessing

def my_function():
    time.sleep(3)
    print "my_function is done!"

if __name__ == '__main__':
    process = multiprocessing.Process(target=my_function)
    process.start()
    print "__main__ is done!"

If you run the above code you will see the output as:

__main__ is done!
my_function is done!

You can see that __main__ finished before the process completed. If you need to wait for the process to finish first you can simply use join. Join will block the current process until the process we started finishes.

import time
import multiprocessing

def my_function():
    time.sleep(3)
    print "my_function is done!"

if __name__ == '__main__':
    process = multiprocessing.Process(target=my_function)
    process.start()
    process.join()
    print "__main__ is done!"

Results:

my_function is done!
__main__ is done!

If you don’t want your target to be a function you can easily subclass multiprocessing.Process like so:

import multiprocessing

class My_Class(multiprocessing.Process):

    def run(self):
        print 'Hello from a class'
        return

if __name__ == '__main__':
    process = My_Class()
    process.start()

Output:

Hello from a class

There is a lot more I didn’t cover so make sure to read the docs for more information.

For a better idea on comparing threads and processes in Python please check out this presentation: http://www.slideshare.net/pvergain/multiprocessing-with-python-presentation

Post to Twitter

This entry was posted in Open Source, Python. Bookmark the permalink.