Getting Values from Multiprocessing Processes

Welcome to part 11 of the intermediate Python programming tutorial series. In this part, we're going to talk more about the built-in library: multiprocessing.

In the previous multiprocessing tutorial, we showed how you can spawn processes. If these processes are fine to act on their own, without communicating with eachother or back to the main program, then this is fine. These processes can also share a common database, or something like that to work together, but, many times, it will make more sense to use multiprocessing to do some processing, and then return results back to the main program. That's what we're going to cover here.

To begin, we're going to import Pool

from multiprocessing import Pool

Pool allows us to create a pool of worker processes

Let's say we want to run a function over each item in an iterable. Let's just do:

def job(num):
    return num * 2

Simple enough, now let's set up the processes:

if __name__ == '__main__':
    p = Pool(processes=20)
    data =, [i for i in range(20)])

In the above case, what we're going to do is first set up the Pool object, which will have 20 processes that we'll allow to do some work.

Next, we're going to map the job function to a list of parameters ([i for i in range(20)]). When done, we close the pool, and then we're printing the result. In this case, we get:

[0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38]

If you raise the range and processes to the 100's, you can see your CPU max out and the processes if you like. ~500 for me seems to do the trick.

The next tutorial:

  • Intermediate Python Programming introduction
  • String Concatenation and Formatting Intermediate Python Tutorial part 2
  • Argparse for CLI Intermediate Python Tutorial part 3
  • List comprehension and generator expressions Intermediate Python Tutorial part 4
  • More on list comprehension and generators Intermediate Python Tutorial part 5
  • Timeit Module Intermediate Python Tutorial part 6
  • Enumerate Intermediate Python Tutorial part 7
  • Python's Zip function
  • More on Generators with Python
  • Multiprocessing with Python intro
  • Getting Values from Multiprocessing Processes
  • Multiprocessing Spider Example
  • Introduction to Object Oriented Programming
  • Creating Environment for our Object with PyGame
  • Many Blobs - Object Oriented Programming
  • Blob Class and Modularity - Object Oriented Programming
  • Inheritance - Object Oriented Programming
  • Decorators in Python Tutorial
  • Operator Overloading Python Tutorial
  • Detecting Collisions in our Game Python Tutorial
  • Special Methods, OOP, and Iteration Python Tutorial
  • Logging Python Tutorial
  • Headless Error Handling Python Tutorial
  • __str__ and __repr_ in Python 3
  • Args and Kwargs
  • Asyncio Basics - Asynchronous programming with coroutines