List comprehension and generator expressions Intermediate Python Tutorial part 4

Welcome to part 4 of the intermediate Python programming tutorial series. In this part, we're going to talk about list comprehension and generators.

To begin, let's show a quick example and reason for both. A generator that is used commonly is Python 3's range() generator (Python 2's xrange).

If you want iterate through something 4 times, you might say something like:

for i in range(4):

What's range() doing here? In this case, it's a generator, so it's generating the values in order on-the-fly. An example of a generator expression:

xyz = (i for i in range(50000000))

An example of list comprehension:

xyz = [i for i in range(50000000)]

These look and appear to act very similarly, but they are quite different under the hood. First, with a generator, the values are generated from an original input, but the values are not copied and instead are generated on-the-fly. This means we will use far less memory, since the entire list is not processed all at once, but also means the process is a bit slower, since things are indeed generated as we go.

The list comprehension puts the entire list into memory, so it is faster, but the penalty is memory use.

Thus, generally, you will use generators for huge ranges/sequences (including infinite ones), and otherwise use list comprehension. Notice that, when we wanted to output the first 5 items in xyz in the case of the generator, we first had to actually convert it to a list, since, it's just a generator object. The list comprehension example was already a list.

In many cases, you wont necessarily be worried about memory or speed, but, if you are working on mobile devices with less power, or maybe you really are working with huge datasets, where you don't actually need the entire list at once, these distinctions matter. Remember too, Python normally works on a single CPU, so it's natively only as powerful as a single CPU.

In the next tutorial, we're going to talk more on list comprehension and generators.

The next tutorial:

  • Intermediate Python Programming introduction
  • String Concatenation and Formatting Intermediate Python Tutorial part 2
  • Argparse for CLI Intermediate Python Tutorial part 3
  • List comprehension and generator expressions Intermediate Python Tutorial part 4
  • More on list comprehension and generators Intermediate Python Tutorial part 5
  • Timeit Module Intermediate Python Tutorial part 6
  • Enumerate Intermediate Python Tutorial part 7
  • Python's Zip function
  • More on Generators with Python
  • Multiprocessing with Python intro
  • Getting Values from Multiprocessing Processes
  • Multiprocessing Spider Example
  • Introduction to Object Oriented Programming
  • Creating Environment for our Object with PyGame
  • Many Blobs - Object Oriented Programming
  • Blob Class and Modularity - Object Oriented Programming
  • Inheritance - Object Oriented Programming
  • Decorators in Python Tutorial
  • Operator Overloading Python Tutorial
  • Detecting Collisions in our Game Python Tutorial
  • Special Methods, OOP, and Iteration Python Tutorial
  • Logging Python Tutorial
  • Headless Error Handling Python Tutorial
  • __str__ and __repr_ in Python 3
  • Args and Kwargs
  • Asyncio Basics - Asynchronous programming with coroutines