Iterators, list comprehensions, and generator expressions

Python Intermediate

In the world of Python programming, mastering the art of iterators, harnessing the elegance of list comprehensions, and unleashing the efficiency of generator expressions are fundamental skills every developer should possess. We will embark on a journey through these essential Python constructs, exploring how they can streamline our code, enhance our data processing capabilities, and empower us to write more concise and readable scripts.

Let's recall the simple ParkingLot class and its constructor from the previous post.

class ParkingLot:
    def __init__(self, spots: int):
        self.__spots: list[Car] = list()
        for _ in range(spots):
            self.__spots.append(None)

The primary purpose of the __init__ method is to initialize the attributes and state of a ParkingLot object when it is created. It takes one parameter, spots, which represents the number of parking spots in the parking lot. Inside the __init__ method the self._spots attribute is initialized as an empty list ([]). This attribute will be used to keep track of the occupancy status of each parking spot. It then uses a for loop to iterate spots times (specified by the spots parameter) and appends None to the self.__spots list for each spot. The result is that the self.__spots list is populated with None values, representing empty parking spots. The number of empty spots is determined by the value of the spots parameter. For the full code example, please follow the link to the previous post. 👇👇👇

But the truth must be told! The for loop in the constructor can be rewritten in a more clean, elegant and concise way using list comprehension.

List comprehension

List comprehension is a concise and powerful feature in Python for creating lists. They provide a compact and readable way to generate new lists by applying an expression to each item in an existing iterable (such as a list, tuple, or range) and optionally filtering items based on a condition.

The basic syntax of list comprehension consists of square brackets [] containing an expression followed by a for clause. Optionally, we can include a if clause for filtering. Here's the general structure.

new_list = [expression for item in iterable]

Let's see a simple example of creating a list with integers from 0 to 9.

>>> ints = [x for x in range(0, 10)]
>>> print(ints)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

In a very similar manner, squares can be calculated.

>>> squares = [x**2 for x in range(0, 10)]
>>> print(squares)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

In the above example, we just substitute x with x**2 to calculate squares.

We can include an if clause to filter items from the iterable based on a condition. Only items that satisfy the condition will be included in the new list.

Let's create a list of even numbers within the range from 0 to 9.

>>> even_numbers = [x for x in range(0, 10) if x % 2 == 0]
>>> print(even_numbers)
[0, 2, 4, 6, 8]

We can use nested list comprehensions to create more complex lists or lists of lists. This is especially useful for working with multi-dimensional data structures.

>>> matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
>>> flattened = [num for row in matrix for num in row]
>>> print(flattened)
[1, 2, 3, 4, 5, 6, 7, 8, 9]

List comprehensions are often more concise than writing equivalent loops, making our code more readable, therefore they express the intent of our code more clearly. They can be more efficient than traditional loops for simple operations.

Now we can easily rewrite the ParkingLot constructor using list comprehension.

class ParkingLot:
    def __init__(self, spots: int):
        self.__spots: list[Car] = [None for _ in range(spots)]

However, that's not everything. We can rewrite the code to make it even more concise! We have already seen that trick in the following post. 👇👇👇

Instead of list comprehension, we can just use the * operator to repeat elements in the list.

class ParkingLot:
    def __init__(self, spots: int):
        self.__spots: list[Car] = spots * [None]

Charming, right? 🥹

List comprehensions are best suited for creating new lists. They are not ideal for modifying existing lists in place. Complex operations might become less readable in list comprehensions. In such cases, using traditional for loops can be more appropriate.

List (and other sequences) can be easily looped over as they provide a unified way of accessing elements named iterators.

Iterators

Iterators are a fundamental concept in Python used to traverse through sequences of data one element at a time. They provide a standardized way to access elements of an iterable object (like lists, tuples, strings, or custom objects) without having to know the underlying details of the data structure.

Python's for loop is designed to work seamlessly with iterators. When we use a for loop to iterate over an iterable, it automatically calls iter() to obtain an iterator and then repeatedly calls next() to retrieve values until a StopIteration exception is raised.

An iterator is an object that implements two methods: __iter__() and __next__(). These methods define how the iterator should behave:

• __iter__(): this method returns the iterator object itself. It's used to initialize or reset the iterator. In most cases, it simply returns self,

• __next__(): this method returns the next value from the iterator. If there are no more items to return, it raises the StopIteration exception.

We can create an iterator from an iterable object using the iter() function.

>>> my_list = [x for x in range(10)]
>>> print(my_list)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> my_iterator = iter(my_list)
>>> print(my_iterator)
<list_iterator object at 0x10bbbfca0>

my_iterator is now an iterator that can be used to traverse through my_list one element at a time. We can use the next() function to retrieve the next item from an iterator.

>>> print(next(my_iterator))
0
>>> print(next(my_iterator))
1
>>> print(next(my_iterator))
2
...
>>> print(next(my_iterator))
9
>>> print(next(my_iterator))
Traceback (most recent call last):
  File "pydevconsole.py", line 364, in runcode
    coro = func()
           ^^^^^^
  File "<input>", line 1, in <module>
StopIteration

The next() function moves the iterator's internal pointer to the next element in the sequence. Once we go past the last element the StopIteration exception is raised.

Let's create a CarShowcase class to display cars. To create a CarShowcase class that implements an iterator, we can define the class with a collection of car objects and provide methods for adding cars and iterating through them.

from typing import List

class CarShowcase:
    def __init__(self):
        self.cars: List[Car] = []
        self.index: int = 0

    def add_car(self, car: Car):
        if isinstance(car, Car):
            self.cars.append(car)
        else:
            raise ValueError(f'Only instances of the Car class can be added to the showcase.')

    def __iter__(self):
        self.index = 0
        return self

    def __next__(self) -> Car:
        if self.index < len(self.cars):
            car = self.cars[self.index]
            self.index += 1
            return car
        raise StopIteration(f'End of showcase reached')

The CarShowcase class maintains a list of cars and includes methods to add cars to the showcase. It implements the iterator protocol by defining __iter__ and __next__ methods. The __iter__ method returns self, and the __next__ method iterates through the list of cars, returning each car one at a time until it reaches the end of the showcase.

Here's how we can use the CarShowcase class:

def main():
    showcase = CarShowcase()

    showcase.add_car(Car(f'Toyota', f'Camry'))
    showcase.add_car(Car(f'Ford', f'Mustang'))
    showcase.add_car(Car(f'Honda', f'Civic'))

    for car in showcase:
        print(car)

if __name__ == "__main__":
    main()

Now we can run it.

❯ python3 main.py
Toyota Camry
Ford Mustang
Honda Civic

We can easily loop through the showcase to see each car.

Lists comprehensions always create lists that obviously consume memory. Generator expressions, on the other hand, create generator objects that produce values lazily and are more memory-efficient for large datasets.

Generator expressions

Generator expressions are a compact way to create generator objects, which are a type of iterable in Python. They are similar to list comprehensions but with one key difference: list comprehensions create lists in memory, while generator expressions create generator objects that yield values on the fly. Generator expressions use parentheses () instead of square brackets [].

The basic syntax of generator expression looks as follows.

generator_object = (expression for item in iterable)

For instance, let's create a generator for the squares of numbers from 0 to 10.

>>> squares_generator = (x**2 for x in range(0, 10))
>>> next(squares_generator)
0
>>> next(squares_generator)
1
>>> next(squares_generator)
4
>>> next(squares_generator)
9
>>> next(squares_generator)
16

We can iterate through squares_generator using a for loop or by explicitly calling the next() function to yield values one at a time. Generator expressions are memory-efficient, making them suitable for large datasets.

We can use generator expressions to generate new cars. Let's see an example.

car_generator = (Car(f'Toyota', model) for model in [f'Camry', f'Corolla', f'Prius'])

for car in car_generator:
    print(car)

The above code efficiently generates car objects with the fixed make "Toyota" and varying models using a generator expression, and it prints the cars within the loop. This approach is memory-efficient and allows us to work with large or dynamic data sets without pre-creating all the objects.

Unlike generator expressions, which are concise and one-liners, generator functions are user-defined functions with a special yield statement. The basic structure is as follows.

def my_generator():
    # Any setup code
    yield value
    # More code

The yield keyword is used to specify the value to yield to the caller. When yield is encountered, the generator's state is saved, and the value is returned to the caller. Execution of the generator function is paused until the next value is requested.

Generator functions are lazily evaluated, meaning they generate values on the fly as we iterate through them. This lazy evaluation is memory-efficient, as it doesn't require storing all values in memory at once, making generator functions ideal for working with large or infinite datasets. Generator functions offer a high degree of flexibility. We can incorporate custom logic, conditions, and computations in our generator, allowing us to generate values based on our specific requirements.

Let's see how we can use a generator function to create the Fibonacci sequence.

def fibonacci_generator(n) -> Generator[int, None, None]:
    a, b = 0, 1
    count = 0
    while count < n:
        yield a
        a, b = b, a + b
        count += 1

The function takes one argument, n, which specifies how many Fibonacci numbers we want to generate. It uses two variables, a and b, to keep track of the current and next Fibonacci numbers. Inside a while loop, it yields the current Fibonacci number (a), updates a and b to calculate the next Fibonacci number, and increments the count until it reaches n.

if __name__ == "__main__":
    for f in fibonacci_generator(5):
        print(f)

Let's generate and print the first 5 Fibonacci numbers.

❯ python3 main.py
0
1
1
2
3

Generator expressions are often more concise for simple data transformations, while generator functions offer more flexibility and are better suited for complex data generation and processing tasks.

Similar to generator expressions, we can use generator functions to generate new cars. The output will show car objects with various makes and models. Let's see an example.

from typing import List, Generator

def car_generator() -> Generator[Car, None, None]:
    makes: List[str] = ["Toyota", "Ford", "Honda"]
    models: List[str] = ["Camry", "Mustang", "Civic"]
    
    for make in makes:
        for model in models:
            yield Car(make, model)

for car in car_generator():
    print(car)

This code demonstrates how to use a generator function to create car objects lazily and efficiently without storing all of them in memory, which can be especially useful for handling large datasets.

❯ python3 main.py
Toyota Camry
Toyota Mustang
Toyota Civic
Ford Camry
Ford Mustang
Ford Civic
Honda Camry
Honda Mustang
Honda Civic

Summary

List comprehensions provide a concise and readable way to create lists by applying an expression to each item in an iterable, making code more efficient and expressive. Iterators are objects that enable sequential access to data one element at a time, facilitating efficient processing of large collections without the need to load everything into memory. Generator expressions, on the other hand, produce memory-efficient and on-the-fly iterable objects, allowing lazy evaluation and ideal for large or infinite data sets. Each of these constructs plays a vital role in Python, offering versatile solutions for various programming needs, from data manipulation to memory optimization.