In the world of Python programming, mastering the art of iterators, harnessing the elegance of list comprehensions, and unleashing the efficiency of generator expressions are fundamental skills every developer should possess. We will embark on a journey through these essential Python constructs, exploring how they can streamline our code, enhance our data processing capabilities, and empower us to write more concise and readable scripts.
Let's recall the simple ParkingLot
class and its constructor from the previous post.
class ParkingLot:
def __init__(self, spots: int):
self.__spots: list[Car] = list()
for _ in range(spots):
self.__spots.append(None)
The primary purpose of the __init__
method is to initialize the attributes and state of a ParkingLot
object when it is created. It takes one parameter, spots, which represents the number of parking spots in the parking lot. Inside the __init__
method the self._spots attribute is initialized as an empty list ([]
). This attribute will be used to keep track of the occupancy status of each parking spot. It then uses a for
loop to iterate spots times (specified by the spots parameter) and appends None
to the self.__spots list for each spot. The result is that the self.__spots
list is populated with None
values, representing empty parking spots. The number of empty spots is determined by the value of the spots parameter. For the full code example, please follow the link to the previous post. 👇👇👇
But the truth must be told! The for
loop in the constructor can be rewritten in a more clean, elegant and concise way using list comprehension.
List comprehension
List comprehension is a concise and powerful feature in Python for creating lists. They provide a compact and readable way to generate new lists by applying an expression to each item in an existing iterable (such as a list, tuple, or range) and optionally filtering items based on a condition.
The basic syntax of list comprehension consists of square brackets []
containing an expression followed by a for
clause. Optionally, we can include a if
clause for filtering. Here's the general structure.
new_list = [expression for item in iterable]
Let's see a simple example of creating a list with integers from 0 to 9.
>>> ints = [x for x in range(0, 10)]
>>> print(ints)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
In a very similar manner, squares can be calculated.
>>> squares = [x**2 for x in range(0, 10)]
>>> print(squares)
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
In the above example, we just substitute x
with x**2
to calculate squares.
We can include an if
clause to filter items from the iterable based on a condition. Only items that satisfy the condition will be included in the new list.
Let's create a list of even numbers within the range from 0 to 9.
>>> even_numbers = [x for x in range(0, 10) if x % 2 == 0]
>>> print(even_numbers)
[0, 2, 4, 6, 8]
We can use nested list comprehensions to create more complex lists or lists of lists. This is especially useful for working with multi-dimensional data structures.
>>> matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
>>> flattened = [num for row in matrix for num in row]
>>> print(flattened)
[1, 2, 3, 4, 5, 6, 7, 8, 9]
List comprehensions are often more concise than writing equivalent loops, making our code more readable, therefore they express the intent of our code more clearly. They can be more efficient than traditional loops for simple operations.
Now we can easily rewrite the ParkingLot
constructor using list comprehension.
class ParkingLot:
def __init__(self, spots: int):
self.__spots: list[Car] = [None for _ in range(spots)]
However, that's not everything. We can rewrite the code to make it even more concise! We have already seen that trick in the following post. 👇👇👇
Instead of list comprehension, we can just use the *
operator to repeat elements in the list.
class ParkingLot:
def __init__(self, spots: int):
self.__spots: list[Car] = spots * [None]
Charming, right? 🥹
List comprehensions are best suited for creating new lists. They are not ideal for modifying existing lists in place. Complex operations might become less readable in list comprehensions. In such cases, using traditional for
loops can be more appropriate.
List (and other sequences) can be easily looped over as they provide a unified way of accessing elements named iterators.
Iterators
Iterators are a fundamental concept in Python used to traverse through sequences of data one element at a time. They provide a standardized way to access elements of an iterable object (like lists, tuples, strings, or custom objects) without having to know the underlying details of the data structure.
Python's for
loop is designed to work seamlessly with iterators. When we use a for
loop to iterate over an iterable, it automatically calls iter()
to obtain an iterator and then repeatedly calls next()
to retrieve values until a StopIteration
exception is raised.
An iterator is an object that implements two methods: __iter__()
and __next__()
. These methods define how the iterator should behave:
__iter__()
: this method returns the iterator object itself. It's used to initialize or reset the iterator. In most cases, it simply returns self,__next__()
: this method returns the next value from the iterator. If there are no more items to return, it raises theStopIteration
exception.
We can create an iterator from an iterable object using the iter()
function.
>>> my_list = [x for x in range(10)]
>>> print(my_list)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> my_iterator = iter(my_list)
>>> print(my_iterator)
<list_iterator object at 0x10bbbfca0>
my_iterator is now an iterator that can be used to traverse through my_list one element at a time. We can use the next()
function to retrieve the next item from an iterator.
>>> print(next(my_iterator))
0
>>> print(next(my_iterator))
1
>>> print(next(my_iterator))
2
...
>>> print(next(my_iterator))
9
>>> print(next(my_iterator))
Traceback (most recent call last):
File "pydevconsole.py", line 364, in runcode
coro = func()
^^^^^^
File "<input>", line 1, in <module>
StopIteration
The next()
function moves the iterator's internal pointer to the next element in the sequence. Once we go past the last element the StopIteration
exception is raised.
Let's create a CarShowcase
class to display cars. To create a CarShowcase
class that implements an iterator, we can define the class with a collection of car objects and provide methods for adding cars and iterating through them.
from typing import List
class CarShowcase:
def __init__(self):
self.cars: List[Car] = []
self.index: int = 0
def add_car(self, car: Car):
if isinstance(car, Car):
self.cars.append(car)
else:
raise ValueError(f'Only instances of the Car class can be added to the showcase.')
def __iter__(self):
self.index = 0
return self
def __next__(self) -> Car:
if self.index < len(self.cars):
car = self.cars[self.index]
self.index += 1
return car
raise StopIteration(f'End of showcase reached')
The CarShowcase
class maintains a list of cars and includes methods to add cars to the showcase. It implements the iterator protocol by defining __iter__
and __next__
methods. The __iter__
method returns self
, and the __next__
method iterates through the list of cars, returning each car one at a time until it reaches the end of the showcase.
Here's how we can use the CarShowcase
class:
def main():
showcase = CarShowcase()
showcase.add_car(Car(f'Toyota', f'Camry'))
showcase.add_car(Car(f'Ford', f'Mustang'))
showcase.add_car(Car(f'Honda', f'Civic'))
for car in showcase:
print(car)
if __name__ == "__main__":
main()
Now we can run it.
❯ python3 main.py
Toyota Camry
Ford Mustang
Honda Civic
We can easily loop through the showcase to see each car.
Lists comprehensions always create lists that obviously consume memory. Generator expressions, on the other hand, create generator objects that produce values lazily and are more memory-efficient for large datasets.
Generator expressions
Generator expressions are a compact way to create generator objects, which are a type of iterable in Python. They are similar to list comprehensions but with one key difference: list comprehensions create lists in memory, while generator expressions create generator objects that yield values on the fly. Generator expressions use parentheses ()
instead of square brackets []
.
The basic syntax of generator expression looks as follows.
generator_object = (expression for item in iterable)
For instance, let's create a generator for the squares of numbers from 0 to 10.
>>> squares_generator = (x**2 for x in range(0, 10))
>>> next(squares_generator)
0
>>> next(squares_generator)
1
>>> next(squares_generator)
4
>>> next(squares_generator)
9
>>> next(squares_generator)
16
We can iterate through squares_generator
using a for
loop or by explicitly calling the next()
function to yield values one at a time. Generator expressions are memory-efficient, making them suitable for large datasets.
We can use generator expressions to generate new cars. Let's see an example.
car_generator = (Car(f'Toyota', model) for model in [f'Camry', f'Corolla', f'Prius'])
for car in car_generator:
print(car)
The above code efficiently generates car objects with the fixed make "Toyota" and varying models using a generator expression, and it prints the cars within the loop. This approach is memory-efficient and allows us to work with large or dynamic data sets without pre-creating all the objects.
Unlike generator expressions, which are concise and one-liners, generator functions are user-defined functions with a special yield
statement. The basic structure is as follows.
def my_generator():
# Any setup code
yield value
# More code
The yield
keyword is used to specify the value to yield to the caller. When yield
is encountered, the generator's state is saved, and the value is returned to the caller. Execution of the generator function is paused until the next value is requested.
Generator functions are lazily evaluated, meaning they generate values on the fly as we iterate through them. This lazy evaluation is memory-efficient, as it doesn't require storing all values in memory at once, making generator functions ideal for working with large or infinite datasets. Generator functions offer a high degree of flexibility. We can incorporate custom logic, conditions, and computations in our generator, allowing us to generate values based on our specific requirements.
Let's see how we can use a generator function to create the Fibonacci sequence.
def fibonacci_generator(n) -> Generator[int, None, None]:
a, b = 0, 1
count = 0
while count < n:
yield a
a, b = b, a + b
count += 1
The function takes one argument, n
, which specifies how many Fibonacci numbers we want to generate. It uses two variables, a
and b
, to keep track of the current and next Fibonacci numbers. Inside a while
loop, it yields the current Fibonacci number (a
), updates a
and b
to calculate the next Fibonacci number, and increments the count
until it reaches n
.
if __name__ == "__main__":
for f in fibonacci_generator(5):
print(f)
Let's generate and print the first 5 Fibonacci numbers.
❯ python3 main.py
0
1
1
2
3
Generator expressions are often more concise for simple data transformations, while generator functions offer more flexibility and are better suited for complex data generation and processing tasks.
Similar to generator expressions, we can use generator functions to generate new cars. The output will show car objects with various makes and models. Let's see an example.
from typing import List, Generator
def car_generator() -> Generator[Car, None, None]:
makes: List[str] = ["Toyota", "Ford", "Honda"]
models: List[str] = ["Camry", "Mustang", "Civic"]
for make in makes:
for model in models:
yield Car(make, model)
for car in car_generator():
print(car)
This code demonstrates how to use a generator function to create car objects lazily and efficiently without storing all of them in memory, which can be especially useful for handling large datasets.
❯ python3 main.py
Toyota Camry
Toyota Mustang
Toyota Civic
Ford Camry
Ford Mustang
Ford Civic
Honda Camry
Honda Mustang
Honda Civic
Summary
List comprehensions provide a concise and readable way to create lists by applying an expression to each item in an iterable, making code more efficient and expressive. Iterators are objects that enable sequential access to data one element at a time, facilitating efficient processing of large collections without the need to load everything into memory. Generator expressions, on the other hand, produce memory-efficient and on-the-fly iterable objects, allowing lazy evaluation and ideal for large or infinite data sets. Each of these constructs plays a vital role in Python, offering versatile solutions for various programming needs, from data manipulation to memory optimization.