Python’s **generators** and the **itertools** module offer powerful techniques for working with iterable objects. By understanding these features, you can write code that is not only efficient but also elegant and concise.

## 1. Exploring Generators.

### 1.1 What is Python Generator.

A Python generator is a special type of iterable, similar to lists or tuples, but with one key difference: they don’t store all the values in memory at once. Instead, they generate values on the fly, as they’re needed. This makes them memory efficient, especially when dealing with large datasets or infinite sequences.

### 1.2 How to Define Python Generator.

Generators are defined using a function that contains one or more `**yield**` statements. When the function is called, it doesn’t execute immediately but returns a generator object. When the generator’s `**next()**` method is called, the function’s execution is resumed until a `**yield**` statement is encountered. The value of the `**yield**` statement is returned, and the function’s state is saved. The next time `**next()**` is called on the generator, execution resumes from where it left off.

Here’s a simple example:

def my_generator(): yield 1 yield 2 yield 3 gen = my_generator() print(type(gen)) print(next(gen)) # Output: 1 print(next(gen)) # Output: 2 print(next(gen)) # Output: 3

Generators are commonly used when you need to iterate over a large collection of items, or when you’re dealing with streams of data that are too large to fit into memory all at once. They’re also useful for creating infinite sequences, like the Fibonacci sequence or prime numbers.

### 1.3 Creating Custom Iterable Objects.

Generators in Python provide a convenient way to create iterable sequences without loading all elements into memory at once. Let’s explore some examples of creating and utilizing generators.

#### 1.3.1 Generating Squares.

Example source code:

# Define a function named 'squares' which takes a default argument n=10 def squares(n=10): # Print a message indicating the range of squares to be generated print(f"Generating squares from 1 to {n ** 2}") # Iterate over the range from 1 to n (inclusive) for i in range(1, n + 1): # Yield the square of the current value of i yield i ** 2 # Create a generator object 'gen' by calling the 'squares' function gen = squares() print(type(gen)) # Iterate over the elements yielded by the generator for x in gen: # Print each yielded value separated by a space print(x, end=" ")

Output:

Generating squares from 1 to 100 1 4 9 16 25 36 49 64 81 100

#### 1.3.2 Generating Fibonacci Sequence.

Example source code.

# Define a function named 'fibonacci' to generate Fibonacci sequence def fibonacci(): # Initialize the first two numbers of the Fibonacci sequence a, b = 0, 1 # Iterate indefinitely to generate Fibonacci numbers while True: # Yield the current Fibonacci number yield a # Update variables to calculate the next Fibonacci number a, b = b, a + b # Create a generator object 'fib_gen' by calling the 'fibonacci' function fib_gen = fibonacci() print(fib_gen) # Iterate 15 times to generate and print 15 Fibonacci numbers for _ in range(15): # Print each generated Fibonacci number separated by a space print(next(fib_gen), end=" ")

Output:

0 1 1 2 3 5 8 13 21 34 55 89 144 233 377

### 1.4 Leveraging Generator Expressions.

Generator expressions are concise expressions for creating generators on the fly. They are especially useful when dealing with large datasets. Let’s see some examples of generator expressions in action.

#### 1.4.1 Calculate sum of Squares using Generator Expression.

In Python, generator expressions are similar to list comprehensions but return an iterator that generates items lazily rather than creating a full list in memory.

In the below code `**sum(x ** 2 for x in range(100))**`, the expression `**(x ** 2 for x in range(100))**` is a generator expression. It generates the square of each number from 0 to 99 lazily as needed. The `**sum()**` function then consumes these values, summing them up.

So, instead of creating a list of 100 elements with their squares and then summing them up, this code calculates the sum directly using a generator expression, which is more memory-efficient.

Example source code.

def generator_expressions_sum(): ''' # Calculate the sum of squares of numbers from 0 to 00 using a generator expression gen = (x ** 2 for x in range(100)) #gen = [x ** 2 for x in range(100)] print(type(gen)) result = sum(gen) ''' result = sum(x ** 2 for x in range(100)) print(type(result)) # Print the result print("Sum of squares:", result)

Output:

Sum of squares: 328350

#### 1.4.2 Creating Dictionary with Generator Expression.

Example source code.

def generator_expressions_dictionary(): # Create a dictionary where keys are numbers from 0 to 4 and values are the squares of the keys gen = ((i, i ** 2) for i in range(5)) print(type(gen)) dictionary = dict(gen) #dictionary = dict((i, i ** 2) for i in range(5)) print(type(dictionary)) ''' dictionary = {} for i in range(5): dictionary[i] = i ** 2 ''' # Print the generated dictionary print("Generated Dictionary:", dictionary)

Output:

Generated Dictionary: {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

#### 1.4.3 Filtering Odd Numbers.

Example source code.

def generator_expressions_odd_number(): # Create a generator expression to generate odd numbers from 0 to 19 odd_gen = (x for x in range(20) if x % 2 != 0) #odd_gen = [x for x in range(20) if x % 2 != 0] print(odd_gen) # Convert the generated odd numbers into a list and print print("Odd Numbers:", list(odd_gen))

Output:

Odd Numbers: [1, 3, 5, 7, 9, 11, 13, 15, 17, 19]

#### 1.4.4 Generating Powers of 2.

**Generating Powers of 2**:

def generator_expressions_power_of_2(): # Create a generator expression to generate powers of 2 from 2^0 to 2^9 powers_of_two = (2 ** i for i in range(10)) #powers_of_two = [2 ** i for i in range(10)] print(powers_of_two) # Convert the generated powers of 2 into a list and print print("Powers of Two:", list(powers_of_two))

Output:

Powers of Two: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]

## 2. Exploring itertools Module.

### 2.1 Unleashing the Power of itertools.

The itertools module provides various functions for creating iterators for efficient iteration. Let’s delve into some examples to understand the versatility of itertools.

#### 2.1.1 Grouping Students by Grade.

Example source code.

import itertools as it # Define a function to extract the grade of a student def get_grade(student): return student[1] def grouping_students_by_grade(): # Define a list of tuples representing students and their grades students = [("Alice", "A"), ("Bob", "B"), ("Charlie", "A"), ("David", "B")] # Group students by their grades, sorted by grade using itertools.groupby sorted_students = sorted(students, key=get_grade) for grade, group in it.groupby(sorted_students, key=get_grade): #for grade, group in it.groupby(sorted(students, key=get_grade), key=get_grade): #for grade, group in it.groupby(students, key=get_grade): # Print the grade print("Grade:", grade) # Print the names of students in the group print("Students:", [student[0] for student in group])

Output:

Grade: A Students: ['Alice', 'Charlie'] Grade: B Students: ['Bob', 'David']

#### 2.1.2 Calculating Running Average.

Example source code.

import itertools as it # Define a function to calculate the running average of numbers in an iterable def running_average(iterable): total = 0 # Initialize the sum of numbers count = 0 # Initialize the count of numbers for item in iterable: total += item # Add the current number to the total sum count += 1 # Increment the count of numbers yield total / count # Yield the running average def calculating_running_average(): numbers = [1, 2, 3, 4, 5] # Define a list of numbers avg_gen = running_average(numbers) # Create a generator for running averages print("Running Average:", list(avg_gen)) # Print the running averages

Output:

Running Average: [1.0, 1.5, 2.0, 2.5, 3.0]

#### 2.1.3 Grouping Names by First Letter.

import itertools as it # Define a function to extract the first letter of a string def first_letter(x): return x[0] def grouping_names_by_first_letter(): # Define a list of names names = ["Alan", "Adam", "Wes", "Will", "Walt", "Albert", "Asiom", "Steven", "Stone"] # Group names by their first letters using itertools.groupby for letter, grouped_names in it.groupby(sorted(names), first_letter): # Print the first letter print(letter, end=" ") # Print the names in the group print(list(grouped_names))

Output:

A ['Alan', 'Adam'] W ['Wes', 'Will', 'Walt'] A ['Albert', 'Asiom'] S ['Steven', 'Stone']

In this code, the names are grouped by their first letters using **itertools.groupby()**. The function **first_letter()** extracts the first letter of each name. Then, the code iterates over the grouped names, printing each letter followed by the list of names starting with that letter.

### 2.2 Key itertools Functions.

Here are additional itertools functions along with their descriptions:

`**accumulate(iterable[, func])**`: Generates accumulated sums or any other binary function results.

`**dropwhile(predicate, iterable)**`: Skips elements from the iterable as long as the predicate is true.

`**takewhile(predicate, iterable)**`: Returns elements from the iterable as long as the predicate is true.

#### 2.2.1 `accumulate(iterable[, func]).

Here’s an example demonstrating the `**accumulate**` function from the `**itertools**` module, along with explanatory comments:

import itertools as it # Example: Accumulating Sums def accumulating_sums(): numbers = [1, 2, 3, 4, 5] # Using accumulate to generate accumulated sums accumulated_sums = it.accumulate(numbers) # Iterating over the accumulated sums print("Accumulated Sums:") for sum_so_far in accumulated_sums: print(sum_so_far, end=" ")

Output.

Accumulated Sums: 1 3 6 10 15

Explanation.

- We import the `itertools` module to access the `accumulate` function. - We have a list `numbers` containing some numeric values. - We apply `itertools.accumulate()` to the `numbers` list, which generates accumulated sums. - Inside the loop, we iterate over the accumulated sums generated by `accumulate`. - Each value in the loop represents the accumulated sum up to that point. - Finally, we print the accumulated sums.

In this example, `**accumulate**` produces the accumulated sums `**[1, 1+2, 1+2+3, 1+2+3+4, 1+2+3+4+5]**`, which are `**[1, 3, 6, 10, 15]**`, respectively.

#### 2.2.2 `dropwhile(predicate, iterable)`.

Here’s an example demonstrating the `**dropwhile**` function from the `**itertools**` module, along with explanatory comments:

import itertools as it # Example: Dropping Elements While Condition is True def drop_while(): numbers = [1, 3, 5, 7, 2, 4, 6, 9, 11, 8] # Defining the predicate function def is_odd(x): return x % 2 != 0 def is_bigger(x): return x > 5 # Using dropwhile to skip odd numbers until a even number is encountered result = it.dropwhile(is_odd, numbers) #result = it.dropwhile(is_bigger, sorted(numbers, reverse=True)) # Iterating over the result print("Result after dropping odd numbers:") for num in result: print(num, end=" ")

Output.

Result after dropping odd numbers: 2 4 6 9 11 8

Explanation.

- We import the `itertools` module to access the `dropwhile` function. - We have a list `numbers` containing both odd and even numbers. - We define a predicate function `is_odd` that returns `True` if the number is odd. - We apply `itertools.dropwhile()` to the `numbers` list along with the `is_odd` function. This function skips elements from the iterable as long as the predicate (in this case, `is_odd`) is true. - Inside the loop, we iterate over the result, which contains elements after the condition of the predicate becomes false. - We print the result after dropping the odd numbers.

In this example, `**dropwhile**` skips elements from the `**numbers**` list until it encounters the first even number (`**2**`). After that, it yields all the elements.

#### 2.2.3 takewhile(predicate, iterable).

Below is an example of how to use the function **takewhile(predicate, iterable)**.

import itertools as it # Example: Taking Elements While Condition is True def take_while(): # Define a predicate function to check if a number is less than 5 def less_than_6(x): return x < 6 # Define an iterable (in this case, a list of numbers) numbers = [1, 3, 5, 7, 2, 4, 6] # Use takewhile to take numbers from the iterable while they satisfy the predicate # The takewhile function stops taking elements as soon as the predicate is False result = it.takewhile(less_than_6, sorted(numbers)) # Convert the result into a list and print it print("Numbers less than 5:", list(result))

Output.

Numbers less than 6: [1, 2, 3, 4, 5]

Explanation.

- We define a predicate function `less_than_5(x)` that checks if a number is less than 5. - We have a list of numbers `[1, 3, 5, 7, 2, 4, 6]`. - We use `itertools.takewhile()` to take elements from the list `numbers` while they satisfy the predicate `less_than_5()`. - As soon as an element fails the predicate (in this case, when we reach 5), `takewhile` stops taking elements. - The result is then converted into a list and printed. So, the output contains numbers less than 5, which are `[1, 3]`.

## 3. Conclusion.

By incorporating generators and the itertools module into your Python code, you can enhance its performance and readability. Experiment with the provided examples and explore the extensive capabilities of these features to become a more proficient Python developer.

## 4. Demo Video for This Article.

**1. Video for how to use Python generator with examples.**

**2. Video for how to use Python itertools module with examples.**