Generators and iterators in Python: saving memory and increasing performance

Imagine that you need to process a log file that is several gigabytes in size. The simplest solution is to load the entire file into memory, split it into lines, and start processing. But what if there is not enough RAM? Or if there are dozens of such files? It is for such situations that Python has generators and iterators — powerful tools that allow you to work with large amounts of data without loading them completely into memory.

What are iterators?

An iterator is an object that allows you to iterate through the elements of a collection one at a time without loading the entire collection into memory at once. In Python, any object that implements the __iter__() and __next__() method is an iterator.

When you write for item in collection, Python implicitly calls iter(collection) to get the iterator, and then repeatedly calls next() to get the next element until the exception StopIteration is thrown.

# Example of a simple iterator
class CountDown:
    def __init__(self, start):
        self.current = start
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

# Use
counter = CountDown(5)
for num in counter:
    print(num)  # Displays: 5, 4, 3, 2, 1

Generators: iterators on steroids

Generators are a simplified way to create iterators. Instead of writing a class with __iter__() and __next__() methods, you simply create a function with the yield keyword. The generator automatically saves its state between calls and resumes execution from where it left off.

def count_down(start):
    while start > 0:
        yield start
        start -= 1

# Use
for num in count_down(5):
    print(num)  # Displays: 5, 4, 3, 2, 1

The advantage is obvious: the code has become more compact and understandable, and the functionality has remained the same.

Why does this save memory?

The main advantage of generators is lazy calculations. Elements are created only when they are needed, and can be processed and forgotten immediately. Let's look at a specific example:

# Bad: we load everything into memory
def read_large_file_bad(file_path):
    with open(file_path, 'r') as file:
        lines = file.readlines()  # The entire file is in memory!
    return [line.strip().upper() for line in lines]

# Good: we process line by line
def read_large_file_good(file_path):
    with open(file_path, 'r') as file:
        for line in file:  # file is already an iterator!
            yield line.strip().upper()

# Use
for processed_line in read_large_file_good('huge_log.txt'):
    # Processing one line at a time
    process(processed_line)

In the first case, if the file is 2 GB, your program will take up at least 2 GB of RAM. In the second case, it will only take a few kilobytes, regardless of the file size.

Practical examples of use.

Processing data from the API

APIs often return data page by page. The generator can hide this complexity:

def fetch_all_users(api_client, page_size=100):
    page = 1
    while True:
        users = api_client.get_users(page=page, size=page_size)
        if not users:
            break
        for user in users:
            yield user
        page += 1

# Use
for user in fetch_all_users(api):
    print(user['name'])  # Processing users one by one

Infinite sequences

Generators allow you to create infinite sequences without the risk of memory overflow:

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# You can take as many elements as you need
from itertools import islice
first_ten = list(islice(fibonacci(), 10))
print(first_ten)  # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Data processing pipelines

Generators are great for creating processing chains:

def read_csv(file_path):
    with open(file_path) as f:
        for line in f:
            yield line.strip().split(',')

def filter_by_status(rows, status):
    for row in rows:
        if row[2] == status:  # status in the third column
            yield row

def extract_emails(rows):
    for row in rows:
        yield row[1]  # email in the second column

# Assembling the conveyor
rows = read_csv('users.csv')
active_users = filter_by_status(rows, 'active')
emails = extract_emails(active_users)

# Only here does the real processing begin
for email in emails:
    send_notification(email)

The beauty of this approach is that each stage of the pipeline is independent and testable, and memory is used for only one element.

Generator expressions

In addition to generator functions, Python supports generator expressions — a compact syntax for creating generators:

# List comprehension — creates a list in memory
squares_list = [x**2 for x in range(1000000)]  # It takes up a lot of memory

# Generator expression — creates a generator
squares_gen = (x**2 for x in range(1000000))  # It hardly takes up any memory

# Can be used in functions
total = sum(x**2 for x in range(1000000))  # Effective!

Please note: round brackets are used instead of square brackets. This is one of the easiest and most effective changes you can make to your code to save memory.

When should you NOT use generators?

Despite all the advantages, generators are not always suitable:

When you need multiple access to data: the generator can only be passed once. If you need to access the data several times, it is better to use a list.

When access speed is important: getting an element by index from the list is instantaneous, and the generator will have to iterate from the beginning.

When data is placed in memory: if you have a small data set, the overhead of creating a generator may not be justified.

When you need to know the length: len() does not work with generators, and calculating the length requires a complete iteration.

Advanced techniques.

Delegating generators

With the help of yield from, you can delegate the execution to another generator:

def read_multiple_files(*file_paths):
    for path in file_paths:
        with open(path) as f:
            yield from f  # Delegate line-by-line reading

# Read multiple files as a single stream
for line in read_multiple_files('log1.txt', 'log2.txt', 'log3.txt'):
    process(line)

Two-way communication with generators

Generators can not only return values through yield, but also accept them through the send() method:

def running_average():
    total = 0
    count = 0
    average = None
    while True:
        value = yield average
        total += value
        count += 1
        average = total / count

# Use
avg = running_average()
next(avg)  # Starting the generator
print(avg.send(10))  # 10.0
print(avg.send(20))  # 15.0
print(avg.send(30))  # 20.0

Measuring efficiency

Let's look at the real difference in memory usage:

import sys

# List
numbers_list = [x for x in range(1000000)]
print(f"List: {sys.getsizeof(numbers_list) / 1024 / 1024:.2f} MB")

# Generator
numbers_gen = (x for x in range(1000000))
print(f"Generator: {sys.getsizeof(numbers_gen) / 1024:.2f} KB")

On my machine, the result shows a difference of thousands of times: the list takes about 8 MB, and the generator takes less than 0.1 KB.

Conclusion

Generators and iterators are not just syntactic sugar, but a fundamental tool for working with large amounts of data. They allow you to write more efficient code that scales to data of any size. Understanding when and how to use generators distinguishes an experienced developer from a beginner.

Start simple: replace list comprehensions with generator expressions where you don't need multiple access to data. Then try to write a generator function for processing files. And gradually you will find that you are thinking about data streams, not about collections in memory.

Appendix Code offers structured courses for beginner developers covering Python, JavaScript, HTML, CSS and many other technologies.

Join our Telegram channel, where you will find community support, additional materials, and answers to questions from experienced developers.

Learn programming effectively and with pleasure!