How to chunk a list into equal parts – Python

When working with lists in Python, you might often encounter the need to divide them into smaller, more manageable parts. This process is called “chunking.” Chunking a list, particularly into equal parts, is a common task in data processing, parallel computing, and various other programming scenarios.

This article will explore several effective methods for chunking lists in Python, ensuring each chunk has (approximately) the same number of elements. We’ll cover approaches using list comprehensions, the itertools module, and NumPy, providing clear examples and explanations for each. Learn how to use itertools.zip_longest, numpy.array_split and other techniques for efficient and easy list chunking in Python.

Here’s a simple example showcasing the desired output:

# Input List
my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9]

# Chunked into parts of size 3
# Expected Output: [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

Method 1: Using List Comprehension

List comprehension offers a concise and Pythonic way to chunk a list. This method is straightforward and easy to understand, making it a good choice for simple chunking tasks. We will create chunks of size n using the power of list comprehension.

def chunk_list_comprehension(input_list, n):
    """Chunks a list into parts of size n using list comprehension."""
    return [input_list[i:i + n] for i in range(0, len(input_list), n)]

# Example usage:
my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
chunk_size = 3
chunked_list = chunk_list_comprehension(my_list, chunk_size)
print(chunked_list)

[[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]

Explanation:

The code defines a function chunk_list_comprehension that takes an input_list and a chunk size n as arguments. Inside the function, a list comprehension is used to iterate through the input_list with a step of n. For each step, a slice of the list from i to i + n is created, forming a chunk. The resulting list of chunks is then returned. If the length of the original list is not a multiple of the chunk size, the last chunk will contain the remaining elements. This avoids index out of bounds exceptions.

Method 2: Using itertools.zip_longest

The itertools module provides powerful tools for working with iterators. itertools.zip_longest (or itertools.izip_longest in Python 2) can be used to chunk a list, especially when you need to handle cases where the list length is not evenly divisible by the chunk size. We’ll use it with a clever iterator trick.

import itertools

def chunk_list_itertools(input_list, n):
    """Chunks a list into parts of size n using itertools.zip_longest."""
    args = [iter(input_list)] * n
    return [list(filter(None, chunk)) for chunk in itertools.zip_longest(*args, fillvalue=None)]

# Example usage:
my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
chunk_size = 3
chunked_list = chunk_list_itertools(my_list, chunk_size)
print(chunked_list)

[[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]

Explanation:

This code uses itertools.zip_longest to chunk the list. The core idea is creating n (chunk size) iterators over the same list. zip_longest then groups elements from these iterators together. fillvalue=None ensures shorter chunks are padded with None. The filter(None, chunk) part removes these None values, giving us clean chunks. Finally, a list comprehension converts each zipped tuple into a list.

Method 3: Using NumPy

NumPy, the numerical computing library, provides efficient array manipulation capabilities. NumPy’s array_split function is particularly useful for chunking arrays (and, by extension, lists) into a specified number of sub-arrays.

import numpy as np

def chunk_list_numpy(input_list, n):
    """Chunks a list into n parts using NumPy's array_split."""
    arr = np.array(input_list)
    chunks = np.array_split(arr, n)
    return [list(chunk) for chunk in chunks]

# Example usage:
my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
num_chunks = 3 # Number of chunks
chunked_list = chunk_list_numpy(my_list, num_chunks)
print(chunked_list)

[[1, 2, 3, 4], [5, 6, 7], [8, 9, 10]]

Explanation:

First, the input list is converted into a NumPy array. Then, np.array_split(arr, n) divides the array into n approximately equal sub-arrays. The critical thing to note here is that n represents the *number* of chunks, not the *size* of each chunk. The function automatically handles the distribution of elements if the list’s length isn’t divisible by `n`. The result is a list of NumPy arrays, which are then converted back into lists using a list comprehension.

Method 4: Chunking with a Generator

Using a generator is an efficient way to chunk a list, especially for very large lists, as it avoids creating intermediate lists in memory. This approach yields chunks one at a time.

def chunk_list_generator(input_list, n):
    """Chunks a list into parts of size n using a generator."""
    for i in range(0, len(input_list), n):
        yield input_list[i:i + n]

# Example usage:
my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
chunk_size = 3
chunked_list = list(chunk_list_generator(my_list, chunk_size))  # Convert generator to a list
print(chunked_list)

[[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]

Explanation:

The chunk_list_generator function iterates through the input list with a step of n, similar to the list comprehension method. However, instead of creating a list, it `yield`s a slice of the list (a chunk) at each iteration. The `yield` keyword makes this function a generator. To get the final list of chunks, the generator is converted into a list using list(). This method is memory-efficient as it generates chunks on demand.