How to chunk a list into smaller Python lists

Need to divide a large list into smaller, more manageable sublists? Chunking, or batching, is a common task in Python, useful for processing data in segments, improving performance, or working with APIs that have size limitations.

This article provides a detailed, practical guide to chunking lists in Python using various methods, including list comprehensions, loops, and the itertools library. Learn different ways to chunk lists and find the best solution for your specific needs.

Let’s say we have a list of numbers:

 [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 

We want to chunk it into sublists of size 3, resulting in:

 [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
 

Method 1: Using a Loop and range()

This method iterates through the list with a specified step size, creating sublists of the desired chunk size.

 def chunk_list_loop(input_list, chunk_size):
     """Chunks a list into sublists of specified size using a loop."""
     result = []
     for i in range(0, len(input_list), chunk_size):
         result.append(input_list[i:i + chunk_size])
     return result

 my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 chunk_size = 3
 chunked_list = chunk_list_loop(my_list, chunk_size)
 print(chunked_list)
 

 [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
 

Explanation: The chunk_list_loop function takes an input_list and a chunk_size. The range(0, len(input_list), chunk_size) creates a sequence of indices starting from 0, incrementing by chunk_size. In each iteration, a slice of the original list (input_list[i:i + chunk_size]) is appended to the result list. This slice represents a chunk of the desired size. Chunking lists effectively and handling edge cases. This is the basic chunking lists method.

Method 2: Using List Comprehension

List comprehension provides a more concise way to achieve the same result as the loop-based method.

 def chunk_list_comprehension(input_list, chunk_size):
     """Chunks a list into sublists using list comprehension."""
     return [input_list[i:i + chunk_size] for i in range(0, len(input_list), chunk_size)]

 my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 chunk_size = 3
 chunked_list = chunk_list_comprehension(my_list, chunk_size)
 print(chunked_list)
 

 [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
 

Explanation: This method uses a list comprehension to create the same result as the previous method in a single line of code. It iterates through the same index range using range() and creates slices of the input list.

Method 3: Using itertools.zip_longest()

The itertools library offers a powerful tool for chunking lists, especially when dealing with potentially incomplete chunks at the end.

 from itertools import zip_longest

 def chunk_list_itertools(input_list, chunk_size):
     """Chunks a list using itertools.zip_longest."""
     args = [iter(input_list)] * chunk_size
     return [list(filter(None, group)) for group in zip_longest(*args, fillvalue=None)]

 my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 chunk_size = 3
 chunked_list = chunk_list_itertools(my_list, chunk_size)
 print(chunked_list)
 

 [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
 

Explanation:
* iter(input_list) creates an iterator from the list.
* [iter(input_list)] * chunk_size creates multiple references to the same iterator.
* zip_longest(*args, fillvalue=None) groups elements from the iterators until all elements from the longest iterator are exhausted. The fillvalue=None ensures that shorter iterators are padded with None.
* filter(None, group) removes the None values.
* list(...) converts the filtered group into a list.
This is more complex but very useful when you want to handle lists of varying sizes and ensure correct chunking behavior. The chunking lists is done effeciently by using itertools.

Method 4: Using NumPy (for Numerical Data)

If you’re working with numerical data and have NumPy available, this library provides efficient array manipulation tools.

 import numpy as np

 def chunk_list_numpy(input_list, chunk_size):
     """Chunks a list using NumPy."""
     arr = np.array(input_list)
     return np.array_split(arr, np.ceil(len(input_list) / chunk_size))

 my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 chunk_size = 3
 chunked_list = chunk_list_numpy(my_list, chunk_size)
 print([list(x) for x in chunked_list]) # Convert NumPy arrays back to lists for display
 

 [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
 

Explanation:
* np.array(input_list) converts the list into a NumPy array.
* np.ceil(len(input_list) / chunk_size) calculates the number of chunks required, rounding up to the nearest integer.
* np.array_split(arr, ...) splits the array into the calculated number of sub-arrays. It handles uneven splits gracefully. Note that the elements are NumPy arrays, so we use list comprehension to convert it to normal python lists.

Method 5: Generator Function

This method uses a generator, which is memory-efficient for very large lists.

 def chunk_list_generator(input_list, chunk_size):
     """Chunks a list using a generator."""
     for i in range(0, len(input_list), chunk_size):
         yield input_list[i:i + chunk_size]

 my_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 chunk_size = 3
 chunked_list = list(chunk_list_generator(my_list, chunk_size)) # Consume the generator into a list
 print(chunked_list)
 

 [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]
 

Explanation: The chunk_list_generator function is a generator. Instead of returning a list, it yields each chunk. This means the chunks are produced on demand, which can be more memory-efficient for large lists. We then convert the generator into a list using list() to view the result.