Selecting and Filtering
loc vs iloc — Label vs Position Indexing
This guide covers the fundamental distinction between label-based and position-based data access in Pandas. Mastering these accessors is essential for writing predictable code that avoids common indexing errors. Use these tools whenever you need to slice, filter, or retrieve subsets of your data structures.
The Core Philosophy of Indexing
To understand Pandas indexing, you must first recognize that a DataFrame is effectively a collection of axes. Each row and column has both a physical integer position and a specific label identifier. The 'loc' accessor is designed strictly for label-based selection, meaning it looks for the names you have assigned to your indices or columns. If you request a label that does not exist, Pandas will raise a KeyError, providing immediate feedback that your data structure lacks the expected identifying information. This is inherently robust because it forces you to reference the specific business logic of your labels rather than arbitrary memory offsets. Conversely, 'iloc' ignores these labels entirely, operating solely on the underlying integer position. It treats the DataFrame as a zero-indexed grid, making it ideal for algorithms or tasks where the structural location is more important than the semantic identifier of the data points.
import pandas as pd
# Setup a DataFrame with named indices
df = pd.DataFrame({'price': [10, 20]}, index=['A', 'B'])
# loc looks for the label 'A'
print(df.loc['A'])
# iloc looks for index position 0
print(df.iloc[0])Understanding loc: Label-Based Access
The 'loc' accessor is the primary way to interact with data when you know the specific identifiers, such as date-times or specific entity names. A critical property of 'loc' is that it is inclusive of both the start and the end point when using slices. If you define a range such as 'row1':'row5', Pandas will return everything from row1 up to and including row5. This is distinct from standard Python list slicing behavior, which is exclusive of the end index. Because labels are not strictly required to be unique, if your index contains duplicate labels, 'loc' will return a DataFrame containing all rows matching that label. This makes 'loc' a powerful tool for grouping and filtering datasets based on categorical keys. By relying on labels, your code remains readable even if the underlying data is reordered, as you are accessing data by its defined identity.
df = pd.DataFrame({'val': [1, 2, 3]}, index=['a', 'b', 'c'])
# Selecting a range of labels (inclusive of 'b')
print(df.loc['a':'b'])
# Selecting multiple non-contiguous rows
print(df.loc[['a', 'c']])Understanding iloc: Position-Based Access
The 'iloc' accessor provides a consistent way to slice data based on its physical location in the memory layout, regardless of what the index labels happen to be. Unlike 'loc', 'iloc' follows standard Python indexing conventions, where the end index of a slice is exclusive. If you perform a slice like 0:3, it will return indices 0, 1, and 2, but exclude index 3. This is essential for operations that depend on order, such as grabbing the first ten rows of a dataset or selecting every second row using step notation. Because it uses integer positions, 'iloc' is inherently safe from changes in index names or data types within your labels. It is the most reliable tool to use when you are writing general-purpose functions that must process data by its relative position rather than by its domain-specific metadata identifiers.
import numpy as np
data = pd.DataFrame(np.random.randn(5, 2))
# Slice using standard Python exclusive end index
print(data.iloc[0:2])
# Using step to select every other row
print(data.iloc[::2])Selecting Specific Subsets
Both accessors support two-dimensional selection, allowing you to filter by both rows and columns simultaneously using a comma-separated format. The pattern is always [row_selector, column_selector]. When you combine labels or positions, you can extract precise rectangular subsets of your data. For example, using 'loc' with labels for both axes allows you to isolate a specific metric for a specific entity. Using 'iloc' with integers allows you to extract sub-grids by coordinates. If you use a colon ':' in place of a selector, you are essentially telling Pandas to select all available items along that axis. This syntax is highly efficient and avoids the need for intermediate temporary variables, which helps minimize memory overhead when dealing with large DataFrames during data cleaning or feature engineering phases in a pipeline.
df = pd.DataFrame({'x': [1, 2], 'y': [3, 4]}, index=['r1', 'r2'])
# Subset using labels for row and column
print(df.loc['r1', 'x'])
# Subset using integer positions
print(df.iloc[0, 1])Advanced Masking and Alignment
Beyond simple slices, these accessors support advanced logic including Boolean masking and callable functions. A Boolean mask is an array of True and False values that acts as a filter; when passed to 'loc', it returns only the rows where the mask is True. This is arguably the most common way to filter data based on value thresholds. When using 'iloc', you cannot pass labels or column names; you must use integer masks, which are less common but useful for coordinate-based transformations. Furthermore, both accessors accept callable functions, which receive the DataFrame as an argument and return a valid indexer. This allows you to chain selection logic in a single fluent expression. By using these advanced techniques, you can write highly compact and expressive code that remains efficient by avoiding expensive loops over your datasets.
df = pd.DataFrame({'score': [50, 80, 95]})
# Using a boolean mask with loc
print(df.loc[df['score'] > 75])
# Using a callable function to select rows
print(df.loc[lambda d: d.score > 75])Key points
- The loc accessor is strictly for label-based indexing of data.
- The iloc accessor is strictly for integer-positional indexing of data.
- Slicing with loc is inclusive of both the start and end indices.
- Slicing with iloc is exclusive of the end index, following standard Python rules.
- The syntax for both accessors is always row selection followed by column selection.
- Labels in loc can be non-unique, which returns all matching records during selection.
- Integer position indexing with iloc is the most reliable way to perform coordinate-based slices.
- Both accessors support boolean masking and callables to enable complex, chained filtering workflows.
Common mistakes
- Mistake: Expecting loc to behave like standard Python slicing. Why it's wrong: loc is inclusive of the endpoint, while Python slicing is exclusive. Fix: Remember that loc includes the stop index, whereas iloc follows standard Python behavior (exclusive stop).
- Mistake: Using iloc with label names. Why it's wrong: iloc only accepts integer positions, not index labels. Fix: Use loc when referencing indices by name.
- Mistake: Trying to use loc for column positional indexing. Why it's wrong: loc requires the column label name, not its integer order. Fix: Use iloc for position-based column selection or get the column name from df.columns.
- Mistake: Assuming that index order is always the same as integer position. Why it's wrong: If the index is sorted differently (e.g., non-numeric or shuffled), label-based loc will find a specific index regardless of its row position. Fix: Use loc for semantic lookup and iloc for absolute row position.
- Mistake: Passing a list of integers to loc and getting unexpected results. Why it's wrong: If your index is numeric (e.g., integers), loc interprets those integers as labels, not positions. Fix: If you truly want position-based selection, use iloc.
Interview questions
What is the fundamental difference between loc and iloc in Pandas?
The fundamental difference lies in how they access data: loc is label-based, while iloc is integer position-based. When using loc, you provide the index labels of the rows or columns you want to select, such as specific strings or names. Conversely, iloc requires integer indices ranging from 0 to n-1, similar to standard Python list indexing. This distinction is crucial because relying on position can lead to errors if the data structure changes, whereas labels remain consistent with the data itself.
How does slicing behavior differ between loc and iloc?
Slicing behaves differently because of how their index types are interpreted. In iloc, the stop index is exclusive, following standard Python slicing conventions where the end point is not included. In contrast, loc is inclusive of the stop label. For example, if you slice with df.loc['A':'C'], the resulting selection includes 'C'. This is a common point of confusion for beginners, but it exists because label-based indexing does not have a concept of 'next' in the same way integer ordering does.
Under what circumstances would you choose loc over iloc for selecting data?
You should choose loc when your DataFrame has meaningful index labels, such as dates, ID strings, or categories. Using loc makes your code more readable and self-documenting; for example, df.loc['2023-01-01'] is much clearer than trying to remember which integer index corresponds to that date. Furthermore, loc is safer when your data might be reordered or subsetted, as it allows you to access specific records regardless of their current physical position in the table.
Compare the two approaches: when is it better to use iloc versus loc for performance and robustness?
The choice between these two approaches depends on your specific workflow. Use iloc when performing iterative operations or when you need the first or last 'n' rows regardless of the index labels, as it is often slightly faster because it avoids label lookups. However, loc is more robust for data analysis scripts where you rely on specific feature names. If you update the dataset or change sorting, iloc might grab the wrong row, whereas loc will consistently retrieve the record associated with your specific criteria.
How can you use loc to perform conditional filtering alongside label selection?
You can use loc to pass a boolean mask, which allows for powerful conditional filtering while simultaneously selecting specific columns. For instance, df.loc[df['price'] > 100, ['product', 'sku']] will filter rows where the price exceeds 100 and return only the product and sku columns. This is highly efficient because it performs the boolean check and the column slicing in a single, clean step, ensuring the code remains performant and readable even with large datasets.
Explain the potential pitfalls of using positional indexing (iloc) on a DataFrame that might have been filtered or sorted.
The primary pitfall of using iloc on a dynamic DataFrame is the loss of data integrity if the underlying data changes. If you write code expecting row 5 to contain a specific customer, but a sort operation or a filter has re-indexed the rows, iloc will blindly return whatever currently occupies that integer position. This leads to silent bugs where your analysis processes the wrong data. Always prefer loc for row access in production code unless you are specifically operating on positions, such as selecting the very first row.
Check yourself
1. Given a DataFrame 'df' with an index of [10, 20, 30], what does df.loc[10:20] return?
- A.Only the row with index 10
- B.The rows with indices 10 and 20
- C.The rows with indices 10, 20, and 30
- D.An error, because 10:20 is not a valid slice
Show answer
B. The rows with indices 10 and 20
loc is label-inclusive, so it includes both the start and end labels. Option 0 is wrong because it ignores the end. Option 2 is wrong because 30 is outside the slice. Option 3 is wrong because numeric indices are valid in loc.
2. What is the primary difference between df.loc[0] and df.iloc[0] if the DataFrame index is a string?
- A.They both return the first row
- B.iloc returns the first row; loc will raise a KeyError if 0 is not a label
- C.loc returns the first row; iloc raises an error
- D.They return the same thing because Pandas aliases them
Show answer
B. iloc returns the first row; loc will raise a KeyError if 0 is not a label
iloc uses positional index (0 is always the first row). loc looks for the literal string '0' as an index label; if that label doesn't exist, it raises a KeyError. The other options are incorrect because they ignore the strict distinction between position and label.
3. You want to select the first 3 rows and first 2 columns of a DataFrame. Which is the correct approach?
- A.df.loc[0:2, 0:1]
- B.df.iloc[0:3, 0:2]
- C.df.iloc[0:2, 0:1]
- D.df.loc[0:3, 0:2]
Show answer
B. df.iloc[0:3, 0:2]
iloc uses integer positions and is exclusive of the stop index (0 to 2 for rows means index 0, 1, 2). Option 0 and 3 use loc, which would look for literal labels '0' and '2'. Option 2 only selects 2 rows, not 3.
4. If you have a DataFrame where the index is [5, 4, 3, 2, 1], what does df.iloc[0:2] return?
- A.Rows with indices 5 and 4
- B.Rows with indices 1 and 2
- C.Rows with indices 5, 4, and 3
- D.An error, because the index is not sorted
Show answer
A. Rows with indices 5 and 4
iloc relies purely on the internal integer position of the data. The first two rows in the current order are 5 and 4, regardless of their label values. The other options confuse position with label sorting.
5. Which of the following is true regarding label-based selection using loc?
- A.It is always faster than iloc for large datasets
- B.It supports boolean arrays for masking
- C.It only accepts single labels, never lists
- D.It automatically converts integer labels to positions
Show answer
B. It supports boolean arrays for masking
loc is specifically designed to handle boolean masks where True values select the corresponding rows. Option 0 is false as performance varies. Option 2 is false as loc accepts lists. Option 3 is false as loc never treats labels as positions.