Narwhals: write DataFrame code once and run it on pandas, Polars and PyArrow

Anyone who has worked with data in Python for a while has lived this: one project standardized on pandas, another moved to Polars for performance, a third consumes data via PyArrow. Then you need a cleaning function — trim strings, filter invalid rows, aggregate by key — and you find that each library's API is different enough that you end up writing the same transformation two or three times.

Worse still when you maintain an internal library reused by several teams: if it depends on pandas, you force pandas on everyone; if it depends on Polars, you force Polars. There's no elegant middle ground — until you meet Narwhals.

What Narwhals is

Narwhals is a lightweight, extensible compatibility layer between DataFrame libraries. The core idea is simple: you write the logic once, using a Polars-style API (expressions, nw.col(...), chained methods), and Narwhals translates it into native operations of whatever backend you pass in — pandas, Polars, PyArrow, Modin, Dask or cuDF (GPU).

Two design principles make the library especially useful for data engineering:

1. Backend-agnostic with type-faithful return. If you passed in a pandas DataFrame, you get pandas back. If you passed in Polars, you get Polars. The library doesn't push you toward a single format.
2. Zero required dependencies. Narwhals lazily imports the backend it finds at the call site. A project that ships Narwhals does not pull in pandas, Polars or PyArrow unless one of them is already installed. That keeps your dependency tree lean.

The usage pattern in 3 steps

Narwhals' canonical flow always follows the same shape:

import narwhals as nw

def clean(df):
    # 1) Wrap any native DataFrame in a Narwhals object
    df = nw.from_native(df)

    # 2) Write the logic with the Narwhals API (Polars-style)
    out = (
        df.filter(nw.col("value") > 0)
          .with_columns(nw.col("name").str.to_uppercase())
    )

    # 3) Return the original native type (pandas -> pandas, etc.)
    return out.to_native()

The same clean() function now works with any of these:

import pandas as pd
import polars as pl
import pyarrow as pa

clean(pd.read_parquet("sales.parquet"))          # returns pandas.DataFrame
clean(pl.read_parquet("sales.parquet"))          # returns polars.DataFrame
clean(pa.parquet.read_table("sales.parquet"))    # returns pyarrow.Table

Not a single line of logic changes between the three cases. That's the win.

A slightly more real example: aggregation

A group_by is also written just once:

import narwhals as nw

@nw.narwhalify        # decorator: does from_native/to_native automatically
def revenue_by_state(df):
    return (
        df.group_by("state")
          .agg(nw.col("value").sum().alias("revenue"))
          .sort("revenue", descending=True)
    )

The @nw.narwhalify decorator saves you from calling from_native/to_native manually — it wraps the input arguments and unwraps the output for you.

Why it matters in practice

• Less code to maintain. One transformation codebase instead of one per library. For teams that maintain internal utilities, it's the difference between maintaining one codebase or five.
• No library lock-in. You can start on pandas and migrate critical parts to Polars without rewriting the business logic — you just swap the input DataFrame.
• Minimal overhead. Narwhals is essentially a thin translation to the native API; it doesn't reimplement the execution engine. You keep the performance of the chosen backend (Polars stays fast, cuDF stays on the GPU).
• Not a risky bet. Widely used tools have already adopted Narwhals internally to accept any DataFrame without forcing a conversion to pandas: Plotly Express, Altair, scikit-learn, marimo, shiny and Hugging Face datasets. When libraries of that size bet on a compatibility layer, it's a sign the problem is real and the solution is solid.

When (not) to use it

Narwhals is, by design, a tool aimed at people who build tools — library authors and engineers who write utilities reused across several projects or several backends. If your project is a single application that already standardized on Polars and will never leave Polars, writing straight to the Polars API is perfectly adequate — you don't gain much by adding a layer.

The value shows up when the input is uncertain or plural: a public function that might receive pandas or Polars, a pipeline that consumes from different sources, an internal library that doesn't want to impose a backend on its consumers. In those scenarios, Narwhals more than pays for itself.

How to get started

pip install narwhals
# or, if you use uv:
uv add narwhals

No extra dependencies: install only the backends you already use. The official documentation has the quickstart guide and the full list of supported operations.