Polars streaming: process data larger than RAM — without Spark

There's a common reflex on data teams: as soon as a file grows past a few gigabytes and pandas starts blowing up memory with a MemoryError, the default answer is "move it to Spark". But a good chunk of these cases isn't a real big data problem — it's a problem of data that doesn't fit in RAM all at once. And that's exactly the scenario Polars solves in a single Python process, with no cluster, no JVM, no extra infrastructure.

The difference is in how the data is processed. pandas is eager: it loads everything into memory and runs operation by operation. Polars has a streaming engine that works on the data in small batches, keeping a constant memory footprint — no matter whether the file is 1 GB or 500 GB.

The three concepts that make it work

1. Lazy API: scan_parquet reads nothing

When you call pl.read_parquet(), Polars loads the whole file into memory — eager behavior, just like pandas. pl.scan_parquet(), on the other hand, does something different: it doesn't read the data. It creates a LazyFrame, which is just a logical plan describing the operations you intend to run.

Nothing happens until you explicitly call .collect() (materialize into memory) or .sink_parquet() (write to disk). That's what opens the door to larger-than-RAM processing.

import polars as pl

# nothing is read here — only a lazy plan is created
lf = pl.scan_parquet("sales/*.parquet")

2. Query planner: optimization before execution

Because Polars knows the whole plan before running it, it optimizes the query — in a way an eager approach can't:

• Projection pushdown: reads from disk only the columns you actually use.
• Predicate pushdown: pushes filters down to read time, discarding rows before loading them.
• Reordering and fusion of operations to minimize passes over the data.

You write the code declaratively and chained; the planner takes care of the rest.

res = (
    lf.filter(pl.col("state") == "SP")
      .group_by("store")
      .agg(pl.col("amount").sum())
)

3. Streaming sinks: write results larger than RAM

The final step is where the magic happens. Instead of materializing everything with .collect(), you use a sink:

res.sink_parquet("summary.parquet")

sink_parquet() evaluates the query in streaming mode and writes the result straight to disk, batch by batch. This guarantees constant memory usage regardless of dataset size — and lets the final result be larger than the available RAM.

Under the hood, operators like group_by, sort, and the build and probe sides of an equi-join are spillable: when they detect memory pressure, they write the accumulated state to temporary files, freeing RAM. The file sinks (sink_parquet, sink_csv, sink_ipc) are spillable by nature — they already write to disk by definition.

The complete pipeline

Putting it all together, a pipeline that processes 100 GB of data on a machine with 8 GB of RAM looks like this:

import polars as pl

# 1. lazy plan — nothing is loaded
lf = pl.scan_parquet("sales/*.parquet")

# 2. transformations optimized by the query planner
res = (
    lf.filter(pl.col("state") == "SP")
      .group_by("store")
      .agg(pl.col("amount").sum())
)

# 3. streaming execution, writing to disk batch by batch
res.sink_parquet("output.parquet")

No cluster. No MemoryError. No rewriting anything in PySpark.

Why this matters (beyond speed)

Polars is built in Rust on top of Apache Arrow and typically delivers 5–50x gains over pandas on real workloads. But the argument here isn't only performance — it's architecture:

• Less infrastructure: a Python script replaces a cluster that has to be provisioned, monitored, and paid for.
• Less cost: no idle nodes, no distributed-orchestration overhead for a volume that runs comfortably on one machine.
• More portability: the same code runs on your laptop, in a small container, or on a VM — with no ecosystem dependencies.

And the ecosystem fits well: DuckDB can query a Polars DataFrame directly in memory via the Arrow interface, with zero serialization overhead. Many teams use both in the same pipeline — Polars for DataFrame manipulation and feature engineering, DuckDB for SQL analysis over the same data.

When Spark is still worth it

Being fair to the right tool matters. Spark still makes sense when:

• you need real horizontal scale, with data distributed across multiple nodes;
• the volume exceeds what a single machine (even with streaming and disk spill) can process in a reasonable time;
• you already have the ecosystem in place (Databricks, Unity Catalog governance, production jobs) and operational consistency weighs more than simplicity.

For the single-node larger-than-RAM processing that shows up on most teams — aggregations, joins, cleaning and transforming large files — the Polars streaming engine covers the case with room to spare. And the roadmap reinforces the direction: Polars 2.0 brings a redesigned streaming engine (morsel-driven parallelism + async state machines in Rust), and Polars Cloud extends the open-source engine to serverless execution, horizontal scale over partitioned data, and fault tolerance.

Summary in three steps

1. scan_parquet creates a lazy plan — nothing is read until you ask.
2. Chain filter / group_by / agg; the query planner optimizes the whole query.
3. sink_parquet runs in streaming and writes to disk batch by batch — results larger than RAM, with constant memory and no cluster.

Before provisioning a cluster on your next pipeline, measure whether a scan_parquet + sink_parquet doesn't solve it. Most of the time, it does.