Delta Lake in pure Python: write Delta tables without Spark using the deltalake package (delta-rs)

There's a silent assumption in many data teams: "Delta Lake is a Spark thing". It makes sense — Delta was born inside the Databricks/Spark ecosystem, and most of the documentation assumes a running cluster. But that assumption is expensive when the work is small. Spinning up a Spark cluster to write a few hundred thousand rows into a Delta table is like renting a truck to carry a single box: it works, but you pay startup, infrastructure cost and a JVM dependency that didn't need to exist.

The deltalake package — the Python interface of the delta-rs project — undoes that assumption. It implements the Delta Lake protocol with a Rust and Apache Arrow core underneath, and delivers it all in Python without touching a JVM, Spark or Java. And the game-changing part for anyone already living in the Databricks world: the table it writes is a legitimate Delta table — the same one your Databricks workspace and Unity Catalog read afterward. It's not a "near-Delta"; it's real Delta, in the same file format and transaction log.

This article shows the what, the how and the when — and where the boundary with Spark really lies.

The concrete problem

A good part of a data team's workloads is not big data. They're API ingestions, files arriving hourly, exports from transactional systems, webhook dumps. Volumes that fit comfortably in the memory of a modest machine. For those cases, the "Spark writes to Delta" pattern has three problems:

1. Startup time. A cluster takes seconds to minutes to be ready. For a job that processes in 4 seconds, the overhead dominates.
2. Cost. An on cluster is money running, even idle between micro-batches.
3. Operational weight. A JVM, Spark versions, connector dependencies — a whole stack to maintain just to be able to do an append.

The right question isn't "Spark or no Spark forever?". It's: why pay for Spark when the work doesn't ask for Spark?

The solution: deltalake (delta-rs)

delta-rs is a Delta Lake implementation written in Rust, with Python bindings. Because it's Arrow-native, it talks naturally — and often with zero copy — to pandas, Polars, PyArrow, DuckDB, Dask and Daft. You read from any of those engines, write to Delta, and read it back in any other.

Step 1 — Install (one line)

pip install deltalake

No JDK, no SPARK_HOME, no connector to configure. The Rust binary ships inside the package.

Step 2 — Write from any DataFrame

from deltalake import write_deltalake, DeltaTable
import pandas as pd

df = pd.DataFrame({"day": ["2026-09-10"], "store": [42], "sales": [1830.0]})

# writes (creates the table if it doesn't exist)
write_deltalake("s3://lake/sales", df, mode="append")

The mode parameter accepts append (adds), overwrite (replaces) and works with schema_mode for schema evolution when new columns appear. It works on local disk (/data/sales), on S3, ADLS/Azure and GCS — you just pass credentials via storage_options or environment variables.

Swapping pandas for Polars is direct — Polars even has df.write_delta(...) on top of the same engine:

import polars as pl
pl.read_parquet("raw/*.parquet").write_delta("s3://lake/sales", mode="append")

Step 3 — Read, maintain and travel in time

dt = DeltaTable("s3://lake/sales")

# compacts small files and co-locates related data
dt.optimize.compact()
dt.optimize.z_order(["day"])

# read into whichever engine you want
df = dt.to_pandas()                 # pandas
ds = dt.to_pyarrow_dataset()        # Arrow / DuckDB / Polars lazy

# time travel: the table as it was at version 0
v0 = DeltaTable("s3://lake/sales", version=0)

Here's the value you don't have to build by hand:

• ACID commits. Every write is an atomic transaction recorded in _delta_log. No half-written file, no dirty read.
• Schema checking and evolution. The table rejects incompatible data; and you explicitly allow new columns when you want.
• Time travel. Every commit becomes a queryable version — auditing, reprocessing and debugging become trivial.
• Layout maintenance. optimize.compact() solves the "small files" problem; z_order([...]) improves data skipping on multi-column filters.

Bonus — Upsert (MERGE) and lightweight CDC

Need an incremental load that updates existing records? delta-rs exposes merge:

(
    dt.merge(
        source=new_df,
        predicate="target.id = source.id",
        source_alias="source",
        target_alias="target",
    )
    .when_matched_update_all()
    .when_not_matched_insert_all()
    .execute()
)

It's CDC (Change Data Capture) without standing up a Spark Structured Streaming pipeline for a volume that doesn't need one.

Where it shines — and where Spark still wins

Adopting delta-rs is not abandoning Spark. It's choosing the tool by the workload.

Use delta-rs when:

• The load fits (or nearly fits) in the memory of one machine: small and medium ingestions.
• You run in serverless functions or small containers, where standing up Spark is infeasible or expensive.
• You want local tests that write a Delta identical to production's.
• You're taking the first leap from a pandas prototype to a governed table, without rewriting everything in PySpark.

Stay with Spark when:

• There are heavy shuffles, joins between giant tables or aggregations that blow past RAM.
• You need high-throughput streaming with exactly-once guarantees at scale.
• The processing already lives on a Databricks cluster and distribution is the real bottleneck.

A governance note: on Databricks/Unity Catalog, managed tables get predictive optimization — OPTIMIZE runs on its own — and the current recommendation for layout is liquid clustering instead of ZORDER/partitions. In other words: use delta-rs's z_order where you handle maintenance yourself; let Unity Catalog handle it when the table is managed by it.

Conclusion

Delta Lake is no longer a synonym for Spark. With the deltalake package, a ten-line Python script writes, reads, optimizes and versions Delta tables — with ACID, schema checking and time travel out of the box — and delivers exactly the format the rest of your Databricks/Azure platform already consumes. The gain isn't swapping one technology for another: it's stopping paying for a cluster when the work doesn't ask for one, and shortening the distance between a prototype and a governed production table.

If you keep small ingestions running on Spark today, it's worth measuring: how much time and cost disappears when write_deltalake takes over?