Auto Partitioning in the Fabric Data Factory Copy job: move giant tables in minutes, with no partition setup

One-line summary: the Fabric Data Factory Copy job now partitions large tables automatically — it picks the column, computes the boundaries and runs parallel reads, all from a single toggle in Advanced settings.

The bottleneck everyone has felt

If you've ever built an ingestion from a large relational database into the lakehouse, you know the scene: the initial load of a transactions table with hundreds of millions of rows locks the processing window for hours. The reason is simple — by default, a copy reads the source on a single thread. A sequential read of 500 million rows is slow by nature, no matter how robust the destination is.

The traditional solution was to partition manually: pick a column with good distribution (an id or a date), compute value ranges that split the table into balanced chunks, and configure how many parallel reads to run. It works, but it has three problems. First, it's work, and it has to be redone for each table. Second, it's easy to get wrong — a column with skewed distribution produces unbalanced partitions, and then one thread carries 80% of the data while the others sit idle. Third, when the volume changes, the configuration that was good becomes a bottleneck again.

What changed: Auto Partitioning (Preview)

Auto Partitioning in the Fabric Data Factory Copy job hands that work to the platform itself. Instead of you defining the partitioning strategy, the engine does it on its own:

1. Analyzes the schema and the data characteristics at the source.
2. Selects the appropriate partitioning column.
3. Computes balanced boundaries between the partitions.
4. Runs multiple concurrent reads — with no input from you.

Partitioning means breaking a large set into smaller chunks that can be read and written at the same time. It's that concurrency that changes the throughput bracket: the difference between a single-thread read and a partitioned parallel read is, literally, the difference between hours and minutes.

Adaptive strategy: it scales with the data

The most elegant part is that the strategy adapts to the table size. Larger tables get more partitions; small tables run without the partitioning overhead. Whether it's a 100-row lookup table or a 500-million-row transaction log, the Copy job applies the right strategy automatically. You don't have to decide from what size partitioning "is worth it" — the engine decides.

How to enable it (it's a toggle)

In practice, the configuration is minimal. In your Copy job, under Advanced settings, you flip the Auto-partitioning toggle. That's it.

Conceptually, the job definition looks like this:

// Copy job → Advanced settings
{
  "source": "AzureSqlDatabase",
  "table": "dbo.transactions",   // ~500M rows
  "copyMode": "incremental",
  "watermark": "modified_at",
  "autoPartition": true          // ← the whole trick
}
// The engine picks the column and the boundaries,
// and runs N reads in parallel. No tuning.

Note that Auto Partitioning is compatible with incremental watermark loading — that is, it works both on the initial full copy and on the subsequent incremental loads. You combine the benefit of parallel partitioning with the incremental logic you already use to avoid reprocessing the whole table every day.

Where it works

Right now (Preview), Auto Partitioning is supported for watermark-based loading scenarios (initial full + incremental) on the main relational connectors:

• Azure SQL Database
• SQL Server / Azure SQL Managed Instance
• Amazon RDS for SQL Server
• Azure Synapse Analytics
• Oracle
• SAP HANA
• Fabric Data Warehouse
• Fabric Lakehouse tables

Since it's Preview, it's worth validating in a development environment before taking it to production, and tracking the limits of each connector in the official documentation.

What this changes day to day

Three concrete gains for the data engineer:

1. Less manual tuning. You stop choosing a partition column, computing ranges and adjusting the degree of parallelism. That work — repetitive and error-prone — disappears.

2. Shorter load windows. Tables that used to lock up the night now fit into minutes. That frees up room for more dependencies in the pipeline or for more aggressive SLAs.

3. Configuration that doesn't age. Because the strategy is adaptive, it follows the table's growth without needing a review. The job you configure today keeps doing the right thing when the volume doubles.

After nearly two decades building data pipelines, I've learned to be suspicious of features that promise "magic". But automatic adaptive partitioning isn't magic — it's the platform finally taking on a heavy job that should never have been manual. The time you save on tuning is time that goes back to modeling, quality and governance, which is where data engineering really delivers value.

Next steps

If you use the Copy job to bring data from relational databases into Fabric, the test is cheap: flip the toggle on a large development table and compare the run time with and without Auto Partitioning. The gain tends to be most significant precisely on the tables that hurt the most today.