Partitioning vs Bucketing in Spark: The Simple Interview Guide

“What is the difference between partitioning and bucketing in Spark?” This is one of the most common Data Engineer interview questions in India, from service companies to product companies. Many people answer “partitioning makes folders, bucketing makes files” and stop there. That is a start, but interviewers want to know why each one exists and when to pick which. So, let’s break it down in simple words, with small code examples you can run yourself. This guide explains partitioning vs bucketing in Spark in simple words.

Quick Answer: Partitioning vs Bucketing in Spark

Partitioning splits a table into folders, one folder per value of a column (like order_date=2026-10-31). It helps Spark skip data when you filter on that column. Bucketing splits data into a fixed number of files using a hash of a column. It helps Spark avoid shuffles in joins and group-bys on that column. Use partitioning for low-cardinality filter columns, and bucketing for high-cardinality join keys.

What Is Partitioning in Spark?

Here we mean table partitioning on disk, done with partitionBy() while writing. This is different from in-memory partitions that you change with repartition(). (If that topic confuses you, read my repartition vs coalesce guide first.)

When you write data with partitionBy("country"), Spark creates one sub-folder for each country. Think of a school almirah with one shelf per class. If the principal wants only Class 10 records, nobody has to open the other shelves.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("partition-demo").getOrCreate()

orders = spark.createDataFrame(
    [(1, 101, "IN", 500.0), (2, 102, "US", 700.0), (3, 101, "IN", 300.0)],
    ["order_id", "customer_id", "country", "amount"],
)

(orders.write
    .mode("overwrite")
    .partitionBy("country")
    .parquet("/lake/orders"))

# Folder layout on disk:
# /lake/orders/country=IN/part-....parquet
# /lake/orders/country=US/part-....parquet

Now, when you read with a filter on country, Spark only opens the matching folder. This is called partition pruning. You can see it in the plan under PartitionFilters:

india = spark.read.parquet("/lake/orders").filter("country = 'IN'")
india.explain()   # look for PartitionFilters: [... (country = IN)]

The big danger: too many partitions

Partitioning works well only when the column has a small number of distinct values, such as country, year or month. If you partition by customer_id with 10 lakh customers, you get 10 lakh folders full of tiny files. Listing and reading them becomes painfully slow. The Spark docs themselves say partitionBy has limited use for high-cardinality columns.

What Is Bucketing in Spark?

Bucketing takes a different path. You choose a column and a fixed number, for example 8 buckets. Spark hashes the column value and sends each row to one of those 8 buckets. So every row of customer 101 always lands in the same bucket, no matter how many customers you have.

Think of a post office with 8 sorting boxes based on the last digit of a PIN code. The number of boxes never grows, even if lakhs of letters come in.

(orders.write
    .mode("overwrite")
    .bucketBy(8, "customer_id")
    .sortBy("customer_id")
    .saveAsTable("orders_bucketed"))

spark.sql("DESCRIBE EXTENDED orders_bucketed").show(50, truncate=False)
# Look for: Num Buckets = 8, Bucket Columns = [`customer_id`]

Interview trap: bucketBy() works only with saveAsTable(). If you try .bucketBy(8, "id").parquet("/path"), Spark throws an error. Bucket details live in the metastore (table catalog), not in plain files, so a path-only write cannot remember them.

Why Bucketing Speeds Up Joins

In a normal sort-merge join, Spark must shuffle both tables so that matching keys meet on the same executor. Shuffles move data over the network, and they are often the slowest part of a job.

If both tables are already bucketed on the join key with the same number of buckets, the matching keys are already sitting in matching buckets. So Spark can skip the shuffle (the Exchange step in the plan). You pay the cost once while writing, and every later join on that key gets cheaper.

customers = spark.createDataFrame(
    [(101, "Asha"), (102, "Ravi")], ["customer_id", "name"]
)
(customers.write.mode("overwrite")
    .bucketBy(8, "customer_id").sortBy("customer_id")
    .saveAsTable("customers_bucketed"))

# Turn off broadcast join so the demo shows a sort-merge join
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)

j = spark.table("orders_bucketed").join(
    spark.table("customers_bucketed"), "customer_id"
)
j.explain()   # no Exchange hashpartitioning(customer_id) before the join

Small note: on real data where one table is tiny, Spark would just use a broadcast join instead. My broadcast join guide explains that case.

The Same Thing in Spark SQL

If you mostly write SQL, here is how you create both kinds of tables. You can even combine them in one table.

-- Partitioned table
CREATE TABLE sales_part (
    order_id   INT,
    amount     DOUBLE,
    order_date DATE
) USING parquet
PARTITIONED BY (order_date);

-- Bucketed table
CREATE TABLE sales_bucket (
    order_id    INT,
    customer_id INT,
    amount      DOUBLE
) USING parquet
CLUSTERED BY (customer_id) SORTED BY (customer_id) INTO 8 BUCKETS;

-- Both together: folders by month, buckets by customer inside each folder
CREATE TABLE sales_both (
    order_id    INT,
    customer_id INT,
    amount      DOUBLE,
    order_month STRING
) USING parquet
PARTITIONED BY (order_month)
CLUSTERED BY (customer_id) INTO 8 BUCKETS;

Partitioning vs Bucketing in Spark: Quick Comparison

Partitioning vs Bucketing in Spark on Delta Lake and Databricks

This is where many tutorials are out of date. Delta Lake tables do not support bucketing. If you try, you get an error saying Delta bucketed tables are not supported.

Even partitioning is used less now. Databricks’ own guidance says most tables under 1 TB do not need partitioning at all, and each partition should hold at least about 1 GB of data. For new tables, Databricks recommends liquid clustering (CLUSTER BY) instead of partitioning or Z-ordering, because it handles both low and high cardinality columns without the small-files problem.

-- Delta table with liquid clustering (Databricks / recent Delta Lake)
CREATE TABLE orders_delta (
    order_id    INT,
    customer_id INT,
    amount      DOUBLE,
    order_date  DATE
) USING DELTA
CLUSTER BY (customer_id);

So in an interview, after explaining partitioning and bucketing, add one line about liquid clustering. It shows you know how modern lakehouse teams actually work.

Common Mistakes with Partitioning vs Bucketing in Spark

• Partitioning by an ID column. You will create lakhs of folders and tiny files.
• Bucketing only one side of the join. To skip the shuffle, both tables should be bucketed on the join key, ideally with the same bucket count.
• Too many buckets with many write tasks. Each write task can create a file per bucket, so 200 tasks × 100 buckets can mean up to 20,000 small files. Repartition by the bucket column before writing to keep this under control.
• Using bucketBy with a path. It needs saveAsTable().
• Ignoring skew. If one customer has most of the orders, one bucket becomes huge. See my data skew in Spark guide for fixes.

Partitioning vs Bucketing in Spark: How to Answer in an Interview (30 Seconds)

“Partitioning splits a table into folders by column value, so filters on that column can skip data through partition pruning. It suits low-cardinality columns like date. Bucketing hashes a column into a fixed number of buckets, so joins and aggregations on that key can avoid a shuffle. It suits high-cardinality join keys and needs saveAsTable. On Delta Lake, bucketing is not supported, and Databricks now recommends liquid clustering for most new tables.”

Key Takeaways on Partitioning vs Bucketing in Spark

• Filter often on date or region? Partition by it.
• Join often on a big ID column? Bucket both tables by it.
• Bucketing needs a metastore table (saveAsTable).
• On Delta / Databricks? Skip bucketing and look at liquid clustering.
• Always check explain() for PartitionFilters and missing Exchange steps.

Frequently Asked Questions (FAQ)

1. Can I use partitioning and bucketing together?

Yes. For example, partition by month and bucket by customer_id. Each month folder then holds a fixed set of bucket files.

2. Is partitionBy() the same as repartition()?

No. repartition() changes in-memory partitions during a job. partitionBy() decides the folder layout on disk when you write.

3. How many buckets should I choose?

There is no single magic number. Pick a count that gives reasonably sized files for your data volume, and use the same count on tables you plan to join. Changing it later means rewriting the table.

4. Why does my bucketed join still show a shuffle?

Common reasons: only one table is bucketed, the bucket counts differ, the join key is not the bucket column, or you read the files by path instead of through the table.

5. Does Delta Lake support bucketing?

No. Use liquid clustering (or Z-ordering on older setups) for similar benefits on Delta tables.

Sources

• Apache Spark Docs – Generic Load/Save Functions (Bucketing, Sorting and Partitioning)
• Apache Spark Docs – DataFrameWriter.bucketBy
• Databricks Docs – When to partition tables
• Databricks Community – Bucketing on Delta tables