Narrow vs Wide Transformations in Spark: The Simple Interview Guide

Narrow vs wide transformations in Spark is a favorite data engineering interview topic. “Explain narrow and wide transformations in Spark.” If you are preparing for a Data Engineer interview in India, you will almost surely hear this question. It looks simple, but a good answer connects three ideas: transformations, shuffle and stages. Once you understand these, many other Spark topics like joins, data skew and performance tuning become much easier. So, let’s break it down in simple words, with small PySpark and SQL examples you can run yourself.

Quick Answer: Narrow vs Wide Transformations in Spark

A narrow transformation is one where each output partition depends on just one input partition, so no data moves between executors. Examples: select, filter, withColumn. A wide transformation needs data of many input partitions, so Spark must shuffle data across the network. Examples: groupBy, join, distinct, orderBy, repartition. Every shuffle creates a new stage in the Spark job.

First, What Is a Transformation? (And Lazy Evaluation)

In Spark, operations are of two types:

• Transformations create a new DataFrame (or RDD) out of an existing one. For example, filter, select, groupBy, join.
• Actions actually run the work and return a result or write output. For example, show, count, collect, write.

Transformations are lazy. Spark does not run them immediately. It only remembers the steps, like writing a recipe. When you call an action, Spark looks at the full recipe, optimises it, and then cooks everything at once. This is why a long chain of transformations finishes in milliseconds, but the final show() takes time.

What Is a Narrow Transformation? (Narrow vs Wide Transformations)

Remember that Spark splits data into partitions, and each task works on one partition. In a narrow transformation, every partition can do its job alone, without talking to other partitions.

Think of an exam hall with 10 rooms. If the rule is “remove answer sheets with no roll number”, each invigilator can do it inside their own room. Nobody needs to walk to another room. That is narrow.

Common narrow transformations:

• select, filter / where, withColumn, drop
• map, flatMap, mapPartitions (RDD API)
• union (it just puts partitions side by side)
• coalesce(n) when reducing partitions (it merges nearby partitions without a full shuffle)

Spark is smart here. It joins many narrow steps together into one pipeline and runs them in memory, in a single stage. This is called pipelining, and it is very fast. Learn more in our guide to cache vs persist in PySpark.

What Is a Wide Transformation? (Narrow vs Wide Transformations)

Now imagine a new rule: “Collect all answer sheets of Class 10 in one room.” Sheets of Class 10 are spread across all 10 rooms, so every invigilator has to send some sheets to other rooms. That movement is the shuffle, and the operation is wide.

Common wide transformations:

• groupBy().agg(), distinct, dropDuplicates
• join (except a broadcast join, which avoids shuffling the big table)
• orderBy / sort (global sort)
• repartition(n) or repartition("col")
• Window functions with partitionBy
• groupByKey, reduceByKey (RDD API)

Why Is Shuffle So Expensive in Narrow vs Wide Transformations?

The official Spark documentation explains that a shuffle involves disk I/O, data serialization and network I/O. In simple words, during a shuffle Spark has to:

1. Convert rows into bytes (serialize them).
2. Write shuffle files to local disk.
3. Send those blocks over the network to other executors.
4. Read them back and convert bytes into rows again.

If data does not fit in memory, Spark also spills to disk, which adds more delay. That is why, in most slow Spark jobs, the shuffle is the first place to look.

How Narrow vs Wide Transformations Create Stages

When you call an action, Spark creates a job. The job is split into stages, and each stage is split into tasks (one task per partition).

The rule is easy: a shuffle is a stage boundary. All narrow steps before a shuffle run together in one stage. The stage after the shuffle can start only when the shuffle data is ready. So, for a simple chain, more shuffles usually mean more stages and more waiting. In the physical plan, a shuffle appears as an Exchange node.

PySpark Example: See Narrow and Wide in the Plan

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("narrow-wide-demo").getOrCreate()

orders = spark.createDataFrame(
    [(1, "Delhi", 500.0), (2, "Mumbai", 700.0),
     (3, "Delhi", 300.0), (4, "Pune", 200.0)],
    ["order_id", "city", "amount"],
)

# Narrow: each partition works alone, no data movement
big_orders = (orders
    .filter(F.col("amount") > 250)
    .withColumn("amount_gst", F.col("amount") * 1.18)
    .select("order_id", "city", "amount_gst"))

# Wide: rows of the same city must come together (shuffle)
city_total = big_orders.groupBy("city").agg(
    F.sum("amount_gst").alias("total")
)

city_total.explain()   # look for: Exchange hashpartitioning(city, ...)
city_total.show()      # action: only now does Spark run the job

In the explain() output, you will see the filter and project steps at the bottom, then a HashAggregate (partial), then Exchange hashpartitioning(city, ...), and then the final HashAggregate. That Exchange line is the shuffle. Notice that Spark does a partial sum before the shuffle, so it sends less data over the network. This is the same idea that makes reduceByKey better than groupByKey in the RDD world.

Narrow vs Wide Transformations in Spark SQL

orders.createOrReplaceTempView("orders")

spark.sql("""
    EXPLAIN
    SELECT city, SUM(amount * 1.18) AS total
    FROM orders
    WHERE amount > 250
    GROUP BY city
""").show(truncate=False)

The WHERE part is narrow. The GROUP BY part is wide and shows an Exchange. SQL and DataFrame code give the same plan because both go through the same Catalyst optimizer.

How to Reduce Shuffle in Narrow vs Wide Transformations

• Filter and select early. Less data before the shuffle means less data moved.
• Broadcast small tables in joins. The big table is not shuffled. See my broadcast join guide.
• Avoid unnecessary distinct and orderBy. Sort only when the output really needs order.
• Use coalesce instead of repartition to reduce files. Details in my repartition vs coalesce guide.
• Tune shuffle partitions. spark.sql.shuffle.partitions defaults to 200. Adaptive Query Execution (AQE), enabled by default since Spark 3.2.0, can merge small shuffle partitions automatically.
• Watch for skew. After a shuffle, one huge key can slow the whole stage. Read my data skew in Spark guide.
• Cache reused results. If a shuffled DataFrame is used many times, cache or persist it so Spark does not repeat the work.

Narrow vs Wide Transformations: Quick Comparison

How to Answer in an Interview (30 Seconds)

“Transformations in Spark are lazy and run only when an action is called. In a narrow transformation like filter or select, each output partition depends on just one input partition, so Spark pipelines them in one stage with no data movement. In a wide transformation like groupBy, join or distinct, data of many partitions is needed, so Spark shuffles it across the network. Each shuffle creates a stage boundary and shows as an Exchange in the plan. Since shuffles are costly, I filter early, broadcast small tables and let AQE tune shuffle partitions.”

Key Takeaways on Narrow vs Wide Transformations

• Narrow = no data movement, same stage, fast.
• Wide = shuffle, new stage, slower.
• Nothing runs until an action is called (lazy evaluation).
• Look for Exchange in explain() to find shuffles.
• Fewer and smaller shuffles usually mean faster jobs.

Frequently Asked Questions (FAQ)

Sources