Adaptive Query Execution (AQE) is one of the most asked Spark performance topics in data engineering interviews, and this simple guide explains it step by step. You wrote a PySpark join. It ran fine on test data. Then in production it suddenly took 40 minutes, with 200 tiny tasks and one task that never finishes. Sounds familiar? This is exactly the kind of problem Adaptive Query Execution (AQE) in Spark was built to fix.
⚡ Quick summary: Adaptive Query Execution
- Adaptive Query Execution re-plans a Spark query while it runs, using real statistics from finished stages.
- It merges tiny shuffle partitions, switches join strategies and splits skewed partitions.
- Know the key settings and be ready with a 30-second interview answer.

AQE is also a favourite question in data engineering interviews at product companies and service companies alike. Let’s break it down in simple words, with small code examples you can try yourself.
What is Adaptive Query Execution (AQE)?
Normally, Spark makes a plan for your query before it runs. At that time, Spark only has rough guesses about data size. If the guess is wrong, the plan is wrong too.
AQE changes this. It lets Spark re-plan the query while it is running, using real numbers (runtime statistics) collected from the stages that have already finished.
Think of it like Google Maps. The old way: you pick a route at home and follow it no matter what. The AQE way: the app sees a traffic jam after you start and changes your route on the way.
How does AQE work? (Query stages) – Adaptive Query Execution
Whenever Spark needs to shuffle data (for joins, groupBy, window functions) or broadcast a table, the work is split into pieces called query stages. Each stage writes its output before the next stage begins.
That pause between stages is the magic moment. Spark now knows the actual size of every shuffle partition. AQE looks at these numbers and, if needed, changes the plan for the remaining stages.
Quick fact for interviews: AQE features came in Spark 3.0, and AQE is turned on by default since Spark 3.2.0. On Databricks it is also on by default.
The 3 big things AQE does – Adaptive Query Execution
1. Coalescing (merging) small shuffle partitions
By default, spark.sql.shuffle.partitions is 200. If your data after a shuffle is only 500 MB, you get 200 tiny partitions and 200 tiny tasks. Too many small tasks waste time on scheduling.
AQE checks the real sizes and merges small neighbouring partitions into bigger ones, aiming for a target size (the advisory size, 64 MB by default). So you might end up with around 8 healthy tasks instead of 200 tiny ones.
2. Switching join strategy at runtime
Before running, Spark may think both tables are big and pick a sort-merge join (slow, needs shuffle and sort on both sides). But after a filter, one side may become very small.
AQE sees the real size and can switch to a broadcast hash join, which is much faster. It can also switch to a shuffled hash join in some cases. If you want to revise broadcast joins first, read our Broadcast Join guide.
3. Handling skewed joins
Data skew means one key has far more rows than others (for example, one city has 80% of all orders). One task gets a huge partition and everyone waits for it.
AQE detects such a partition and splits it into smaller pieces, so many tasks share the work. A partition is treated as skewed only if it is bigger than both 5 times the median partition size and 256 MB (the default values). Our older post on Data Skew in Spark explains manual fixes like salting too.
Bonus: Databricks documentation also lists a fourth trick. AQE can detect parts of a query that return no rows and simplify the plan.
Important AQE settings (cheat sheet) – Adaptive Query Execution
spark.sql.adaptive.enabled– main on/off switch (default: true)spark.sql.adaptive.coalescePartitions.enabled– merge small partitions (default: true)spark.sql.adaptive.advisoryPartitionSizeInBytes– target partition size (default: 64 MB)spark.sql.adaptive.skewJoin.enabled– fix skewed joins (default: true)spark.sql.adaptive.skewJoin.skewedPartitionFactor– default: 5spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes– default: 256 MBspark.sql.adaptive.autoBroadcastJoinThreshold– broadcast limit used by AQE (falls back tospark.sql.autoBroadcastJoinThreshold, usually 10 MB)
Code example: checking and using AQE in PySpark – Adaptive Query Execution
Here is a short, practical example of the Adaptive Query Execution in action.
First, check whether AQE is on, and set it explicitly if you want to be sure:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("aqe-demo").getOrCreate()
print(spark.conf.get("spark.sql.adaptive.enabled")) # 'true' on Spark 3.2+
# Turn on AQE and its main features explicitly
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes", "64m")Now a simple join followed by an aggregation. This creates shuffles, so AQE gets a chance to act:
from pyspark.sql import functions as F
orders = spark.range(0, 5_000_000).select(
F.col("id").alias("order_id"),
(F.col("id") % 1000).alias("customer_id"),
(F.rand() * 500).alias("amount"),
)
customers = spark.range(0, 1000).select(
F.col("id").alias("customer_id"),
F.concat(F.lit("city_"), (F.col("id") % 20).cast("string")).alias("city"),
)
result = (
orders.join(customers, "customer_id")
.groupBy("city")
.agg(F.sum("amount").alias("total_sales"))
)
result.explain() # shows AdaptiveSparkPlan isFinalPlan=false (plan before running)
result.show() # runs the query; AQE can now re-plan using real sizesBefore running, the plan starts with AdaptiveSparkPlan isFinalPlan=false. After the query runs, open the SQL / DataFrame tab in Spark UI to see the final plan. Signs that AQE worked:
- A different join type than the initial plan (for example, BroadcastHashJoin instead of SortMergeJoin).
- An
AQEShuffleReadnode markedcoalesced(older versions call itCustomShuffleReader). - A SortMergeJoin with
isSkew=truewhen skew handling kicked in.
Using Spark SQL instead? The same switch works there:
SET spark.sql.adaptive.enabled = true;
SELECT c.city, SUM(o.amount) AS total_sales
FROM orders o
JOIN customers c ON o.customer_id = c.customer_id
GROUP BY c.city;Limitations you should know
Every tool has limits, and the Adaptive Query Execution is no exception.
- Needs a shuffle or subquery: a simple read-filter-write job with no exchange gives AQE nothing to re-plan.
- Not for streaming: AQE applies to batch (non-streaming) queries.
- No join reordering: AQE does not change the order of your joins.
- Broadcast may come late: AQE may switch to broadcast only after both sides are already shuffled. If you know a table is small, a
broadcast()hint can still be faster. - Skew fix has limits: it works only for shuffle-based joins, and full outer joins do not get skew optimization.
How to answer AQE in an interview (30-second version)
Use this short script to explain the Adaptive Query Execution with confidence.
“AQE is a Spark SQL feature that re-optimizes the query plan at runtime using real statistics from finished query stages. It does three main things: coalesces small shuffle partitions, switches sort-merge join to broadcast join when one side is small, and splits skewed partitions in joins. It came in Spark 3.0 and is enabled by default from Spark 3.2. We control it with spark.sql.adaptive.enabled.”
Then add one real example from your project, like a job where AQE reduced 200 small tasks to a handful. Interviewers love practical examples.
In short, Adaptive Query Execution lets Spark fix a weak plan at runtime, so you get faster joins and fewer tiny tasks without manual tuning. Related: read our explainer on PySpark UDF vs Pandas UDF.
Key takeaways on the Adaptive Query Execution
- AQE = Spark changes its plan during the run, not just before it.
- It works at stage boundaries created by shuffles and broadcasts.
- Three main features: merge small partitions, switch join type, fix skewed joins.
- On by default since Spark 3.2.0, so check your Spark version first.
- AQE helps a lot, but it is not a replacement for good data modelling and smart join hints.
FAQ on the Adaptive Query Execution
These quick answers cover the most common questions about the Adaptive Query Execution.
Is AQE enabled by default?
Yes, from Apache Spark 3.2.0 onwards, and on Databricks. On Spark 3.0 and 3.1 you need to set spark.sql.adaptive.enabled to true yourself.
Do I still need to tune spark.sql.shuffle.partitions?
Less often. With partition coalescing on, you can set a fairly high starting number and let AQE merge partitions down to a good size.
Does AQE replace salting for data skew?
For many skewed joins, AQE is enough. But skew in aggregations, broadcast joins or full outer joins may still need manual fixes like salting.
Why does explain() show isFinalPlan=false?
Because the query has not run yet. The final plan is decided during execution. Check the Spark UI after the job finishes to see the final plan.
Does AQE work with Structured Streaming?
No. AQE is meant for batch (non-streaming) queries.
Related reading: PySpark UDF vs Pandas UDF vs Built-in Functions: The Simple Interview Guide · Delta Lake Time Travel in PySpark: VERSION AS OF, RESTORE and VACUUM – The Simple Interview Guide

Pingback: Delta Lake Time Travel in PySpark: Simple Interview Guide