Cache vs Persist in PySpark: The Best Simple Interview Guide

Cache vs persist in PySpark is one of the most asked Spark interview topics. If you are preparing for a data engineering interview, one question comes up again and again: “What is the difference between cache() and persist() in PySpark?” Many people answer it in one line and lose marks. In this guide, let’s break it down in simple words, with small code examples, a comparison table and the mistakes that interviewers love to ask about.

Why cache vs persist in PySpark matters: caching in Spark

Spark is lazy. When you write a filter, a join or a withColumn, Spark does not run anything. It only builds a plan. The real work happens when you call an action like count(), show() or write.

Here is the catch: every action runs the full plan again from the start. So if you read a big file, clean it, and then run three different reports on the cleaned data, Spark may read and clean that file three times.

Caching solves this. You tell Spark: “Keep this result ready after the first time, so the next actions can reuse it.” Think of it like keeping your notes on the desk instead of going back to the library every time you need them.

What is cache()?

cache() is the simple, no-options version. It takes no arguments. For a DataFrame, it stores the data using the default storage level, which is MEMORY_AND_DISK_DESER in modern Spark versions. In simple words: keep the data in memory, and if memory is not enough, put the extra part on disk.

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("cache-demo").getOrCreate()

orders = spark.read.parquet("/data/orders")

clean = (orders
         .filter(F.col("status") == "DELIVERED")
         .withColumn("order_date", F.to_date("order_ts")))

clean.cache()    # only MARKS it for caching, nothing is stored yet
clean.count()    # first action: reads, filters and fills the cache

daily = clean.groupBy("order_date").agg(F.sum("amount").alias("revenue"))
by_city = clean.groupBy("city").count()

daily.show()     # uses the cached data
by_city.show()   # uses the cached data again

Notice the important point: cache() is also lazy. Spark stores nothing until the first action runs. That is why many engineers call count() right after caching, to “warm up” the cache.

What is persist()?

persist() does the same job, but it lets you choose where and how the data is stored. You pass a StorageLevel. If you pass nothing, a DataFrame’s persist() uses the same default as cache() (MEMORY_AND_DISK_DESER).

from pyspark import StorageLevel

customers = spark.read.parquet("/data/customers")

enriched = clean.join(customers, "customer_id", "left")

# Very large result, memory is tight: keep it on disk only
enriched.persist(StorageLevel.DISK_ONLY)
enriched.count()                 # fills the storage

print(enriched.is_cached)        # True
print(enriched.storageLevel)     # shows the storage level being used

# ... run your reports here ...

enriched.unpersist()             # free the space when you are done

So the one-line answer is: cache() is persist() with the default storage level. But a good interview answer goes a little deeper, as shown below.

Cache vs persist in PySpark: storage levels in simple words

• MEMORY_ONLY – keep data in RAM only. If it does not fit, Spark recomputes the missing part when needed. Fastest when data is small.
• MEMORY_AND_DISK – keep in RAM, spill the extra part to disk. A safe choice for most jobs.
• MEMORY_AND_DISK_DESER – like the one above, but the in-memory part stays in deserialized form (ready to use, but takes more RAM). This is the DataFrame default.
• DISK_ONLY – store only on local disk. Slower than memory, but still faster than recomputing a heavy join.
• Levels ending with _2 or _3 (for example MEMORY_ONLY_2, DISK_ONLY_2) – keep 2 or 3 copies on different nodes, so you do not lose the cache if one executor dies.
• OFF_HEAP – store outside the JVM heap. You must enable off-heap memory first; rarely used by beginners.

Interview bonus point: for the old RDD API, rdd.cache() uses MEMORY_ONLY, not memory-and-disk. So “the default level” depends on whether you are talking about an RDD or a DataFrame.

cache() vs persist(): quick comparison

Cache vs persist in PySpark with Spark SQL

If your team writes more SQL than Python, you can cache tables directly. One difference to remember: CACHE TABLE in SQL is eager by default (it caches immediately). Add LAZY if you want it to cache only on first use. If you do not set a storage level here, Spark SQL uses MEMORY_AND_DISK.

-- cache the result of a query right now
CACHE TABLE delivered_orders AS
SELECT * FROM orders WHERE status = 'DELIVERED';

-- cache only when the table is first used
CACHE LAZY TABLE customers;

-- choose a storage level
CACHE TABLE big_report OPTIONS ('storageLevel' 'DISK_ONLY')
SELECT * FROM sales;

-- remove from cache
UNCACHE TABLE IF EXISTS delivered_orders;

To clear everything from the cache in one go from PySpark, use spark.catalog.clearCache().

When should you use cache vs persist in PySpark (and when not)?

Cache when:

• Two or more actions use the same DataFrame (many reports, many writes).
• The DataFrame is expensive to build – heavy joins, big aggregations, or reading from a slow source.
• You are running loops or iterative logic (like ML training) over the same data.

Do not cache when:

• You use the DataFrame only once. Caching then just adds extra work.
• The data is huge and memory is small – you may push useful data out and slow the whole job.
• Reading the source is already cheap (for example, a small, well-partitioned Parquet table).

Common mistakes interviewers ask about

1. Thinking cache() runs immediately. It does not. Without an action, Spark stores nothing.
2. Caching too early. Cache after filters and column selection, not the raw full table. Smaller data = smaller cache.
3. Forgetting unpersist(). Cached data keeps using executor memory until you remove it or Spark pushes it out. In long jobs, always clean up.
4. Caching everything “just in case”. This often makes jobs slower, not faster.
5. Not checking the Spark UI. The Storage tab shows what is cached, how much is in memory and how much is on disk. In explain(), a cached DataFrame shows an InMemoryTableScan step.

Note for Databricks users: on Databricks serverless compute, caching behaves differently – the Databricks docs say persist() acts like cache() and ignores the storage level there, and they suggest writing intermediate results to a Delta table instead. Check the latest Databricks docs for your setup.

How to answer cache vs persist in PySpark in an interview (30-second version)

“Both cache() and persist() store a DataFrame so Spark does not recompute it for every action. cache() takes no arguments and uses the default storage level – MEMORY_AND_DISK_DESER for DataFrames and MEMORY_ONLY for RDDs. persist() lets me choose the level, like DISK_ONLY or a replicated level. Both are lazy, so Spark stores the data on the first action. I cache only DataFrames that are reused and expensive, and I call unpersist() when done.”

Key takeaways

• cache() = persist() with the default storage level.
• DataFrame default: MEMORY_AND_DISK_DESER. RDD default: MEMORY_ONLY. SQL CACHE TABLE default: MEMORY_AND_DISK.
• Both are lazy; the first action fills the cache. SQL CACHE TABLE is eager unless you add LAZY.
• Cache only reused, expensive data – and always unpersist() when finished.

FAQ

1. Is cache() faster than persist()?
No. With the default level they do exactly the same thing. Speed depends on the storage level you pick, not on the function name.

2. Does cache() return a new DataFrame?
It returns the same DataFrame, now marked as cached. So both df.cache() and df = df.cache() work.

3. What happens if the cached data does not fit in memory?
With a memory-and-disk level, the extra part goes to disk. With MEMORY_ONLY, Spark recomputes the missing partitions when needed.

4. Is the cache lost when one executor fails?
The cached blocks on that executor are lost, but Spark can rebuild them from the lineage. Use a “_2” level if you want a backup copy.

5. How do I check if a DataFrame is cached?
Use df.is_cached and df.storageLevel in PySpark, or open the Storage tab in the Spark UI.

6. Is caching the same as checkpointing?
No. Caching keeps the lineage and is temporary. Checkpointing writes data to reliable storage and cuts the lineage. That is a good topic for another day!

Related reading: see our guides on our companion caching guide, broadcast join in PySpark and data skew in Spark.

Sources

• Apache Spark docs – pyspark.sql.DataFrame.cache
• Databricks docs – DataFrame.persist
• Azure Databricks docs – DataFrame.cache
• Apache Spark docs – pyspark.RDD.cache
• Apache Spark SQL reference – CACHE TABLE