PySpark cache() vs persist(): The Best Interview Guide with Examples

PySpark cache vs persist is one of the most common Spark interview topics. If you are preparing for a Data Engineer interview, one PySpark question comes up again and again: “What is the difference between cache() and persist()?” Many candidates answer “cache stores in memory, persist stores on disk” — and that answer is only half right. Let’s break it down in simple words, with small code examples you can run yourself.

Why PySpark cache vs persist matters: caching in Spark

Spark is lazy. When you write transformations like filter(), join() or groupBy(), Spark does not run anything. It only builds a plan. The real work starts when you call an action such as count(), show() or write.

Here is the catch: if you call two actions on the same DataFrame, Spark runs the full plan two times — reading the files again, joining again, filtering again. For a big pipeline, this wastes a lot of time and cluster money.

Caching tells Spark: “Once you compute this DataFrame, keep the result. Next time, reuse it instead of recomputing.” The official Spark guide says reusing persisted data can make later actions much faster, often more than 10x.

PySpark cache vs persist: the simple answer

• cache() = persist() with the default storage level. You cannot choose where data goes.
• persist() = same idea, but you can pass a storage level (memory only, disk only, memory and disk, replicated, etc.).
• Both are lazy. Spark stores nothing until the first action runs.
• You remove both with unpersist().

So the real difference is control. cache() is a shortcut; persist() gives you options. That is the whole idea behind PySpark cache vs persist, and it is what interviewers want to hear.

PySpark cache vs persist: default storage levels (RDD vs DataFrame)

This is where interviewers catch people. The default is not the same for RDDs and DataFrames:

• RDD.cache() uses MEMORY_ONLY. If a partition does not fit in memory, Spark simply does not cache it and recomputes it when needed.
• DataFrame.cache() in PySpark uses MEMORY_AND_DISK_DESER (changed in Spark 3.0 to match Scala). Partitions that do not fit in memory spill to disk.
• SQL CACHE TABLE uses MEMORY_AND_DISK by default if you do not set a storage level.

So saying “cache() is memory only” is correct for RDDs but wrong for DataFrames. Mentioning this difference in an interview shows real hands-on knowledge.

PySpark cache vs persist storage levels explained in simple words

• MEMORY_ONLY – keep in RAM only. Fastest, but Spark recomputes extra partitions.
• MEMORY_AND_DISK – keep in RAM; what does not fit goes to local disk.
• DISK_ONLY – store only on disk. Saves memory, slower to read.
• _2 / _3 versions (like MEMORY_AND_DISK_2) – same as above, but Spark keeps copies on 2 or 3 nodes for faster recovery if a node fails.
• OFF_HEAP – stores data outside the JVM heap (experimental; you must enable it).

Note for Python users: the Spark docs say Python always serializes stored RDD objects with pickle, so the _SER levels do not apply to PySpark RDDs.

PySpark cache vs persist, example 1: cache() in PySpark

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("CacheDemo").getOrCreate()

orders = spark.read.parquet("/data/orders")

# Expensive step: filter + aggregate
daily_sales = (orders
    .filter(F.col("status") == "DELIVERED")
    .groupBy("order_date")
    .agg(F.sum("amount").alias("total_sales")))

daily_sales.cache()          # only marks it - nothing stored yet
daily_sales.count()          # first action: computes AND stores the cache

daily_sales.show(10)         # reuses the cache, no re-read of files
daily_sales.write.mode("overwrite").parquet("/out/daily_sales")

print(daily_sales.is_cached)     # True
print(daily_sales.storageLevel)  # shows the storage level used

daily_sales.unpersist()      # free memory when done

PySpark cache vs persist, example 2: persist() with a storage level

from pyspark import StorageLevel

# Large lookup data that may not fit in executor memory
customers = spark.read.parquet("/data/customers")

customers.persist(StorageLevel.DISK_ONLY)   # keep it on disk only
customers.count()                           # materialize

# Reuse in many joins
gold = orders.join(customers, "customer_id")
silver = orders.join(customers, "customer_id").filter(F.col("country") == "IN")

customers.unpersist()

Code example 3: caching in Spark SQL

-- Cache immediately (eager)
CACHE TABLE sales_2026 AS
SELECT * FROM sales WHERE year = 2026;

-- Cache only when first used (lazy), on disk
CACHE LAZY TABLE big_logs OPTIONS ('storageLevel' 'DISK_ONLY')
SELECT * FROM raw_logs;

-- Remove from cache
UNCACHE TABLE sales_2026;

Remember: CACHE TABLE in SQL is eager by default (it caches right away), while df.cache() in PySpark is lazy. Add LAZY to the SQL command if you want lazy behaviour.

When should you use PySpark cache vs persist, and when not?

Good cases to cache:

• The same DataFrame is used by two or more actions (for example, count + write, or many joins).
• The DataFrame is expensive to build (big joins, heavy filters, wide aggregations).
• Iterative work like machine-learning loops that scan the same data many times.

Avoid caching when:

• The DataFrame is used only once. Caching adds cost with no benefit.
• The data is huge and will push other useful data out of memory. Spark removes old cached partitions using an LRU (least recently used) policy.
• You are on Databricks serverless. Databricks docs say persist() behaves like cache() there and ignores the storage level, and Databricks recommends writing intermediate results to a Delta table instead of relying on DataFrame.cache().

Common mistakes freshers make

1. Calling cache() and expecting instant storage. It is lazy. Trigger an action (like count()) to fill the cache.
2. Caching the wrong DataFrame. df.filter(...).cache() caches the filtered result, not df. Assign it to a variable and reuse that variable.
3. Never calling unpersist(). Long-running jobs keep eating executor memory. Clean up, or use spark.catalog.clearCache() to clear everything.
4. Using show() to materialize. show() may compute only a few partitions, so the cache may fill only partly. count() touches all partitions.

How to answer this in an interview (30-second version)

“Both cache() and persist() keep a computed DataFrame or RDD so Spark does not recompute it for every action. cache() uses the default storage level — MEMORY_ONLY for RDDs and MEMORY_AND_DISK (deserialized) for DataFrames. persist() lets me choose the level, like DISK_ONLY or MEMORY_AND_DISK_2. Both are lazy, so Spark stores the data on the first action, and I release it with unpersist(). I cache only when a DataFrame is reused and expensive to compute.”

Key takeaways

• cache() is a shortcut for persist() with the default storage level.
• Defaults differ: RDD = MEMORY_ONLY, DataFrame = MEMORY_AND_DISK_DESER, SQL CACHE TABLE = MEMORY_AND_DISK.
• Both are lazy; an action fills the cache. SQL CACHE TABLE is eager unless you add LAZY.
• Cache only reused, expensive DataFrames, and always unpersist() when done.

FAQ

1. Is cache() faster than persist()?
No. With the same storage level they do the same thing. cache() just uses the default level.

2. Does cache() run immediately?
No. In PySpark it is lazy. Spark stores data when the first action runs.

3. What happens if cached data does not fit in memory?
With MEMORY_ONLY, Spark recomputes extra partitions when needed. With MEMORY_AND_DISK, they spill to local disk.

4. How do I check if a DataFrame is cached?
Use df.is_cached and df.storageLevel, or look for InMemoryTableScan in df.explain(). The Storage tab in Spark UI also shows cached data.

5. Is cached data lost if an executor fails?
The cached copy on that executor is lost, but Spark can recompute it from the lineage. Replicated levels (like MEMORY_AND_DISK_2) help recovery go faster.

6. Cache vs checkpoint — are they the same?
No. Cache keeps the lineage and stores data temporarily. Checkpoint writes data to reliable storage and cuts the lineage. That is a topic for another post!

Related reading: learn how broadcast join in PySpark and data skew in Spark affect performance too.

Sources

• Apache Spark – RDD Programming Guide (RDD Persistence)
• Apache Spark SQL Reference – CACHE TABLE
• Databricks – PySpark DataFrame.cache()
• Databricks – PySpark DataFrame.persist()