PySpark UDF vs Pandas UDF interview guide thumbnail
23
Views

PySpark UDF vs Pandas UDF vs built-in functions is a classic Spark interview question, and this simple guide explains when to use each. If you are preparing for a data engineering interview, there is one question that comes up again and again: “What is the difference between a UDF, a Pandas UDF and a built-in function in PySpark? Which one should you use?”

⚡ Quick summary: PySpark UDF vs Pandas UDF

  • Plain Python UDFs are slow because data moves row by row between the JVM and Python.
  • Pandas UDFs work on batches of data, so they are usually much faster.
  • Built-in functions are fastest, so use them first.
PySpark UDF vs Pandas UDF interview guide thumbnail
PySpark UDF vs Pandas UDF vs built-in functions: the simple interview guide

Many freshers answer, “UDFs are slow.” That is true, but it is only half the answer. The interviewer wants to know why they are slow and what you would do instead. Let’s break it down in simple words, with small code examples you can run yourself.

First, what is a UDF? – PySpark UDF vs Pandas UDF

UDF means User-Defined Function. It is your own Python function that Spark runs on every row of a DataFrame column.

Spark already gives you hundreds of ready-made functions in pyspark.sql.functions – like upper(), trim(), when(), regexp_replace() and date_format(). These are called built-in functions. A UDF is what you write when no built-in function can do the job.

There are three main ways to run custom Python logic in PySpark today:

  • Normal Python UDF – works one row at a time.
  • Arrow-optimized Python UDF – still one row at a time in your code, but data moves faster between Spark and Python (added in Spark 3.5).
  • Pandas UDF (vectorized UDF) – works on a whole batch of rows as a pandas Series.

Why are normal Python UDFs slow? – PySpark UDF vs Pandas UDF

Think of Spark as a factory that runs inside the JVM (Java Virtual Machine). Your Python function lives in a different building – a separate Python process on each worker.

When you use a normal Python UDF, Spark has to:

  1. Pack (serialize) each row in the JVM.
  2. Send it to the Python process.
  3. Run your function.
  4. Pack the result and send it back to the JVM.

This back-and-forth trip is the main cost. On top of that, Spark’s query optimizer (Catalyst) cannot look inside your Python code. For Spark, a UDF is a “black box”, so it cannot apply many of its smart optimizations to that part of the query.

Built-in functions do not have this problem. They run directly inside the JVM, so there is no data transfer to Python at all.

Example 1: Built-in function vs Python UDF – PySpark UDF vs Pandas UDF

Let’s convert employee names to upper case, both ways.

from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.types import StringType

spark = SparkSession.builder.appName("udf-demo").getOrCreate()

df = spark.createDataFrame(
    [("ravi", 52000), ("anita", 61000), (None, 45000)],
    ["name", "salary"]
)

# Way 1: Python UDF (slower)
@F.udf(returnType=StringType())
def to_upper(s):
    return s.upper() if s is not None else None

df.withColumn("name_upper", to_upper("name")).show()

# Way 2: Built-in function (faster, preferred)
df.withColumn("name_upper", F.upper("name")).show()

Both give the same output. But the built-in F.upper() never leaves the JVM. Notice also that in the UDF we had to handle None ourselves – if you forget this, your job can fail with an AttributeError on null rows. Built-in functions handle nulls for you.

Example 2: Pandas UDF (vectorized UDF) – PySpark UDF vs Pandas UDF

A Pandas UDF uses Apache Arrow, a fast columnar format, to send data in batches. Your function receives a whole pandas Series instead of one value, so you can use fast pandas/NumPy operations.

import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("double")
def add_bonus(salary: pd.Series) -> pd.Series:
    # Works on the full batch at once
    return salary * 1.10

df.withColumn("salary_with_bonus", add_bonus("salary")).show()

Here the type hints (pd.Series -> pd.Series) tell Spark this is a “Series to Series” Pandas UDF. The output must have the same number of rows as the input.

Of course, in real life you would just write F.col("salary") * 1.10 for this. We use it only to show the syntax.

Example 3: Arrow-optimized Python UDF (Spark 3.5+)

From Apache Spark 3.5, you can keep your simple row-by-row function but ask Spark to use Arrow for the data transfer. Just add useArrow=True:

@F.udf(returnType="string", useArrow=True)
def to_upper_arrow(s):
    return s.upper() if s is not None else None

df.withColumn("name_upper", to_upper_arrow("name")).show()

# Or turn it on for all Python UDFs in the session
spark.conf.set("spark.sql.execution.pythonUDF.arrow.enabled", "true")

Databricks reported that Arrow-optimized UDFs were about 1.6x to 1.9x faster than normal pickled UDFs in their tests. Newer Databricks runtimes have also introduced a native @arrow_udf decorator that skips the pandas conversion step entirely – check your platform’s documentation before using it.

How much faster is built-in, really?

AWS published a simple benchmark in its AWS Glue tuning guide. It ran the same calculation on 10 million rows in three ways:

  • Normal Python UDF: about 3.20 seconds
  • Pandas (vectorized) UDF: about 0.59 seconds (around 5x faster)
  • Spark SQL built-in: about 0.087 seconds (around 37x faster)

Your numbers will differ based on cluster size and data, but the order stays the same: built-in > Pandas/Arrow UDF > normal Python UDF.

How to spot a UDF in the query plan

A great interview tip: run df.explain() and look at the physical plan.

df.withColumn("name_upper", to_upper("name")).explain()
# Look for: BatchEvalPython  -> normal Python UDF

df.withColumn("salary_with_bonus", add_bonus("salary")).explain()
# Look for: ArrowEvalPython  -> Pandas / Arrow UDF

If you see neither, your logic is running fully inside the JVM – which is what you want.

Using a UDF in Spark SQL

You can also register a Python function so you can call it from SQL:

spark.udf.register("to_upper_sql", to_upper)
df.createOrReplaceTempView("employees")

spark.sql("""
    SELECT name, to_upper_sql(name) AS name_upper
    FROM employees
""").show()

Quick comparison table

  • Built-in functions: Run in JVM, fully optimized by Catalyst, null-safe. Use first, always.
  • Pandas UDF: Batch processing via Arrow, good for pandas/NumPy logic or ML scoring. Use when built-ins cannot express the logic.
  • Arrow-optimized Python UDF: Simple row-wise code with faster transfer. A quick upgrade for old UDFs on Spark 3.5+.
  • Normal Python UDF: Easiest to write, slowest to run. Fine for small data or quick exploration.
  • Scala/Java UDF: Runs inside the JVM, so generally faster than Python UDFs. Useful when your team can maintain Scala code.

What should you do? (Key takeaways)

Here is what to do about the PySpark UDF vs Pandas UDF, step by step.

  • Always search for a built-in function first. Combinations of when(), regexp_extract(), transform() and expr() can replace most UDFs.
  • If you must write Python logic, prefer a Pandas UDF or set useArrow=True on Spark 3.5+.
  • Avoid normal Python UDFs in large, scheduled ETL or streaming jobs. Databricks itself recommends Spark methods over UDFs for these workloads.
  • Handle nulls inside every UDF and always set the correct return type.
  • Use explain() to confirm whether a UDF is hiding in your plan.

FAQ on the PySpark UDF vs Pandas UDF

These quick answers cover the most common questions about the PySpark UDF vs Pandas UDF.

1. Is a Pandas UDF the same as a vectorized UDF?

Yes. Both names mean the same thing – a UDF that processes data in batches using Apache Arrow and pandas.

2. Do I need to enable Arrow for Pandas UDFs?

Pandas UDFs always use Arrow. The setting spark.sql.execution.arrow.pyspark.enabled is mainly for speeding up toPandas() and createDataFrame() from a pandas DataFrame.

3. Why did my UDF return null for every row?

Usually the declared return type does not match what your function returns – for example, returning an int when you declared StringType(). Normal pickled UDFs quietly return null in such cases, so always double-check the return type.

4. Are Scala UDFs faster than Python UDFs?

Generally yes, because they run inside the JVM and avoid the Python data transfer. But built-in functions are still the best choice.

5. When is a normal Python UDF okay?

For small datasets, one-time data cleaning or quick exploration in a notebook, the simplicity is often worth it.

Related reading: Adaptive Query Execution (AQE) in Spark: The Simple Interview Guide · Delta Lake Time Travel in PySpark: VERSION AS OF, RESTORE and VACUUM – The Simple Interview Guide

Sources for the PySpark UDF vs Pandas UDF

In short, for the PySpark UDF vs Pandas UDF decision, try built-in functions first, then a Pandas UDF, and use a plain UDF last. Related: read our explainer on Adaptive Query Execution.

Article Categories:
Educations

Leave a Reply

Your email address will not be published. Required fields are marked *