Query fails when passing column to rlike() function in Pyspark

Use the expr() function.

Written by manikandan.ganesan

Last published at: October 29th, 2025

Problem

You attempt to pass a column to the rlike() function using PySpark code. The following code provides an example. 

df3 = df1.join(df2, (df1.name.rlike(df2.template)), how="left")

 

Your code fails with the following error.

“[NOT_ITERABLE] Column is not iterable" 

 

You may notice the equivalent Apache Spark SQL query works as expected.

df3 = spark.sql("""
SELECT *
FROM df1_view a
LEFT JOIN df2_view b
ON a.name RLIKE b.template
""") 

 

Cause

PySpark's rlike() function does not support passing a column instance as an argument. Instead, it expects a string literal.

 

Solution

Use the expr() function, which allows SQL-like expressions within DataFrame transformations. This enables you to write expressions as strings, making it possible to use rlike() with a column comparison.

  1. Wrap the rlike() function with the expr() function.
  2. Write the comparison expression as a string within the expr() function. The following code updates the example from the problem statement to demonstrate. 
%python
df3 = df1.join(df2, F.expr("name rlike template"), how="left")