Calculating a new column in spark df based on another spark df without an explicit join column

I have the df1 and df2 without a common crossover column. Now I need to add a new column in df1 from df2 if a condition based on columns df2 is met. I will try to explain myself better with an example:

df1:

+--------+----------+
|label   |    raw   |
+--------+----------+
|0.0     |-1.1088619|
|0.0     |-1.3188809|
|0.0     |-1.3051535|
+--------+----------+

df2:

+--------------------+----------+----------+
|    probs           |    minRaw|    maxRaw|
+--------------------+----------+----------+
|                 0.1|-1.3195256|-1.6195256|
|                 0.2|-1.6195257|-1.7195256|
|                 0.3|-1.7195257|-1.8195256|
|                 0.4|-1.8195257|-1.9188809|

The expected output will be a new column in df1 that get the df2.probs if df1.raw value is between df2.minRaw and df2.maxRaw .

My first aproach has been try to explode the range minRaw and maxRaw, and then joined dataframes, but those columns are continuous. The second idea is an udflike this:

def get_probabilities(raw):
    df= isotonic_prob_table.filter((F.col("min_raw")>=raw)& \
                                    (F.col("max_raw")<=raw))\
                           .select("probs")
    df.show()
    #return df.select("probabilidad_bin").value()
    #return df.first()["probabilidad_bin"]

But it takes a long time in my large dataframe, and give me this alerts:

23/02/13 22:02:20 WARN org.apache.spark.sql.execution.window.WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
23/02/13 22:02:20 WARN org.apache.spark.sql.execution.window.WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
[Stage 82:>                 (0 + 1) / 1][Stage 83:====>            (4 + 3) / 15]23/02/13 22:04:36 WARN org.apache.spark.sql.catalyst.expressions.RowBasedKeyValueBatch: Calling spill() on RowBasedKeyValueBatch. Will not spill but return 0.
23/02/13 22:04:36 WARN org.apache.spark.sql.catalyst.expressions.RowBasedKeyValueBatch: Calling spill() on RowBasedKeyValueBatch. Will not spill but return 0.

If value is'n't between minRaw and maxRaw, the output expected is null and df1 can have duplicates.

I have spark version 2.4.7 and I'm not a pyspark expert. Thank you in advance for read!

I think you can just join those dataframes with the condition between.

df1.join(df2, f.col('raw').between(f.col('maxRaw'), f.col('minRaw')), 'left').show(truncate=False)

+-----+-----+-----+----------+----------+
|label|raw  |probs|minRaw    |maxRaw    |
+-----+-----+-----+----------+----------+
|0.0  |-1.1 |null |null      |null      |
|0.0  |-1.1 |null |null      |null      |
|0.0  |-1.32|0.1  |-1.3195256|-1.6195256|
|0.0  |-1.32|0.1  |-1.3195256|-1.6195256|
|0.0  |-1.73|0.3  |-1.7195257|-1.8195256|
|0.0  |-1.88|0.4  |-1.8195257|-1.9188809|
+-----+-----+-----+----------+----------+

You can perform a crossjoin between df1 and df2, and apply a filter so that you're only selecting rows where df1.raw is between df2.minRaw and df2.maxRaw – this should be more performant than a udf.

Note: Since df1 can have duplicates, we want to deduplicate df1 before crossjoining with df2 so that after we apply the filter we don't have any duplicate rows, but still have the minimum information we need. Then we can right join on df1 to ensure we have all of the original rows in df1.

I've also modified your df1 slightly to include duplicates for the purpose of demonstrating the result:

df1 = spark.createDataFrame(
    [
        (0.0,-1.10),
        (0.0,-1.10),
        (0.0,-1.32),
        (0.0,-1.32),
        (0.0,-1.73),
        (0.0,-1.88)
    ],
    ['label','raw']
)

df2 = spark.createDataFrame(
    [
        (0.1, -1.3195256, -1.6195256),
        (0.2, -1.6195257, -1.7195256),
        (0.3, -1.7195257, -1.8195256),
        (0.4, -1.8195257, -1.9188809)
    ],
    ['probs','minRaw','maxRaw']
)

This is the result when you crossjoin df1 and df2 and remove duplicates:

df1.drop_duplicates().crossJoin(df2).show()

+-----+-----+-----+----------+----------+
|label|  raw|probs|    minRaw|    maxRaw|
+-----+-----+-----+----------+----------+
|  0.0| -1.1|  0.1|-1.3195256|-1.6195256|
|  0.0|-1.32|  0.1|-1.3195256|-1.6195256|
|  0.0|-1.73|  0.1|-1.3195256|-1.6195256|
|  0.0|-1.88|  0.1|-1.3195256|-1.6195256|
...
|  0.0| -1.1|  0.4|-1.8195257|-1.9188809|
|  0.0|-1.32|  0.4|-1.8195257|-1.9188809|
|  0.0|-1.73|  0.4|-1.8195257|-1.9188809|
|  0.0|-1.88|  0.4|-1.8195257|-1.9188809|
+-----+-----+-----+----------+----------+

Then we can apply the filter and right join with df1 to make sure all of the original rows exist:

df1.crossJoin(df2).filter(
    (F.col('raw') > F.col('maxRaw')) & (F.col('raw') < F.col('minRaw'))
).select(
    'label','raw','probs'
).join(
    df1, on=['label','raw'], how='right'
)

+-----+-----+-----+
|label|  raw|probs|
+-----+-----+-----+
|  0.0| -1.1| null|
|  0.0| -1.1| null|
|  0.0|-1.32|  0.1|
|  0.0|-1.32|  0.1|
|  0.0|-1.73|  0.3|
|  0.0|-1.88|  0.4|
+-----+-----+-----+

Use range between in a sql expression

df2.createOrReplaceTempView('df2')

df1.createOrReplaceTempView('df1')

%sql
SELECT minRaw,maxRaw,raw
FROM df1 JOIN df2 ON df1.raw BETWEEN df2.minRaw and df2.maxRaw