Python how to find duplicate

When I try to use duplicated function does not work.

Here is my query example:

query = """
SELECT
  variable1,
  variable2,
  variable3,
  variable4

FROM source.table
"""
df = spark.sql(query)

When I try to find dups using this function, it does not work:

# Selecting duplicate rows based
# on list of column names
duplicate = df [df .duplicated(['variable1', 'variable2'])]
 
print("Duplicate Rows based on variable1 and variable2:")
 
# Print the resultant Dataframe
duplicate

I got this error:

AttributeError: 'DataFrame' object has no attribute 'duplicated'

Do you know why? And how I can create a pd dataframe based on my current DF?

Spark dataframes and pandas dataframes (and pandas-on-spark dataframes) are different. If you have pandas installed and available, you may be interested in converting to a pandas on spark dataframe and then using pandas on spark duplicated. I believe the code would be something like this:

import pyspark.pandas as ps
ps_df = df.to_pandas_on_spark()
ps_df[ps_df.duplicated(['variable1', 'variable2'])]