Matching two dataframes with texts

I have a dataframe called df_profanity_en which contains a set of words. I have also another dataframe called df_ed with three columns (id, text, emotion_predicted).

I would like to create a new dataframe called df_filtered that contains only the texts of df_ed where words from the df_profanity_en are detected.

I have implemented the following code, but as you can see the word assure shouln't be there. Any idea to solve this issue?

import pandas as pd

# Create a sample df_profanity_en DataFrame
df_profanity_en = pd.DataFrame({
    'word': ['bad', 'offensive', 'curse', 'vulgar', 'ass']
})

# Create a sample df_ed DataFrame
df_ed = pd.DataFrame({
    'id': [1, 2, 3, 4, 5],
    'text': ['This is a clean text. Lets assure', 'There is a bad word here', 'No profanity in this one', 'Watch your language!', 'Another clean text'],
    'predicted_emotion': ['happy', 'sad', 'neutral', 'angry', 'happy']
})

# Create a list of profane words from df_profanity_en
profane_words = df_profanity_en['word'].tolist()

# Filter the texts in df_ed that contain profane words
df_filtered = df_ed[df_ed['text'].str.contains('|'.join(profane_words), case=False, na=False)]

# Add a new column containing the profanity word
df_filtered['profane_word'] = df_filtered['text'].apply(lambda text: next((word for word in profane_words if word.lower() in text.lower()), None))

# Optional: Reset the index if needed
df_filtered.reset_index(drop=True, inplace=True)

# Print the filtered DataFrame
df_filtered

Output:

   id	text	                            predicted_emotion      profane_word
	1	This is a clean text. Lets assure	happy	               ass
	2	There is a bad word here	        sad	                   bad

IIUC, you have to add word boundaries (\b) to match the entire word :

pat = r"{}".format("|".join(r"\b{}\b".format(word) for word in df_profanity_en["word"]))
#'\\bbad\\b|\\boffensive\\b|\\bcurse\\b|\\bvulgar\\b|\\bass\\b'

m = df_ed["text"].str.contains(pat)

df_filtered = df_ed.loc[m]

Output :

print(df_filtered)

   id                      text predicted_emotion
1   2  There is a bad word here               sad

With extract to make the profane_word column :

df_filtered = (
    df_ed.assign(profane_word= df_ed["text"].str.extract(f"({pat})"))
            .dropna(subset="profane_word")
)

Another variant (with a clearer pattern), used by @mozway :

import re

pat = fr"\b({'|'.join(map(re.escape, df_profanity_en["word"]))})\b"
#'\\b(bad|offensive|curse|vulgar|ass)\\b'

df_filtered = (
    df_ed.assign(profane_word= df_ed["text"].str.extract(pat))
            .dropna(subset="profane_word")
)

Output :

print(df_filtered)

   id                      text predicted_emotion profane_word
1   2  There is a bad word here               sad          bad