英文:
Python - Fuzzy matching result in new column for category based on ratio over 80
问题
I would like to scan a folder to pick up all the files end with '.txt' and then create a data frame by creating a new column for categorization with similar file names (partial score of ratio >=80)
import os
path = '../../../files'
text_files = [f for f in os.listdir(path) if f.endswith('.txt')]
text_files
from fuzzywuzzy import fuzz
from fuzzywuzzy import process
s1 = "programmi.txt"
s2 = "programmi-2.txt"
fuzz.ratio(s1, s2)
The result I expect to see is like below:
英文:
I would like to scan a folder to pick up all the files end with '.txt' and then create a data frame by creating a new column for categorization with similar file names (partial score of ratio >=80)
import ospath = '../../../files'text_files = [f for f in os.listdir(path) if f.endswith('.txt')]text_filesfrom fuzzywuzzy import fuzzfrom fuzzywuzzy import processs1 = "programmi.txt"s2 = "programmi-2.txt"fuzz.ratio(s1, s2)
The result I expect to see is like below:
答案1
得分: 1
这是一个使用两个for循环比较每个文本与其他所有文本以获得所需模糊比率的解决方案。
import pandas as pdfrom fuzzywuzzy import fuzzfrom fuzzywuzzy import processtxt_list = ["programmi.txt","readl-001.txt","dict_class124.txt","readl-002.txt","programmi-2.txt","programmi-re.txt","readl-003.txt","dict_class125.txt","dict_class1264.txt","hello world"]list_categorised_texts = []txt_category = []category_index = 0threshold = 80# two for loops since we need to compare each text to all the othersfor txt_1 in txt_list:if txt_1 not in list_categorised_texts: # if the first text of the current pair is not yet categorised, add as a new categorycategory_index += 1list_categorised_texts.append(txt_1)txt_category.append(category_index)for txt_2 in txt_list:if txt_1 == txt_2: # we don't want to compare the same textscontinueelif txt_2 in list_categorised_texts: # skip already classified textscontinueelse: # if txt_2 is similar, add to the list of classified texts with the corresponding categorysimilarity = fuzz.ratio(txt_1, txt_2)if similarity >= threshold:list_categorised_texts.append(txt_2)txt_category.append(category_index)data = {'texts': list_categorised_texts,'category': txt_category}df = pd.DataFrame(data)print(df.to_markdown())
结果：
| | texts | category ||---:|:-------------------|-----------:|| 0 | programmi.txt | 1 || 1 | programmi-2.txt | 1 || 2 | programmi-re.txt | 1 || 3 | readl-001.txt | 2 || 4 | readl-002.txt | 2 || 5 | readl-003.txt | 2 || 6 | dict_class124.txt | 3 || 7 | dict_class125.txt | 3 || 8 | dict_class1264.txt | 3 || 9 | hello world | 4 |
警告：
请注意，此方法具有顺序依赖性：在下面的示例中，将dict_cl.txt与其他名称进行比较只导致一个匹配，而将dict_class12.txt与所有其他名称进行比较导致3个匹配。对于您的用例，我们假设每个组与其他组非常不同，这不应该是问题。但是，此示例显示在更复杂的情况下，成对比较有点棘手。
print(fuzz.ratio('dict_cl.txt', 'dict_class125.txt')) # 79 -> not same categoryprint(fuzz.ratio('dict_cl.txt', 'dict_class1264.txt')) # 76 -> not same categoryprint(fuzz.ratio('dict_cl.txt', 'dict_class12.txt')) # 81 -> same categoryprint("###")print(fuzz.ratio('dict_class12.txt', 'dict_cl.txt')) # 81 -> same categoryprint(fuzz.ratio('dict_class12.txt', 'dict_class125.txt')) # 97 -> same categoryprint(fuzz.ratio('dict_class12.txt', 'dict_class1264.txt')) # 94 -> same category
英文:
Here's a solution which uses two for loops to compare each text to all the others to obtain the fuzz ratio needed for the categorisations.
import pandas as pdfrom fuzzywuzzy import fuzzfrom fuzzywuzzy import processtxt_list = ["programmi.txt","readl-001.txt","dict_class124.txt","readl-002.txt","programmi-2.txt","programmi-re.txt","readl-003.txt","dict_class125.txt","dict_class1264.txt","hello world"]list_categorised_texts = []txt_category = []category_index = 0threshold = 80# two for loops since we need to compare each text to all the othersfor txt_1 in txt_list:if txt_1 not in list_categorised_texts: # if the first text of the current pair is not yet categorised, add as new categorycategory_index += 1list_categorised_texts.append(txt_1)txt_category.append(category_index)for txt_2 in txt_list:if txt_1 == txt_2: # we don't want to compare the same textscontinueelif txt_2 in list_categorised_texts: # skip already classified textscontinueelse: # if the txt_2 is similar, add to list of classified texts with corresponding categorysimilarity = fuzz.ratio(txt_1, txt_2)if similarity >= threshold:list_categorised_texts.append(txt_2)txt_category.append(category_index)data = {'texts': list_categorised_texts,'category': txt_category}df = pd.DataFrame(data)print(df.to_markdown())
Result:
| | texts | category ||---:|:-------------------|-----------:|| 0 | programmi.txt | 1 || 1 | programmi-2.txt | 1 || 2 | programmi-re.txt | 1 || 3 | readl-001.txt | 2 || 4 | readl-002.txt | 2 || 5 | readl-003.txt | 2 || 6 | dict_class124.txt | 3 || 7 | dict_class125.txt | 3 || 8 | dict_class1264.txt | 3 || 9 | hello world | 4 |
Warning:
Please note that this approach has an order-dependency: In the example below, comparing dict_cl.txt to the other names only leads to one match, while comparing dict_class12.txt to all other names leads to 3 matches. For your use case, where we assume that each group is very distinct from each other, this should not be a problem. However, this example shows that pairwise comparisons are a bit tricky in more sophisticated situations.
print(fuzz.ratio('dict_cl.txt', 'dict_class125.txt')) # 79 -> not same categoryprint(fuzz.ratio('dict_cl.txt', 'dict_class1264.txt')) # 76 -> not same categoryprint(fuzz.ratio('dict_cl.txt', 'dict_class12.txt')) # 81 -> same categoryprint("###")print(fuzz.ratio('dict_class12.txt', 'dict_cl.txt')) # 81 -> same categoryprint(fuzz.ratio('dict_class12.txt', 'dict_class125.txt')) # 97 -> same categoryprint(fuzz.ratio('dict_class12.txt', 'dict_class1264.txt')) # 94 -> same category
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论