实现多线程在Python中读取文件中的行并检查是否匹配给定的字符串。

huangapple
go评论152阅读模式
英文:

Implementing multi threading in python to to read the lines in a file and check whether the line matches the given string

问题

我正在尝试在Python3中实现多线程以读取特定文件中的行并检查是否与给定字符串匹配。这让我感到有些困惑。请帮助我解决这个问题，还告诉我如果在这里使用多进程而不是多线程。请检查下面的代码

还有一件事。我正在尝试在基于Linux的系统上实现这个

  1. import threading 
  2. from subprocess import check_output
  4. def thread_task(line): 
  6.     try:
  7.         if line=="hello123":
  8.             print("Found")
  9.             init_.close()
  10.             exit()
  11.     except:
  12.         pass
  14. def check_point(lock,file1):
  15.     lock.acquire()
  16.     print(file1)
  17.     with open(file1,"r",encoding="utf-8",errors = "ignore") as data:
  18.         for line in data:
  19.             line = line[:-1]
  20.             thread_task(line)
  21.     lock.release()
  23. def main_task(wordlist):
  24.     lock = threading.Lock() 
  26.     # creating threads
  27.     t = list(range(0,len(div)))
  28.     for i in t: 
  29.         t[i] = threading.Thread(target=check_point, args=(lock,div[i],))
  31.     # start threads 
  32.     for i in range(0,len(div)):	
  33.         t[i].start()
  35.     # wait until threads finish their job
  36.     for i in range(0,len(div)):	
  37.         t[i].join()
  39. if __name__ == "__main__": 
  40.     wordlist = input("Enter the File location : ")
  41.     div = check_output(['split','-l','100000',wordlist,"Temp/"])
  42.     div = check_output(["ls",'Temp/']).decode("utf").split("\n")
  43.     div = div[:-1]	
  44.     main_task(div)
  45.     print("String Not found")

希望这对你有所帮助。

英文:

I am trying to implement multi-threading in python3 to read the lines in a particular file and check whether the line matches the given string. This is causing me some confusion. Please help me with this and also let me know if multi-processing used here instead of multiprocessing. Please check the code below

One more thing. I am trying to implement this on Linux based system

  1. import threading 
  2. from subprocess import check_output
  4. def thread_task(line): 
  6. 	 
  7. 	try:
  8. 		if line=="hello123":
  9. 		    print("Found")
  10. 		    init_.close()
  11. 		    exit()
  12. 	except:
  13.                                 #print("Somethings gotta")
  14. 		pass
  15. 	
  17. def check_point(lock,file1):
  18. 	lock.acquire()
  19. 	print(file1)
  20. 	with open(file1,"r",encoding="utf-8",errors = "ignore") as data:
  21. 			for line in data:
  22. 				line = line[:-1]
  23. 				thread_task(line)
  24. 	lock.release()
  26. def main_task(wordlist):
  27. 	lock = threading.Lock() 
  29. 	# creating threads
  30. 	t = list(range(0,len(div)))
  31. 	for i in t: 
  32. 			t[i] = threading.Thread(target=check_point, args=(lock,div[i],))
  33. 	
  34. 	# start threads 
  35. 	for i in range(0,len(div)):	
  36. 		t[i].start()
  38. 	# wait until threads finish their job
  39. 	for i in range(0,len(div)):	
  40. 		t[i].join()
  42. if __name__ == "__main__": 
  43. 	wordlist = input("Enter the File location : ")
  44. 	div = check_output(['split','-l','100000',wordlist,"Temp/"])
  45. 	div = check_output(["ls",'Temp/']).decode("utf").split("\n")
  46. 	div = div[:-1]	
  47. 	main_task(div)
  48. 	print("String Not found")

答案1

得分: 1

In most OS the bottleneck is the disk access. Therefore splitting your large file into many small files to read them simultaneously is just a waste of time.

Please, read your file in your python script, one line at a time, check for the word being included or not. Multiprocessing won't help you here.

If I were you, I'd write something like this:

  1. if __name__ == "__main__":
  2.     wordlist = set()
  3.     wordlist_name = input("Enter the File location: ")
  4.     with open(wordlist_name) as fin:
  5.         wordlist = set(fin.read().split())
  7.     if 'hello123' in wordlist:
  8.         print("Found")
  9.     else:
  10.         print("String Not found")
英文:

In most OS the bottleneck is the disk access. Therefore splitting your large file into many small files to read them simultaneously is just a waste of time.

Please, read your file in your python script, one line at a time, check for the word being included or not. Multiprocessing won't help you here.

If I were you, I'd write something like this:

  1. if __name__ == "__main__":
  2.     wordlist = set()
  3.     wordlist_name = input("Enter the File location : ")
  4.     with open( wordlist_name ) as fin :
  5.         wordlist = set( fin.read().split() )
  7.     if 'hello123' in wordlist :
  8.         print( "Found" )
  9.     else :
  10.         print("String Not found")

答案2

得分: -1

If you need to choose between multiprocessing and multi-threading you may be better off using multiprocessing in this case. Your program would then look something like this-

  1. from multiprocessing import Pool
  3. def process_line(line):
  4.     if line.strip() == "hello123":
  5.         return True
  6.     return False
  8. if __name__ == "__main__":
  9.     pool = Pool(4)
  10.     wordlist = input("Enter the File location : ")
  11.     with open(wordlist) as source_file:
  12.         results = pool.map(process_line, source_file, 4)
  13.     if True in results:
  14.         print("Found")
  15.     else:
  16.         print("string not found")

Though depending on your system this might not help at all since in this case your reading from a file you tend to be I/O bound.

英文:

If you need to choose between multiprocessing and multi-threading you may be better off using multiprocessing in this case. Your program would then look something like this-

  1. from multiprocessing import Pool
  3. def process_line(line):
  4. 	if(line.strip() == "hello123"):
  5. 		return True
  6. 	return False
  8. if __name__ == "__main__":
  9. 	pool = Pool(4)
  10. 	wordlist = input("Enter the File location : ")
  11. 	with open(wordlist) as source_file:
  12. 		results = pool.map(process_line, source_file, 4)
  13.     if True in results:
  14.         print("Found")
  15.     else:
  16.         print("string not found")

Though depending on your system this might not help at all since in this case your reading from a file you tend to be I/O bound

huangapple
  • 本文由 发表于 2020年1月6日 16:50:33
  • 转载请务必保留本文链接：https://go.coder-hub.com/59609084.html
  • multithreading
  • python
  • python-3.x
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定