哪个更好,多进程还是子进程适合这个ping脚本?

huangapple go评论74阅读模式
英文:

Which one would be better, multiprocessing or sub processing for this ping script

问题

以下是您要翻译的部分:

为了工作需要,我被要求重新创建一个用于检查机器的ping脚本。旧的脚本是用Perl编写的,但我更习惯Python,所以我用Python编写了一个ping脚本。我在研究多进程和子进程之间的一些区别时,我认为多进程可能更好,因为我们有很多主机名,我认为最好使用多个线程,每个线程ping一组主机名。但由于我需要使用子进程来禁止控制台输出,所以我无法完全理解多进程的指南。我的问题是,如果我同时使用子进程,我应该如何使用多线程?

我的工作代码如下:

import subprocess
from datetime import datetime
import re

hostfile = open("hosts.txt", "r")
lines = hostfile.readlines()

# 创建一个输出文件以写入输出
with open('Ping_Results.txt', 'a') as f:
   
    # 在最近打开的文件末尾附加当天日期
    today = datetime.now()
    dateStr = str(today)
    appendDate = "\n------------------------------------" + " " + dateStr + " " + "-----------------------------------\n"
    f.write(appendDate)

    for i in lines:
        # 使用ping命令对hostfile上的每个设备进行ping测试,发送1个大小为10字节的数据包。
        # shell=True用于启用ping命令的多个参数。capture_output用于捕获输出,text=True用于文本输出。
        p1 = subprocess.run(['ping', '-n', '1', '-l', '10', i], shell=True, capture_output=True)
    
        # 检查目标主机不可达与主机不存在的错误
        l = p1.stdout.decode()
        unreach = re.search(r'Destination host unreachable', l)
        
        if unreach:
            status = i.rstrip() + " was not reachable"
            unreachable = (p1.stdout.decode())
            f.write(unreachable)
        elif ( p1.returncode == 0 ):
            # 如果主机可以ping通并且响应正常
            status = i.rstrip() + " was reachable"
            reachable = (p1.stdout.decode())
            f.write(reachable)
        else:   
            # 如果主机不可达
            status = i.rstrip() + " was not found. Check logs for more information"
            notFound = (p1.stdout.decode())
            f.write(notFound)

这个代码运行良好,但在有10-20个主机名时速度较慢。我尝试将子进程拆分,但无法弄清楚。我一直找到人们使用以下命令的示例
p1 = subprocess.run(['ping', '-n', '1', '-l', '10', i], shell=True, capture_output=True)
这确实有效,我喜欢它的工作方式,但我需要一种加速它以处理200多个主机的方法。对于正确指南的任何链接或对这个主题的任何解释都将不胜感激。谢谢。

我尝试过各种多进程的指南,但我想要禁止控制台输出,而子进程似乎是唯一可以与我当前设置一起工作的方法。我可能对子进程的工作方式有所误解。在我看来,每个子进程就像一个线程,但我似乎无法弄清楚。

英文:

So for work, I was asked to recreate a ping script that we were using to check on machines. The old one was written in Perl, and I am more comfortable with Python, so I wrote a ping script in Python. I was looking over some of the differences between multiprocessing and subprocesses, and I think multiprocessing would work better, since we have a lot of host names, I think it would be better to use multiple threads, each to ping a set number of host names. But I couldn't quite follow the multiprocessing guides since I was needing to use subprocesses to quiet the console output. My question is how would I go about using multi-threading if I am also using subprocesses?

My working code is as follows:

import subprocess
from datetime import datetime
import re


hostfile = open("hosts.txt", "r")
lines = hostfile.readlines()

#Create an output file to write output to
with open('Ping_Results.txt', 'a') as f:
   
   
    #Append the current day to the end of the recently opened file
    today = datetime.now()
    dateStr = str(today)
    appendDate = "\n------------------------------------" + " " + dateStr + " " + "-----------------------------------\n"
    f.write(appendDate)

    for i in lines:
        #Ping each device on hostfile with 1 packet with size of 10 bytes. 
        #shell=True to enable multiple arguments for the ping command. capture_output is to capture the output, and text=True. 
        p1 = subprocess.run(['ping' ,'-n', '1', '-l', '10', i], shell= True, capture_output=True)
    
        #Error checking for host unreachable vs host does not exist. Bit buggy
        l = p1.stdout.decode()
        unreach = re.search(r'Destination host unreachable', l)
        
        if unreach:
            status = i.rstrip() + " was not reachable"
            unreachable = (p1.stdout.decode())
            f.write(unreachable)
            #print(status) 
        elif ( p1.returncode ==0 ):
            #if host is able to be pinged and is responsive.
            status = i.rstrip() + " was reachable"
            reachable=(p1.stdout.decode())
            f.write(reachable)
            print(status)
            print(reachable)
       
        else:   
            #If host is not reachable. 
            status = i.rstrip() + " was not found. Check logs for more information"

            #Decode the string to get the readable console output infomation. 
            notFound= (p1.stdout.decode())

            #Writes to the Ping_Results file.
            f.write(notFound)
            #print(status)`

This works fine, but it is a lot slower when I have 10-20 hostnames. I tried splitting up the sub-processes, but I couldn't figure it out. I kept finding examples of people using
p1 = subprocess.run(['ping' ,'-n', '1', '-l', '10', i], shell= True, capture_output=True)
and that does work, and I like how it works, but I need a way to speed it up to be able to handle 200+ hosts. Any links to the proper guides would be appreciated or any clarity on the topic in general. Thank you.

I have tried various multiprocessing guides, but I was wanting to quiet the console output, and subprocesses were the only way that I had found that worked with how I had it set up. I may be misunderstanding exactly how subprocesses work. In my mind, each sub-process acts like a thread, but I can't seem to figure it out.

答案1

得分: 1

以下是您请求的中文翻译部分:

您需要使用subprocess来运行外部的ping命令。至于并行运行多个ping命令,multiprocessing模块实现了线程和进程,使用非常相似的接口使它们易于互换。由于您的代码主要受I/O限制,并且ping已经在单独的进程中执行,因此线程池是一个不错的选择。您可以选择一个池的大小,让它自行处理如何运行线程。

将所有的ping代码放入一个函数中,该函数返回结果而不是写入文件。然后,使用该函数让池将输入数据映射到ping输出,然后将其写入文件。

import subprocess
from datetime import datetime
import re
import multiprocessing as mp
import multiprocessing.pool

def ping_host(host):
    # 使用ping命令对主机进行ping,发送1个大小为10字节的数据包。
    # 使用shell=True以启用ping命令的多个参数,capture_output=True以捕获输出,text=True以获取文本输出。
    p1 = subprocess.run(['ping', '-n', '1', '-l', '10', host], shell=True, capture_output=True)

    # 检查主机不可达与主机不存在的错误。有点小问题。
    l = p1.stdout.decode()
    unreach = re.search(r'Destination host unreachable', l)

    if unreach:
        status = host.rstrip() + " 无法访问"
        unreachable = (p1.stdout.decode())
        return unreachable

    if (p1.returncode == 0):
        # 如果主机可以ping通且响应。
        status = host.rstrip() + " 可以访问"
        reachable = (p1.stdout.decode())
        print(f"{status}\n{reachable}")
        return reachable

    # 如果主机不可达。
    status = host.rstrip() + " 未找到。请检查日志以获取更多信息"

    # 解码字符串以获取可读的控制台输出信息。
    notFound = (p1.stdout.decode())

    # 写入Ping_Results文件。
    return notFound

hostfile = open("hosts.txt", "r")
lines = hostfile.readlines()

# 创建一个输出文件以写入输出
with open('Ping_Results.txt', 'a') as f:
    # 将当前日期附加到最近打开的文件末尾
    today = datetime.now()
    dateStr = str(today)
    appendDate = "\n------------------------------------" + " " + dateStr + " " + "-----------------------------------\n"
    f.write(appendDate)

    with mp.pool.ThreadPool(min(len(lines), 20)) as pool:
        hostfile.writelines(pool.map(ping_host, lines))

希望这对您有所帮助!如果您有任何其他问题,请随时提出。

英文:

You need subprocess to run the external ping command. As for running multiple pings in parallel, the multiprocessing module implements and thread and processes using very similar interfaces to make them easy to interchange. Since you code is mostly I/O bound and the ping is already being done in a separate process, a thread pool would be a good choice. You pick a size for the pool and let it figure out the details of how to make the threads run.

Put all of the ping code in a function that returns a result instead of writing the file. Have the pool use that function to map your input data to the ping output, then write that to a file.

import subprocess
from datetime import datetime
import re
import multiprocessing as mp
import multiprocessing.pool

def ping_host(host):
    #Ping each device on hostfile with 1 packet with size of 10 bytes. 
    #shell=True to enable multiple arguments for the ping command. capture_output is to capture the output, and text=True. 
    p1 = subprocess.run(['ping' ,'-n', '1', '-l', '10', i], shell= True, capture_output=True)

    #Error checking for host unreachable vs host does not exist. Bit buggy
    l = p1.stdout.decode()
    unreach = re.search(r'Destination host unreachable', l)
    
    if unreach:
        status = i.rstrip() + " was not reachable"
        unreachable = (p1.stdout.decode())
        return unreachable

    if ( p1.returncode ==0 ):
        #if host is able to be pinged and is responsive.
        status = i.rstrip() + " was reachable"
        reachable=(p1.stdout.decode())
        print(f"{status}\n{reachable}")
        return reachable
    
    #If host is not reachable. 
    status = i.rstrip() + " was not found. Check logs for more information"

    #Decode the string to get the readable console output infomation. 
    notFound= (p1.stdout.decode())

    #Writes to the Ping_Results file.
    #print(status)`
    return notFound



hostfile = open("hosts.txt", "r")
lines = hostfile.readlines()

#Create an output file to write output to
with open('Ping_Results.txt', 'a') as f:
    #Append the current day to the end of the recently opened file
    today = datetime.now()
    dateStr = str(today)
    appendDate = "\n------------------------------------" + " " + dateStr + " " + "-----------------------------------\n"
    f.write(appendDate)

    with mp.pool.ThreadPool(min(len(lines), 20)) as pool:
        hostfile.writelines(pool.map(ping_host, lines))

huangapple
  • 本文由 发表于 2023年5月17日 09:41:31
  • 转载请务必保留本文链接:https://go.coder-hub.com/76268063.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定