Go/Python异步桥接

huangapple go评论76阅读模式
英文:

Go/Python asynchronous bridge

问题

我写了一个客户端来处理低级别的TLS连接参数,比如ClientHello等。

我使用Go语言来完成这个任务,因为在Go中更容易实现。我的主程序(网页爬虫)是用Python编写的。我通过ctypes将Go源代码连接到我的Python文件中的DLL。到目前为止,我的网页爬虫结构是异步的,可以同时处理多个连接。

不幸的是,我的Go客户端不是异步的。有没有办法在Python中对它进行更改,使其异步等待ctypes指针的响应,直到响应到达为止?目前它正在等待响应,但同时阻塞了所有其他代码的执行。

编辑:
下面是代码示例:

async def request(self, method, url, headers, body=None, rawBody=None, pseudoHeaderOrder=["method", "authority", "scheme", "path"]):
        global httpLib
        global initFunc
        global requestFunc
        global changeProxyFunc
        global freePointerFunc
        config = {
            "id": self.cid,
            "method": method.upper(),
            "timeout": 20000,
            "url": url,
            "pseudoHeaderOrder": pseudoHeaderOrder,
            "headers": headers
        }
        #Critical
        if body:
            config["body"] = body
        if rawBody:
            rawBody = [b for b in bytes(rawBody, "utf-8")]
            config["rawBody"] = rawBody
        config = json.dumps(config)
        #print(config)
        #THIS PART CASTS THE REQUEST
        ptr = requestFunc(config.encode('utf-8'))
        string = ctypes.cast(ptr, ctypes.c_char_p).value.decode("utf-8")
        #THIS PART CLEARS THE POINTER
        freePointerFunc(ptr)
        #...
英文:

I wrote a client to handle lower-level TLS connection parameters like ClientHellos etc.

I did this in Go since it's a lot easier there. My main program (webscraper) is in Python. I connected the Go source to my Python file through a DLL via ctypes. My webscraper structure was asynchronous so far, to handle multiple connections at a time.

Unfortunately, my Go client is not asynchronous. Is there a way I can change it in Python so it asynchronously awaits a response from the ctypes pointer until it is there? Right now it's waiting for a response but blocks all other code executions in the meantime obviously.

EDIT:
Code example below

async def request(self, method, url, headers, body=None, rawBody=None, pseudoHeaderOrder=["method", "authority", "scheme", "path"]):
        global httpLib
        global initFunc
        global requestFunc
        global changeProxyFunc
        global freePointerFunc
        config = {
            "id": self.cid,
            "method": method.upper(),
            "timeout": 20000,
            "url": url,
            "pseudoHeaderOrder": pseudoHeaderOrder,
            "headers": headers
        }
        #Critical
        if body:
            config["body"] = body
        if rawBody:
            rawBody = [b for b in bytes(rawBody, "utf-8")]
            config["rawBody"] = rawBody
        config = json.dumps(config)
        #print(config)
        #THIS PART CASTS THE REQUEST
        ptr = requestFunc(config.encode('utf-8'))
        string = ctypes.cast(ptr, ctypes.c_char_p).value.decode("utf-8")
        #THIS PART CLEARS THE POINTER
        freePointerFunc(ptr)
        #...

答案1

得分: 1

你可以使用执行器(executor)将阻塞调用移动到单独的线程或进程中。

类似这样的代码应该可以工作:

async def request(self, method, url, headers, body=None, rawBody=None, pseudoHeaderOrder=["method", "authority", "scheme", "path"]):
        global httpLib
        global initFunc
        global requestFunc
        global changeProxyFunc
        global freePointerFunc
        config = {
            "id": self.cid,
            "method": method.upper(),
            "timeout": 20000,
            "url": url,
            "pseudoHeaderOrder": pseudoHeaderOrder,
            "headers": headers
        }
        #关键部分
        if body:
            config["body"] = body
        if rawBody:
            rawBody = [b for b in bytes(rawBody, "utf-8")]
            config["rawBody"] = rawBody
        config = json.dumps(config)
        
        # 将阻塞代码移动到单独的函数中
        def blocking_io():
          ptr = requestFunc(config.encode('utf-8'))
          string = ctypes.cast(ptr, ctypes.c_char_p).value.decode("utf-8")
          freePointerFunc(ptr)
          return string

        # 异步等待结果
        loop = asyncio.get_running_loop()
        string = await loop.run_in_executor(None, blocking_io)
        
        #...

你可以参考这里了解更多关于执行代码在线程或进程池中的信息。

英文:

You can move the blocking call to a separate thread/process using an executor.

Something like this should work,

async def request(self, method, url, headers, body=None, rawBody=None, pseudoHeaderOrder=["method", "authority", "scheme", "path"]):
        global httpLib
        global initFunc
        global requestFunc
        global changeProxyFunc
        global freePointerFunc
        config = {
            "id": self.cid,
            "method": method.upper(),
            "timeout": 20000,
            "url": url,
            "pseudoHeaderOrder": pseudoHeaderOrder,
            "headers": headers
        }
        #Critical
        if body:
            config["body"] = body
        if rawBody:
            rawBody = [b for b in bytes(rawBody, "utf-8")]
            config["rawBody"] = rawBody
        config = json.dumps(config)
        
        # Move blocking code to separate function
        def blocking_io():
          ptr = requestFunc(config.encode('utf-8'))
          string = ctypes.cast(ptr, ctypes.c_char_p).value.decode("utf-8")
          freePointerFunc(ptr)
          return string

        # Aschronously wait on the result
        loop = asyncio.get_running_loop()
        string = await loop.run_in_executor(None, blocking_io)
        
        #...

huangapple
  • 本文由 发表于 2021年8月21日 18:06:47
  • 转载请务必保留本文链接:https://go.coder-hub.com/68872029.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定