Asyncssh在使用多个进程时无法创建通道。

huangapple go评论69阅读模式
英文:

Asyncssh fails to create channel with multiple processes

问题

我正在尝试使用asyncsshasyncio.gather来执行多个并发的SSH命令。例如:

import asyncio, asyncssh, sys

async def run(ssh, cmd):
    return (await ssh.run(cmd, check=True)).stdout

async def main():
    host = sys.argv[1]

    ssh = await asyncssh.connect(host)
    nproc = int(await run(ssh, 'nproc'))
    cmds = [run(ssh, 'cpufreq-info -c{} -p'.format(core)) for core in range(nproc)]
    for ret in await asyncio.gather(*cmds):
        print(ret)

if __name__ == '__main__':
    asyncio.get_event_loop().run_until_complete(main())

如果我将nproc的值强制设置为一个小数字(<10),程序可以正常运行,但是使用我的机器上的实际值(12)时,会出现以下错误:

Traceback (most recent call last):
  File "./mwe.py", line 17, in <module>
    asyncio.get_event_loop().run_until_complete(main())
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "./mwe.py", line 13, in main
    for ret in await asyncio.gather(*cmds):
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "./mwe.py", line 5, in run
    return (await ssh.run(cmd, check=True)).stdout
  File "/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py", line 3103, in run
    process = await self.create_process(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py", line 3009, in create_process
    *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py", line 2927, in create_session
    bool(self._agent_forward_path))
  File "/usr/local/lib/python3.5/dist-packages/asyncssh/channel.py", line 1012, in create
    packet = await self._open(b'session')
  File "/usr/local/lib/python3.5/dist-packages/asyncssh/channel.py", line 633, in _open
    return await self._open_waiter
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
asyncssh.misc.ChannelOpenError: open failed

是否有并发命令的限制?如果有,我该如何找到它并如何使用asyncio控制它?

英文:

I am trying to use asyncssh and asyncio.gather to execute multiple concurrent ssh commands. For example:

import asyncio, asyncssh, sys

async def run(ssh, cmd):
    return (await ssh.run(cmd, check=True)).stdout

async def main():
    host = sys.argv[1]

    ssh = await asyncssh.connect(host)
    nproc = int(await run(ssh, &#39;nproc&#39;))
    cmds = [run(ssh, &#39;cpufreq-info -c{} -p&#39;.format(core)) for core in range(nproc)]
    for ret in await asyncio.gather(*cmds):
        print(ret)

if __name__ == &#39;__main__&#39;:
    asyncio.get_event_loop().run_until_complete(main())

If I force the value of nproc to a small number (<10) the program works correctly, but with the real value on my machine (12), I get the following error:

Traceback (most recent call last):
  File &quot;./mwe.py&quot;, line 17, in &lt;module&gt;
    asyncio.get_event_loop().run_until_complete(main())
  File &quot;/usr/lib/python3.5/asyncio/base_events.py&quot;, line 387, in run_until_complete
    return future.result()
  File &quot;/usr/lib/python3.5/asyncio/futures.py&quot;, line 274, in result
    raise self._exception
  File &quot;/usr/lib/python3.5/asyncio/tasks.py&quot;, line 241, in _step
    result = coro.throw(exc)
  File &quot;./mwe.py&quot;, line 13, in main
    for ret in await asyncio.gather(*cmds):
  File &quot;/usr/lib/python3.5/asyncio/futures.py&quot;, line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File &quot;/usr/lib/python3.5/asyncio/tasks.py&quot;, line 296, in _wakeup
    future.result()
  File &quot;/usr/lib/python3.5/asyncio/futures.py&quot;, line 274, in result
    raise self._exception
  File &quot;/usr/lib/python3.5/asyncio/tasks.py&quot;, line 241, in _step
    result = coro.throw(exc)
  File &quot;./mwe.py&quot;, line 5, in run
    return (await ssh.run(cmd, check=True)).stdout
  File &quot;/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py&quot;, line 3103, in run
    process = await self.create_process(*args, **kwargs)
  File &quot;/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py&quot;, line 3009, in create_process
    *args, **kwargs)
  File &quot;/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py&quot;, line 2927, in create_session
    bool(self._agent_forward_path))
  File &quot;/usr/local/lib/python3.5/dist-packages/asyncssh/channel.py&quot;, line 1012, in create
    packet = await self._open(b&#39;session&#39;)
  File &quot;/usr/local/lib/python3.5/dist-packages/asyncssh/channel.py&quot;, line 633, in _open
    return await self._open_waiter
  File &quot;/usr/lib/python3.5/asyncio/futures.py&quot;, line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File &quot;/usr/lib/python3.5/asyncio/tasks.py&quot;, line 296, in _wakeup
    future.result()
  File &quot;/usr/lib/python3.5/asyncio/futures.py&quot;, line 274, in result
    raise self._exception
asyncssh.misc.ChannelOpenError: open failed

Is there a limitation to the number of concurrent commands? If so, how do I find it and how do I control it with asyncio?

答案1

得分: 1

基于Mikhail在项目问题上的建议和答案(链接:https://github.com/ronf/asyncssh/issues/244),似乎问题的原因是SSH服务器允许的会话最大数量。由于我正在使用OpenSSH服务器,我的解决方法尝试使用/etc/ssh/sshd_config的基本解析器来读取最大会话数,以查找MaxSessions设置。使用这个设置,它创建了一个信号量,限制了run调用的未完成数量,以防止程序达到服务器的限制。

#!/usr/bin/python3
import asyncio, asyncssh, sys

async def run(ssh, sem, cmd):
    async with sem:
        return (await ssh.run(cmd, check=True)).stdout

async def main():
    host = sys.argv[1]

    ssh = await asyncssh.connect(host)
    max_sessions = (await ssh.run(r'sed -n "s/^MaxSessions\s*\([[:digit:]]*\)//p" ' \
                                  '/etc/ssh/sshd_config', check=True)).stdout
    max_sessions = max_sessions or 10
    print('MaxSessions {}'.format(max_sessions))
    sem = asyncio.Semaphore(max_sessions)
    nproc = int(await run(ssh, sem, 'nproc'))
    cmds = [run(ssh, sem, 'cpufreq-info -c{} -p'.format(core)) for core in range(nproc)]
    for ret in await asyncio.gather(*cmds):
        print(ret)

if __name__ == '__main__':
    asyncio.get_event_loop().run_until_complete(main())
英文:

Based on Mikhail's advice and answers on the project issue it seems that the cause is the maximum number of sessions allowed by the ssh server. As I am using an OpenSSH server, my workaround tries to read the maximum number of sessions with a basic parser of /etc/ssh/sshd_config looking for the MaxSessions setting. Using this it creates a semaphore that limits the number of outstanding run calls to prevent the program from reaching the server's limit.

#!/usr/bin/python3
import asyncio, asyncssh, sys

async def run(ssh, sem, cmd):
    async with sem:
        return (await ssh.run(cmd, check=True)).stdout

async def main():
    host = sys.argv[1]

    ssh = await asyncssh.connect(host)
    max_sessions = (await ssh.run(r&#39;sed -n &quot;s/^MaxSessions\s*\([[:digit:]]*\)//p&quot; &#39; \
                                  &#39;/etc/ssh/sshd_config&#39;, check=True)).stdout
    max_sessions = max_sessions or 10
    print(&#39;MaxSessions {}&#39;.format(max_sessions))
    sem = asyncio.Semaphore(max_sessions)
    nproc = int(await run(ssh, sem, &#39;nproc&#39;))
    cmds = [run(ssh, sem, &#39;cpufreq-info -c{} -p&#39;.format(core)) for core in range(nproc)]
    for ret in await asyncio.gather(*cmds):
        print(ret)

if __name__ == &#39;__main__&#39;:
    asyncio.get_event_loop().run_until_complete(main())

huangapple
  • 本文由 发表于 2020年1月6日 20:12:14
  • 转载请务必保留本文链接:https://go.coder-hub.com/59611897.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定