英文:
Asyncssh fails to create channel with multiple processes
问题
我正在尝试使用asyncssh
和asyncio.gather
来执行多个并发的SSH命令。例如:
import asyncio, asyncssh, sys
async def run(ssh, cmd):
return (await ssh.run(cmd, check=True)).stdout
async def main():
host = sys.argv[1]
ssh = await asyncssh.connect(host)
nproc = int(await run(ssh, 'nproc'))
cmds = [run(ssh, 'cpufreq-info -c{} -p'.format(core)) for core in range(nproc)]
for ret in await asyncio.gather(*cmds):
print(ret)
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(main())
如果我将nproc
的值强制设置为一个小数字(<10),程序可以正常运行,但是使用我的机器上的实际值(12)时,会出现以下错误:
Traceback (most recent call last):
File "./mwe.py", line 17, in <module>
asyncio.get_event_loop().run_until_complete(main())
File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "./mwe.py", line 13, in main
for ret in await asyncio.gather(*cmds):
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "./mwe.py", line 5, in run
return (await ssh.run(cmd, check=True)).stdout
File "/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py", line 3103, in run
process = await self.create_process(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py", line 3009, in create_process
*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py", line 2927, in create_session
bool(self._agent_forward_path))
File "/usr/local/lib/python3.5/dist-packages/asyncssh/channel.py", line 1012, in create
packet = await self._open(b'session')
File "/usr/local/lib/python3.5/dist-packages/asyncssh/channel.py", line 633, in _open
return await self._open_waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
asyncssh.misc.ChannelOpenError: open failed
是否有并发命令的限制?如果有,我该如何找到它并如何使用asyncio控制它?
英文:
I am trying to use asyncssh
and asyncio.gather
to execute multiple concurrent ssh commands. For example:
import asyncio, asyncssh, sys
async def run(ssh, cmd):
return (await ssh.run(cmd, check=True)).stdout
async def main():
host = sys.argv[1]
ssh = await asyncssh.connect(host)
nproc = int(await run(ssh, 'nproc'))
cmds = [run(ssh, 'cpufreq-info -c{} -p'.format(core)) for core in range(nproc)]
for ret in await asyncio.gather(*cmds):
print(ret)
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(main())
If I force the value of nproc
to a small number (<10) the program works correctly, but with the real value on my machine (12), I get the following error:
Traceback (most recent call last):
File "./mwe.py", line 17, in <module>
asyncio.get_event_loop().run_until_complete(main())
File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "./mwe.py", line 13, in main
for ret in await asyncio.gather(*cmds):
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "./mwe.py", line 5, in run
return (await ssh.run(cmd, check=True)).stdout
File "/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py", line 3103, in run
process = await self.create_process(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py", line 3009, in create_process
*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/asyncssh/connection.py", line 2927, in create_session
bool(self._agent_forward_path))
File "/usr/local/lib/python3.5/dist-packages/asyncssh/channel.py", line 1012, in create
packet = await self._open(b'session')
File "/usr/local/lib/python3.5/dist-packages/asyncssh/channel.py", line 633, in _open
return await self._open_waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
asyncssh.misc.ChannelOpenError: open failed
Is there a limitation to the number of concurrent commands? If so, how do I find it and how do I control it with asyncio?
答案1
得分: 1
基于Mikhail在项目问题上的建议和答案(链接:https://github.com/ronf/asyncssh/issues/244),似乎问题的原因是SSH服务器允许的会话最大数量。由于我正在使用OpenSSH服务器,我的解决方法尝试使用/etc/ssh/sshd_config
的基本解析器来读取最大会话数,以查找MaxSessions
设置。使用这个设置,它创建了一个信号量,限制了run
调用的未完成数量,以防止程序达到服务器的限制。
#!/usr/bin/python3
import asyncio, asyncssh, sys
async def run(ssh, sem, cmd):
async with sem:
return (await ssh.run(cmd, check=True)).stdout
async def main():
host = sys.argv[1]
ssh = await asyncssh.connect(host)
max_sessions = (await ssh.run(r'sed -n "s/^MaxSessions\s*\([[:digit:]]*\)//p" ' \
'/etc/ssh/sshd_config', check=True)).stdout
max_sessions = max_sessions or 10
print('MaxSessions {}'.format(max_sessions))
sem = asyncio.Semaphore(max_sessions)
nproc = int(await run(ssh, sem, 'nproc'))
cmds = [run(ssh, sem, 'cpufreq-info -c{} -p'.format(core)) for core in range(nproc)]
for ret in await asyncio.gather(*cmds):
print(ret)
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(main())
英文:
Based on Mikhail's advice and answers on the project issue it seems that the cause is the maximum number of sessions allowed by the ssh server. As I am using an OpenSSH server, my workaround tries to read the maximum number of sessions with a basic parser of /etc/ssh/sshd_config
looking for the MaxSessions
setting. Using this it creates a semaphore that limits the number of outstanding run
calls to prevent the program from reaching the server's limit.
#!/usr/bin/python3
import asyncio, asyncssh, sys
async def run(ssh, sem, cmd):
async with sem:
return (await ssh.run(cmd, check=True)).stdout
async def main():
host = sys.argv[1]
ssh = await asyncssh.connect(host)
max_sessions = (await ssh.run(r'sed -n "s/^MaxSessions\s*\([[:digit:]]*\)//p" ' \
'/etc/ssh/sshd_config', check=True)).stdout
max_sessions = max_sessions or 10
print('MaxSessions {}'.format(max_sessions))
sem = asyncio.Semaphore(max_sessions)
nproc = int(await run(ssh, sem, 'nproc'))
cmds = [run(ssh, sem, 'cpufreq-info -c{} -p'.format(core)) for core in range(nproc)]
for ret in await asyncio.gather(*cmds):
print(ret)
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(main())
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论