从脚本而不是从Flask路由调用时，Asyncio函数有效 [英] Asyncio function works when called from script but not from Flask route
问题描述
我是Python和这些库/模块的新手.我正在编写一个简单的ping测试网络扫描仪作为一个学习项目.
I'm a novice with Python and these libraries/modules. I'm writing a simple ping-test network scanner as a learning project.
我首先使用asyncio开发了一个脚本来ping网络上的地址
I first developed a script using asyncio to ping addresses on a network
#ip_test.py
import asyncio
import ipaddress
async def ping(addr):
proc = await asyncio.create_subprocess_exec(
'ping','-W','1','-c','3',addr,
stdout=asyncio.subprocess.PIPE
)
await proc.wait()
return proc.returncode
async def pingMain(net):
#hosts() returns list of Ipv4Address objects
result = await asyncio.gather(*(ping(str(addr)) for addr in net.hosts()))
return result
def getHosts(net_): #net_ is an Ipv4Network object
return asyncio.run(pingMain(net_))
#Returns list of response codes which I then zip with the list of searched ips
当我打开python并运行以下命令时，它会按预期工作:
When I open python and run the following, it works as expected:
import ip_test as iptest
import ipaddress
print(iptest.getHosts(ipaddress.ip_network('192.168.1.0/29')))
#prints: [0, 0, 0, 1, 1, 1] as expected on this network
但是，最终目标是通过表单输入从用户那里获取输入(结果被记录到数据库中，为便于说明，这是一个简化的示例).我通过烧瓶路径收集输入:
However, the ultimate goal is to take input from the user via form input (the results are recorded to a database, this is a simplified example for illustrative purposes). I collect the input via a flask route:
@app.route("/newscan",methods=['POST'])
def newScan():
form = request.form
networkstring = form.get('network') + "/" + form.get('mask')
result = iptest.getHosts(ipaddress.ip_network(networkstring))
return result
以这种方式调用模块时，出现错误:运行时错误:无法添加子处理程序，子监视程序没有附加的循环.
When I call the module this way, I get an error: Runtime Error: Cannot add child handler, the child watcher does not have a loop attached.
为什么当我从命令行导入模块并运行该函数时却无法正常工作，而当我从烧瓶路径中使用相同的输入调用它时却为什么不起作用?
Why does this work when I import the module and run the function from the command line, but not when I call it with the same input from a flask route?
回溯:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
response = self.handle_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/app/app.py", line 41, in newScan
result = iptest.getHosts(ipaddress.ip_network(networkstring))
File "/app/ip_test.py", line 22, in getHosts
res = asyncio.run(pingMain(net_))
File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
return future.result()
File "/app/ip_test.py", line 15, in pingMain
result = await asyncio.gather(*(ping(str(addr)) for addr in net.hosts()))
File "/app/ip_test.py", line 7, in ping
stdout=asyncio.subprocess.PIPE
File "/usr/local/lib/python3.7/asyncio/subprocess.py", line 217, in create_subprocess_exec
stderr=stderr, **kwds)
File "/usr/local/lib/python3.7/asyncio/base_events.py", line 1529, in subprocess_exec
bufsize, **kwargs)
File "/usr/local/lib/python3.7/asyncio/unix_events.py", line 193, in _make_subprocess_transport
self._child_watcher_callback, transp)
File "/usr/local/lib/python3.7/asyncio/unix_events.py", line 930, in add_child_handler
"Cannot add child handler, "
RuntimeError: Cannot add child handler, the child watcher does not have a loop attached
推荐答案
您正在尝试从主线程以外的线程运行异步子进程.这需要从主线程进行一些初始设置，请参见 asyncio Subprocesses 文档的子进程和线程部分:
You are trying to run your async subprocess from a thread other than the main thread. This requires some initial setup from the main thread, see the Subprocesses and Threads section of the asyncio Subprocesses documentation:
标准asyncio事件循环支持从不同线程运行子进程，但是存在局限性:
Standard asyncio event loop supports running subprocesses from different threads, but there are limitations:
- 事件循环必须在主线程中运行.
- 在从其他线程执行子进程之前，必须在主线程中实例化子监视程序.在主线程中调用
get_child_watcher()函数以实例化子监视程序.
- An event loop must run in the main thread.
- The child watcher must be instantiated in the main thread before executing subprocesses from other threads. Call the
get_child_watcher()function in the main thread to instantiate the child watcher.
这里发生的是您的WSGI服务器正在使用多个线程来处理传入的请求，因此请求处理程序不在 main 线程上运行.但是您的代码使用 asyncio.run()来启动新的事件循环，因此您的 asyncio.create_subprocess_exec()调用将失败，因为主控件上没有子监视程序线程.
What is happening here is that your WSGI server is using multiple threads to handle incoming requests, so the request handler is not running on the main thread. But your code uses asyncio.run() to start a new event loop, and so your asyncio.create_subprocess_exec() call will fail as there is no child watcher on the main thread.
You'd have to start a loop (and not stop it) from the main thread, and call asyncio.get_child_watcher() on that thread, for your code not to fail:
# to be run on the main thread, set up a subprocess child watcher
assert threading.current_thread() is threading.main_thread()
asyncio.get_event_loop()
asyncio.get_child_watcher()
注意:此限制仅适用于Python 3.7之前的Python版本，限制已在Python 3.8中取消.
Note: this restriction only applies to Python versions up to Python 3.7, the restriction has been lifted in Python 3.8.
但是，仅运行一堆子进程并等待它们完成，使用 asyncio 实在是太过分了；您的OS可以并行运行子进程.只需使用 subprocess.Popen() ，然后通过 Popen.poll检查每个进程()方法:
However, just to run a bunch of subprocesses and wait for these to complete, using asyncio is overkill; your OS can run subprocesses in parallel just fine. Just use subprocess.Popen() and check each process via the Popen.poll() method:
import subprocess
def ping_proc(addr):
return subprocess.Popen(
['ping', '-W', '1', '-c', '3', addr],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL
)
def get_hosts(net):
# hosts() returns list of Ipv4Address objects
procs = [ping_proc(str(addr)) for addr in net.hosts()]
while any(p.poll() is None for p in procs):
time.sleep(0.1)
return [p.returncode for p in procs]
Popen.poll()不会阻止；如果尚未设置 Popen.returncode ，它将使用
Popen.poll() does not block; if Popen.returncode is not yet set it checks for the process status with the OS with waitpid([pid], WNOHANG) and returns either None if the process is still running, or the now-available returncode value. The above just checks for those statuses in a loop with a short sleep in between to avoid thrashing.
asyncio 子进程包装器(至少在POSIX上)使用 SIGCHLD 信号处理程序通知子进程已退出，或者(在Python 3.8中)使用单独的线程每个子进程在创建的每个子进程上使用阻塞的 waitpid()调用.您可以实现相同的信号处理程序，但要考虑到信号处理程序只能在主线程上注册，因此您必须跳过多个步骤才能在右侧传达输入的 SIGCHLD 信号信息线程.
The asyncio subprocess wrapper (on POSIX at least) either uses a SIGCHLD signal handler to be notified of child processes exiting or (in Python 3.8) uses a separate thread per child process to use a blocking waitpid() call on each subprocess created. You could implement the same signal handler, but take into account that signal handlers can only be registered on the main thread, so you'd have to jump through several hoops to communicate incoming SIGCHLD signal information to the right thread.
这篇关于从脚本而不是从Flask路由调用时，Asyncio函数有效的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
