将Flask请求/应用上下文复制到另一个进程 [英] Copy flask request/app context to another process

问题描述

如何序列化Flask应用程序或请求上下文，或该上下文的子集(即可以成功序列化的任何 )，以便可以从以下位置访问该上下文另一个进程，而不是线程?

How can I serialise a Flask app or request context, or a subset of that context (i.e. whatever can be successfully serialised) so that I can access that context from another process, rather than a thread?

我有一些功能需要访问要在后台运行的Flask请求上下文或App上下文.

I have some functions that require access to the Flask request context, or the App context, that I want to run in the background.

Flask具有内置的 @copy_current_request_context 装饰器，用于将函数包装在请求上下文的副本中，因此您可以在其他线程中运行它:

Flask has a built-in @copy_current_request_context decorator to wrap a function in a copy of the request context, so you can run it in a different thread:

from threading import Thread
from flask import Flask, request, copy_current_request_context

app = Flask(__name__)

@app.route('/')
def index():
    request.foo = 'bar'
    @copy_current_request_context
    def baz():
        print(request.foo)
    thr = Thread(target=baz)
    thr.start()
    return 'ok'

虽然Flask没有提供内置的装饰器来复制应用程序上下文，但它提供了执行此操作的机制-

Whilst Flask doesn't provide a built-in decorator to copy the app context, it provides the machinery to do it - a solution is described on Access flask.g inside greenlet

首先，我希望和Pickle一起解决.这就是 concurrent.futures.ProcessPoolExecutor 使用的内容.不幸的是，由于在应用程序上下文中存在线程锁定对象，因此Pickle失败了:

First of all, I was hoping to solve with Pickle. This is what is used by concurrent.futures.ProcessPoolExecutor. Unfortunately, Pickle fails due to the presence of thread lock objects inside the app context:

>>> with app.test_request_context('/'):
...     pickle.dumps(appctx)
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: can't pickle _thread.lock objects

它也不能复制修饰/包装的函数:

It also can't copy decorated/wrapped functions:

>>> with app.test_request_context('/'):
...     bar = copy_current_request_context(pow)
...     pickle.dumps(bar)
...
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
_pickle.PicklingError: Can't pickle <function pow at 0x110bee8c8>: it's not the same object as builtins.pow

接下来，我尝试了 dill .它通过了以上两个测试，但是无法序列化可能在上下文中出现的许多其他事情.应用程序上下文尤其容易在扩展中链接自身:SQLAlchemy是一个很好的例子.当您使用SQLAlchemy并尝试序列化您的应用上下文时，会发生以下情况:

Next I tried dill. It passes the above two tests, but it can't serialise a lot of other things that might end up in a context. App contexts especially are prone to having extensions link themselves in: SQLAlchemy is a great example. Here's what happens when you're using SQLAlchemy and you try to serialise your app context:

>>> from flask_sqlalchemy import SQLAlchemy
>>> db = SQLAlchemy(app)
>>> with app.test_request_context('/'):
...     from flask.globals import _app_ctx_stack
...     appctx = _app_ctx_stack.top
...     dill.dumps(appctx)
...
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 294, in dumps
    dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 287, in dump
    pik.dump(obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 816, in save_list
    self._batch_appends(obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(tmp[0])
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1386, in save_function
    obj.__dict__), obj=obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 786, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1129, in save_cell
    pickler.save_reduce(_create_cell, (f,), obj=obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
    save(element)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
    save(v)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1330, in save_type
    StockPickler.save_global(pickler, obj)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 957, in save_global
    (obj, module_name, name)) from None
_pickle.PicklingError: Can't pickle <class 'sqlalchemy.orm.session.SignallingSession'>: it's not found as sqlalchemy.orm.session.SignallingSession

所以:有什么办法解决这个问题?我曾经以为是及时"分叉一个新流程(即在我要运行后台任务之前)可能意味着我什至根本不需要复制上下文，只要它已经存在就可以了.分支在设置上下文后发生.我不知道如何使用 ProcessPoolExecutor 来实现此目的，但是进程是一个长期存在的池.我唯一的想法是以某种方式强制工作者进程在每个任务结束时关闭，但是我很确定这只会导致池中断.

So: is there any way around this? One thought I had was that forking a new process "just in time" - i.e. immediately before I want to run my background task - may mean I don't even need to copy a context at all, it should already have it as long as the fork happens after the context is set up. I don't know how to make this work with a ProcessPoolExecutor however, where processes are a long lived pool. The only idea I have there is to somehow force worker processes to shut down at the end of each task but I'm pretty sure that this will just result in a broken pool.

推荐答案

不确定是否适合您，但是可以使用其他进程中的Flask应用程序上下文.它要求您使用应用程序工厂模式.

Not sure if this is suitable for you, but there is a possibility to use Flask application context from other processes. It requires you to utilize the application factory pattern.

# your_project/app.py

def create_app(config_file):
    # Load the config file somehow
    # Create your app instance
    app = Flask(__name__)
    # Initialize your extensions
    db.init_app(app)
    # Mount routes, etc
    app.register_blueprint(...)
    # Return your app
    return app

# your_project/wsgi.py
from .app import create_app

app = create_app("./config/local.cfg")

您定义了一个应用程序工厂，并用它来创建您的应用程序作为 wsgi.py 或其他方便位置的入口点，以后您可以从后台脚本访问该应用程序实例，该脚本将运行在另一个过程中.

You define an application factory and use it to create your app as an entry point in wsgi.py or another convenient location, and you can later access this app instance from your background script which would run in another process.

# your_project/background_script.py
from .wsgi import app

with app.app_context():
    # Do some stuff

顺便说一句，例如，这也是将Flask与Celery结合使用的一种方式，如

BTW, this is also a way to use Flask with Celery, for example, as described in this question. Hope this helps you.

这篇关于将Flask请求/应用上下文复制到另一个进程的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！