exec()调用中的超大型python函数定义使Django崩溃，但不使直接执行的Python代码崩溃 [英] Very large python function definition in exec() call crashes Django but does not crash directly executed Python code

问题描述

我有一个非常大的Python函数(约40万行)，我试图通过 exec()调用来定义.如果我运行以下Python脚本:

I have a very large (~400k lines) Python function that I am attempting to define through an exec() call. If I run the following Python script:

exec("""def blah()
# 400k lines of IF/THEN/ELSE
""", globals())
blah()

通过从命令行调用Python，它可以正常工作.

By calling Python from the command line, it works fine.

但是，如果我在Django实例中执行相同的操作，它将使服务器崩溃，而没有任何错误消息或堆栈跟踪，我只能认为这是由于分段错误所致.

However, if I do the same within a Django instance, it crashes the server without any error message or stack trace, which I can only assume is due to a segmentation fault.

Django runserver和上面的脚本都在相同的Conda环境中运行，并且两者都具有可用的无限堆栈(通过在Django中打印出 resource.getrlimit 来确认).

Both Django runserver and the above script are run from the same Conda enviroment, and both have unlimited stack available (confirmed by printing out resource.getrlimit in Django).

这是我完整的 ulimit -a 输出:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515017
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

启动Django服务器的命令序列如下:

The command sequence to launch the Django server is as follows:

source activate <conda env name>
python manage.py runserver

这是导致崩溃的shell输入/输出:

This is the shell input/output leading to the crash:

(faf) [pymaster@t9dpyths3 faf]$ python manage.py runserver 9000
Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
August 04, 2020 - 08:25:19
Django version 3.0.3, using settings 'faf.settings'
Starting development server at http://127.0.0.1:9000/
Quit the server with CONTROL-C.
[04/Aug/2020 08:25:25] "GET /projects/ HTTP/1.1" 200 13847
[04/Aug/2020 08:26:49] "PUT /projects/projectname/ HTTP/1.1" 200 76  # This event triggers the exec
(faf) [pymaster@t9dpyths3 faf]$

推荐答案

问题可能是由于 int(s)，浮点数和其他内容可能会导致分段错误

此处:

请尝试设置环境标志 PYTHONMALLOC = debug

这可能允许您的代码运行而不会遇到分段错误，如果仍然出现错误，则应该可以使用捕获它.

This might allow your code to run without running into segmentation errors, if you do still get an error you should be able to catch it using.

PYTHONMALLOC=debug python3 -X tracemalloc=10

您可能还想签出:故障处理程序

此模块包含显式转储Python跟踪的功能，故障，超时或用户信号时.称呼faulthandler.enable()为SIGSEGV安装故障处理程序，SIGFPE，SIGABRT，SIGBUS和SIGILL信号.您也可以启用它们在启动时，通过设置PYTHONFAULTHANDLER环境变量或通过使用-X Faulthandler命令行选项.

This module contains functions to dump Python tracebacks explicitly, on a fault, after a timeout, or on a user signal. Call faulthandler.enable() to install fault handlers for the SIGSEGV, SIGFPE, SIGABRT, SIGBUS, and SIGILL signals. You can also enable them at startup by setting the PYTHONFAULTHANDLER environment variable or by using the -X faulthandler command line option.

添加此内容是为了使其更加清晰，因为它与之相关；以下内容摘自Darrrrrren提供的答案，是对使Faulthandler在线程django应用程序上运行的一项调整:

Adding this for more clarity since it's related; the following is taken from the answer provided by Darrrrrren and is a tweak to make faulthandler run on threaded django applications:

所以我能够通过使用以下命令初始化Python获得堆栈跟踪故障处理程序，但另外我还必须运行 manage.py runserver -nothreading --noreload -由于某些原因，如果您不使用Django禁用线程，即使Faulthandler也不会打印堆栈跟踪.

So I was able to get a stack trace by initializing Python with faulthandler, but additionally I had to run manage.py runserver --nothreading --noreload - for some reason if you do not disable threading with Django, even faulthandler will not print a stack trace.

这篇关于exec()调用中的超大型python函数定义使Django崩溃，但不使直接执行的Python代码崩溃的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！