致命的Python错误和`BufferedWriter` [英] Fatal Python error and `BufferedWriter`

问题描述

我在文档中看到了这一段:

I've came across this paragraph in the Documentations which says:

二进制缓冲对象(BufferedReader，BufferedWriter，BufferedRandom和BufferedRWPair的实例)使用锁来保护其内部结构；因此，可以安全地一次从多个线程中调用它们.

Binary buffered objects (instances of BufferedReader, BufferedWriter, BufferedRandom and BufferedRWPair) protect their internal structures using a lock; it is therefore safe to call them from multiple threads at once.

我不确定在GIL发挥作用的情况下为什么他们需要保护"其内部结构.谁在乎?在我发现此锁具有一定意义之前，我并不在乎，请考虑以下代码:

I'm not sure why they need to "protect" their internal structures given the GIL is in action. Who cares? I didn't care much until I found out that this lock has some significance, consider this piece of code:

from _thread import start_new_thread
import time

def start():
    for i in range(10):
        print("SPAM SPAM SPAM!")

for i in range(10):
    start_new_thread(start, ())

time.sleep(0.0001)
print("main thread exited")

在Python 3.X上运行时的输出:

Output when run on Python 3.X:

...many SPAM...
SPAM SPAM SPAM!
SPAM SPAM SPAM!
SPAM SPAM SPAM!
main thread exited
SPAM SPAM SPAM!
SPAM SPAM SPAM!
SPAM SPAM SPAM!
SPAM SPAM SPAM!
SPAM SPAM SPAM!
Fatal Python error: could not acquire lock for 
<_io.BufferedWritername='<stdout>'> at interpreter shutdown, possibly due to daemon threads

在Python 2.7下，没有错误.我不知道为什么会这样，但是，我一直在 bufferedio.c .另一个行为与上面在Python 3.X上测试过的代码段相似的代码，有时我得到了Fatal Python error，有时却没有.任何带有循环加std[out][err].write的线程功能都会导致此致命错误.确实很难定义此错误的特征，据我所知，文档中未提及任何错误.我不确定哪怕是一个错误，我希望不是.

Under Python 2.7, no errors. I'm not aware why would this happen, however, I've been looking around in bufferedio.c. Another code that behaves similarly to the above snippet that was tested on Python 3.X, sometimes I got Fatal Python error and sometimes I did not. Any threaded function with a loop plus std[out][err].write causes this fatal error. It's really hard to define the characteristics of this error and to the best of my knowledge the Documentation doesn't mention anything about it. I'm not sure even if it's a bug, I hope not.

我对这种行为的解释是这样的，*我可能是完全错误的: 主线程按住sys.stdout.buffer锁时退出.但是，这似乎与在运行Python，Linux的系统上主线程退出时线程终止的事实相反.

My explanation of this behavior goes like this, *I could be totally wrong: The main thread exited while its holding the lock of sys.stdout.buffer. However, this seems contrary to the fact that threads are terminated when the main thread exits on the system on which I'm running Python, Linux.

我将其作为答案发布，只是在评论部分无法完成.

I'm posting this as answer, it just can't be done in the comment section.

此行为不仅限于write，它还会影响read以及对那些对象BufferedReader，BufferedWriter，BufferedRandom和BufferedRWPair的flush调用.

This behavior isn't just limited to write it affects read as well as flush calls on those objects BufferedReader, BufferedWriter, BufferedRandom and BufferedRWPair.

1)在Linux以及可能在Windows上，当主线程退出时，其子线程也会终止.这对上述行为有何影响?如果主线程能够在其时间片内退出，则在与另一个线程进行上下文切换之前，不会发生致命错误，因为所有线程都会被终止.但是，不能保证主线程将在启动后立即退出.

1) On Linux and probably on Windows too, when the main thread exits, its child threads are terminated. How does this affect the mentioned behavior in question? If the main thread was able to exit during its time slice, before being context switched with another thread, no fatal error occurs as all threads get terminated. Nothing guarantees however, that the main thread will exit as soon as it starts.

2)致命错误发生在解释器的完成过程(关闭)与read，write或flush调用以及可能对Buffered*对象进行的其他操作之间.终结过程将获取这些对象的锁定，例如，任何write对BufferedWriter对象的锁定都将导致Fatal Python error.

2) The fatal error takes place between the finalization process (shutdown) of the interpreter and the read, write, or flush call and possibly other operations on the Buffered* object. The finalization process acquires the lock of the those objects, any write for example to the BufferedWriter object results in Fatal Python error.

os._exit终止了解释器而没有完成步骤，因此解释器将不会拥有我们正在谈论的对象的锁，这是另一个示例:

os._exit terminates the interpreter without the finalization steps and hence the interpreter will not own the lock of the object that we are talking about, this is another example:

from _thread import start_new_thread
import time, sys, os

def start(myId):
    for i in range(10):
        sys.stdout.buffer.write(b"SPAM\n")

for i in range(2):
    start_new_thread(start, (i,))

x = print("main thread")
print(x)

#os._exit(0)

在上面的代码中，如果主线程在启动后立即退出，就不会发生致命错误，并且所有生成的线程都会立即终止(至少在Linux中是终止的)，但这取决于平台.如果您不够幸运，并且另一个线程在主线程退出之前就开始在该字段上播放，而没有调用os._exit(0)，则解释器将经历其正常的终结周期以获取sys.stdout.buffer锁，这将导致致命错误.多次运行此代码，以注意其不同的行为.

In above code, if the main thread exits as soon as it starts, that's it, no fatal error occurs and all spawned threads are terminated immediately (at least in Linux) this is platform-dependent though. If you're unlucky enough and another thread started to play on the field before the main threads exits, without os._exit(0) call the interpreter goes through its normal cycle of finalization to acquire the lock of sys.stdout.buffer which results in fatal error. Run this code multiple times to notice its different behaviors.

推荐答案

TL; DR

您的问题与锁的内容并不严格相关，但是实际上您正在尝试使用daemon thread编写不再存在的stdout.

TL;DR

Your issue is not strictly related to lock stuff, but with the fact you are trying to write to a no-more-existent stdout with a daemon thread.

运行主脚本时，Python解释器将启动并执行代码，打开stdout文件描述符.

When you run your main script the Python interpreter starts and execute your code opening the stdout file descriptor.

当脚本结束而没有等待线程完成时:

When your script ends without waiting for threads to finish:

• 所有线程从非守护程序切换到 daemons
• 解释器退出，调用 finalize 函数，该函数会扫描线程的全局变量，包括stdout
• 现在后台驻留程序的线程尝试获取stdout的锁，由于上一步，该锁不再可访问

• all the threads switch from non-daemons to daemons
• the interpreter exits calling a finalize function which wypes the threads' globals including the stdout
• the now-daemon threads try to aquire lock for stdout which is no more accessible due to the previous step

为避免此问题，您可以写入文件而不是stdout(如守护程序线程应该做的那样)，或者只是等待线程完成，例如:

To avoid this issue you could write to file instead of stdout (as a daemon thread should do) or just wait for threads to finish with something like:

from threading import Thread
import time

def start():
    for i in range(10):
        print("SPAM SPAM SPAM!")

# create a thread list (you'll need it later)
threads = [Thread(target=start, args=()) for i in range(10)]

# start all the threads
for t in threads:
    t.start()
# or [t.start() for t in threads] if you prefer the inlines

time.sleep(0.0001)

# wait for threads to finish
for t in threads:
    t.join()
# or [t.join() for t in threads] for the inline version

print("main thread exited")

这篇关于致命的Python错误和`BufferedWriter`的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！