在内部线程中调用时，Python队列内存泄漏 [英] Python Queues memory leaks when called inside thread
问题描述
我有python TCP客户端，需要循环将media(.mpg)文件发送到"C" TCP服务器.
I have python TCP client and need to send media(.mpg) file in a loop to a 'C' TCP server.
我有以下代码，其中在单独的线程中，我读取文件的10K块并将其发送并在循环中重新进行一次，我认为这是因为我实现了线程模块或tcp send .我正在使用队列在我的GUI(Tkinter)上打印日志，但是一段时间后，它内存不足了..
I have following code, where in separate thread I am reading the 10K blocks of file and sending it and doing it all over again in loop, I think it is because of my implementation of thread module, or tcp send. I am using Queues to print the logs on my GUI ( Tkinter ) but after some times it goes out of memory..
更新1-根据要求添加了更多代码
用于创建发送数据的线程的线程类"Sendmpgthread"
Thread class "Sendmpgthread" used to create thread to send data
.
.
def __init__ ( self, otherparams,MainGUI):
.
.
self.MainGUI = MainGUI
self.lock = threading.Lock()
Thread.__init__(self)
#This is the one causing leak, this is called inside loop
def pushlog(self,msg):
self.MainGUI.queuelog.put(msg)
def send(self, mysocket, block):
size = len(block)
pos = 0;
while size > 0:
try:
curpos = mysocket.send(block[pos:])
except socket.timeout, msg:
if self.over:
self.pushlog(Exit Send)
return False
except socket.error, msg:
print 'Exception'
return False
pos = pos + curpos
size = size - curpos
return True
def run(self):
media_file = None
mysocket = None
try:
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.connect((self.ip, string.atoi(self.port)))
media_file = open(self.file, 'rb')
while not self.over:
chunk = media_file.read(10000)
if not chunk: # EOF Reset it
print 'resetting stream'
media_file.seek(0, 0)
continue
if not self.send(mysocket, chunk): # If some error or thread is killed
break;
#disabling this solves the issue
self.pushlog('print how much data sent')
except socket.error, msg:
print 'print exception'
except Exception, msg:
print 'print exception'
try:
if media_file is not None:
media_file.close()
media_file = None
if mysocket is not None:
mysocket.close()
mysocket = None
finally:
print 'some cleaning'
def kill(self):
self.over = True
我发现这是由于Queue的错误实现导致评论得以解决
更新2-MainGUI类，从上面的Thread类调用
class MainGUI(Frame):
def __init__(self, other args):
#some code
.
.
#from the above thread class used to send data
self.send_mpg_status = Sendmpgthread(params)
self.send_mpg_status.start()
self.after(100, self.updatelog)
self.queuelog = Queue.Queue()
def updatelog(self):
try:
msg = self.queuelog.get_nowait()
while msg is not None:
self.printlog(msg)
msg = self.queuelog.get_nowait()
except Queue.Empty:
pass
if self.send_mpg_status: # only continue when sending
self.after(100, self.updatelog)
def printlog(self,msg):
#print in GUI
推荐答案
由于printlog已添加到tkinter文本控件中，因此该控件所占用的内存将随每条消息而增加(它必须存储所有日志消息，以便显示它们.)
Since printlog is adding to a tkinter text control, the memory occupied by that control will grow with each message (it has to store all the log messages in order to display them).
除非存储所有日志至关重要，一种常见的解决方案是限制显示的日志行的最大数量.
Unless storing all the logs is critical, a common solution is to limit the maximum number of log lines displayed.
一个简单的实现是在控件达到最大消息数后从头开始消除多余的行.将函数添加到获得控件中的行数，然后在printlog中类似于:
A naive implementation is to eliminate extra lines from the begining after the control reaches a maximum number of messages. Add a function to get the number of lines in the control and then, in printlog something similar to:
while getnumlines(self.edit) > self.maxloglines:
self.edit.delete('1.0', '1.end')
(以上代码未经测试)
更新:一些常规准则
请记住，看起来像内存泄漏的情况并不总是意味着函数为wrong，或者不再可以访问内存.很多时候，缺少用于累积元素的容器的清除代码.
Keep in mind that what might look like a memory leak does not always mean that a function is wrong, or that the memory is no longer accessible. Many times there is missing cleanup code for a container that is accumulating elements.
解决此类问题的基本方法:
A basic general approach for this kind of problems:
- 对可能导致问题的代码部分形成意见
- 通过注释掉该代码进行检查(或保留注释代码，直到找到候选人为止)
- 在负责的代码中查找容器，添加代码以打印其大小
- 确定可以从该容器中安全删除哪些元素，以及何时进行操作
- 测试结果
这篇关于在内部线程中调用时，Python队列内存泄漏的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
