类型错误:无法pickle 'weakref' 对象 [英] TypeError: cannot pickle 'weakref' object

问题描述

这里对多处理非常陌生.我有一个运行两个进程的代码.一种是不断地从服务器接收数据块并将其放入队列中，另一种是从队列中取出数据块并进行处理.

以下是我的客户端代码:

<预><代码>进口插座进口龟导入多处理从多处理导入进程，队列从 tkinter 导入 *类图形用户界面:def __init__(self, master):rec_data = recv_data()self.master = 主人master.title(碰撞检测")self.input_label = Label(root, text=输入所有光栅以nm为单位设置直线波长值")self.input_label.grid(row=0)self.core_string = "核心";self.entries = []self.label_col_inc = 0self.entry_col_inc = 1self.core_range = range(1, 5)对于 self.core_range 中的 y:self.core_text = self.core_string + str(y) + '_' + '25'self.core_label = Label(root, text=self.core_text)self.entry = 条目(根)self.core_label.grid(row=1, column=self.label_col_inc,sticky=E)self.entry.grid(row=1, column=self.entry_col_inc)self.entries.append(self.entry)self.label_col_inc += 2self.entry_col_inc += 2self.threshold_label = Label(root, text =以纳米为单位的阈值")self.entry_threshold = 入口(根)self.threshold_label.grid(row=2,sticky=E)self.entry_threshold.grid(row=2, column=1)self.light_label = Label(root, text='Status')self.light_label.grid(row=3, column=3)self.canvas = Canvas(root, width=150, height=50)self.canvas.grid(row=4, column=3)#绿灯self.green_light = turtle.RawTurtle(self.canvas)self.green_light.shape('圆形')self.green_light.color('灰色')self.green_light.penup()self.green_light.goto(0, 0)# 红灯self.red_light = turtle.RawTurtle(self.canvas)self.red_light.shape('圆圈')self.red_light.color('灰色')self.red_light.penup()self.red_light.goto(40, 0)self.data_button = Button(root, text=获取高于阈值的数据"，command=rec_data.getData)self.data_button.grid(row=5, column=0)类 recv_data:def __init__(self):self.buff_data = multiprocessing.Queue()self.p1 = multiprocessing.Process(target=self.recvData)self.p2 = multiprocessing.Process(target=self.calculate_threshold)self.host = '127.0.0.1'self.port = 5001self.s = socket.socket()self.s.connect((self.host, self.port))# 接收TCP数据块的函数def getData(self):len_message = self.s.recv(4)bytes_length = int(len_message.decode('utf-8')) #为自制服务器recvd_data = self.s.recv(bytes_length)self.buff_data.put(recvd_data)self.p1.start()self.p2.start()self.p1.join()self.p2.join()def recvData(self):len_message = self.s.recv(4)而 len_message:bytes_length = int(len_message.decode('utf-8')) #为自制服务器recvd_data = self.s.recv(bytes_length)self.buff_data.put(recvd_data)len_message = self.s.recv(4)别的:打印('循环外')self.s.close()定义计算阈值(自我):rmv_data = self.buff_data.get()stringdata = rmv_data.decode('utf-8')rep_str = stringdata.replace(",", ".")splitstr = rep_str.split()# 接收波长值公司 = 34wav_threshold = []对于 gui.entries 中的 y:direct_wav = float(y.get())wav = float(splitstr[inc])wav_diff = wav - direct_wav如果 wav_diff <0:wav_diff = wav_diff * (-1)wav_threshold.append(wav_diff)公司 += 56阈值 = 浮点数(gui.entry_threshold.get())对于 wav_threshold 中的 x:如果(x>阈值):gui.red_light.color('红色')gui.green_light.color('灰色')别的:gui.red_light.color('灰色')gui.green_light.color('绿色')# 写入文件的函数def write_file(self, data):使用 open("Output.txt", "a") 作为 text_file:text_file.write('\t'.join(data[0:]))text_file.write('\n')如果 __name__ == '__main__':根 = Tk()gui1 = GUI(根)root.mainloop()

我得到的错误如下所示:

<预><代码>Tkinter 回调中的异常回溯(最近一次调用最后一次):文件C:\Users\AppData\Local\Programs\Python\Python38-32\lib\tkinter\__init__.py"，第 1883 行，在 __call__ 中返回 self.func(*args)文件C:/Users/PycharmProjects/GUI/GUI_multiprocess.py"，第 85 行，在 getData 中self.p2.start()文件C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\process.py"，第 121 行，在开始self._popen = self._Popen(self)文件C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\context.py"，第 224 行，在 _Popen返回_default_context.get_context().Process._Popen(process_obj)文件C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\context.py"，第 326 行，在 _Popen返回 Popen(process_obj)文件C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\popen_spawn_win32.py"，第 93 行，在 __init__ 中减少.转储(process_obj，to_child)转储中的文件C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\reduction.py"，第 60 行ForkingPickler(文件，协议).转储(对象)类型错误:无法pickle 'weakref' 对象回溯(最近一次调用最后一次):文件<string>"，第 1 行，在 <module>文件C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\spawn.py"，第 116 行，在 spawn_main 中exitcode = _main(fd, parent_sentinel)文件C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\spawn.py"，第 126 行，在 _mainself = reduction.pickle.load(from_parent)EOFError:输入不足

我在这里做错了什么，我该如何解决?任何帮助表示赞赏.谢谢！

解决方案

我刚刚遇到相同的回溯并设法解决了它.这是因为一个对象有一个正在运行或退出的 Process 作为变量，并且它正在使用该对象启动另一个进程.

问题

这是产生错误的最少代码:

导入多处理Foo类:def __init__(self):self.process_1 = multiprocessing.Process(target=self.do_stuff1)self.process_2 = multiprocessing.Process(target=self.do_stuff2)def do_multiprocessing(self):self.process_1.start()self.process_2.start()def do_stuff1(self):打印(做1")def do_stuff2(self):打印(做2")如果 __name__ == '__main__':foo = foo()foo.do_multiprocessing()[出去]:回溯(最近一次调用最后一次):文件myfile.py"，第 21 行，在 <module> 中.foo.do_multiprocessing()文件myfile.py"，第 11 行，在 do_multiprocessing 中self.process_2.start()文件...\lib\multiprocessing\process.py"，第 121 行，在开始self._popen = self._Popen(self)文件...\lib\multiprocessing\context.py"，第 224 行，在 _Popen返回_default_context.get_context().Process._Popen(process_obj)文件...\lib\multiprocessing\context.py"，第 327 行，在 _Popen返回 Popen(process_obj)文件...\lib\multiprocessing\popen_spawn_win32.py"，第 93 行，在 __init__ 中减少.转储(process_obj，to_child)转储中的文件...\lib\multiprocessing\reduction.py"，第 60 行ForkingPickler(文件，协议).转储(对象)类型错误:无法pickle 'weakref' 对象做 1回溯(最近一次调用最后一次):文件<string>"，第 1 行，在 <module>文件...\lib\multiprocessing\spawn.py"，第 116 行，在 spawn_main 中exitcode = _main(fd, parent_sentinel)文件...\lib\multiprocessing\spawn.py"，第 126 行，在 _main 中self = reduction.pickle.load(from_parent)EOFError:输入不足

所以问题是 Foo 在启动 foo.process_2 时还包含正在运行/退出的进程 foo.process_1.

解决方案 1

将 foo.process_1 设置为 None 或其他.或者将进程存储在 foo 之外的其他地方，以防止在启动 process_2 时被传递.

<代码>...def do_multiprocessing(self):self.process_1.start()self.process_1 = None # 删除退出的进程self.process_2.start()...

解决方案 2

从酸洗中移除有问题的变量 (process_1):

 Foo 类:def __getstate__(self):# 捕获通常腌制的东西state = self.__dict__.copy()# 删除不可选择的/有问题的变量状态['process_1'] = 无返回状态...

这在较新的 Python 版本中似乎是个问题.我自己的代码在 3.7 上运行良好，但由于 3.9 中的这个问题而失败.

我测试了您的代码(来自 recv_data).由于您加入进程并需要它们，因此您应该执行解决方案 2 或将进程存储在 recv_data 之外的其他位置.不确定您的代码还有哪些其他问题.

Quite new to multiprocessing here. I have a code that runs two processes. One to continuously receive data blocks from the server and put it inside a queue and the other to remove the data blocks from the queue and process it.

Below is my client code:

import socket
import turtle
import multiprocessing
from multiprocessing import Process, Queue
from tkinter import *


class GUI:

    def __init__(self, master):
        rec_data = recv_data()
        self.master = master
        master.title("Collision Detection")

        self.input_label = Label(root, text="Input all the gratings set straight wavelength values in nm")
        self.input_label.grid(row=0)

        self.core_string = "Core "
        self.entries = []


        self.label_col_inc = 0
        self.entry_col_inc = 1
        self.core_range = range(1, 5)

        for y in self.core_range:
            self.core_text = self.core_string + str(y) + '_' + '25'
            self.core_label = Label(root, text=self.core_text)
            self.entry = Entry(root)
            self.core_label.grid(row=1, column=self.label_col_inc, sticky=E)
            self.entry.grid(row=1, column=self.entry_col_inc)
            self.entries.append(self.entry)
            self.label_col_inc += 2
            self.entry_col_inc += 2

        self.threshold_label = Label(root, text="Threshold in nm")
        self.entry_threshold = Entry(root)

        self.threshold_label.grid(row=2, sticky=E)
        self.entry_threshold.grid(row=2, column=1)

        self.light_label = Label(root, text='Status')
        self.light_label.grid(row=3, column=3)

        self.canvas = Canvas(root, width=150, height=50)
        self.canvas.grid(row=4, column=3)

        # Green light
        self.green_light = turtle.RawTurtle(self.canvas)
        self.green_light.shape('circle')
        self.green_light.color('grey')
        self.green_light.penup()
        self.green_light.goto(0, 0)

        # Red light
        self.red_light = turtle.RawTurtle(self.canvas)
        self.red_light.shape('circle')
        self.red_light.color('grey')
        self.red_light.penup()
        self.red_light.goto(40, 0)

        self.data_button = Button(root, text="Get data above threshold", command=rec_data.getData)
        self.data_button.grid(row=5, column=0)


class recv_data:
    def __init__(self):
        self.buff_data = multiprocessing.Queue()
        self.p1 = multiprocessing.Process(target=self.recvData)
        self.p2 = multiprocessing.Process(target=self.calculate_threshold)
        self.host = '127.0.0.1'
        self.port = 5001
        self.s = socket.socket()
        self.s.connect((self.host, self.port))

    # function to receive TCP data blocks
    def getData(self):
        len_message = self.s.recv(4)
        bytes_length = int(len_message.decode('utf-8'))  # for the self-made server
        recvd_data = self.s.recv(bytes_length)
        self.buff_data.put(recvd_data)
        self.p1.start()
        self.p2.start()

        self.p1.join()
        self.p2.join()

    def recvData(self):
        len_message = self.s.recv(4)
        while len_message:
            bytes_length = int(len_message.decode('utf-8'))  # for the self-made server
            recvd_data = self.s.recv(bytes_length)
            self.buff_data.put(recvd_data)
            len_message = self.s.recv(4)

        else:
            print('out of loop')
        self.s.close()

    def calculate_threshold(self):
        rmv_data = self.buff_data.get()
        stringdata = rmv_data.decode('utf-8')
        rep_str = stringdata.replace(",", ".")
        splitstr = rep_str.split()

        # received wavelength values
        inc = 34
        wav_threshold = []
        for y in gui.entries:
            straight_wav = float(y.get())
            wav = float(splitstr[inc])
            wav_diff = wav - straight_wav
            if wav_diff < 0:
                wav_diff = wav_diff * (-1)
            wav_threshold.append(wav_diff)
            inc += 56

        threshold = float(gui.entry_threshold.get())

        for x in wav_threshold:
            if (x > threshold):
                gui.red_light.color('red')
                gui.green_light.color('grey')

            else:
                gui.red_light.color('grey')
                gui.green_light.color('green')
    

        # function to write into the file

    def write_file(self, data):
        with open("Output.txt", "a") as text_file:
            text_file.write('\t'.join(data[0:]))
            text_file.write('\n')

if __name__ == '__main__':
    root = Tk()
    gui1 = GUI(root)

    root.mainloop()

The error I get is shown below:

Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\AppData\Local\Programs\Python\Python38-32\lib\tkinter\__init__.py", line 1883, in __call__
    return self.func(*args)
  File "C:/Users/PycharmProjects/GUI/GUI_multiprocess.py", line 85, in getData
    self.p2.start()
  File "C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\context.py", line 326, in _Popen
    return Popen(process_obj)
  File "C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'weakref' object
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\AppData\Local\Programs\Python\Python38-32\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

What am I doing wrong here and how can I fix it? Any help is appreciated. Thank you!

解决方案

I just came to the same traceback and managed to solve it. It was due to that an object had a running or exited Process as a variable and it was starting another Process using that object.

Problem

This is a minimal code to produce your error:

import multiprocessing

class Foo:
    def __init__(self):
        self.process_1 = multiprocessing.Process(target=self.do_stuff1)
        self.process_2 = multiprocessing.Process(target=self.do_stuff2)

    def do_multiprocessing(self):
        self.process_1.start()
        self.process_2.start()

    def do_stuff1(self):
        print("Doing 1")

    def do_stuff2(self):
        print("Doing 2")

if __name__ == '__main__':
    foo = Foo()
    foo.do_multiprocessing()

[out]:
Traceback (most recent call last):
  File "myfile.py", line 21, in <module>
    foo.do_multiprocessing()
  File "myfile.py", line 11, in do_multiprocessing
    self.process_2.start()
  File "...\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "...\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "...\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "...\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "...\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'weakref' object
Doing 1
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "...\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "...\lib\multiprocessing\spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

So the issue is that Foo contains also the running/exited process foo.process_1 when it starts foo.process_2.

Solution 1

Set foo.process_1 to None or something else. Or store the Processes somewhere else than in foo to prevent being passed when starting process_2.

...
def do_multiprocessing(self):
    self.process_1.start()
    self.process_1 = None # Remove exited process
    self.process_2.start()
...

Solution 2

Remove the problematic variable (process_1) from pickling:

class Foo:
    def __getstate__(self):
        # capture what is normally pickled
        state = self.__dict__.copy()

        # remove unpicklable/problematic variables 
        state['process_1'] = None
        return state
...

This seems to be problem in newer Python versions. My own code worked fine for 3.7 but failed due to this issue in 3.9.

I tested your code (from recv_data). Since you join the processes and need them, you should do the solution 2 or store the processes somewhere else than in recv_data. Not sure what other problems your code has.

这篇关于类型错误:无法pickle 'weakref' 对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！