我如何腌制pyEphem对象以进行多处理? [英] How do I pickle pyEphem objects for multiprocessing?

问题描述

我正在尝试计算卫星的某些值，数据生成需要很长时间，因此我想使用多重处理来实现.

I am trying to calculate some values of satellites, the data-generation takes quite long so I want to implement this using multiprocessing.

问题是我从pyEphem TypeError: can't pickle ephem.EarthSatellite objects收到此错误. pyEphem对象未在我要并行化的函数中使用.

The problem is that I get this error from pyEphem, TypeError: can't pickle ephem.EarthSatellite objects. The pyEphem objects are not used in the functions that I want to parallelize.

这是我的代码的示例文件(已最小化).

This is an example file of my code (minimized).

这是我的主文件:

main.py

import ephem
import numpy
import math
import multiprocessing as mp
from SampleSats import Sats


GPS_Satellites = []

SFrames = 1
TLE = ["GPS BIIR-3  (PRN 11)",
       "1 25933U 99055A   18090.43292845 -.00000054  00000-0  00000+0 0  9994",
       "2 25933  51.8367  65.0783 0165007 100.2058 316.9161  2.00568927135407"]
# PRN TLE file from CelesTrak
GPS_Satellites.append(Sats(TLE))
Position = ephem.Observer()
Position.date = '2018/3/31 00:00'  # 1st January 2018 at 00:00 UTC
Position.lon, Position.lat = "36.845663", "-37.161123"   # Coordinates for desired Position

# Calculate Satellites
for Frames in range(SFrames):
    print("Generate Signals for Time: ", Position.date)
    for Sats in GPS_Satellites:  # par
        Sats.compute(Position)

        if ((float(repr(Sats.ephemeris.alt)) * 180 / math.pi) < 5) or (  # Calculate angle above horizon
                (float(repr(Sats.ephemeris.alt)) * 180 / math.pi) > 90):
            Sats.visible = 0
        else:
            Sats.visible = 1

    with mp.Pool() as pool:
        for value, obj in zip(pool.map(Sats.genSignal, GPS_Satellites), GPS_Satellites):
            obj.Signal = value

    Position.date = Position.date + 6*ephem.second  # 1 Subframe is 6 seconds long

这是我写的Sats类:

This is the Sats class that i wrote:

sats.py :

import ephem
import numpy

class Sats:
    """Save Satellites as Objects"""

    def __init__(self, tle):
        """:param tle: Two Line Element for ephemeris data also used to get PRN Number from name"""
        self.ephemeris = ephem.readtle(tle[0], tle[1], tle[2])
        self.visible = 1
        self.subframes = 0
        self.CAseq = [x for x in range(1023)]
        self.Out = []
        self.Signal = numpy.zeros(int(300*20*1023), dtype=numpy.int8)

    def compute(self, pos):
        self.ephemeris.compute(pos)
        self.Out.append(numpy.arange(0, 299, 1))
        self.subframes += 1

    def calcData(self, bit, prn):
         return (self.Out[self.subframes - 1].item(0)[0][bit] + self.CAseq[prn]) % 2

    def genSignal(self):
        if(self.visible == 1):
            for bit in range(300):  # 1 Subframe is 300 Bit long
                for x in range(20):  # The PRN Sequence reoccurs every ms -> 20 times per pit
                    for prn in range(1023):  # length of the prn sequence
                        self.Signal[bit*x*prn] = (-1 if (self.calcData(bit, prn))==0 else 1)
        else:
            self.Signal = numpy.zeros(300*20*1023)
        return self.Signal

跟踪:

Traceback (most recent call last):
  File "C:/Users/PATH_TO_PROJECT/SampleTest.py", line 33, in <module>
    for value, obj in zip(pool.map(Sats.genSignal, GPS_Satellites), GPS_Satellites):
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 644, in get
    raise self._value
  File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 424, in _handle_tasks
    put(task)
  File "C:\Program Files\Python36\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Program Files\Python36\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle ephem.EarthSatellite objects

推荐答案

原因是这样的...当您尝试对函数进行酸洗时，它可以尝试对globals()进行酸洗，因此无论您在全局变量中拥有什么还对名称空间进行了腌制(以防万一您的函数引用了globals()中的某些内容，是的，这是意料之外的，但事实就是如此).因此，一个简单的解决方法是将您要腌制的功能隔离到另一个文件中–在这种情况下，将multiprocessing内容放入一个文件中，而将另一个代码放入另一个文件中...因此，globals()中的内容较少苦苦挣扎的腌制者.可能有帮助的另一件事是使用multiprocess而不是multiprocessing-multiprocess使用dill序列化器而不是pickle，因此您有更大的机会序列化将在Pool.

The reason is something like this... when you try to pickle a function, it can attempt to pickle globals(), so whatever you have in your global namespace is also pickled (just in case your function has a reference to something in globals() -- yes, that's unexpected, but that's how it is). So, an easy fix is to isolate the function you want to pickle in another file -- in this case, put the multiprocessing stuff in one file and the other code in another file... so there's less in globals() for the pickler to struggle with. Another thing that might help is to use multiprocess instead of multiprocessing -- multiprocess uses the dill serializer instead of pickle, so you have a better chance of serializing objects that will be sent across the workers in the Pool.

这篇关于我如何腌制pyEphem对象以进行多处理?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！