通过子进程在python脚本之间传输腌制的对象输出. [英] Transmitting a pickled object output between python scripts through a subprocess.communicate

问题描述

我有两个python脚本:object_generator.py，它们使给定的对象腌制并打印.另一个脚本object_consumer.py通过subprocess.communicate选择第一个脚本的输出，并尝试使用pickle.loads对其进行修补.我在使这个简单的方案起作用时遇到了麻烦.这是我的代码:

I have two python scripts:object_generator.py which pickles a given object and prints it. Another script object_consumer.py picks the output of the first script through a subprocess.communicate and tries to unpickle it using pickle.loads. I am having trouble making this simple scenario work. This is my code:

import pickle
import base64

o = {'first':1,'second':2,'third':3,'ls':[1,2,3]}
d = pickle.dumps(o)
print(d)

#Various Approaches I had tried, none of which worked. Ignore this part.
#s = base64.b64decode(d)
#encoded_str = str(d).encode('ascii')
#print('encoded str is :')
#print(encoded_str)
#decoded_str = encoded_str.decode('ascii')
#print('decoded str is :')
#print(decoded_str)
#unpickled_obj = pickle.loads(bytes(decoded_str))
#print(unpickled_obj)
#print(type(d))
#print(codecs.decode(d))

object_consumer.py

import pickle
import subprocess
import os

dr = '"' + os.path.dirname(os.path.abspath(__file__)) + '\\object_generator.py"'

cmd = 'python -u ' + dr

proc = subprocess.Popen(cmd,stdout=subprocess.PIPE)

try:
    outs, errs = proc.communicate(timeout=15)
except TimeoutExpired:
    proc.kill()
    outs, errs = proc.communicate()

# 'out' at this point is something like this : 
# b"b'\\x80\\x03}q\......x05K\\x03u.'\r\n"
# DO SOMETHING WITH outs to get back the bytes which can then be 
# unpickled using pickle.loads

obj = pickle.loads(outs)
print(obj)

很明显，我需要删除尾随的\ r \ n，这很容易，但是接下来应该怎么做?

Clearly, I need to strip off the trailing \r\n which is easy but what should be done next?

推荐答案

这里有两个问题.首先，您要在object_generator.py中打印一个bytes对象.在Python 3.x中，这将导致调用str(obj)，这意味着将打印b'yourbyteshere'.您不希望前导b'或尾随'.要解决此问题，您需要将bytes对象编码为字符串. pickle使用'latin-1'编码，因此我们可以使用它将bytes对象解码为str.另一个问题是Windows默认使用的sys.stdout编码实际上并不支持打印解码的pickle字符串.因此，我们需要将sys.stdout *的默认编码更改为'latin-1'，以便该字符串将以正确的编码进入父进程.

There are a couple of issues going on here. First, you're printing a bytes object in object_generator.py. In Python 3.x, that's going to result in str(obj) being called, which means b'yourbyteshere' gets printed. You don't want the leading b' or trailing '. To fix that, you need to encode the bytes object as a string. pickle uses the 'latin-1' encoding, so we can use that to decode the bytes object to a str. The other issue is that the encoding Windows uses by default for sys.stdout doesn't actually support printing decoded pickle strings. So, we need to change the default encoding for sys.stdout* to 'latin-1', so the string will make it to the parent process with the correct encoding.

import pickle
import base64
import codecs

o = {'first':1,'second':2,'third':3,'ls':[1,2,3]}
d = pickle.dumps(o)
sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding='latin-1')
print(d.decode('latin-1'), end='', flush=True)  # end='' will remove that extra \r\n

进行这些更改，它应该可以正常工作.

Make those changes, and it should work fine.

另一种选择是在父进程中将PYTHONIOENCODING环境变量设置为'latin-1':

Another option would be to set the PYTHONIOENCODING environment variable to 'latin-1' from the parent process:

env = os.environ.copy()
env['PYTHONIOENCODING'] = 'latin-1'
proc = subprocess.Popen(['python3', 'async2.py'] ,stdout=subprocess.PIPE, env=env)

_{*有关更改Python 3中sys.stdout编码的更多信息，请参见此问题.我说的是这里提到的.}

_{* See this question for more info on changing the sys.stdout encoding in Python 3. Both approaches I show here are mentioned there.}

这篇关于通过子进程在python脚本之间传输腌制的对象输出.的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！