如何使用POST方法发送 pandas 数据帧并在Hug/其他REST API框架中接收它? pickle.loads发送后无法释放 [英] How to send a pandas dataframe using POST method and receive it in Hug/other REST API framework? pickle.loads fails to unpickle after sending

问题描述

如何使用POST方法发送熊猫DataFrame?

How to send a pandas DataFrame using a POST method?

例如，以下拥抱服务器监听POST请求并以腌制的熊猫作为响应数据框:

For example, the following hug server listens to a POST requests and responds with a pickled pandas DataFrame:

import hug
import pickle
import traceback
import pandas as pd

@hug.post()
def call(pickle_dump):
    print(type(pickle_dump))
    try:
        df = pickle.loads(pickle_dump)
        return pickle.dumps(df.iloc[0])
    except:
        print(traceback.format_exc())
        return pickle.dumps(pd.DataFrame())

发出以下POST请求时:

import requests
import pandas as pd

df = pd.DataFrame(pd.np.random.randn(10,20))
r = requests.post('http://localhost:8000/call', data = {'pickle_dump':pickle.dumps(df)})
pickle.loads(r.text)

服务器返回此回溯:

<class 'str'>
Traceback (most recent call last):
  File "post.py", line 9, in call
    df = pickle.loads(pickle_dump)
TypeError: a bytes-like object is required, not 'str'

127.0.0.1 - - [23/Jul/2018 17:12:12] "POST /call HTTP/1.1" 200 10

同样，客户返回:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-292-956952cbfca9> in <module>()
      5 r = requests.post('http://localhost:8000/call', data = {'pickle_dump':pickle.dumps(df)})
      6 
----> 7 pickle.loads(r.text)

TypeError: a bytes-like object is required, not 'str'

这似乎与以下事实有关:当将字节对象发送到hug api时，字节将通过以下方式转换为str:

This seems to be related to the fact that when a byte object is sent to the hug api, the bytes are converted to a str in the following way:

例如，pickle.dumps(b'test')在客户端上返回b'\x80\x03C\x04testq\x00.'. 在拥抱服务器中收到该消息后，该消息将变为str('\x80\x03C\x04testq\x00.')(缺少b).可以使用pickle.loads('\x80\x03C\x04testq\x00.'.encode()[1:])将对象解码回其原始形式.

For example pickle.dumps(b'test') returns b'\x80\x03C\x04testq\x00.' on the client. When it is received in the hug server, this becomes str('\x80\x03C\x04testq\x00.') (missing b). The object can be decoded back to it's original form using pickle.loads('\x80\x03C\x04testq\x00.'.encode()[1:]).

在DataFrame上应用上述过程将导致UnpicklingError:

Applying the above process on a DataFrame results in an UnpicklingError:

> pickle.dumps(pd.DataFrame())
b'\x80\x03cpandas.core.frame\nDataFrame\nq\x00)\x81q\x01}q\x02(X\t\x00\x00\x00_metadataq\x03]q\x04X\x04\x00\x00\x00_typq\x05X\t\x00\x00\x00dataframeq\x06X\x05\x00\x00\x00_dataq\x07cpandas.core.internals\nBlockManager\nq\x08)\x81q\t(]q\n(cpandas.core.indexes.base\n_new_Index\nq\x0bcpandas.core.indexes.base\nIndex\nq\x0c}q\r(X\x04\x00\x00\x00nameq\x0eNX\x04\x00\x00\x00dataq\x0fcnumpy.core.multiarray\n_reconstruct\nq\x10cnumpy\nndarray\nq\x11K\x00\x85q\x12C\x01bq\x13\x87q\x14Rq\x15(K\x01K\x00\x85q\x16cnumpy\ndtype\nq\x17X\x02\x00\x00\x00O8q\x18K\x00K\x01\x87q\x19Rq\x1a(K\x03X\x01\x00\x00\x00|q\x1bNNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK?tq\x1cb\x89]q\x1dtq\x1ebu\x86q\x1fRq h\x0bh\x0c}q!(h\x0eNh\x0fh\x10h\x11K\x00\x85q"h\x13\x87q#Rq$(K\x01K\x00\x85q%h\x1a\x89]q&tq\'bu\x86q(Rq)e]q*]q+}q,X\x06\x00\x00\x000.14.1q-}q.(X\x06\x00\x00\x00blocksq/]q0X\x04\x00\x00\x00axesq1h\nustq2bub.'

倒泡菜

pickle.loads('\x80\x03cpandas.core.frame\nDataFrame\nq\x00)\x81q\x01}q\x02(X\t\x00\x00\x00_metadataq\x03]q\x04X\x04\x00\x00\x00_typq\x05X\t\x00\x00\x00dataframeq\x06X\x05\x00\x00\x00_dataq\x07cpandas.core.internals\nBlockManager\nq\x08)\x81q\t(]q\n(cpandas.core.indexes.base\n_new_Index\nq\x0bcpandas.core.indexes.base\nIndex\nq\x0c}q\r(X\x04\x00\x00\x00nameq\x0eNX\x04\x00\x00\x00dataq\x0fcnumpy.core.multiarray\n_reconstruct\nq\x10cnumpy\nndarray\nq\x11K\x00\x85q\x12C\x01bq\x13\x87q\x14Rq\x15(K\x01K\x00\x85q\x16cnumpy\ndtype\nq\x17X\x02\x00\x00\x00O8q\x18K\x00K\x01\x87q\x19Rq\x1a(K\x03X\x01\x00\x00\x00|q\x1bNNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK?tq\x1cb\x89]q\x1dtq\x1ebu\x86q\x1fRq h\x0bh\x0c}q!(h\x0eNh\x0fh\x10h\x11K\x00\x85q"h\x13\x87q#Rq$(K\x01K\x00\x85q%h\x1a\x89]q&tq\'bu\x86q(Rq)e]q*]q+}q,X\x06\x00\x00\x000.14.1q-}q.(X\x06\x00\x00\x00blocksq/]q0X\x04\x00\x00\x00axesq1h\nustq2bub.'.encode()[1:])

结果:

---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
<ipython-input-314-7082d60a5569> in <module>()
----> 1 pickle.loads('\x80\x03cpandas.core.frame\nDataFrame\nq\x00)\x81q\x01}q\x02(X\t\x00\x00\x00_metadataq\x03]q\x04X\x04\x00\x00\x00_typq\x05X\t\x00\x00\x00dataframeq\x06X\x05\x00\x00\x00_dataq\x07cpandas.core.internals\nBlockManager\nq\x08)\x81q\t(]q\n(cpandas.core.indexes.base\n_new_Index\nq\x0bcpandas.core.indexes.base\nIndex\nq\x0c}q\r(X\x04\x00\x00\x00nameq\x0eNX\x04\x00\x00\x00dataq\x0fcnumpy.core.multiarray\n_reconstruct\nq\x10cnumpy\nndarray\nq\x11K\x00\x85q\x12C\x01bq\x13\x87q\x14Rq\x15(K\x01K\x00\x85q\x16cnumpy\ndtype\nq\x17X\x02\x00\x00\x00O8q\x18K\x00K\x01\x87q\x19Rq\x1a(K\x03X\x01\x00\x00\x00|q\x1bNNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK?tq\x1cb\x89]q\x1dtq\x1ebu\x86q\x1fRq h\x0bh\x0c}q!(h\x0eNh\x0fh\x10h\x11K\x00\x85q"h\x13\x87q#Rq$(K\x01K\x00\x85q%h\x1a\x89]q&tq\'bu\x86q(Rq)e]q*]q+}q,X\x06\x00\x00\x000.14.1q-}q.(X\x06\x00\x00\x00blocksq/]q0X\x04\x00\x00\x00axesq1h\nustq2bub.'.encode()[1:])

UnpicklingError: 

我愿意使用任何允许使用HTTP请求发送和接收熊猫DataFrame的框架.

I am open to using any framework which will allow me to send and receive a pandas DataFrame using HTTP requests.

服务器和客户端都在具有相同软件包版本的相同环境中运行.

Both the server and the client are run in the same environment with identical package versions.

如何使用HTTP方法发送和接收熊猫DataFrame?

How to send and receive a pandas DataFrame using HTTP methods?

推荐答案

似乎b64编码腌制字符串似乎可以缓解此问题.为简洁起见，我将使用一个示例进行演示.

It seems like b64 encoding the pickled string seems to alleviate the issue. For brevity, I will use an example to demonstrate.

假设我具有以下数据框:

Suppose I have the following dataframe:

>>> import pandas as pd
>>> df = pd.DataFrame({['a': [0, 1, 2, 3]})
>>> df
   a
0  0
1  1
2  2
3  3

现在，让我们将对象腌制为类似字节的字符串，然后b64对腌制的字符串进行编码:

Now, let's pickle the object to a bytes-like string, and then b64encode the pickled string:

>>> pickled = pickle.dumps(df)
>>> import base64
>>> pickled_b64 = base64.b64encode(pickled)
>>> pickled_b64
b'gANjcGFuZGFzLmNvcmUuZnJhbWUKRGF0YUZyYW1lCnEAKYFxAX1xAihYCQAAAF9tZXRhZGF0YXEDXXEEWAQAAABfdHlwcQVYCQAAAGRhdGFmcmFtZXEGWAUAAABfZGF0YXEHY3BhbmRhcy5jb3JlLmludGVybmFscwpCbG9ja01hbmFnZXIKcQgpgXEJKF1xCihjcGFuZGFzLmNvcmUuaW5kZXhlcy5iYXNlCl9uZXdfSW5kZXgKcQtjcGFuZGFzLmNvcmUuaW5kZXhlcy5iYXNlCkluZGV4CnEMfXENKFgEAAAAbmFtZXEOTlgEAAAAZGF0YXEPY251bXB5LmNvcmUubXVsdGlhcnJheQpfcmVjb25zdHJ1Y3QKcRBjbnVtcHkKbmRhcnJheQpxEUsAhXESQwFicROHcRRScRUoSwFLAYVxFmNudW1weQpkdHlwZQpxF1gCAAAATzhxGEsASwGHcRlScRooSwNYAQAAAHxxG05OTkr/////Sv////9LP3RxHGKJXXEdWAEAAABhcR5hdHEfYnWGcSBScSFoC2NwYW5kYXMuY29yZS5pbmRleGVzLnJhbmdlClJhbmdlSW5kZXgKcSJ9cSMoaA5OWAUAAABzdGFydHEkSwBYBAAAAHN0b3BxJUsEWAQAAABzdGVwcSZLAXWGcSdScShlXXEpaBBoEUsAhXEqaBOHcStScSwoSwFLAUsEhnEtaBdYAgAAAGk4cS5LAEsBh3EvUnEwKEsDWAEAAAA8cTFOTk5K/////0r/////SwB0cTJiiUMgAAAAAAAAAAABAAAAAAAAAAIAAAAAAAAAAwAAAAAAAABxM3RxNGJhXXE1aAtoDH1xNihoDk5oD2gQaBFLAIVxN2gTh3E4UnE5KEsBSwGFcTpoGoldcTtoHmF0cTxidYZxPVJxPmF9cT9YBgAAADAuMTQuMXFAfXFBKFgGAAAAYmxvY2tzcUJdcUN9cUQoWAgAAABtZ3JfbG9jc3FFY2J1aWx0aW5zCnNsaWNlCnFGSwBLAUsBh3FHUnFIWAYAAAB2YWx1ZXNxSWgsdWFYBAAAAGF4ZXNxSmgKdXN0cUtidWIu'

因此64编码的字符串也是类似字节的字符串，但是它不包含十六进制转义序列，因此当将其转换为字符串时，在将其编码为字节时仍保留该字符串.

So the 64encoded string is also a bytes-like string, but it doesn't contain the hex escape sequences so when it gets converted to a string, the string is still preserved when encoding it to bytes.

现在，让我们模仿一下hug对字符串所做的操作，如您所述:

Now, let's mimic what hug does to the string, as you have noted:

>>> hug_pickled_str = pickled_b64.decode('utf-8')
>>> hug_pickled
'gANjcGFuZGFzLmNvcmUuZnJhbWUKRGF0YUZyYW1lCnEAKYFxAX1xAihYCQAAAF9tZXRhZGF0YXEDXXEEWAQAAABfdHlwcQVYCQAAAGRhdGFmcmFtZXEGWAUAAABfZGF0YXEHY3BhbmRhcy5jb3JlLmludGVybmFscwpCbG9ja01hbmFnZXIKcQgpgXEJKF1xCihjcGFuZGFzLmNvcmUuaW5kZXhlcy5iYXNlCl9uZXdfSW5kZXgKcQtjcGFuZGFzLmNvcmUuaW5kZXhlcy5iYXNlCkluZGV4CnEMfXENKFgEAAAAbmFtZXEOTlgEAAAAZGF0YXEPY251bXB5LmNvcmUubXVsdGlhcnJheQpfcmVjb25zdHJ1Y3QKcRBjbnVtcHkKbmRhcnJheQpxEUsAhXESQwFicROHcRRScRUoSwFLAYVxFmNudW1weQpkdHlwZQpxF1gCAAAATzhxGEsASwGHcRlScRooSwNYAQAAAHxxG05OTkr/////Sv////9LP3RxHGKJXXEdWAEAAABhcR5hdHEfYnWGcSBScSFoC2NwYW5kYXMuY29yZS5pbmRleGVzLnJhbmdlClJhbmdlSW5kZXgKcSJ9cSMoaA5OWAUAAABzdGFydHEkSwBYBAAAAHN0b3BxJUsEWAQAAABzdGVwcSZLAXWGcSdScShlXXEpaBBoEUsAhXEqaBOHcStScSwoSwFLAUsEhnEtaBdYAgAAAGk4cS5LAEsBh3EvUnEwKEsDWAEAAAA8cTFOTk5K/////0r/////SwB0cTJiiUMgAAAAAAAAAAABAAAAAAAAAAIAAAAAAAAAAwAAAAAAAABxM3RxNGJhXXE1aAtoDH1xNihoDk5oD2gQaBFLAIVxN2gTh3E4UnE5KEsBSwGFcTpoGoldcTtoHmF0cTxidYZxPVJxPmF9cT9YBgAAADAuMTQuMXFAfXFBKFgGAAAAYmxvY2tzcUJdcUN9cUQoWAgAAABtZ3JfbG9jc3FFY2J1aWx0aW5zCnNsaWNlCnFGSwBLAUsBh3FHUnFIWAYAAAB2YWx1ZXNxSWgsdWFYBAAAAGF4ZXNxSmgKdXN0cUtidWIu'

现在可以在服务器端使用该字符串了:

Now to make the string consumable on the server-side:

>>> ss_df = pickle.loads(base64.b64decode(hug_pickled.encode())
>>> ss_df
   a
0  0
1  1
2  2
3  3

因此，您需要对腌制的字符串进行base64编码，并将该字符串作为数据传递到API.

Therefore, you would need to base64 encode your pickled string and pass that string as the data to your API.

这篇关于如何使用POST方法发送 pandas 数据帧并在Hug/其他REST API框架中接收它? pickle.loads发送后无法释放的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！