pyspark Hive上下文-使用UTF-8编码读取表 [英] pyspark Hive Context -- read table with UTF-8 encoding

问题描述

我在蜂巢中有一张桌子，而我正在pyspark df_sprk_df中读取该桌子

I have a table in hive, And I am reading that table in pyspark df_sprk_df

from pyspark import SparkContext
from pysaprk.sql import HiveContext
sc = SparkContext()
hive_context = HiveContext(sc)
df_sprk_df = hive_context.sql('select * from databasename.tablename')
df_pandas_df = df_sprk_df.toPandas()
df_pandas_df = df_pandas_df.astype('str')

但是当我尝试将df_pandas_df转换为str的类型时.但出现类似错误

but when I try to convert df_pandas_df to astype of str. but I get error like

UnicodeEnCodeError:'ascii'编解码器无法在位置编码字符u'\ u20ac'

UnicodeEnCodeError :'ascii' codec cant encode character u'\u20ac' in position

即使我尝试将

for cols in df_pandas_df.columns:
    df_pandas_df[cols] = df_pandas_df[cols].str.encode('utf-8')

但是没有运气，所以基本上我该如何将配置单元表以utf-8编码导入到数据帧中

but no luck, so basically how can I import hive table to dataframe in utf-8 encoding

推荐答案

因此，此变通办法通过更改会话的默认编码来帮助解决此问题

So this workaround helped to solve this, By changing the default encoding for the session

import sys
reload(sys)
sys.setdefaultencoding('UTF-8')

然后

df_pandas_df = df_pandas_df.astype(str)

将整个数据帧转换为字符串df.

converts whole dataframe as string df.

这篇关于pyspark Hive上下文-使用UTF-8编码读取表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！