AWS EMR-ModuleNotFoundError:没有名为'pyarrow'的模块 [英] AWS EMR - ModuleNotFoundError: No module named 'pyarrow'
问题描述
我遇到了带有Apache Arrow Spark Integration的问题.
I am running into this problem w/ Apache Arrow Spark Integration.
使用带有Spark 2.4.3的AWS EMR
Using AWS EMR w/ Spark 2.4.3
在本地spark单机实例和Cloudera集群上都测试了此问题,并且一切正常.
Tested this problem on both local spark single machine instance and a Cloudera cluster and everything works fine.
export PYSPARK_PYTHON=python3
export PYSPARK_PYTHON_DRIVER=python3
在spark shell中确认了这一点
spark.version
2.4.3
sc.pythonExec
python3
SC.pythonVer
python3
使用apache箭头集成运行基本的pandas_udf会导致错误
from pyspark.sql.functions import pandas_udf, PandasUDFType
df = spark.createDataFrame(
[(1, 1.0), (1, 2.0), (2, 3.0), (2, 5.0), (2, 10.0)],
("id", "v"))
@pandas_udf("id long, v double", PandasUDFType.GROUPED_MAP)
def subtract_mean(pdf):
# pdf is a pandas.DataFrame
v = pdf.v
return pdf.assign(v=v - v.mean())
df.groupby("id").apply(subtract_mean).show()
aws emr上的错误[在cloudera和本地计算机上没有错误]
ModuleNotFoundError: No module named 'pyarrow'
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)
at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:172)
at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:122)
at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:291)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:283)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
任何人都知道发生了什么事吗?一些可能的想法...
Anyone have an idea what is going on? some possible ideas ...
PYTHONPATH是否会引起问题,因为我没有使用anaconda
?
Could PYTHONPATH be causing a problem because I am not using anaconda
?
它与Spark版本和Arrow版本有关吗?
Does it have to do with the Spark Version and Arrow Version?
这是最奇怪的事情,因为我在所有3个平台(本地桌面,cloudera,emr)中使用相同的版本,并且只有EMR无法正常工作...
This is the strangest thing because I am using the same versions across within all 3 platforms [local desktop, cloudera, emr] and only EMR is not working ...
我登录了所有4个EMR EC2数据节点,并测试了是否可以导入pyarrow
,它完全可以正常工作,但是在尝试与spark
I logged into all 4 EMR EC2 data nodes and tested that I can importpyarrow
and it works totally fine but not when trying to use it with spark
# test
import numpy as np
import pandas as pd
import pyarrow as pa
df = pd.DataFrame({'one': [20, np.nan, 2.5],'two': ['january', 'february', 'march'],'three': [True, False, True]},index=list('abc'))
table = pa.Table.from_pandas(df)
推荐答案
在EMR中,默认情况下不解析python3.您必须使其明确.一种方法是在创建集群时传递config.json
文件.它在AWS EMR UI的Edit software settings
部分中可用.一个示例json文件看起来像这样.
In EMR python3 is not resolved by default. You have to make it explicit. One way to do it is to pass a config.json
file as you're creating the cluster. It's available in the Edit software settings
section in AWS EMR UI. A sample json file looks something like this.
[
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"Properties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}
]
},
{
"Classification": "yarn-env",
"Properties": {},
"Configurations": [
{
"Classification": "export",
"Properties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}
]
}
]
此外,您还需要在所有核心节点(不仅是主节点)中安装pyarrow
模块.为此,您可以在AWS中创建集群时使用引导脚本.同样,示例引导脚本可以像这样简单:
Also you need to have the pyarrow
module installed in all core nodes, not only in the master. For that you can use a bootstrap script while creating the cluster in AWS. Again, a sample bootstrap script can be as simple as something like this:
#!/bin/bash
sudo python3 -m pip install pyarrow==0.13.0
这篇关于AWS EMR-ModuleNotFoundError:没有名为'pyarrow'的模块的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!