带有 DataFrame API 的 Apache Spark MLlib 在 createDataFrame() 或 read().csv(...) 时给出 java.net.URISyntaxException [英] Apache Spark MLlib with DataFrame API gives java.net.URISyntaxException when createDataFrame() or read().csv(...)
问题描述
在独立应用程序中(在 java8、Windows 10 上运行,使用 spark-xxx_2.11:2.0.0 作为 jar 依赖项)下一个代码给出错误:
In a standalone application (runs on java8, Windows 10 with spark-xxx_2.11:2.0.0 as jar dependencies) next code gives an error:
/* this: */
Dataset<Row> logData = spark_session.createDataFrame(Arrays.asList(
new LabeledPoint(1.0, Vectors.dense(4.9,3,1.4,0.2)),
new LabeledPoint(1.0, Vectors.dense(4.7,3.2,1.3,0.2))
), LabeledPoint.class);
/* or this: */
/* logFile: "C:\files\project\file.csv", "C:\\files\\project\\file.csv",
"C:/files/project/file.csv", "file:/C:/files/project/file.csv",
"file:///C:/files/project/file.csv", "/file.csv" */
Dataset<Row> logData = spark_session.read().csv(logFile);
异常:
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:C:/files/project/spark-warehouse
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.<init>(Path.java:172)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.makeQualifiedPath(SessionCatalog.scala:114)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase(SessionCatalog.scala:145)
at org.apache.spark.sql.catalyst.catalog.SessionCatalog.<init>(SessionCatalog.scala:89)
at org.apache.spark.sql.internal.SessionState.catalog$lzycompute(SessionState.scala:95)
at org.apache.spark.sql.internal.SessionState.catalog(SessionState.scala:95)
at org.apache.spark.sql.internal.SessionState$$anon$1.<init>(SessionState.scala:112)
at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:112)
at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:111)
at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:373)
at <call in my line of code>
如何将 csv 文件从 java 代码加载到 Dataset
中?
How can I load csv file into Dataset<Row>
from java code?
推荐答案
文件系统路径存在一些问题.请参阅 jira https://issues.apache.org/jira/browse/SPARK-15899一>.对于解决方法,您可以在 SparkSession 中设置spark.sql.warehouse.dir",如下所示.
There is some issue with file system path. See jira https://issues.apache.org/jira/browse/SPARK-15899. For workaround you can set "spark.sql.warehouse.dir" in SparkSession like below.
SparkSession spark = SparkSession
.builder()
.appName("JavaALSExample")
.config("spark.sql.warehouse.dir", "/file:C:/temp")
.getOrCreate();
这篇关于带有 DataFrame API 的 Apache Spark MLlib 在 createDataFrame() 或 read().csv(...) 时给出 java.net.URISyntaxException的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!