创建Spark DataFrame.无法推断类型的架构:< type'float'> [英] Create Spark DataFrame. Can not infer schema for type: <type 'float'>

问题描述

有人可以帮我解决Spark DataFrame遇到的这个问题吗?

Could someone help me solve this problem I have with Spark DataFrame?

当我执行myFloatRDD.toDF()时，我得到一个错误:

When I do myFloatRDD.toDF() I get an error:

TypeError:无法推断类型的架构:输入'float'

TypeError: Can not infer schema for type: type 'float'

我不明白为什么...

I don't understand why...

示例:

myFloatRdd = sc.parallelize([1.0,2.0,3.0])
df = myFloatRdd.toDF()

谢谢

推荐答案

SparkSession.createDataFrame(在引擎盖下使用)需要Row/tuple/list/的RDD/list dict *或pandas.DataFrame，除非提供了带有DataType的架构.尝试将float转换为元组，如下所示:

SparkSession.createDataFrame, which is used under the hood, requires an RDD / list of Row/tuple/list/dict* or pandas.DataFrame, unless schema with DataType is provided. Try to convert float to tuple like this:

myFloatRdd.map(lambda x: (x, )).toDF()

甚至更好:

from pyspark.sql import Row

row = Row("val") # Or some other column name
myFloatRdd.map(row).toDF()

要从标量列表创建DataFrame，您必须直接使用SparkSession.createDataFrame并提供模式***:

To create a DataFrame from a list of scalars you'll have to use SparkSession.createDataFrame directly and provide a schema***:

from pyspark.sql.types import FloatType

df = spark.createDataFrame([1.0, 2.0, 3.0], FloatType())

df.show()

## +-----+
## |value|
## +-----+
## |  1.0|
## |  2.0|
## |  3.0|
## +-----+

但对于简单范围，最好使用SparkSession.range:

but for a simple range it would be better to use SparkSession.range:

from pyspark.sql.functions import col

spark.range(1, 4).select(col("id").cast("double"))

---

*不再受支持.

---

* No longer supported.

** Spark SQL还为暴露__dict__的Python对象上的模式推断提供了有限的支持.

** Spark SQL also provides a limited support for schema inference on Python objects exposing __dict__.

***仅在Spark 2.0或更高版本中受支持.

*** Supported only in Spark 2.0 or later.

这篇关于创建Spark DataFrame.无法推断类型的架构:< type'float'>的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！