将向量列添加到pyspark DataFrame [英] Adding a Vectors Column to a pyspark DataFrame

问题描述

如何在pyspark数据框中添加 Vectors.dense 列？

How do I add a Vectors.dense column to a pyspark dataframe?

import pandas as pd
from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.ml.linalg import DenseVector

py_df = pd.DataFrame.from_dict({"time": [59., 115., 156., 421.], "event": [1, 1, 1, 0]})

sc = SparkContext(master="local")
sqlCtx = SQLContext(sc)
sdf = sqlCtx.createDataFrame(py_df)
sdf.withColumn("features", DenseVector(1))

给出文件 anaconda3 / lib / python3.6 / site-packages / pyspark / sql中的错误/dataframe.py ，第1848行：

Gives an error in file anaconda3/lib/python3.6/site-packages/pyspark/sql/dataframe.py, line 1848:

AssertionError: col should be Column

它不喜欢 DenseVector 类型作为列。本质上，我有一个熊猫数据框，希望将其转换为pyspark数据框，并添加一个 Vectors.dense 类型的列。还有另一种方法吗？

It doesn't like the DenseVector type as a column. Essentially, I have a pandas dataframe that I'd like to transform to a pyspark dataframe and add a column of the type Vectors.dense. Is there another way of doing this?

推荐答案

常量向量不能是作为文字添加。您必须使用pyspark.sql.functions import中的 udf ：

Constant Vectors cannot be added as literal. You have to use udf:

from pyspark.sql.functions import udf
from pyspark.ml.linalg import VectorUDT

one = udf(lambda: DenseVector([1]), VectorUDT())
sdf.withColumn("features", one()).show()

但我不确定您为什么需要它。如果要将现有的列转换为 Vectors ，请使用适当的 pyspark.ml 工具，例如 VectorAssembler -在PySpark中编码和组合多个功能

But I am not sure why you need that at all. If you want to transform existing columns into Vectors use appropriate pyspark.ml tools, like VectorAssembler - Encode and assemble multiple features in PySpark

from pyspark.ml.feature import VectorAssembler

VectorAssembler(inputCols=["time"], outputCol="features").transform(sdf)

这篇关于将向量列添加到pyspark DataFrame的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！