如何在 spark 中使用 Regexp_replace [英] how to use Regexp_replace in spark

问题描述

我对 spark 还很陌生，想对数据帧的一列执行操作，以便将列中的所有 , 替换为 .

假设有一个数据框 x 和列 x4

x41,34351,6566-0,34435

我希望输出为

x41.34351.6566-0.34435

我使用的代码是

import org.apache.spark.sql.Columndef replace = regexp_replace((x.x4,1,6566:String,1.6566:String)x.x4)

但我收到以下错误

import org.apache.spark.sql.Column<控制台>:1: 错误: ')' 预期但 '.'成立.def 替换 = regexp_replace((train_df.x37,0,160430299:String,0.160430299:String)train_df.x37)

对语法、逻辑或任何其他合适方式的任何帮助将不胜感激

解决方案

这是一个可重现的示例，假设 x4 是一个字符串列.

import org.apache.spark.sql.functions.regexp_replaceval df = spark.createDataFrame(Seq((1, "1,3435"),(2, "1,6566"),(3, "-0,34435"))).toDF("Id", "x4")

语法是regexp_replace(str,pattern,replacement)，转换为:

df.withColumn("x4New", regexp_replace(df("x4"), "\\,", ".")).show+---+--------+--------+|编号|x4|x4新|+---+--------+--------+|1|1,3435|1.3435||2|1,6566|1.6566||3|-0,34435|-0.34435|+---+--------+--------+

I am pretty new to spark and would like to perform an operation on a column of a dataframe so as to replace all the , in the column with .

Assume there is a dataframe x and column x4

x4
1,3435
1,6566
-0,34435

I want the output to be as

x4
1.3435
1.6566
-0.34435

The code I am using is

import org.apache.spark.sql.Column
def replace = regexp_replace((x.x4,1,6566:String,1.6566:String)x.x4)

But I get the following error

import org.apache.spark.sql.Column
<console>:1: error: ')' expected but '.' found.
       def replace = regexp_replace((train_df.x37,0,160430299:String,0.160430299:String)train_df.x37)

Any help on the syntax, logic or any other suitable way would be much appreciated

解决方案

Here's a reproducible example, assuming x4 is a string column.

import org.apache.spark.sql.functions.regexp_replace

val df = spark.createDataFrame(Seq(
  (1, "1,3435"),
  (2, "1,6566"),
  (3, "-0,34435"))).toDF("Id", "x4")

The syntax is regexp_replace(str, pattern, replacement), which translates to:

df.withColumn("x4New", regexp_replace(df("x4"), "\\,", ".")).show
+---+--------+--------+
| Id|      x4|   x4New|
+---+--------+--------+
|  1|  1,3435|  1.3435|
|  2|  1,6566|  1.6566|
|  3|-0,34435|-0.34435|
+---+--------+--------+

这篇关于如何在 spark 中使用 Regexp_replace的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！