Sparksql 在特殊位置后剪切字符串 [英] Sparksql cut String after special position

问题描述

您好，我想做的是剪切一个 URL，使其全部采用特定格式.目前我的网址看起来像这样.

Hello what i want to do is to cut a URL so it is all in a specific Format. At the Moment my URL Looks like this.

[https://url.com/xxxxxxx/xxxxx/xxxxxx]

我只想在第三个/之后删除所有内容，然后计算我的数据，以便了解我的数据中有多少 URL.

I just want to cut everything after the third / and just Count my data so that i have an overview how much URLs i have in my data.

希望有人能帮帮我

推荐答案

用户定义函数 (UDF) 正是您所需要的.假设您有以下输入:

User-defined functions (UDFs) is what you need. Assume you have following input:

case class Data(url: String)
val urls = sqlContext.createDataFrame(Seq(Data("http://google.com/q=dfsdf"), Data("https://fb.com/gsdgsd")))
urls.registerTempTable("urls")

现在您可以定义仅从 URL 获取主机名的 UDF:

Now you can define UDF that gets only hostname from URL:

def getHost(url: String) = url.split('/')(2) //naive implementation, for example only
sqlContext.udf.register("getHost", getHost _)

并使用 SQL 转换您的数据:

And get your data transformed using SQL:

val hosts = sqlContext.sql("select getHost(url) as host from urls")
hosts.show()

结果:

+----------+
|      host|
+----------+
|google.com|
|    fb.com|
+----------+

如果你更喜欢 Scala DSL，你也可以使用你的 UDF:

If you prefer Scala DSL, you can use your UDF too:

import org.apache.spark.sql.functions.udf
val getHostUdf = udf(getHost _)
val urls = urls.select(getHostUdf($"url") as "host")

结果将完全相同.

这篇关于Sparksql 在特殊位置后剪切字符串的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！