Sparksql在特殊位置后剪切字符串 [英] Sparksql cut String after special position

问题描述

您好，我想做的就是剪切一个URL，以便所有这些都采用特定的格式. 瞬间，我的URL看起来像这样.

Hello what i want to do is to cut a URL so it is all in a specific Format. At the Moment my URL Looks like this.

[https://url.com/xxxxxxx/xxxxx/xxxxxx]

我只想剪切掉第三个/之后的所有内容，然后只计算我的数据，这样我就可以大致了解我的数据中有多少个URL.

I just want to cut everything after the third / and just Count my data so that i have an overview how much URLs i have in my data.

我希望有人能帮助我

推荐答案

您需要的是用户定义函数(UDF).假设您输入以下内容:

User-defined functions (UDFs) is what you need. Assume you have following input:

case class Data(url: String)
val urls = sqlContext.createDataFrame(Seq(Data("http://google.com/q=dfsdf"), Data("https://fb.com/gsdgsd")))
urls.registerTempTable("urls")

现在，您可以定义仅从URL获取主机名的UDF:

Now you can define UDF that gets only hostname from URL:

def getHost(url: String) = url.split('/')(2) //naive implementation, for example only
sqlContext.udf.register("getHost", getHost _)

并使用SQL转换数据:

And get your data transformed using SQL:

val hosts = sqlContext.sql("select getHost(url) as host from urls")
hosts.show()

结果:

+----------+
|      host|
+----------+
|google.com|
|    fb.com|
+----------+

如果您喜欢Scala DSL，也可以使用UDF:

If you prefer Scala DSL, you can use your UDF too:

import org.apache.spark.sql.functions.udf
val getHostUdf = udf(getHost _)
val urls = urls.select(getHostUdf($"url") as "host")

结果将完全相同.

这篇关于Sparksql在特殊位置后剪切字符串的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！