在 Spark SQL DSL 中计算字符串长度 [英] compute string length in Spark SQL DSL

问题描述

这是一个关于 Spark 1.2 的老问题

this is an old question concerning Spark 1.2

为了 orderBy 的目的，我一直在尝试动态计算 SchemaRDD 中字符串列的长度.我正在学习 Spark SQL，所以我的问题是关于使用 DSL 或 Spark SQL 公开的 SQL 接口，或者了解它们的局限性.

I've been trying to compute on the fly the length of a string column in a SchemaRDD for orderBy purposes. I am learning Spark SQL so my question is strictly about using the DSL or the SQL interface that Spark SQL exposes, or to know their limitations.

我的第一次尝试是使用集成的关系查询，例如

My first attempt has been to use the integrated relational queries, for instance

notes.select('note).orderBy(length('note))

编译失败:

error: not found: value length

(这让我想知道在哪里可以找到此 DSL 实际可以解析的表达式".例如，它解析列添加的+".)

(Which makes me wonder where to find what "Expression" this DSL can actually resolve. For instance, it resolves "+" for column additions.)

然后我尝试了

sql("SELECT note, length(note) as len FROM notes")

这失败了

java.util.NoSuchElementException: key not found: length

(然后我重读了这个(我正在运行 1.2.0)http://spark.apache.org/docs/1.2.0/sql-programming-guide.html#supported-hive-features并想知道 Spark SQL 在何种意义上支持列出的 hive 功能.)

(Then I reread this (I'm running 1.2.0) http://spark.apache.org/docs/1.2.0/sql-programming-guide.html#supported-hive-features and wonder in what sense Spark SQL supports the listed hive features.)

问题:表达式和/或 SQL 语句中真的支持长度运算符吗?如果是，语法是什么?(奖励:是否有关于 Spark SQL 表达式中解析内容的特定文档，以及一般的语法是什么?)

Questions: is the length operator really supported in Expressions and/or in SQL statements? If yes, what is the syntax? (bonus: is there a specific documentation about what is resolved in Spark SQL Expressions, and what would be the syntax in general?)

谢谢！

推荐答案

在 Spark Shell 中试试这个:

Try this in Spark Shell:

case class Note(id:Int,text:String)
val notes=List(Note(1,"One"),Note(2,"Two"),Note(3,"Three"))
val notesRdd=sc.parallelize(notes)
import org.apache.spark.sql.hive.HiveContext
val hc=new HiveContext(sc)
import hc.createSchemaRDD
notesRdd.registerTempTable("note")
hc.sql("select id, text, length(text) from note").foreach(println)

它通过设置工作(开箱即用的 spark 1.2.1 with hadoop 2.4):

It works on by setup (out of the box spark 1.2.1 with hadoop 2.4):

[2,Two,3]
[1,One,3]
[3,Three,5]

这篇关于在 Spark SQL DSL 中计算字符串长度的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！