Spark SQL-使用SQL语句而不是表名使用JDBC加载数据 [英] Spark SQL - load data with JDBC using SQL statement, not table name

问题描述

我认为我缺少了一些东西，但无法弄清楚是什么. 我想使用特定的sql语句使用SQLContext和JDBC加载数据 喜欢

I think I am missing something but can't figure what. I want to load data using SQLContext and JDBC using particular sql statement like

select top 1000 text from table1 with (nolock)
where threadid in (
  select distinct id from table2 with (nolock)
  where flag=2 and date >= '1/1/2015' and  userid in (1, 2, 3)
)

我应该使用哪种SQLContext方法?我看到的示例总是指定表名以及上下边距.

Which method of SQLContext should I use? Examples I saw always specify table name and lower and upper margin.

谢谢.

推荐答案

您应传递有效的子查询作为dbtable参数.例如在Scala中:

You should pass a valid subquery as a dbtable argument. For example in Scala:

val query = """(SELECT TOP 1000 
  -- and the rest of your query
  -- ...
) AS tmp  -- alias is mandatory*"""   

val url: String = ??? 

val jdbcDF = sqlContext.read.format("jdbc")
  .options(Map("url" -> url, "dbtable" -> query))
  .load()

---

* Hive语言手册子查询: https://cwiki.apache.org/confluence/display/Hive/LanguageManual + SubQueries

这篇关于Spark SQL-使用SQL语句而不是表名使用JDBC加载数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！