Livy Server:以JSON格式返回数据帧? [英] Livy Server: return a dataframe as JSON?

问题描述

我正在通过使用localhost:8998/sessions/0/statements的HTTP POST调用在Livy Server中执行以下语句，

I am executing a statement in Livy Server using HTTP POST call to localhost:8998/sessions/0/statements, with the following body

{
  "code": "spark.sql(\"select * from test_table limit 10\")"
}

我想要以下格式的答案

(...)
"data": {
  "application/json": "[
    {"id": "123", "init_date": 1481649345, ...},
    {"id": "133", "init_date": 1481649333, ...},
    {"id": "155", "init_date": 1481642153, ...},
  ]"
}
(...)

但是我得到的是

(...)
"data": {
  "text/plain": "res0: org.apache.spark.sql.DataFrame = [id: string, init_date: timestamp ... 64 more fields]"
}
(...)

哪个是数据框的toString()版本.

是否有某种方法可以使用Livy Server将数据帧作为JSON返回?

Is there some way to return a dataframe as JSON using the Livy Server?

找到了解决该问题的JIRA问题: https://issues.cloudera.org/browse /LIVY-72

Found a JIRA issue that addresses the problem: https://issues.cloudera.org/browse/LIVY-72

通过评论可以说Livy不支持并且不支持这种功能吗?

By the comments one can say that Livy does not and will not support such feature?

推荐答案

我对Livy经验不足，但据我所知，该端点用作交互式外壳，输出将是字符串并带有外壳显示的实际结果.因此，考虑到这一点，我可以想到一种模拟所需结果的方法，但这可能不是最佳方法:

I don't have a lot of experience with Livy, but as far as I know this endpoint is used as an interactive shell and the output will be a string with the actual result that would be shown by a shell. So, with that in mind, I can think of a way to emulate the result you want, but It may not be the best way to do it:

{
  "code": "println(spark.sql(\"select * from test_table limit 10\").toJSON.collect.mkString(\"[\", \",\", \"]\"))"
}

然后，您将在字符串中包装一个JSON，以便您的客户端可以对其进行解析.

Then, you will have a JSON wrapped in a string, so your client could parse it.

这篇关于Livy Server:以JSON格式返回数据帧?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！