从Spark导出JSON并将其输入到R中 [英] Export JSON from Spark and input into R

问题描述

我已将一些数据从Spark写入JSON文件，并且正在努力将其导入R.

I've written some data from Spark to a JSON file and I am struggling to import it into R.

我无法将其与R中的任何传统JSON包一起导入:

I cannot import it with any of the traditional JSON packages in R:

library("jsonlite")
bids <- fromJSON("win_rate_sample.json")

我收到以下错误:

Error in feed_push_parser(readBin(con, raw(), n), reset = TRUE) : 
 parse error: trailing garbage 
      X","domain":"ifunny_premium"}{"win":0,"bid_price":0.75,"size
                 (right here) ------^

如何将这个文件放到R中?

How can I get this file into R?

推荐答案

事实证明，Spark会导出流json文件，如以下问题中所讨论的:

It turns out that Spark exports streaming json files like those discussed in the following question: Error parsing JSON file with the jsonlite package

解决方案是使用jsonlite的流功能:

The solution is to use jsonlite's streaming function:

library(jsonlite)
json_file <- stream_in(file("win_rate_sample.json"))

这篇关于从Spark导出JSON并将其输入到R中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！