如何在Zeppelin中的％pyspark解释器和％python解释器之间传递数据集? [英] How can I pass datasets between %pyspark interpreter and %python interpreters in Zeppelin?

问题描述

我正在编写代码，在其中使用内部库和％pyspark解释器来获取数据集. 但是我无法将数据集传递给％python解释器.我尝试使用字符串变量，它可以正常工作，但是对于数据集，我正在使用以下代码将数据集放入齐柏林飞艇上下文中-z.put("input_data",input_data)

I'm writing a code where I'm fetching a dataset using an internal library and %pyspark interpreter. However I am unable to pass the dataset to %python interpreter. I tried using string variables and it is working fine, but with dataset I'm using the following code to put dataset in a zeppelin context- z.put("input_data",input_data)

并引发以下错误:

AttributeError: 'DataFrame' object has no attribute '_get_object_id'.

您能告诉我我该怎么做吗?预先感谢.

Can you please tell me how can I do this? Thanks in advance.

推荐答案

您可以通过将结果打印到％table来将结果放入ResourcePool.

You can put the result in ResourcePool via print it to %table.

%python
print('%table a\tb\n408+\t+408\n0001\t++99\n40817810300001453030\t0000040817810300001453030')

然后以这种方式获得.

%spark.pyspark
ic = z.getInterpreterContext()
pool = ic.getResourcePool()
paragraphId = "20180828-093109_1491500809"
t = pool.get(ic.getNoteId(), paragraphId, "zeppelin.paragraph.result.table").get().toString()

print(t)

这种方式最多可以传输50-100兆字节的原始数据.

This way allows to transfer up to 50-100 megabytes of raw data.

无论如何，我建议遵循@zjffdu仅使用这些解释器之一.

Anyway I recommend to follow @zjffdu to use only one of these interpreters.

这篇关于如何在Zeppelin中的％pyspark解释器和％python解释器之间传递数据集?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！