如何检索PCollection的内容并将其分配给普通变量? [英] How to retrieve the content of a PCollection and assign it to a normal variable?

问题描述

我正在将Apache-Beam与Python SDK配合使用.

I am using Apache-Beam with the Python SDK.

当前，我的管道读取多个文件，对其进行解析，并根据其数据生成熊猫数据帧. 然后，将它们分组为单个数据框.

Currently, my pipeline reads multiple files, parse them and generate pandas dataframes from its data. Then, it groups them into a single dataframe.

我现在想要的是检索单个胖数据帧，并将其分配给普通的Python变量.

What I want now is to retrieve this single fat dataframe, assigning it to a normal Python variable.

有可能吗?

推荐答案

PCollection只是执行图中的逻辑节点，其内容不一定实际存储在任何地方，因此这不可能直接实现.

PCollection is simply a logical node in the execution graph and its contents are not necessarily actually stored anywhere, so this is not possible directly.

但是，您可以要求管道将PCollection写入文件(例如，将元素转换为字符串并将WriteToText与num_shards=1一起使用)，运行管道并等待其完成，然后阅读主程序中的文件.

However, you can ask your pipeline to write the PCollection to a file (e.g. convert elements to strings and use WriteToText with num_shards=1), run the pipeline and wait for it to finish, and then read that file from your main program.

这篇关于如何检索PCollection的内容并将其分配给普通变量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！