如何检索 PCollection 的内容并将其分配给普通变量? [英] How to retrieve the content of a PCollection and assign it to a normal variable?

问题描述

我将 Apache-Beam 与 Python SDK 结合使用.

I am using Apache-Beam with the Python SDK.

目前，我的管道读取多个文件，解析它们并从其数据生成 Pandas 数据帧.然后，它将它们组合成一个单个数据帧.

Currently, my pipeline reads multiple files, parse them and generate pandas dataframes from its data. Then, it groups them into a single dataframe.

我现在想要的是检索这个单一的胖数据帧，并将其分配给一个普通的 Python 变量.

What I want now is to retrieve this single fat dataframe, assigning it to a normal Python variable.

可以吗?

推荐答案

PCollection 只是执行图中的一个逻辑节点，其内容不一定实际存储在任何地方，因此无法直接实现.

PCollection is simply a logical node in the execution graph and its contents are not necessarily actually stored anywhere, so this is not possible directly.

但是，您可以要求您的管道将 PCollection 写入文件(例如将元素转换为字符串并使用 WriteToText 和 num_shards=1>)，运行管道并等待它完成，然后从主程序中读取该文件.

However, you can ask your pipeline to write the PCollection to a file (e.g. convert elements to strings and use WriteToText with num_shards=1), run the pipeline and wait for it to finish, and then read that file from your main program.

这篇关于如何检索 PCollection 的内容并将其分配给普通变量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！