将数据从Postgres传输到Python [英] Streaming data from Postgres into Python

问题描述

我正在寻找有关将数据从Postgres表逐步流式传输到Python的有效方法的建议。我正在实施在线学习算法，我想从数据库表中读取成批的训练示例到内存中进行处理。有什么想法可以最大化吞吐量？感谢您的建议。

I'm looking for advice on efficient ways to stream data incrementally from a Postgres table into Python. I'm in the process of implementing an online learning algorithm and I want to read batches of training examples from the database table into memory to be processed. Any thoughts on good ways to maximize throughput? Thanks for your suggestions.

推荐答案

如果使用的是psycopg2，则需要使用命名的游标，否则它将尝试读取整个查询数据立即存入内存。

If you are using psycopg2, then you will want to use a named cursor, otherwise it will try to read the entire query data into memory at once.

cursor = conn.cursor("some_unique_name")
cursor.execute("SELECT aid FROM pgbench_accounts")
for record in cursor:
    something(record)

这将以2000个批次（默认值 itersize ）从服务器中提取记录，然后一次将它们打包到循环中。

This will fetch the records from the server in batches of 2000 (default value of itersize) and then parcel them out to the loop one at a time.

这篇关于将数据从Postgres传输到Python的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！