tqdm 可以与数据库读取一起使用吗? [英] Can tqdm be used with Database Reads?

问题描述

当从 SQL 数据库读取大关系到 Pandas 数据帧时，最好有一个进度条，因为元组的数量是静态已知的，并且可以估计 I/O 率.看起来 tqdm 模块有一个函数 tqdm_pandas 它将报告列上映射函数的进度，但默认情况下调用它不会报告 I/像这样哦.是否可以使用 tqdm 在调用 pd.read_sql 时制作进度条?

While reading large relations from a SQL database to a pandas dataframe, it would be nice to have a progress bar, because the number of tuples is known statically and the I/O rate could be estimated. It looks like the tqdm module has a function tqdm_pandas which will report progress on mapping functions over columns, but by default calling it does not have the effect of reporting progress on I/O like this. Is it possible to use tqdm to make a progress bar on a call to pd.read_sql?

推荐答案

答案可能具有误导性 - chunksize 对操作的数据库端没有影响.请参阅下面的评论.

Answer may be misleading - chunksize has no effect on database side of the operation. See comments below.

你可以使用 chunksize 参数来做这样的事情:

You could use the chunksize parameter to do something like this:

chunks = pd.read_sql('SELECT * FROM table', con=conn, chunksize=100)

df = pd.DataFrame()
for chunk in tqdm(chunks):
    df = pd.concat([df, chunk])

我认为这也会使用更少的内存.

I think this would use less memory as well.

这篇关于tqdm 可以与数据库读取一起使用吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！