快速获取文件夹的总大小 [英] very quickly getting total size of folder
问题描述
我想使用python快速找到任何文件夹的总大小.
import os
from os.path import join, getsize, isfile, isdir, splitext
def GetFolderSize(path):
TotalSize = 0
for item in os.walk(path):
for file in item[2]:
try:
TotalSize = TotalSize + getsize(join(item[0], file))
except:
print("error with file: " + join(item[0], file))
return TotalSize
print(float(GetFolderSize("C:\\")) /1024 /1024 /1024)
这是我编写的用于获取文件夹总大小的简单脚本,耗时约60秒(+ -5秒).通过使用多处理,我在四核计算机上将时间缩短到23秒.
使用Windows文件资源管理器仅需约3秒钟(右键单击->属性即可自行查看).那么,有没有一种更快的方法来找到文件夹的总大小,使其接近Windows可以达到的速度?
Windows 7,python 2.6(虽然进行了搜索,但大多数时候人们使用的方法与我自己的方法非常相似) 预先感谢.
您处于不利地位.
Windows资源管理器几乎可以肯定使用 FindFirstFile
/ FindNextFile
都遍历目录结构和一次性收集大小信息(通过lpFindFileData
),从而基本上每个文件进行一次系统调用.
不幸的是,在这种情况下,Python不是您的朋友.因此,
-
os.walk
首先调用os.listdir
(内部调用FindFirstFile
/FindNextFile
)- 从现在开始进行的任何其他系统调用只会使您比Windows Explorer慢
-
os.walk
然后为os.listdir
返回的每个文件调用isdir
(内部调用GetFileAttributesEx
)
每个文件的系统调用比Windows资源管理器多3倍,再加上内存分配和操作开销.
您可以使用Anurag的解决方案,也可以尝试直接递归地调用FindFirstFile
/FindNextFile
(这应该与 cygwin
或其他 win32端口 du -s some_directory
.)>
请参考 os.py
os.walk
的实现, posixmodule.c
用于实现listdir
和win32_stat
(由isdir
和getsize
调用.)
请注意,Python的 os.walk
在所有平台(Windows和* nices)(包括Python3.1以及更高版本)上都不理想.在Windows和* nices上,os.walk
都可以在不调用isdir
的情况下一次遍历,因为FindFirst
/FindNext
(Windows)和opendir
/readdir
(* nix)都已经通过返回了文件类型. lpFindFileData->dwFileAttributes
(Windows)和dirent::d_type
(* nix).
在大多数现代配置(例如Win7和NTFS,甚至某些SMB实现)上,可能会违反常理,GetFileAttributesEx
是单个文件的FindFirstFile
的两倍(可能比迭代慢)在具有FindNextFile
的目录上.)
更新:Python 3.5包括新的 PEP 471 os.scandir()
函数,该函数通过返回文件属性以及文件名.此新功能用于加快内置的os.walk()
(在Windows和Linux上).您可以在PyPI上使用 scandir模块来获得针对旧版Python版本(包括2.x)的这种行为.
I want to quickly find the total size of any folder using python.
import os
from os.path import join, getsize, isfile, isdir, splitext
def GetFolderSize(path):
TotalSize = 0
for item in os.walk(path):
for file in item[2]:
try:
TotalSize = TotalSize + getsize(join(item[0], file))
except:
print("error with file: " + join(item[0], file))
return TotalSize
print(float(GetFolderSize("C:\\")) /1024 /1024 /1024)
That's the simple script I wrote to get the total size of the folder, it took around 60 seconds (+-5 seconds). By using multiprocessing I got it down to 23 seconds on a quad core machine.
Using the Windows file explorer it takes only ~3 seconds (Right click-> properties to see for yourself). So is there a faster way of finding the total size of a folder close to the speed that windows can do it?
Windows 7, python 2.6 (Did searches but most of the time people used a very similar method to my own) Thanks in advance.
You are at a disadvantage.
Windows Explorer almost certainly uses FindFirstFile
/FindNextFile
to both traverse the directory structure and collect size information (through lpFindFileData
) in one pass, making what is essentially a single system call per file.
Python is unfortunately not your friend in this case. Thus,
os.walk
first callsos.listdir
(which internally callsFindFirstFile
/FindNextFile
)- any additional system calls made from this point onward can only make you slower than Windows Explorer
os.walk
then callsisdir
for each file returned byos.listdir
(which internally callsGetFileAttributesEx
-- or, prior to Win2k, aGetFileAttributes
+FindFirstFile
combo) to redetermine whether to recurse or notos.walk
andos.listdir
will perform additional memory allocation, string and array operations etc. to fill out their return value- you then call
getsize
for each file returned byos.walk
(which again callsGetFileAttributesEx
)
That is 3x more system calls per file than Windows Explorer, plus memory allocation and manipulation overhead.
You can either use Anurag's solution, or try to call FindFirstFile
/FindNextFile
directly and recursively (which should be comparable to the performance of a cygwin
or other win32 port du -s some_directory
.)
Refer to os.py
for the implementation of os.walk
, posixmodule.c
for the implementation of listdir
and win32_stat
(invoked by both isdir
and getsize
.)
Note that Python's os.walk
is suboptimal on all platforms (Windows and *nices), up to and including Python3.1. On both Windows and *nices os.walk
could achieve traversal in a single pass without calling isdir
since both FindFirst
/FindNext
(Windows) and opendir
/readdir
(*nix) already return file type via lpFindFileData->dwFileAttributes
(Windows) and dirent::d_type
(*nix).
Perhaps counterintuitively, on most modern configurations (e.g. Win7 and NTFS, and even some SMB implementations) GetFileAttributesEx
is twice as slow as FindFirstFile
of a single file (possibly even slower than iterating over a directory with FindNextFile
.)
Update: Python 3.5 includes the new PEP 471 os.scandir()
function that solves this problem by returning file attributes along with the filename. This new function is used to speed up the built-in os.walk()
(on both Windows and Linux). You can use the scandir module on PyPI to get this behavior for older Python versions, including 2.x.
这篇关于快速获取文件夹的总大小的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!