在linux上创建/修改稀疏文件 [英] creating/modifying sparse files on linux

问题描述

python中的稀疏文件处理是否有任何特殊支持？我的

初始搜索没有带来太多（不是彻底的搜索）。我写了以下代码：/ b
$ b options.size = 6442450944

options.ranges = [" 4096， 1024"，" 30000,314572800"]

fd = open（" testfile"，" w"）

fd.seek（options.size-1）

fd.write（" a"）

for appenge in options.ranges：

off = int（drange.split（"） ，"）[0]）

len = int（drange.split（"，"）[1]）

print" off ="，关闭， len ="，len

fd.seek（off）

for x in range（len）：

fd.write（" a）


fd.close（）


这段代码需要很长时间，实际上我不得不将其作为

linux系统开始进行大量交换。我在做什么

错了吗？有没有更好的方法来创建/修改稀疏文件？


谢谢，

Raghu。

Hi,

Is there any special support for sparse file handling in python? My
initial search didn''t bring up much (not a thorough search). I wrote
the following pice of code:

options.size = 6442450944
options.ranges = ["4096,1024","30000,314572800"]
fd = open("testfile", "w")
fd.seek(options.size-1)
fd.write("a")
for drange in options.ranges:
off = int(drange.split(",")[0])
len = int(drange.split(",")[1])
print "off =", off, " len =", len
fd.seek(off)
for x in range(len):
fd.write("a")

fd.close()

This piece of code takes very long time and in fact I had to kill it as
the linux system started doing lot of swapping. Am I doing something
wrong here? Is there a better way to create/modify sparse files?

Thanks,
Raghu.

推荐答案

[dr ******* @ gmail.com写道]

[dr*******@gmail.com wrote]

你好，

有吗在python中对稀疏文件处理的任何特殊支持？我的初始搜索没有带来太多（不是彻底的搜索）。我写了以下代码：

options.size = 6442450944
options.ranges = [" 4096,1024"，" 30000,314572800"]
fd = open（" testfile"，" w"）
fd.seek（options.size-1）
fd.write（" a"）
for drange in options.ranges：
off = int（drange.split（"，"）[0]）
len = int（drange.split（"，"）[1]）
print" off ="，off，" len ="，len
fd.seek（off）
for x in range（len）：
fd.write（" a"）

fd .close（）

这段代码需要很长时间，事实上我不得不杀掉它，因为linux系统开始进行大量的交换。我在这做错了吗？有没有更好的方法来创建/修改稀疏文件？

Hi,

Is there any special support for sparse file handling in python? My
initial search didn''t bring up much (not a thorough search). I wrote
the following pice of code:

options.size = 6442450944
options.ranges = ["4096,1024","30000,314572800"]
fd = open("testfile", "w")
fd.seek(options.size-1)
fd.write("a")
for drange in options.ranges:
off = int(drange.split(",")[0])
len = int(drange.split(",")[1])
print "off =", off, " len =", len
fd.seek(off)
for x in range(len):
fd.write("a")

fd.close()

This piece of code takes very long time and in fact I had to kill it as
the linux system started doing lot of swapping. Am I doing something
wrong here? Is there a better way to create/modify sparse files?

Python测试套件中的test_largefile.py可以做这种事情和

我不需要花很长时间在Linux上运行（SuSE 9.0盒子）。


Trent


-

Trent Mick
Tr****@ActiveState.com

在< 11 ********************** @ f14g2000cwb.googlegroups .com> ;,
dr ******* @ gmail.com 写道：

In <11**********************@f14g2000cwb.googlegroups .com>,
dr*******@gmail.com wrote:

options.size = 6442450944
options.ranges = [" 4096,1024"，" 30000,314572800"]
fd = open（" testfile"，" w"）
fd.seek（options） .size-1）
fd.write（" a"）
for options in options.ranges：
off = int（drange.split（"，"）[0] ）
len = int（drange.split（"，"）[1]）
print" off ="，off，" len ="，len
fd.seek（off）
for x in range（len）：
fd.write（" a"）

fd .close（）

这段代码需要很长时间，事实上我不得不杀掉它，因为linux系统开始进行大量的交换。我在这做错了吗？有没有更好的方法来创建/修改稀疏文件？

options.size = 6442450944
options.ranges = ["4096,1024","30000,314572800"]
fd = open("testfile", "w")
fd.seek(options.size-1)
fd.write("a")
for drange in options.ranges:
off = int(drange.split(",")[0])
len = int(drange.split(",")[1])
print "off =", off, " len =", len
fd.seek(off)
for x in range(len):
fd.write("a")

fd.close()

This piece of code takes very long time and in fact I had to kill it as
the linux system started doing lot of swapping. Am I doing something
wrong here? Is there a better way to create/modify sparse files?

`range（len）`在内存中创建一个大小为`len` *的列表*所以你正在尝试

建立一个包含314,572,800个数字的列表。这似乎占用了你所有的RAM

并导致交换。


您可以使用`xrange（len）`而不是使用恒定的内存量。

但是要准备好等一段时间，因为现在你正在将314,572,800

个字符*逐个*写入文件中。在每一步写出更大的

字符串会更快。


Ciao，

Marc''BlackJack''Rintsch

`range(len)` creates a list of size `len` *in memory* so you are trying to
build a list with 314,572,800 numbers. That seems to eat up all your RAM
and causes the swapping.

You can use `xrange(len)` instead which uses a constant amount of memory.
But be prepared to wait some time because now you are writing 314,572,800
characters *one by one* into the file. It would be faster to write larger
strings in each step.

Ciao,
Marc ''BlackJack'' Rintsch

< dr ******* @ gmail.com>在消息中写道

news：11 ********************** @ f14g2000cwb.googlegr oups.com ...

<dr*******@gmail.com> wrote in message
news:11**********************@f14g2000cwb.googlegr oups.com...

对python中的稀疏文件处理有什么特别的支持吗？


由于我几年没有听说过这种情况，我怀疑没有。 CPython，正常编译的
，使用标准的C stdio lib。如果你的系统+ C有一个

sparseIO lib，你可能需要专门编译才能使用它。

options.size = 6442450944
options.ranges = [" 4096,1024"，"&n; 30000,314572800"]


options.ranges = [（4096,1024），（30000,314572800）]＃使得下面更好

fd = open（" testfile"，" w"）
fd.seek（options.size-1）
fd.write（" a"）
for appenge in options.ranges：
off = int（drange.split（"，"）[0]）
len = int（drange.split（"，"）[1 ]）


off，len = map（int，drange.split（"，"））＃或

off，len = [int（ s）对于s in drange.split（"，"）]＃或对于元组的建议

以上

off，len = drange

print" off ="，off，" len ="，len
fd.seek（off）
for x in range（len）：


如果我读了上面的右边，第二个len是300,000,000+使得范围列表需要几GB的空间需要
。我怀疑这是你b $ b开始捶打的地方;-)。相反：

x in xrange（len）中的
：＃这是xrange的用途;-)

fd.write（" a"）


如果没有缩进，这是语法错误，所以如果您的代码完全运行，那么这个

不能是精确的副本。即使使用xrange修复，300,000,000次写入也将是b $ b b慢。我希望一个真正的应用程序应该创建或累积大于单个字符的
块。

fd.close（）

这段代码需要很长时间，事实上我不得不杀死它，因为linux系统开始进行大量的交换。我在这做错了吗？


见上文

有没有更好的方法来创建/修改稀疏文件？

Is there any special support for sparse file handling in python?
Since I have not heard of such in several years, I suspect not. CPython,
normally compiled, uses the standard C stdio lib. If your system+C has a
sparseIO lib, you would probably have to compile specially to use it.
options.size = 6442450944
options.ranges = ["4096,1024","30000,314572800"]
options.ranges = [(4096,1024),(30000,314572800)] # makes below nicer
fd = open("testfile", "w")
fd.seek(options.size-1)
fd.write("a")
for drange in options.ranges:
off = int(drange.split(",")[0])
len = int(drange.split(",")[1])
off,len = map(int, drange.split(",")) # or
off,len = [int(s) for s in drange.split(",")] # or for tuples as suggested
above
off,len = drange
print "off =", off, " len =", len
fd.seek(off)
for x in range(len):
If I read the above right, the 2nd len is 300,000,000+ making the space
needed for the range list a few gigabytes. I suspect this is where you
started thrashing ;-). Instead:

for x in xrange(len): # this is what xrange is for ;-)
fd.write("a")
Without indent, this is syntax error, so if your code ran at all, this
cannot be an exact copy. Even with xrange fix, 300,000,000 writes will be
slow. I would expect that an real application should create or accumulate
chunks larger than single chars.
fd.close()

This piece of code takes very long time and in fact I had to kill it as
the linux system started doing lot of swapping. Am I doing something
wrong here?
See above
Is there a better way to create/modify sparse files?

除非您可以访问建筑设施，否则请创建您自己的地图索引。


Terry J. Reedy

Unless you can access builting facilities, create your own mapping index.

Terry J. Reedy

这篇关于在linux上创建/修改稀疏文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！