文件流 - 性能越来越慢 - 为什么？ [英] File Stream - Performance is getting slower and slower - why?
问题描述
您好。我正在尝试将334,386行的文件拆分为
每个
50,000的单独文件。
这是我正在运行的代码：
Dim intFragmentRawIndex作为整数
Dim swRawDataFile作为StreamWriter
Dim intNewRawIndex作为整数
Dim strNewRawDataFileName As String
Dim intFragmentCallCount As Integer = 0
Dim strHeaderLine As String
Dim blnFileClosed As Boolean = False
strHeaderLine = colRawDataFile（1）
CreateRawDataFragment（intParentRawIndex，intNewRawIndex，
strNewRawDataFileName）
将myFileStream调暗为新
System.IO.FileStream（strTempDirectoryPath& strNewRawDataFileName，_
FileMode.OpenOrCreate，FileAccess.Write，FileShare.None）
>
swRawDataFile = New StreamWriter（myFileStream）
swRawDataFile.WriteLine（strHeaderLine）
对于i as Integer = 2 to colRawD ataFile.Count
如果intFragmentCallCount = 50000那么
''清除流编写器缓冲区
swRawDataFile.Flush（ ）
''关闭文件
swRawDataFile.Close（）
''根据原始数据设置呼叫计数文件
SetFragmentCallCount（intNewRawIndex，intFragmentCallCount）
''重置通话数
intFragmentCallCount = 0
''如果不在原始数据文件的最后一行......
如果我<> colRawDataFile.Count然后
CreateRawDataFragment（intParentRawIndex，
intNewRawIndex，strNewRawDataFileName）
myFileStream = New
System.IO.FileStream（strTempDirectoryPath& strNewRawDataFileName，_
FileMode.OpenOrCreate，FileAccess.Write，FileShare.None）
swRawDataFile =新的StreamWriter（myFileStream）
swRawDataFile.WriteLine（strHeaderLine）
否则
blnFileClosed = True
结束如果
结束如果
swRawDataFile.WriteLine（ colRawDataFile（i））
intFragmentCallCount + = 1
下一页
如果不是blnFileClosed那么
''关闭最后一个片段
swRawDataFile.Close（）
''针对最后片段设置呼叫计数
SetFragmentCallCount（intNewRawIndex，intFragmentCallCount）
结束如果
第一个文件crea测试3分钟。
第二个文件在11分钟内创建。
第三个文件在18分钟内创建。
我还在等待第四个要创建的文件。
我正在为每个文件写入相同数量的记录，那么为什么写入文件需要时间
相同的大小每次需要更长的时间吗？
我认为调用流的flush方法会保持
的性能，但这似乎不是这样的！我做错了什么？
-
欢迎来到mooon！
Hello. I am trying to split a file with 334,386 lines into seperate files of
50,000 each.
This is the code i am running:
Dim intFragmentRawIndex As Integer
Dim swRawDataFile As StreamWriter
Dim intNewRawIndex As Integer
Dim strNewRawDataFileName As String
Dim intFragmentCallCount As Integer = 0
Dim strHeaderLine As String
Dim blnFileClosed As Boolean = False
strHeaderLine = colRawDataFile(1)
CreateRawDataFragment(intParentRawIndex, intNewRawIndex,
strNewRawDataFileName)
Dim myFileStream As New
System.IO.FileStream(strTempDirectoryPath & strNewRawDataFileName, _
FileMode.OpenOrCreate, FileAccess.Write, FileShare.None)
swRawDataFile = New StreamWriter(myFileStream)
swRawDataFile.WriteLine(strHeaderLine)
For i As Integer = 2 To colRawDataFile.Count
If intFragmentCallCount = 50000 Then
''Clear Stream Writer Buffer
swRawDataFile.Flush()
''Close file
swRawDataFile.Close()
''Set Call Count against raw data file
SetFragmentCallCount(intNewRawIndex, intFragmentCallCount)
''Reset call count
intFragmentCallCount = 0
''If not on final line of raw data file....
If i <> colRawDataFile.Count Then
CreateRawDataFragment(intParentRawIndex,
intNewRawIndex, strNewRawDataFileName)
myFileStream = New
System.IO.FileStream(strTempDirectoryPath & strNewRawDataFileName, _
FileMode.OpenOrCreate, FileAccess.Write, FileShare.None)
swRawDataFile = New StreamWriter(myFileStream)
swRawDataFile.WriteLine(strHeaderLine)
Else
blnFileClosed = True
End If
End If
swRawDataFile.WriteLine(colRawDataFile(i))
intFragmentCallCount += 1
Next
If Not blnFileClosed Then
''Close last fragment
swRawDataFile.Close()
''Set call count against last fragment
SetFragmentCallCount(intNewRawIndex, intFragmentCallCount)
End If
The first file creates in 3 mins.
The second file creates in 11 minutes.
The third file creates in 18 minutes.
I am still waiting for the forth file to create.
I am writing the same number of records to each file, so why would the time
it takes to write the file of the same size take longer each time?
I thought that calling the flush method of the stream would maintain
performance but this does not seem to be the case! What am i doing wrong?
--
welcome to the mooon !
推荐答案
这些例程是什么：
- SetFragmentCallCount
- CreateRawDataFragment
这个变量是什么：
colRawDataFile ...
??
What are these routines :
- SetFragmentCallCount
- CreateRawDataFragment
What is this variable :
colRawDataFile ...
??
>> SetFragmentCallCount
>>SetFragmentCallCount
CreateRawDataFragment
不要担心这些 - 基本数据库操作
colRawDataFile
CreateRawDataFragment Don''t worry about these - basic database operations
colRawDataFile
这是此例程中的关键点 - 它基本上是整个文件，
加载到一个集合中 - 因此，这个集合在
中有334,386个条目。
-
欢迎来到mooon！
" olrt"写道：
这些例程是什么：
- SetFragmentCallCount
- CreateRawDataFragment
这个变量是什么：
colRawDataFile。 ..
??
This is the key point in this routine - it''s basically the entire file,
loaded into a collection - therefore, this collection has 334,386 entries in
it.
--
welcome to the mooon !
"olrt" wrote:
What are these routines :
- SetFragmentCallCount
- CreateRawDataFragment
What is this variable :
colRawDataFile ...
??
好吧我以为我会尝试不同的方法，那么什么我现在正在尝试的是
从集合中追加50,000行到一个stringbuilder，然后
将整个stringbuilder写入文件。
但是，看看这个日志：
21/04/2006 14:09:06：构建字符串开始
21/04 / 2006 14:09:14：向stringbuilder追加10,000行
21/04/2006 14:09:39：向stringbuilder追加10,000行
21/04 / 2006 14:10:20：向stringbuilder追加10,000行
21/04/2006 14:11:20：向stringbuilder追加10,000行
21/04 / 2006 14:12:36：向stringbuilder追加10,000行
21/04/2006 14:12:36：附加50,000行t o来自stringbuilder的文件
完成
21/04/2006 14:12:36：构建字符串开始
21/04/2006 14： 14:05：向stringbuilder追加10,000行
21/04/2006 14:16:00：向stringbuilder追加10,000行
21/04/2006 14： 18:36：向stringbuilder追加10,000行
21/04/2006 14:21:18：向stringbuilder追加10,000行
21/04/2006 14： 23:58：向stringbuilder添加10,000行
21/04/2006 14:23:59：从stringbuilder追加50,000行文件
完成
21/04/2006 14:23:59：构建字符串开始
我清除使用此代码附加到文件之间的字符串构建器：
sbFileContent =新的StringBuilder
然而，显然还是有一个很大的减速，为什么会这样？
-
欢迎来到mooon！
" m00nm0nkey"写道：
Ok well i thought i''d try a different approach, so what I''m now trying is
appending 50,000 lines from the collection to a stringbuilder, and then
writing that entire stringbuilder to a file.
However, look at this log:
21/04/2006 14:09:06: Building String Start
21/04/2006 14:09:14: appended 10,000 lines to the stringbuilder
21/04/2006 14:09:39: appended 10,000 lines to the stringbuilder
21/04/2006 14:10:20: appended 10,000 lines to the stringbuilder
21/04/2006 14:11:20: appended 10,000 lines to the stringbuilder
21/04/2006 14:12:36: appended 10,000 lines to the stringbuilder
21/04/2006 14:12:36: append of 50,000 lines to file from stringbuilder
complete
21/04/2006 14:12:36: Building String Start
21/04/2006 14:14:05: appended 10,000 lines to the stringbuilder
21/04/2006 14:16:00: appended 10,000 lines to the stringbuilder
21/04/2006 14:18:36: appended 10,000 lines to the stringbuilder
21/04/2006 14:21:18: appended 10,000 lines to the stringbuilder
21/04/2006 14:23:58: appended 10,000 lines to the stringbuilder
21/04/2006 14:23:59: append of 50,000 lines to file from stringbuilder
complete
21/04/2006 14:23:59: Building String Start
I clear the stringbuilder between appending to the file using this code:
sbFileContent = New StringBuilder
However, there''s still obviously a big slow down, why is this?
--
welcome to the mooon !
"m00nm0nkey" wrote:
你好。我试图将334,386行的文件拆分成每个单独的50,000个文件。
这是我正在运行的代码：
Dim intFragmentRawIndex As Integer
Dim swRawDataFile As StreamWriter
Dim intNewRawIndex As Integer
Dim strNewRawDataFileName As String
Dim intFragmentCallCount As Integer = 0
Dim strHeaderLine As String
Dim blnFileClosed As Boolean =错误
strHeaderLine = colRawDataFile（1）
CreateRawDataFragment（intParentRawIndex，intNewRawIndex，
strNewRawDataFileName）
Dim myFileStream As New
System.IO.FileStream（strTempDirectoryPath& strNewRawDataFileName，_
FileMode.OpenOrCreate，FileAccess.Write，FileShare.None）
swRawDataFile = New StreamWriter（myFileStream）
swRawDataFile.WriteLine（strHeaderLine ）
对于我作为整数= 2到colRawDataFile.Count
如果intFragmentCallCount = 50000那么
''清除流编写器缓冲区
swRawDataFile.Flush（）
''关闭文件
swRawDataFile.Close（）
''针对原始数据文件设置调用计数
SetFragmentCallCount（intNewRawIndex ，intFragmentCallCount）
''重置通话数
intFragmentCallCount = 0
''如果不是原始数据文件的最后一行......
如果我<> colRawDataFile.Count然后
CreateRawDataFragment（intParentRawIndex，
intNewRawIndex，strNewRawDataFileName）
myFileStream = New
System.IO.FileStream（strTempDirectoryPath& strNewRawDataFileName， _文件管理器。文件管理器文件。
其他
blnFileClosed =真如果
结束如果
swRawDataFile。 WriteLine（colRawDataFile（i））
intFragmentCallCount + = 1
如果没有blnFileClosed那么
''关闭最后一个片段
swRawDataFile.Close（）
''设置最后一个片段的呼叫计数
SetFragmentCallCount（intNewRawIndex，intFragmentCallCount）
结束如果
第一个文件在3分钟内创建。
第二个文件文件在11分钟内创建。
第三个文件在18分钟内创建。
我还在等待第四个文件创建。
我正在写相同数量的记录到每个文件，那么为什么每次写相同大小的文件需要花费更长的时间？
我认为调用流的flush方法会保持表现，但似乎并非如此！我做错了什么？
-
欢迎来到mooon！
Hello. I am trying to split a file with 334,386 lines into seperate files of
50,000 each.
This is the code i am running:
Dim intFragmentRawIndex As Integer
Dim swRawDataFile As StreamWriter
Dim intNewRawIndex As Integer
Dim strNewRawDataFileName As String
Dim intFragmentCallCount As Integer = 0
Dim strHeaderLine As String
Dim blnFileClosed As Boolean = False
strHeaderLine = colRawDataFile(1)
CreateRawDataFragment(intParentRawIndex, intNewRawIndex,
strNewRawDataFileName)
Dim myFileStream As New
System.IO.FileStream(strTempDirectoryPath & strNewRawDataFileName, _
FileMode.OpenOrCreate, FileAccess.Write, FileShare.None)
swRawDataFile = New StreamWriter(myFileStream)
swRawDataFile.WriteLine(strHeaderLine)
For i As Integer = 2 To colRawDataFile.Count
If intFragmentCallCount = 50000 Then
''Clear Stream Writer Buffer
swRawDataFile.Flush()
''Close file
swRawDataFile.Close()
''Set Call Count against raw data file
SetFragmentCallCount(intNewRawIndex, intFragmentCallCount)
''Reset call count
intFragmentCallCount = 0
''If not on final line of raw data file....
If i <> colRawDataFile.Count Then
CreateRawDataFragment(intParentRawIndex,
intNewRawIndex, strNewRawDataFileName)
myFileStream = New
System.IO.FileStream(strTempDirectoryPath & strNewRawDataFileName, _
FileMode.OpenOrCreate, FileAccess.Write, FileShare.None)
swRawDataFile = New StreamWriter(myFileStream)
swRawDataFile.WriteLine(strHeaderLine)
Else
blnFileClosed = True
End If
End If
swRawDataFile.WriteLine(colRawDataFile(i))
intFragmentCallCount += 1
Next
If Not blnFileClosed Then
''Close last fragment
swRawDataFile.Close()
''Set call count against last fragment
SetFragmentCallCount(intNewRawIndex, intFragmentCallCount)
End If
The first file creates in 3 mins.
The second file creates in 11 minutes.
The third file creates in 18 minutes.
I am still waiting for the forth file to create.
I am writing the same number of records to each file, so why would the time
it takes to write the file of the same size take longer each time?
I thought that calling the flush method of the stream would maintain
performance but this does not seem to be the case! What am i doing wrong?
--
welcome to the mooon !
这篇关于文件流 - 性能越来越慢 - 为什么？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
