写一个100万（10万卢比）记录的文本文件 [英] writing a text file of 1 million (10 lakhs) records

问题描述

亲爱的所有



我正在银行业领域工作并撰写一份100万（10万卢比）的文本文件

和写入需要14个小时，

i使用5个foreach循环，数据使用

stringbuilder逐个写入。



任何人都可以快速写出文字文件



谢谢。

Dear All

I am working on a banking domain and writing a text file of 1 million (10 lakhs) records
and it will take near by 14 hours to write ,
i have use five foreach loop with in it ,data write one by one basis by using
stringbuilder .

can any one tell fast way to write a text file

thanks.

For Each dRow As DataRow In dsetSegmentSelect.Tables("ConsumerBranches").Rows
                intRecCnt = 0 'Uncomment For Previous
                strBrCode = dRow.Item("BranchCode")

                If dsetSegmentSelect.Tables.Contains("ConsumerBorrowerSegment") Then
                    For Each dCBSBrRow As DataRow In dsetSegmentSelect.Tables("ConsumerBorrowerSegment").Select("BranchCode='" & strBrCode & "'")
                        strBrCode = dCBSBrRow.Item("BranchCode")
                        intCustomerID = dCBSBrRow.Item("CustomerID")
                        intcustData = 0
                        intRecCnt = intRecCnt + 1
                        If dsetSegmentSelect.Tables.Contains("ConsumerAccountSegment") Then
                            For Each dCBABrRow As DataRow In dsetSegmentSelect.Tables("ConsumerAccountSegment").Select("BranchCode='" & strBrCode & "' AND CustomerID='" & intCustomerID & "'")
                                intAcID = dCBABrRow.Item("Acid")
                                intcustData = intcustData + 1
                                blnJoint = False
                                For Each dBPNBrRow As DataRow In dsetSegmentSelect.Tables("BPNBranches").Select("BranchCode='" & strBrCode & "' AND CustomerID='" & intCustomerID & "'")

                                    dsetSegmentSelect.Tables("BPNBranches").DefaultView.RowFilter = "BranchCode='" & strBrCode & "' AND CustomerID='" & intCustomerID & "'"
                                    intCIBILPGID = dBPNBrRow.Item("CIBILPGID")

                                    If dsetSegmentSelect.Tables("BPNBranches").DefaultView.ToTable().Rows.Count > 1 Then
                                        blnJoint = True
                                    Else
                                        blnJoint = False
                                    End If
                                    If dsetSegmentSelect.Tables.Contains("BorrowerNameSegment") Then
                                        For Each dBNSRow As DataRow In dsetSegmentSelect.Tables("BorrowerNameSegment").Select("BranchCode='" & strBrCode & "' AND CustomerID='" & intCustomerID & "' AND CIBILPGID='" & intCIBILPGID & "'")
                                            If blnDSFromNEXP = True Then
                                                'cnnVision.Execute(" Update CustomerID Set CustomerMark ='Y' Where Branchcode='" & rstCustomer!BranchCode & "' and CustomerId = " & rstCustomer!CustomerID)

                                            End If
                                            Call ConsumerTableToString(dBNSRow, FileFullPathName)
                                        Next
                                    End If

                                    If dsetSegmentSelect.Tables.Contains("BorrowerIDSegment") Then
                                        For Each dBASRow As DataRow In dsetSegmentSelect.Tables("BorrowerIDSegment").Select("BranchCode='" & strBrCode & "' AND CustomerID='" & intCustomerID & "' AND CIBILPGID='" & intCIBILPGID & "'", "IDSegmentTag Asc")
                                            Call ConsumerTableToString(dBASRow, FileFullPathName)
                                        Next
                                    End If
                                    If
NEXT

Private Sub ConsumerTableToString(ByRef DR As DataRow, ByVal FileFullPathName As String, Optional ByVal NoOfRows As Integer = 1)
       Dim strTables As New StringBuilder
       Dim intlen As Integer
       Dim I As Integer = 0
       If DR.Table.TableName.ToString = "HeaderSegment" Then
           intlen = 0
       Else
           intlen = 4
       End If
       Try

           For I = 4 To DR.ItemArray.Length - 1

               If Len(DR.Item(I).ToString) > intlen Then
                   If DR.Table.TableName.ToString = "AccountSegment" Then
                       If blnJoint = True Then
                           If DR.Table.Columns(I).ColumnName.ToString = "OwnerShip" Then
                               strField = "05014"
                           Else
                               strField = DR.Item(I).ToString
                           End If
                       End If
                       If blnGuarantor = True Then
                           If DR.Table.Columns(I).ColumnName.ToString = "OwnerShip" Then
                               strField = "05013"
                           Else
                               strField = DR.Item(I).ToString
                           End If
                       End If
                       If blnJoint = False And blnGuarantor = False Then strField = DR.Item(I).ToString
                   Else
                       strField = DR.Item(I).ToString
                   End If
               Else
                   strField = ""
               End If
               strRowConsumer.Append(strField)

           Next
       Catch ex As Exception
           MessageBox.Show(ex.Message, objCommonPar.strpackagename)
       End Try
   End Sub

推荐答案

您的申请方式你的循环是需要花时间的。 Visual Studio Premium附带了Profiling工具，您还可以从Red Gate下载演示分析工具。这将显示您的代码花费在所有时间的位置。



理想情况下，您希望编写单个选择语句或存储过程，以平面列表的形式输出需要写入文件的数据。让数据库做得很辛苦。



实体关系数据库维护索引，这意味着它将实现你的逻辑，哪些行写出的速度比任何.Net代码要快得多。



然后有一个循环将每行输出到文件。



你还为每个低效的项目创建一个新的字符串构建器。当您写入文件时，最有效的方法是写入文件。使用Stream Writer并忘记构建器。



您希望将输出的流编写器传递给文本文件，以便将所有输出直接写入文件。否则你将花费大量时间在内存中移动字符串。

The way you're applying your loops is what's going to take the time. Visual Studio Premium comes with Profiling tools and you can also download demo profiling tools from Red Gate. This will show you where your code is spending all of its time.

Ideally you want to write a single select statement or stored procedure which outputs, as a flat list, the data which needs to be written to the file. Make the database do the hard work.

Entity relational databases maintain indexes which means it will implement your logic for which rows to write out much faster than any .Net code will.

Then have a single loop to output each of the rows to the file.

You also create a new string builder for each item which in inefficient. When you're writing to a file the most efficient thing to do is to write to the file. Use the Stream Writer and forget about builders.

You want to pass the stream writer around which outputs to the text file so all output is written directly to the file. Otherwise you're going to spend a lot of time just moving strings around in memory.

这篇关于写一个100万（10万卢比）记录的文本文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！