查询非常大的.out(平面文件) [英] Quering very large .out (flat file)

问题描述

有人可以帮我解决以下问题吗?
我必须从具有数百万条记录的平面文件中读取特定字段.平面文件的扩展名是.out.
在性能方面，查询此类文件的最佳方法是哪种?
到目前为止，我们正在考虑LINQ或XSLT解析.
请就上述问题提出您的想法.
谢谢
Prasad

Hi,
Could someone help me on following issue?
I have to read a particular field(s) from a flat file which has millions of records. Extension of flat file is .out.
Which is the best way to query such files on performance wise?
As of now we are thinking LINQ or XSLT parsing.
Please suggest your ideas on above issue.
Thanks
Prasad

推荐答案

根据平面文件的格式，我建议以下其中一项:

1)快速CSV阅读器 [用于平面文件的可移植且高效的通用解析器 [

Depending on the format of the flat file, I would recommend one of the following:

1) A Fast CSV Reader[^] - Handles delimited file formats

2) A Portable and Efficient Generic Parser for Flat Files[^] - Handles delimited and fixed width file formats

Both will allow you to get the data into memory in a quick way. Depending on what you''re doing with it though, you may prefer one of the methods Nijboer has already suggested.

您可以使用LINQ，但首先必须从文件，然后将其解析为列表，然后再使用它.另一个选择是使用SQLServer的批量导入功能，然后以这种方式将数据导入到表中.

http://msdn.microsoft.com/en-us/library/ms187042.aspx [ ^ ]

http://msdn.microsoft.com/en-us/library/ms188365.aspx [ ^ ]

由于记录数量和内存限制，将其读入内存可能是一个问题.当然，这确实取决于行的大小和您实际需要的数据，因为其余部分可以被丢弃.您还可以选择仅将where子句中所需的列保留在内存中，稍后再基于选择返回的记录/行号读取要选择的数据.

祝你好运！

You could use LINQ but you would first have to read the lines from the file and parse them into a list before you could use it. Another option is to use the bulk import function of SQLServer and import the data into a table that way.

http://msdn.microsoft.com/en-us/library/ms187042.aspx[^]

http://msdn.microsoft.com/en-us/library/ms188365.aspx[^]

Reading it into memory could be a problem because of the number of records and memory limitations. This does of course depend on the size of the lines and the data you actually need because the rest can be discarded. You could also choose to only hold in memory the columns you need in your where clause and later on read the data you want to select based on the record/line numbers the select returned.

Good luck!

这篇关于查询非常大的.out(平面文件)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！