加载大型CSV文件时的性能问题（Objective-C） [英] Performance Issues when loading big CSV file (Objective-C)

问题描述

我有一个CSV文件，其中包含超过80,000行和100列。我试图处理加载/访问CSV数据以尽可能最高性能的方式。现在我的CSVParser将数据加载到NSArray，但是它非常慢/缓慢;这是一个问题，因为我希望在移动设备上处理这个解析/加载：iPhone。

I have a CSV file that contains over 80,000 rows and 100 columns. I'm trying to handle loading /accessing the CSV data in the most performance-efficient way possible. Right now my CSVParser loads the data into an NSArray, but it's extremely slow/sluggish; this is a problem as I hope to handle this parsing/loading on a mobile device: the iPhone.

任何建议一个替代的方法将不胜感激。谢谢

Any suggestions for an alternate method would be much appreciated. Thank you

更新：

为了将来参考/讨论，我现在有以下尝试：

For future reference/discussion, I now have the following attempt:

// Mark time the parser starts 
NSTimeInterval start = [NSDate timeIntervalSinceReferenceDate];
// Parse the CSV file
[parser parse];
NSTimeInterval end = [NSDate timeIntervalSinceReferenceDate];

// Print how long the parsing took 
NSLog(@"raw difference: %f", (end-start));

// Copy the allLines array from the parsing delegate 
NSArray *allOfTheRows = [NSArray arrayWithArray:d.allLines]; 
NSLog( @"There are %i lines in the csv file", [allOfTheRows count]); 

NSFileManager *f = [[NSFileManager alloc] init]; 
NSString *filePath = @"/Users/..../rawData"; // This is of course not a literal location...

// Archive the array as NSData 
NSData *someData = [NSKeyedArchiver archivedDataWithRootObject:allOfTheRows];

// Write the data to a file
[f createFileAtPath:filePath contents:someData attributes:nil]; 

/*
 If I were to load the data from the iPhone, i'd copy the newly created someData file above to my application's mainBundle, and then unarchive the NSData to an array on the iPhone
*/
// Read the data back as an array 
NSData *readData = [NSData dataWithContentsOfFile:filePath]; 

NSArray *bigCollectionReadBack = [NSKeyedUnarchiver unarchiveObjectWithData:readData]; 

推荐答案

我最后在Mac上做解析，并写出一个包含结构数据数组的二进制文件。过去需要120秒来解析/加载iPhone 4上的CSV文件，但二进制文件在10毫秒内加载。

I had similar problems with CSV parsing on the iPhone. I ended up doing the parsing on the Mac and writing out a binary file containing the array of struct data. It used to take 120 seconds to parse/load the CSV file on the iPhone 4 but the binary file loads in under 10 milliseconds.

EDIT - 要进一步阐述，在Mac上我读取CSV文件，将数据组织成几个结构体数组，然后写入使用将数据导出为二进制文件fwrite 。在iOS上，我使用读取二进制文件fread （一个读取头获取大小信息，第二个读取数据）转换为正确大小的结构数组。一个较大的文件是2.2MB，使用fread从闪存读入RAM需要66毫秒。

EDIT - To elaborate a bit more, on the Mac I read the CSV file, organize the data into several arrays of structs then write out the data to a binary file using fwrite. On iOS I read the binary file using fread (one read for the header to get size info, and a second read for the data) into an array of structs of the right size. One of the larger files is 2.2MB and it takes 66 msec to read from the flash into RAM using fread.

2011-11-15 17:32:35.304 -[BinFile initWithFile:] 001953f0 file Metro
2011-11-15 17:32:35.370 -[BinFile initWithFile:] read 2217385 bytes (Metro)

这篇关于加载大型CSV文件时的性能问题（Objective-C）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！