从iPad转换/上传大量数据到Dropbox [英] Converting/uploading large amounts of data from iPad to Dropbox

问题描述

我通过在Instruments上运行应用程序并添加大量数据来完善我的应用程序。仪器测试正常，但压力测试是我遇到的问题。无需过多讨论，我为我的应用增加了核心数据事件的数量，需要使用这些事件来推断数据，制作图形并在 MKMapView 实例。我从小规模开始，增加到56000个事件，它能很好地处理所有泄漏或内存警告（我为能处理所有事件而感到自豪）。

我的应用实现Dropbox API，以允许上传和下载模板和数据以进行同步。从我的应用上传的文件从核心数据转换为 NSDictionary ，然后转换为 NSData 。我为数据创建一个临时文件夹，然后将该文件上传到Dropbox，正常工作正常。如果我尝试上传具有56000个事件的数据文件，则它会崩溃。我已经记录下来，看着数据被转换。它到达了最后一个事件，没有任何问题，但是当它应该开始上载到Dropbox时，应用程序崩溃了，我一生都无法找出原因。我在日志中看到内存警告。通常情况下，它将变为Level = 1，Level = 2，Level = 1，Level = 2，然后崩溃，这使我感到困惑，因为它从未达到Level = 3。

我发现的大部分信息都在我对波顿的编辑中。下面是一些相关代码：

 -（void）uploadSurveys：（NSDictionary *）dict {
 NSArray * templateArray = [dict objectForKey：@ templates]; 
 NSArray * dataArray = [dict objectForKey：@ data]; 
 NSString *文件名; 
 NSLog（@上载称为）; 
 if（[[templateArray count] || [dataArray count]）{
 if（[templateArray count]）{
 // irrelevent代码； 
} 
 if（[dataArray count]）{
 SurveyData *调查； 
 for（int i = 0; i< [dataArray count]; i ++）{
 BOOL matchExists = NO; 
 // // ......代码，以确保Dropbox文件夹中没有文件，并在必要时创建新版本； 
 
 dispatch_async（dispatch_get_global_queue（DISPATCH_QUEUE_PRIORITY_DEFAULT，0），^ {
 NSData * data = [self convertSurvey：survey]; 
 dispatch_async（dispatch_get_main_queue（），^ {
 [self uploaddata：data withfilename：filename]; 
 NSLog（@转换并上传）; 
}）; 
}）; 
} 
} 
} 
  

[self convertSurvey：survey] 只是将我的核心数据对象转换为 NSData 。

 -（void）uploadData：（NSData *）data withFilename：（NSString *）filename {
 NSFileManager * manager = [ NSFileManager defaultManager]; 
 NSString * pathComponent = [NSString stringWithFormat：@ tempData。％@，文件名]; 
 NSString * path = [NSTemporaryDirectory（）stringByAppendingPathComponent：pathComponent]; 
 if（[管理器createFileAtPath：path内容：数据属性：无]）{
 [self.restClient uploadFile：filename toPath：[NSString stringWithFormat：@ /％@，currentSearch] fromPath：path] ; 
 NSLog（@上传数据）; 
} 
} 
  

任何帮助将不胜感激，我非常感谢提前。我只是想弄清楚我是对大文件采用了错误的方法，还是根本不允许这样做。如果我必须分割文件，那很好，但是我想知道在解决问题之前，正在发生的事情会阻止我的应用执行此操作。再次感谢您。

更新：由于此问题现在是发布我的应用程序的唯一障碍，因此我为此问题添加了一笔赏金，以期寻求解决方案或解决方法。这将持续一周，然后在给定的时间后，我很可能会在上传文件时将它们分开，以确保未达到此明显的大小限制。这种方法并不理想，这就是为什么一个更好的解决方案受到人们欢迎的原因，但是如果不能带来更多方便的话，这就是我的备份计划。

编辑：看来 NSTemporaryDirectory 根本不起作用。这是新情况。如您在上面的代码中看到的， NSData * data = [self convertSurvey：survey]; 在辅助线程中被调用（这不是问题）。我一直在记录创建的对象，知道它们已经到达了最后一个，但是从没想过要检查一下是否返回了 NSData 文件。事实并非如此。简而言之，我将所有核心数据对象转换为数组，并将其放入字典中（仅适用于要转换的相关测量/数据）。确实可以，并且创建了字典。然后使用 NSData创建一个 NSData 文件* data = [NSKeyedArchiver archivedDataWithRootObject：d]; 其中 d 是我的字典。之后，我立即调用返回数据； 设置 NSData的值* data = [self convertSurvey：survey]; 。在这种情况下，似乎 NSData 或 NSKeyedArchiver 在这里有问题。根据Apple文档：

使用32位Cocoa，数据大小受理论上2GB的限制（实际上，因为其他对象将使用内存，所以此限制会更小）；使用64位Cocoa，数据大小受理论上约8EB的限制（实际上，该限制不应作为一个因素）。

我已经小幅检查了文件大小，以查看发生故障的位置。我已经成功获取了48.2MB的数据，但没有成功获取51.5MB的数据，这使我相信该问题大约发生在50MB左右，远低于 NSData 的理论极限（除非存在在这方面是iOS和OS X之间的差异）。

希望这些新信息将有助于解决此问题

解决方案

在iOS上，NSData的2 GB限制是完全理论上的，即使iPhone 4仅具有512 MB的RAM，iOS（与Mac OS X不同）也无法交换，因此如果您的物理RAM已满，您崩溃了（或者您的应用程序在那之前终止了。）

仅50 MB的 NSData 对象已经很大，并且不是内存中唯一的对象-假设您将数据从核心数据转换为字典表示形式，然后转换为 NSData ，则可能会消耗至少两倍的内存（可能更多）。系统和其他应用程序也需要RAM，因此您可能已达到极限。

尝试在Instruments中运行应用程序以查看实际消耗的内存。

p>

为减少峰值内存使用量，您有两种选择在很大程度上取决于您的数据模型：

• 为 Jason Foreman 在回答中提出了建议，请尽量避免将整个文件立即存储在内存中。使用 NSFileHandle ，您可以将数据块写入文件中，而无需一次将所有数据存储在内存中。当然，这需要您相应地准备数据，以便可以将其拆分为大块。更高级的方法可能是将数据序列化为XML格式，然后可以将其写出为流。如果您的数据格式非常简单，则类似CSV的方法也可能起作用。




• 不要使用 NSData 以上传到Dropbox。而是将数据写入文件（请参见上文），然后将Dropbox SDK指向该文件。 Dropbox SDK使其非常容易实现（ DBRestClient 具有 uploadFile：toPath：fromPath：方法）。 / p>




• 如果您的数据模型难以采用流传输方法，请尝试将数据分段为更易于管理的部分。然后，您可以使用仅对多个文件进行序列化的旧方法。




• 请注意Core Data的内存使用情况。如果可能的话，尝试使用 refreshObject：mergeChanges：重新故障对象，以破坏数据中的循环引用（请参见《核心数据编程指南》 了解详情）




• 在长时间运行的循环中避免使用自动释放池，或者创建单独的 NSAutoreleasePool 会在循环的每次迭代中耗尽。

I'm finishing up my app by running it through Instruments as well as stressing it with large amounts of data. The Instruments tests go fine, but the stress test is where I'm having issues. Without getting into too much detail, I'm giving my app increasing amounts of Core Data events with which it needs to extrapolate data, make graphs, and present locations on a MKMapView instance. I started small and increased to 56000 events, which it handled fine wihtout any leaks or memory warnings (and I was quite proud of it for handling it all).

My app implements the Dropbox API to allow for uploading and downloading templates and data for sync purposes. Files uploaded from my app are converted from Core Data to an NSDictionary, then to NSData. I create a temporary folder for the data, then upload that file to Dropbox, which works fine.....normally. If I try to upload my data file with 56000 events, then it crashes. I've logged it and watched as the data is converted. It reaches the last event with no issues, but when it's supposed to start uploading to Dropbox, the app crashes and I cannot for the life of me figure out why. I see memory warnings pop up on my log. Typically, it will go Level=1, Level=2, Level=1, Level=2, then crash, which confuses me as it never reaches Level=3.

The majority of the information I've found is in my edit at the botton. Below is some relevant code:

- (void)uploadSurveys:(NSDictionary *)dict {
    NSArray *templateArray = [dict objectForKey:@"templates"];
    NSArray *dataArray = [dict objectForKey:@"data"];
    NSString *filename;
    NSLog(@"upload called");
    if ([templateArray count] || [dataArray count]) {
        if ([templateArray count]) {
            // irrelevent code;
        }
        if ([dataArray count]) {
            SurveyData *survey;
            for (int i = 0; i < [dataArray count]; i++) {
                BOOL matchExists = NO;
                // ...... code to make sure no file exists in dropbox folder and creates new version if necessary;

                dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
                NSData *data = [self convertSurvey:survey];
                dispatch_async(dispatch_get_main_queue(), ^{
                    [self uploadData:data withFilename:filename];
                    NSLog(@"converted and uploading");
                });
            });
        }
    }
}

[self convertSurvey:survey] simply converts my Core Data object to NSData.

- (void)uploadData:(NSData *)data withFilename:(NSString *)filename {
    NSFileManager *manager = [NSFileManager defaultManager];
    NSString *pathComponent = [NSString stringWithFormat:@"tempData.%@", filename];
    NSString *path = [NSTemporaryDirectory() stringByAppendingPathComponent:pathComponent];
    if ([manager createFileAtPath:path contents:data attributes:nil]) {
        [self.restClient uploadFile:filename toPath:[NSString stringWithFormat:@"/%@", currentSearch] fromPath:path];
        NSLog(@"uploading data");
    }
}

Any help would be much appreicated and I thoroughly thank you in advance. I'm just trying to figure out if I'm either taking the wrong approach for large files or if it's simply not allowed. If I have to split the files, that is fine, but I'd prefer to know what is going on that prevents my app from performing this action before I try to make a workaround. Thank you again.

UPDATE: As this issue is now the only hinderance to the release of my application, I'm adding a bounty to this question to hopefully get a solution or workaround. It will be up for a week, after which given time I am most likely going to just split up the files as they upload to ensure that this apparent size limit is not reached. This approach is not ideal, which is why a better solution is very welcomed, but is my backup plan if this fails to bring in something more convenient.

EDIT: It appears that NSTemporaryDirectory plays no part in this at all. Here is the new situation. As you can see in the code above, NSData *data = [self convertSurvey:survey]; is called in a secondary thread (which isn't the issue). I have been logging the objects created and knew that they had reached the last one, but never thought to check and see if the NSData file was returned. Turns out, it isn't. In short, I convert all my Core Data objects into arrays and place them into a dictionary (only for the relevant survey/data to be converted). This does indeed work and the dictionary is created. Then I create an NSData file using NSData *data = [NSKeyedArchiver archivedDataWithRootObject:d]; where d is my dictionary. Directly after that, I call return data; to set the value for NSData *data = [self convertSurvey:survey];. This being the case, it appears the NSData or NSKeyedArchiver are at fault here. According to the Apple documentation:

Using 32-bit Cocoa, the size of the data is subject to a theoretical 2GB limit (in practice, because memory will be used by other objects this limit will be smaller); using 64-bit Cocoa, the size of the data is subject to a theoretical limit of about 8EB (in practice, the limit should not be a factor).

I have checked the file sizes in small increments to see where the failure occurs. I have successfully gotten 48.2MB of data through, but not 51.5MB, which leads me to believe that the issue occurs around 50MB, well below the theoretical limit for NSData (unless there is a discrepancy between iOS and OS X in that respect).

Hopefully this new information will help to solve this problem

解决方案

The 2 GB limit for NSData is completely theoretical on iOS, even the iPhone 4 only has 512 MB of RAM and iOS (unlike Mac OS X) cannot swap, so if your physical RAM is full, you crash (or your app is terminated before that).

The 50 MB NSData object alone is already very large and it's not the only object you have in memory – given that you convert the data from Core Data to a dictionary representation and then to NSData, you probably consume at least twice as much memory (likely more). The system and other apps also need RAM, so you're probably reaching a limit.

Try running your app in Instruments to see how much memory you actually consume.

To reduce your peak memory usage, you have a couple of options that largely depend on your data model:

• As Jason Foreman suggested in his answer, try to avoid having your whole file in memory at once. Using NSFileHandle, you can write chunks of data to a file without needing to have the whole data in memory at once. Of course, this requires that you prepare your data accordingly, so that it can be split into chunks. A higher-level approach might be to serialize your data into an XML format that you could write out as a stream. If your data format is very simple, something like CSV might also work.

• Don't use NSData for uploading to Dropbox. Write your data to a file instead (see above) and point the Dropbox SDK to that file. The Dropbox SDK makes it pretty easy to do so (DBRestClient has an uploadFile:toPath:fromPath: method).

• If your data model makes it difficult to take a streaming approach, try to segment the data into more manageable parts. You could then use your old method of serializing dictionaries, just with multiple files.

• Be careful with Core Data's memory usage. Try to re-fault objects using refreshObject:mergeChanges: if possible to break cyclic references within your data (see the Core Data Programming Guide for details).

• Avoid using autorelease pools while you're in a long-running loop or create a separate NSAutoreleasePool that gets drained in each iteration of your loop.

这篇关于从iPad转换/上传大量数据到Dropbox的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！