AFHTTPRequestOperations队列创建内存累积 [英] Queue of AFHTTPRequestOperations creating Memory Buildup
问题描述
我刚刚更新到AFNetworking 2.0，并且正在重新编写代码以下载数据和将其插入Core Data.
我下载JSON数据文件(从10-200mb的任何位置)，将它们写入磁盘，然后将其传递给后台线程以处理数据.以下是下载JSON&的代码将其写入磁盘.如果我只是让它运行(甚至不处理数据)，该应用程序就会用完内存，直到被杀死为止.
我假设随着数据的传入，它被存储在内存中，但是一旦我将其保存到磁盘上，为什么它仍会保留在内存中?自动释放池不应该解决这个问题吗?我还设置了responseData，并将downloadData设置为nil.有什么明显的地方是我在这里做错了吗?
@autoreleasepool
{
for(int i = 1; i <= totalPages; i++)
{
NSString *path = ....
NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:path]];
AFHTTPRequestOperation *op = [[AFHTTPRequestOperation alloc] initWithRequest:request];
op.responseSerializer =[AFJSONResponseSerializer serializer];
[op setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject)
{
//convert dictionary to data
NSData *downloadData = [NSKeyedArchiver archivedDataWithRootObject:responseObject];
//save to disk
NSError *saveError = nil;
if (![fileManager fileExistsAtPath:targetPath isDirectory:false])
{
[downloadData writeToFile:targetPath options:NSDataWritingAtomic error:&saveError];
if (saveError != nil)
{
NSLog(@"Download save failed! Error: %@", [saveError description]);
}
}
responseObject = nil;
downloadData = nil;
} failure:^(AFHTTPRequestOperation *operation, NSError *error) {
DLog(@"Error: %@", error);
}];
}
[mutableOperations addObject:op];
}
NSArray *operations = [AFURLConnectionOperation batchOfRequestOperations:mutableOperations progressBlock:^(NSUInteger numberOfFinishedOperations, NSUInteger totalNumberOfOperations) {
DLog(@"%lu of %lu complete", (unsigned long)numberOfFinishedOperations, (unsigned long)totalNumberOfOperations);
} completionBlock:^(NSArray *operations) {
DLog(@"All operations in batch complete");
}];
mutableOperations = nil;
[manager.operationQueue addOperations:operations waitUntilFinished:NO];
谢谢！
编辑#1
在我的完整代码块中添加@autoreleasepool似乎会稍微降低内存使用率，但它仍会累积并最终导致应用程序崩溃.
如果每个JSON文件的大小实际上为10-200mb，则肯定会导致内存问题，因为这种请求会将响应加载到内存中(而是而不是将它们流式传输到持久性存储).更糟糕的是，由于您使用的是JSON，因此我认为问题要严重一倍，因为您要将其加载到字典/数组中，这也会占用内存.因此，如果您正在进行四次100mb的下载，则您的峰值内存使用量可能约为800mb(NSData为100mb，而数组/字典为〜100mb(可能更大)，是四次的四倍)并发请求).您可能很快就会用完内存.
所以，有几个反应:
-
在处理此数据量时，您希望使用流接口(
NSURLConnection或NSURLSessionDataTask，在其中写入数据时将其写入，而不是将其保存在内存中；或使用NSURLSessionDownloadTask为您执行此操作)，它将数据直接写入持久性存储(而不是在下载时试图将其保存在RAM中的NSData中).如果使用
NSURLSessionDownloadTask，这确实很简单.如果您需要支持7.0之前的iOS版本，则不确定AFNetworking是否支持将响应直接流式传输到持久性存储.我敢打赌，您可以编写自己的响应序列化器来执行此操作，但是我还没有尝试过.我一直写自己的NSURLConnectionDataDelegate方法，这些方法直接下载到持久性存储中(例如，像这样的 a>). -
您可能不希望为此使用JSON(因为
NSJSONSerialization会将整个资源加载到内存中，然后将其解析到也在内存中的NSArray/NSDictionary中)，而是使用一种格式，可用于响应的流解析(例如XML)，并在解析时编写一种将数据存储到数据存储区(Core Data或SQLite)的解析器，而不是尝试将整个内容加载到RAM中. /p>请注意，甚至
NSXMLParser的内存效率也令人惊讶(请参阅此问题).在 XMLPerformance 示例中，Apple演示了如何实现使用更麻烦的 LibXML2 可以最大程度地减少XML解析器的内存占用. -
顺便说一句，我不知道您的JSON是否包含您已编码的任何二进制数据(例如base 64等)，但是如果是这样，您可能要考虑一个二进制传输格式，不必进行此转换.使用base-64或uuencode或其他方法可能会增加带宽和内存需求. (如果您不处理已编码的二进制数据，那么请忽略这一点.)
-
顺便说一句，您可能想使用可达性"来确认用户的连接类型(Wifi与蜂窝网络)，因为在蜂窝网络上下载那么多数据被认为是一种不好的形式(至少在未经用户许可的情况下). ，不仅是因为速度问题，而且还有耗尽运营商每月数据计划过多部分的风险.我什至听说苹果从历史上就拒绝过尝试通过蜂窝网络下载过多数据的应用程序.
I just updated to AFNetworking 2.0 and I am re-writing my code to download data & insert it into Core Data.
I download JSON data files (anywhere from 10-200mb files), write them to disk, then pass them off to background threads to process the data. Below is the code that downloads the JSON & write it to disk. If I just let this run (without even processing the data), the app uses up memory until it is killed.
I assume as the data is coming in, it is being stored in memory, but once I save to disk why would it stay in memory? Shouldn't the autorelease pool take care of this? I also set the responseData, and downloadData to nil. Is there something blatantly obvious that I am doing wrong here?
@autoreleasepool
{
for(int i = 1; i <= totalPages; i++)
{
NSString *path = ....
NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:path]];
AFHTTPRequestOperation *op = [[AFHTTPRequestOperation alloc] initWithRequest:request];
op.responseSerializer =[AFJSONResponseSerializer serializer];
[op setCompletionBlockWithSuccess:^(AFHTTPRequestOperation *operation, id responseObject)
{
//convert dictionary to data
NSData *downloadData = [NSKeyedArchiver archivedDataWithRootObject:responseObject];
//save to disk
NSError *saveError = nil;
if (![fileManager fileExistsAtPath:targetPath isDirectory:false])
{
[downloadData writeToFile:targetPath options:NSDataWritingAtomic error:&saveError];
if (saveError != nil)
{
NSLog(@"Download save failed! Error: %@", [saveError description]);
}
}
responseObject = nil;
downloadData = nil;
} failure:^(AFHTTPRequestOperation *operation, NSError *error) {
DLog(@"Error: %@", error);
}];
}
[mutableOperations addObject:op];
}
NSArray *operations = [AFURLConnectionOperation batchOfRequestOperations:mutableOperations progressBlock:^(NSUInteger numberOfFinishedOperations, NSUInteger totalNumberOfOperations) {
DLog(@"%lu of %lu complete", (unsigned long)numberOfFinishedOperations, (unsigned long)totalNumberOfOperations);
} completionBlock:^(NSArray *operations) {
DLog(@"All operations in batch complete");
}];
mutableOperations = nil;
[manager.operationQueue addOperations:operations waitUntilFinished:NO];
Thanks!
EDIT #1
Adding an @autoreleasepool within my complete block seemed to slow the memory usage a bit, but it still builds up and eventually crashes the app.
If your JSON files are really 10-200mb each, this would definitely cause memory problems, because this sort of request is going to load the responses in memory (rather than streaming them to persistent storage). Worse, because your using JSON, I think the problem is twice as bad, because you're going to be loading this into a dictionary/array, which also takes up memory. So, if you have four 100mb downloads going on, your peak memory usage could be of the order of magnitude of 800mb (100mb for the NSData plus ~100mb for the array/dictionary (possibly much larger), times four for the four concurrent requests). You could quickly run out of memory.
So, a couple of reactions:
When dealing with this volume of data, you'd want to pursue a streaming interface (a
NSURLConnectionorNSURLSessionDataTaskwhere you write the data as it comes in, rather than holding it in memory; or useNSURLSessionDownloadTaskwhich does this for you), one that writes the data directly to persistent storage (rather than trying to hold it in aNSDatain RAM as it's being downloaded).If you use
NSURLSessionDownloadTask, this is really simple. If you need to support iOS versions prior to 7.0, I'm not sure if AFNetworking supports streaming of the responses directly to persistent storage. I'd wager you could write your own response serializer that does that, but I haven't tried it. I've always written my ownNSURLConnectionDataDelegatemethods that download directly to persistent storage (e.g. something like this).You might not want to use JSON for this (because
NSJSONSerializationwill load the whole resource into memory, and then parse it to aNSArray/NSDictionary, also in memory), but rather use a format that lends itself to streamed parsing of the response (e.g. XML) and write a parser that stores the data to your data store (Core Data or SQLite) as it's being parsed, rather than trying to load the whole thing in RAM.Note, even
NSXMLParseris surprisingly memory inefficient (see this question). In the XMLPerformance sample, Apple demonstrates how you can use the more cumbersome LibXML2 to minimize the memory footprint of your XML parser.By the way, I don't know if your JSON includes any binary data that you have encoded (e.g. base 64 or the like), but if so, you might want to consider a binary transfer format that doesn't have to do this conversion. Using base-64 or uuencode or whatever can increase your bandwidth and memory requirements. (If you're not dealing with binary data that has been encoded, then ignore this point.)
As an aside, you might want to use Reachability to confirm the user's connection type (Wifi vs cellular), because it is considered bad form to download that much data over cellular (at least not without the user's permission), not only because of speed issues, but also the risk of using up an excessive portion of their carrier's monthly data plan. I've even heard that Apple historically rejected apps that tried to download too much data over cellular.
这篇关于AFHTTPRequestOperations队列创建内存累积的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
