LZMA SDK解压缩iOS（xcode）使用太多的RAM [英] LZMA SDK decompress for iOS (xcode) using too much RAM

问题描述

我想在iPhone / iPad应用程序中使用LZMA SDK，我的起点是Mo Dejong提供的iPhone的LZMA示例项目，可以在这里：

所以直到有人修复iOS的SDK，解决方法是：

1）确定要对文件解压缩操作使用的RAM限制。

2）在归档文件中超过上面的1的限制的任何SINGLE文件必须拆分，你可以使用任何二进制拆分器应用程序，如拆分：
http://www.fourmilab.ch/splits/

3）文件准备好，使用字典/块大小选项创建7z文件，如MoDJ在他的答案中所述，例如24 Mbps限制：
7za a -mx = 9 -md = 24m -ms = 24m CompressedFile。 7z SourceFiles *

4）在您的iOS应用程序中，解压缩文件后，确定已拆分的文件，并将它们连接在一起。代码不是那么复杂（我假设splits.exe使用的命名约定，它是file.001，file.002等）。

  if（iParts> 1）
 {
 //如果这是一个多部分二进制分割文件，我们必须结合所有的零件才能使用
 NSString * finalfilePath = whateveryourfinaldestinationfilenameis 
 NSString * splitfilePath = [finalfilePath stringByAppendingString：@。001]; 
 
 NSFileHandle * myHandle; 
 NSFileManager * fileManager = [NSFileManager defaultManager]; 
 NSError * error; 
 
 //如果目标组合文件已经存在，删除它
 if（[fileManager fileExistsAtPath：finalfilePath]）
 {
 BOOL success = [fileManager removeItemAtPath：finalfilePath错误：&错误]; 
 if（！success）NSLog（@Error：％@，[error localizedDescription]）; 
} 
 
 myHandle = [NSFileHandle fileHandleForUpdatingAtPath：splitfilePath]; 
 NSString * nextPart; 
 //按顺序连接每个片段
 for（int i = 2; i <= iParts; i ++）{
 //假设少于100个片段
 if（i <10 ）nextPart = [splitfilePath stringByReplacingOccurrencesOfString：@。001withString：[NSString stringWithFormat：@。00％d，i]]; 
 else nextPart = [splitfilePath stringByReplacingOccurrencesOfString：@。001withString：[NSString stringWithFormat：@。0％d，i]]; 
 NSData * datapart = [[NSData alloc] initWithContentsOfFile：（NSString *）nextPart]; 
 [myHandle seekToEndOfFile]; 
 [myHandle writeData：datapart]; 
} 
 [myHandle closeFile]; 
 //重命名并置文件
 [fileManager moveItemAtPath：splitfilePath toPath：finalfilePath error：& error]; 
} 
  

I am trying to use the LZMA SDK in an iPhone/iPad app, my starting point was the LZMA example project for iPhone provided by Mo Dejong, available here: https://github.com/jk/lzmaSDK Original was here: http://www.modejong.com/iOS/lzmaSDK.zip (I tried both and I get the same result from both).

The problem is that the extract uses as much RAM as the .7z contains uncompressed. In other words, say I have a 40MB compressed file, the uncompressed file is a binary sqlite DB that is about 250MB, it will slowly use up more and more memory as it uncompresses the file all the way up to 250MB. This will crash an iPad1 or anything before iPhone4 (256MB RAM). I have a feeling a lot of people will eventually run into this same problem, so a resolution now could help a lot of developers.

I originally created the .7z file on a PC using windows based 7-zip (latest version) and a 16MB dictionary size. It should only require 18MB of RAM to uncompress (and that is the case when testing on a PC looking at task manager). I also tried creating the archive using keka (the open source mac archiver), it did not resolve anything, although I can confirm that keka itself only uses 19MB of ram during its extract of the file on a mac which is what I would expect. I guess the next step would be to compare the source code of Keka to the source code of the LZMA SDK.

I played around with different dictionary sizes and other settings when creating the .7z file but nothing helped. I also tried splitting my single binary file into 24 smaller pieces before compressing, but that also did not help (still uses over 250MB of RAM to extract the 24 pieces).

Note that the ONLY change I made to the original code was to use a bigger .7z file. Also note that it does immediately free up the RAM as soon as the extract is finished, but that doesn't help. I feel like it is not freeing up RAM as it extracts like it should, or it is putting the entire contents into RAM until the very end when it is done and only then moving it out of RAM. Also, if I try to extract the same exact file using a mac app, while running instruments, I do not see the same behavior (StuffIt Expander for example maxed out at around 60MB of RAM while extracting the file, Keka, the open source mac archiver maxed out at 19MB of RAM).

I'm not much of a mac/xcode/objective-c developer (yet) so any help with this would be greatly appreciated. I could resort to using zip or rar instead, but I get far superior compression with LZMA so if at all possible I want to stick with this solution but obviously I need to get it to work without crashing.

Thanks!

解决方案

Igor Pavlov, author of 7zip, emailed me, he basically said the observations I made in the original question are a known limitation of the c version of the SDK. The C++ version does not have this limitation. Actual quote:

"7-Zip uses another multithreaded decoder written in C++. That C++ .7z decoder doesn't need to allocate RAM block for whole solid block. Read also this thread:

http://sourceforge.net/projects/sevenzip/forums/forum/45797/topic/5655623 "

So until someone fixes the SDK for iOS, the workaround is to:

1) Decide what RAM limit you want to have for file decompression operations.

2) Any SINGLE file in your archive that exceeds limit from 1 above, must be split, you can do this using any binary spliter app such as splits: http://www.fourmilab.ch/splits/

3) After your files are ready, create the 7z file using the dictionary/block size options as described by MoDJ in his answer, for example with 24 meg limit: 7za a -mx=9 -md=24m -ms=24m CompressedFile.7z SourceFiles*

4) In your iOS app, after you decompress the files, determine what files had been split, and concatenate them back together again. The code for this is not all that complicated (I assume the naming convention that splits.exe uses, which is file.001, file.002, etc.)

    if(iParts>1)
    {
        //If this is a multipart binary split file, we must combine all of the parts before we can use it
        NSString *finalfilePath = whateveryourfinaldestinationfilenameis
        NSString *splitfilePath = [finalfilePath stringByAppendingString:@".001"];

        NSFileHandle *myHandle;
        NSFileManager *fileManager = [NSFileManager defaultManager];
        NSError *error;

        //If the target combined file exists already, remove it
        if ([fileManager fileExistsAtPath:finalfilePath]) 
        {
            BOOL success = [fileManager removeItemAtPath:finalfilePath error:&error];
            if (!success) NSLog(@"Error: %@", [error localizedDescription]);
        }

        myHandle  = [NSFileHandle fileHandleForUpdatingAtPath:splitfilePath];
        NSString *nextPart;
        //Concatenate each piece in order
        for (int i=2; i<=iParts; i++) {
            //Assumes fewer than 100 pieces
            if (i<10) nextPart = [splitfilePath stringByReplacingOccurrencesOfString:@".001" withString:[NSString stringWithFormat:@".00%d", i]];
            else nextPart = [splitfilePath stringByReplacingOccurrencesOfString:@".001" withString:[NSString stringWithFormat:@".0%d", i]];
            NSData *datapart = [[NSData alloc] initWithContentsOfFile:(NSString *)nextPart];
            [myHandle seekToEndOfFile];
            [myHandle writeData:datapart];
        }    
        [myHandle closeFile];
        //Rename concatenated file
        [fileManager moveItemAtPath:splitfilePath toPath:finalfilePath error:&error];
    }

这篇关于LZMA SDK解压缩iOS（xcode）使用太多的RAM的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！