使用 Adobe AIR 解析大型文本文件 [英] Parsing large text files with Adobe AIR
问题描述
我正在尝试在 AIR 中执行以下操作:
I am trying to do the following in AIR:
- 浏览到一个文本文件
- 读取文本文件并将其存储在字符串中(最终存储在数组中)
- 用分隔符\n分割字符串并将结果字符串放入数组中
- 在将数据发送到网站(mysql 数据库)之前对其进行操作
我正在处理的文本文件大小在 100-500mb 之间.到目前为止，我已经能够完成第 1 步和第 2 步，这是我的代码:
The text files I am dealing with will be anywhere from 100-500mb in size. So far, I've been able to to complete steps 1 and 2, here is my code:
<mx:Script>
<![CDATA[
import mx.collections.ArrayCollection;
import flash.filesystem.*;
import flash.events.*;
import mx.controls.*;
private var fileOpened:File = File.desktopDirectory;
private var fileContents:String;
private var stream:FileStream;
private function selectFile(root:File):void {
var filter:FileFilter = new FileFilter("Text", "*.txt");
root.browseForOpen("Open", [filter]);
root.addEventListener(Event.SELECT, fileSelected);
}
private function fileSelected(e:Event):void {
var path:String = fileOpened.nativePath;
filePath.text = path;
stream = new FileStream();
stream.addEventListener(ProgressEvent.PROGRESS, fileProgress);
stream.addEventListener(Event.COMPLETE, fileComplete);
stream.openAsync(fileOpened, FileMode.READ);
}
private function fileProgress(p_evt:ProgressEvent):void {
fileContents += stream.readMultiByte(stream.bytesAvailable, File.systemCharset);
readProgress.text = ((p_evt.bytesLoaded/1048576).toFixed(2)) + "MB out of " + ((p_evt.bytesTotal/1048576).toFixed(2)) + "MB read";
}
private function fileComplete(p_evt:Event):void {
stream.close();
//fileText.text = fileContents;
}
private function process(c:String):void {
if(!c.length > 0) {
Alert.show("File contents empty!", "Error");
}
//var array:Array = c.split(/\n/);
}
]]>
</mx:Script>
这是 MXML
<mx:Text x="10" y="10" id="filePath" text="Select a file..." width="678" height="22" color="#FFFFFF" fontWeight="bold"/>
<mx:Button x="10" y="40" label="Browse" click="selectFile(fileOpened)" color="#FFFFFF" fontWeight="bold" fillAlphas="[1.0, 1.0]" fillColors="[#E2E2E2, #484848]"/>
<mx:Button x="86" y="40" label="Process" click="process(fileContents)" color="#FFFFFF" fontWeight="bold" fillAlphas="[1.0, 1.0]" fillColors="[#E2E2E2, #484848]"/>
<mx:TextArea x="10" y="70" id="fileText" width="678" height="333" editable="false"/>
<mx:Label x="10" y="411" id="readProgress" text="" width="678" height="19" color="#FFFFFF"/>
第 3 步是我遇到一些麻烦的地方.我的代码中有两行被注释掉了，这两行都会导致程序冻结.
step 3 is where I am having some troubles. There are 2 lines in my code commented out, both lines cause the program to freeze.
fileText.text = 文件内容；尝试将字符串的内容放入 textarea
var array:Array = c.split(/\n/);尝试通过分隔符换行符分割字符串
fileText.text = fileContents; attempts to put the contents of the string in a textarea
var array:Array = c.split(/\n/); attempts to split the string by delimiter newline
此时可以使用一些输入...我什至以正确的方式解决这个问题吗?flex/air 可以处理这么大的文件吗?(我会这么认为)这是我第一次尝试做任何类型的弹性工作，如果你看到我做错的其他事情或可以做得更好，我会很感激！
Could use some input at this point... Am i even going about this the right way? Can flex/air handle files this large? (i'd assume so) This is my first attempt at doing any sort of flex work, if you see other things ive done wrong or could be done better, i'd appreciate the heads up!
谢谢！
推荐答案
对 500MB 的文件执行 split 可能不是一个好主意.您可以编写自己的解析器来处理文件，但速度也可能不是很快:
Doing a split on a 500MB file might not be a good idea. You can write your own parser to work on the file but it may not be very fast either:
private function fileComplete(p_evt:Event):void
{
var array:Array = [];
var char:String;
var line:String = "";
while(stream.position < stream.bytesAvailable)
{
char = stream.readUTFBytes(1);
if(char == "\n")
{
array.push(line);
line = "";
}
else
{
line += char;
}
}
// catch the last line if the file isn't terminated by a \n
if(line != "")
{
array.push(line);
}
stream.close();
}
我还没有测试过，但它应该一个字符一个字符地遍历文件.如果字符是新行，则将旧行推入数组，否则将其添加到当前行.
I haven't tested it but it should just step through the file character by character. If the character is a new line then push the old line into the array otherwise add it to the current line.
如果你不希望它在你做的时候阻塞你的 UI，你需要把它抽象成一个基于计时器的想法:
If you don't want it to block your UI while you do it, you'll need to abstract it into a timer based idea:
// pseudo code
private function fileComplete(p_evt:Event):void
{
var array:Array = [];
processFileChunk();
}
private function processFileChunk(event:TimerEvent=null):void
{
var MAX_PER_FRAME:int = 1024;
var bytesThisFrame:int = 0;
var char:String;
var line:String = "";
while( (stream.position < stream.bytesAvailable)
&& (bytesThisFrame < MAX_PER_FRAME))
{
char = stream.readUTFBytes(1);
if(char == "\n")
{
array.push(line);
line = "";
}
else
{
line += char;
}
bytesThisFrame++;
}
// if we aren't done
if(stream.position < stream.bytesAvailable)
{
// declare this in the class
timer = new Timer(100, 1);
timer.addEventListener(TimerEvent.TIMER_COMPLETE, processFileChunk);
timer.start();
}
// we're done
else
{
// catch the last line if the file isn't terminated by a \n
if(line != "")
{
array.push(line);
}
stream.close();
// maybe dispatchEvent(new Event(Event.COMPLETE)); here
// or call an internal function to deal with the complete array
}
}
基本上，您选择一定数量的文件来处理每一帧 (MAX_PER_FRAME)，然后处理那么多字节.如果您超过了字节数，那么只需创建一个计时器以在几帧时间内再次调用进程函数，它应该从停止的地方继续.一旦确定完成，您就可以调度调用另一个函数的事件.
Basically you choose an amount of the file to process each frame (MAX_PER_FRAME) and then process that many bytes. If you go over the number of bytes then just make a timer to call the process function again in a few frames time and it should continue where it left off. You can dispatch an event of call another function once you are sure you are complete.
这篇关于使用 Adobe AIR 解析大型文本文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
