Java PipedInputStream PipedOutputStream大小限制 [英] Java PipedInputStream PipedOutputStream Size limitation

问题描述

我正在使用Java管道将数据(流)从解压缩模块(JavaUncompress类)传递到解析模块(处理程序类)，文件很大，我想先解压缩文件并直接解析而不是保存解压缩的文件，然后解析.但是，它仅适用于较小的文件.当我输入一个1G文件时，似乎只有一部分文件(例如50000行)从解析模块的输出流向输入流输出.

I am using java pipeline to pass the data (outstream) from an unzip module (JavaUncompress class) to a parsing module (handler class), the file is large, I want to unzip the file first and parse directly instead of saving the unzipped file and then parse. However, it only works for file of small size. When I input an 1G file, it seems only part of the file (say 50000 lines) are piplined from the outstream to the inputstream of the parsing module.

我尝试使用String来保存未压缩的文件，并且发生了同样的事情，该String仅包含部分解压缩的文件(停在与piplined版本相同的第50000行).关于发生的事情有什么想法吗?非常感谢.

I tried to use a String to save the uncompressed file, and the same thing happened, the String only contains part of the unzipped file (stopped at the same 50000th line as the piplined version). Is there any idea about what happened? Thank you very much.

   {
   PipedInputStream in = new PipedInputStream(); // to output
   final PipedOutputStream out = new PipedOutputStream(in); // out is something from other

   new Thread(
    new Runnable(){
        public void run(){ 
                JavaUncompress.putDataOnOutputStream(inFile,out); }
        }
        ).start();

   doc = handler.processDataFromInputStream(in);
   }

   public static void putDataOnOutputStream(String inZipFileName, PipedOutputStream out){

   try {
          FileInputStream fis = new FileInputStream(inZipFileName);
          //FilterInputStream ftis = new FilterInputStream;
          ZipInputStream zis = new ZipInputStream(new BufferedInputStream(fis));
          ZipEntry entry;

          while((entry = zis.getNextEntry()) != null) {
             System.out.println("Extracting: " +entry);
             byte data[] = new byte[BUFFER];

             long len = entry.getSize();
             long blk = len/BUFFER;
             int  rem = (int)(len - blk*BUFFER);
             System.out.println(len+" = "+blk +"*BUFFER + "+rem);

             for(long i=0; i!=blk; ++i){
                 if ((zis.read(data, 0, BUFFER)) != -1) {
                     out.write(data);
                 }
             }

             byte dataRem[] = new byte[rem];
             if ((zis.read(dataRem, 0, rem)) != -1) {
                 out.write(dataRem);
                 out.flush();
                 out.close();
             }

          }
          zis.close();

       } catch(Exception e) {
          e.printStackTrace();
       }
   }

推荐答案

PipedOutputStream.write()将在相应的PipedInputStream大于4096或后面的任何字节的情况下阻塞，但是为什么要这样做呢?为什么不将文件解压缩并在同一线程中处理呢?多线程没有任何优势，这只是毫无意义的复杂性.

PipedOutputStream.write() will block if the corresponding PipedInputStream gets more than 4096 or whatever bytes behind it, but why do this at all? Why not just unzip the file and process it in the same thread? There's no advantage to multi-threading it, it's just a pointless complication.

在Java中，每15年我只使用过一次管道，所以很快就将其更改为队列.

I've used pipes exactly once in 15 years in Java and I pretty quickly changed it to a queue.

这篇关于Java PipedInputStream PipedOutputStream大小限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！