将一个大文件拆分为多个InputStream，以Java多线程处理 [英] Split one large file into multiple InputStream(s) to be processed in multithreading in Java

问题描述

我的代码:

public ArrayList<InputStream> getAllInputStreams() {
    ArrayList<InputStream> allStreams = new ArrayList<InputStream>();
    InputStream stream = this.getNext();
    while (stream != null) {
        allStreams.add(stream);
        stream = this.getNext();
    }
    return allStreams;
}

public InputStream getNext() {
    if (done()) {
        return null;
    }
    InputStream segment = createInputStream();
    this.countStream++;
    return segment;
}

protected InputStream createInputStream() {
    BoundedInputStream res = new BoundedInputStream(
            Channels.newInputStream(this.randomAccessFile.getChannel().position(this.countStream * chunkSize)), chunkSize);
    res.setPropagateClose(false) ;
    return res ;
}

我正在尝试将file拆分为多个InputStream(private RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r");.所有InputStream(从getAllInputStreams()获取))要由多个线程处理，看来它们大多数都是空的.为什么?

I am trying to split file into several InputStream(s) (private RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r");. All InputStream(s) (got from getAllInputStreams()) to be processed by multiple threads, it seems that most of them are empty. Why?

欢迎任何提示.谢谢

更新

以下代码似乎正常工作.以下代码段是将文件拆分为多个卡盘的好方法吗?每个卡盘的大小是否应该小于内存大小?

It seems that the following piece of codes working fine. Is the following piece of codes a good way to split the file into several chucks? Should the size of each chuck smaller than memory size?

protected InputStream createInputStream() {
    RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r");
    BoundedInputStream res = new BoundedInputStream(
            Channels.newInputStream(randomAccessFile.getChannel().position(this.countStream * chunkSize)), chunkSize);
    res.setPropagateClose(false) ;
    return res ;
}

推荐答案

我认为这句话:

      res.setPropagateClose(false);

您的"UPDATE"解决方案中的

将导致资源泄漏.您打开的RAF未与其他任何人共享.当BoundedInputStream关闭时，您 需要关闭以传播到RAF.

in your "UPDATE" solution will result in a resource leak. The RAF that you have opened is not shared with anything else. When the BoundedInputStream is closed, you need the close to propagate to the RAF.

这篇关于将一个大文件拆分为多个InputStream，以Java多线程处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！