openCSV没有读取我的整个文件 [英] openCSV not reading my entire file

问题描述

我在Java中有一个应用程序，我使用openCSV来读取文件（非常大）。然后，我将第4个（最终将有另外一列或两个添加，如果有所不同）列放入HashSet并将其输出到新文件。这一切似乎工作正常，但我发现它只是读取文件的一部分（131,544行272,948）。这是openCSV或Java的一般限制还是有办法解决这个问题？

I have an application in Java that I am using openCSV to read a file (very large). I am then putting the 4th (Eventually this will have another column or two added if that makes a difference) column into a HashSet and outputting that to a new file. This all seems to work fine but I discovered it is only reading part of the file (131,544 lines of 272,948). Is this a limitation of the openCSV or Java in general or is there a way to get around this?

我的参考代码：

public static void main(String[] args) throws IOException {
    String itemsFile = new String();        
    String outFile = new String();
    itemsFile = "items.txt";        
    outFile = "so.txt";
    CSVReader reader = null;
    try {
        reader = new CSVReader(new FileReader(itemsFile), '\t');
    } catch (FileNotFoundException e) {
        System.out.println(e.getMessage());
        e.printStackTrace();
    }

    String[] nextLine;
    HashSet<String> brands = new HashSet<>();               
    while ((nextLine = reader.readNext()) != null) {
        brands.add(nextLine[4]);            
    }               

    String[] brandArray = new String[brands.size()];
    Iterator<String> it = ((HashSet<String>) brands).iterator();
    int listNum = 0;
    while (it.hasNext()) {
        Object brand = (Object) it.next();
        brandArray[listNum] = (String) brand;
        listNum++;
    }

    CSVWriter writer = new CSVWriter(new FileWriter(outFile), '\n');
    writer.writeNext(brandArray);           
    writer.close();
}

如果我的代码很乱，我很抱歉这是我的第一个真正的已完成的Java应用。非常感谢任何帮助。

I apologize if my code is messy this is my first real "Completed" Java application. Any assistance is much appreciated.

我甚至尝试从txt文件中删除这些行，以确保它不会挂在某些字符或其他东西上，但似乎停止无论如何在那条线上

I've even tried removing those lines from the txt file to make sure it's not hanging up on some character or something but it seems to stop on that line anyway

推荐答案

好的，我想通过聊天用户@Michael来解决这个问题。显然openCSV无法处理如此大的文件，因为它不是流式传输。所以我查看了流式传输此文件并且效果很好。

OK I figured this out thanks to user @Michael in chat. Apparently openCSV can't handle such a large file because it is not streaming. SO I looked into streaming this file and it works great.

以下是结束代码：

public static void main(String[] args) throws IOException {

    String fileName = new String();
    fileName = "items.txt";
    String outputFile = new String();
    outputFile = "so.txt";      
    String thisLine;
    HashSet<String> brand = new HashSet<>();
    FileInputStream fis = new FileInputStream(fileName);
    @SuppressWarnings("resource")
    BufferedReader myInput = new BufferedReader(new InputStreamReader(fis));
    while ((thisLine = myInput.readLine()) != null) {
        String[] line = thisLine.split("\t");
        if (line[20].equals("1")) {
            if (!line[2].equals("") && !line[2].equals(" ")
                    && !line[2].equals(null)) {                 
                if(line[2].indexOf("'") > -1){
                    System.out.println(line[2]);
                    line[2] = line[2].replace("'", "\'");
                    System.out.println(line[2]);
                }

                brand.add(line[2]);
            }
        }
        if (!line[3].equals("") && !line[3].equals(" ")
                && !line[3].equals(null)) {             
                line[3] = line[3].replace("'", "\'");               
            brand.add(line[3]);
        }
        if (!line[4].equals("") && !line[4].equals(" ")
                && !line[4].equals(null)) {
            if(line[4].indexOf("'") > -1){
                System.out.println(line[4]);
                line[4] = line[4].replace("'", "\'");
                System.out.println(line[4]);
            }


            brand.add(line[4]);
        }
    }

    String[] brands = brand.toArray(new String[brand.size()]);

    try {
        FileWriter fstream = new FileWriter(outputFile);
        BufferedWriter bw = new BufferedWriter(fstream);
        for (int i = 0; i < brands.length; i++) {

            if (i == 0) {
                bw.write("'" + brands[i] + "'");
            } else {
                bw.write(",'" + brands[i] + "'");
            }
        }           

        bw.close();
    } catch (Exception e) {
        System.out.println(e.getMessage());
        e.printStackTrace();
    }
}

感谢大家的帮助。

这篇关于openCSV没有读取我的整个文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！