如何通过使用OpenCsv记录无效记录的错误来创建有效CSV记录的列表? [英] How to create a list of valid CSV records by logging the error of invalid record using OpenCsv?

问题描述

在我的CSV中，数据不一致，因此必须在记录重复记录时记录错误，并希望创建有效CSV记录的POJO列表.我正在使用OpenCsv处理CSV文件. 我添加了try-catch块，以记录错误，如果在 while(iterator.hasNext())上没有任何错误，则由于下一条记录包含一些格式错误的数据，因此存在异常/错误.

In my CSV I have inconsistent data so I have to log the error while iterating record and want to create a list of POJO of valid CSV records. I am using OpenCsv to process the CSV file. I have added try-catch block to log the error if there is any but at while(iterator.hasNext()) there is the exception/error as next record has some incorrect formatted data.

那么如何记录错误并继续处理下一条记录?

So how to log the error and continue to process the next record?

List<UserProvisioning> list = new ArrayList<>();
CsvToBean<UserProvisioning> beans = new CsvToBeanBuilder<UserProvisioning>(
        new FileReader(file.getAbsolutePath())).withType(UserProvisioning.class)
                .withIgnoreQuotations(true).build();
Iterator<UserProvisioning> iterator = beans.iterator();
while (iterator.hasNext()) {
    try {
        UserProvisioning userProvisioning = (UserProvisioning) iterator.next();
        System.out.println(userProvisioning.getFIRST_NAME());
        list.add(userProvisioning);
    } catch (Exception e) {
        // TODO Auto-generated catch block
        logger.error("Error occured...)
    }
}

如果我使用 .withThrowExceptions(false)，我可以处理所有有效记录，但无法记录错误.

If I use .withThrowExceptions(false) I can process all valid records but not able to log the error.

错误

java.lang.RuntimeException: com.opencsv.exceptions.CsvRequiredFieldEmptyException: Number of data fields does not match number of headers.
    at com.opencsv.bean.concurrent.ProcessCsvLine.run(ProcessCsvLine.java:101)
    at com.opencsv.bean.CsvToBean$CsvToBeanIterator.readLineWithPossibleError(CsvToBean.java:551)
    at com.opencsv.bean.CsvToBean$CsvToBeanIterator.readSingleLine(CsvToBean.java:571)
    at com.opencsv.bean.CsvToBean$CsvToBeanIterator.next(CsvToBean.java:591)
    at com.apds.partner.nycdoc.main.NycDocApplication.main(NycDocApplication.java:90)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49)
Caused by: com.opencsv.exceptions.CsvRequiredFieldEmptyException: Number of data fields does not match number of headers.
    at com.opencsv.bean.HeaderColumnNameMappingStrategy.verifyLineLength(HeaderColumnNameMappingStrategy.java:110)
    at com.opencsv.bean.AbstractMappingStrategy.populateNewBean(AbstractMappingStrategy.java:313)
    at com.opencsv.bean.concurrent.ProcessCsvLine.processLine(ProcessCsvLine.java:132)
    at com.opencsv.bean.concurrent.ProcessCsvLine.run(ProcessCsvLine.java:85)
    ... 9 more

如何通过使用OpenCsv记录无效记录的错误来创建有效的csv记录列表?

How to create a list of valid csv records by logging the error of invalid record using OpenCsv?

根据我的理解，iterator.hasNext()尝试通过将csv记录列映射到POJO字段来检查下一个元素是否存在，并且由于csv记录标题中的数据无效与记录文件不匹配，因此错误

As per my understanding iterator.hasNext() tries to check whether next element is present or not by mapping csv record column to POJO fields and as there is invalid data in csv record headers count do not matches record files hence error java.lang.RuntimeException: com.opencsv.exceptions.CsvRequiredFieldEmptyException: Number of data fields does not match number of headers.

编辑

OpenCSV 4.6版

OpenCSV version 4.6

样本记录:

ID1,ID2,FIRST_NAME,LAST_NAME,BIRTH_DATE,HA1,HA2,TYPE,STATUS,DT,LEVEL
3491905454,04572538R,L,L,1964-08-01,RNDC,M4SL,GP  ,DEP,,
3491901894,04353902J,TO,TO,1962-10-20,AMKC,QUAD-L3,GP  ,DE,,
3491903493,01940960Y,JAM"ES,TO,1985-03-12,GRVC,13A,PS  ,DPV,,
8951900652,08661334Z,"ROT,TEST",RODRIGUEZ,1971-09-17,AMKC,1 TOP,GP  ,DE,,
4411801431,02661015Y,CET,TEC,1964-06-21,RNDC,M4NU,GP  ,DE,,
9801900155,06467584H,RAT,BAT,1969-12-01,GRVC,8A,GP  ,DE,,GSL3

第四行和第五行的数据不一致

4th and 5th lines has inconsistant data

推荐答案

1. 添加到CsvToBeanBuilder .withThrowExceptions(false)至 忽略运行时异常
2. 解析bean
3. 调用getCapturedExceptions()以获取所有 在导入过程中会被抛出，但是被抑制了
4. 遍历CsvException数组(解析后)并登录 例外

1. Add to CsvToBeanBuilder .withThrowExceptions(false) to ignore runtime exceptions
2. Parse the bean
3. Invoke getCapturedExceptions() to get all exceptions that would have been thrown during the import, but were suppressed
4. Iterate through CsvException array (after parsing) and log exceptions

请参见下面的代码段:

final CsvToBean<UserProvisioning> beans = 
    new CsvToBeanBuilder<UserProvisioning>(new FileReader("c:\\test.csv"))
        .withType(UserProvisioning.class)
        .withIgnoreQuotations(true)
        .withThrowExceptions(false) //1
        .build();

    final List<UserProvisioning> users = beans.parse();//2
    users.stream().forEach((user) -> {
        logger.info("Parsed data:" + user.toString());
    });

    beans.getCapturedExceptions().stream().forEach((exception) -> { //3
        logger.error("Inconsistent data:" + 
                      String.join("", exception.getLine()), exception);//4
    });

这篇关于如何通过使用OpenCsv记录无效记录的错误来创建有效CSV记录的列表?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！