实施spring-integration-aws的问题 [英] Issues implementing spring-integration-aws

问题描述

我正在使用Spring Integration AWS轮询S3资源并从S3存储桶中获取文件，并使用Spring Integration处理它们. 以下是我所拥有的:

I'm using spring integration aws to poll an S3 resource and get files from the S3 bucket and process them using spring integration. Below is what i have :

AmazonS3 amazonS3 = new AmazonS3Client(new BasicAWSCredentials(accessKey, secretKey));

@Bean
IntegrationFlow fileReadingFlow() {
    return IntegrationFlows
              .from(s3InboundFileSynchronizingMessageSource(),
                      e -> e.poller(p -> p.fixedDelay(30, TimeUnit.SECONDS)))
            .handle(receiptProcessor())
            .get();
}
@Bean
public S3InboundFileSynchronizer s3InboundFileSynchronizer() {
    S3InboundFileSynchronizer synchronizer = new S3InboundFileSynchronizer(amazonS3);
    synchronizer.setDeleteRemoteFiles(false);
    synchronizer.setPreserveTimestamp(true);
    synchronizer.setRemoteDirectory(s3BucketName.concat("/").concat(s3InboundFolder));
    synchronizer.setFilter(new S3RegexPatternFileListFilter(".*\\.dat\\.{0,1}\\d{0,2}"));
    return synchronizer;
}

@Bean
public S3InboundFileSynchronizingMessageSource s3InboundFileSynchronizingMessageSource() {
    S3InboundFileSynchronizingMessageSource messageSource =
            new S3InboundFileSynchronizingMessageSource(s3InboundFileSynchronizer());
    messageSource.setAutoCreateLocalDirectory(false);
    messageSource.setLocalDirectory(new File(inboundDir));
    messageSource.setLocalFilter(new AcceptOnceFileListFilter<File>());
    return messageSource;
}

我的S3存储桶和密钥是:

and my S3 bucket and key is :

bucketName = shipmentReceipts
key = receipts/originalReceipts/inbound/receipt1.dat

因此，我在此实现过程中面临2个问题:
1. inboundDir文件夹名称被重命名为另一个路径名称，并附加了s3key，从而导致FileNotFoundException.我将此跟踪到AbstractInboundFileSynchronizer.java文件中的以下代码:

So i'm facing 2 issues with this implementation:
1. The inboundDir folder name is being renamed to a different path name with the s3key appended to it thus causing a FileNotFoundException. I traced this to the the below code in AbstractInboundFileSynchronizer.java file:

    protected void copyFileToLocalDirectory(String remoteDirectoryPath, F remoteFile, File localDirectory,
        Session<F> session) throws IOException {
    String remoteFileName = this.getFilename(remoteFile);
    String localFileName = **this.generateLocalFileName(remoteFileName);**
    String remoteFilePath = remoteDirectoryPath != null
            ? (remoteDirectoryPath + this.remoteFileSeparator + remoteFileName)
            : remoteFileName;
    if (!this.isFile(remoteFile)) {
        if (this.logger.isDebugEnabled()) {
            this.logger.debug("cannot copy, not a file: " + remoteFilePath);
        }
        return;
    }

    **File localFile = new File(localDirectory, localFileName);**
    if (!localFile.exists()) {........

因此，它最终找到了找不到的文件路径C:\ SpringAws \ S3inbound \ receipts \ originalReceipts \ inbound \ receipt1.dat并给出了FileNotFoundException错误.相反，它应该只是复制到本地文件夹C:\ SpringAws \ S3inbound \ receipt1.dat

So it ends up looking for a file path C:\SpringAws\S3inbound\receipts\originalReceipts\inbound\receipt1.dat which it doesnt find and gives that FileNotFoundExceptionerror. Instead it should just be copying to local folder C:\SpringAws\S3inbound\receipt1.dat

1. 在拉动S3对象时，我注意到它拉动了shipmentReceipts/receipts而不是shipmentReceipts/receipts/originalReceipts/inbound下的所有对象 进一步调试时，我发现S3Session.java中的以下代码段对此负责:

1. While pulling the S3 objects i noticed it was pulling all objects under shipmentReceipts/receipts instead of shipmentReceipts/receipts/originalReceipts/inbound On debugging further i found that the below code snippet in S3Session.java is responsible for it:

@Override
public S3ObjectSummary[] list(String path) throws IOException {
Assert.hasText(path, "'path' must not be empty String.");
String[] bucketPrefix = path.split("/");
Assert.state(bucketPrefix.length > 0 && bucketPrefix[0].length() >= 3,
        "S3 bucket name must be at least 3 characters long.");

String bucket = resolveBucket(bucketPrefix[0]);

ListObjectsRequest listObjectsRequest = new ListObjectsRequest()
        .withBucketName(bucket);
if (bucketPrefix.length > 1) {
    **listObjectsRequest.setPrefix(bucketPrefix[1]);**
}

/*
For listing objects, Amazon S3 returns up to 1,000 keys in the response.
If you have more than 1,000 keys in your bucket, the response will be truncated.
You should always check for if the response is truncated.
*/
ObjectListing objectListing;
List<S3ObjectSummary> objectSummaries = new ArrayList<>();
do {......

它将遇到的第一个正斜杠/之后的所有内容设置为前缀. 我该如何缓解这些问题?谢谢！

It sets the prefix to everything after the first forward slash / it encounters. How do i mitigate these? Thanks!

推荐答案

对嵌套路径的首要关注是一个已知问题，并且已在最新的5.0 M3中修复:

The first concern for nested path is a know issue and has been fixed in the latest 5.0 M3: https://spring.io/blog/2017/04/05/spring-integration-5-0-milestone-3-available with the RecursiveDirectoryScanner.

同时您必须将LocalFilenameGeneratorExpression指定为:

Expression expression = PARSER.parseExpression("#this.contains('/') ? #this.substring(#this.lastIndexOf('/') + 1) : #this");
synchronizer.setLocalFilenameGeneratorExpression(expression);

S3ObjectSummary包含key作为没有bucket的完整路径.

The S3ObjectSummary contains key as a full path without a bucket.

第二个嵌套路径"问题已通过以下方式解决: https ://github.com/spring-projects/spring-integration-aws/issues/45 .该修复程序可在1.1.0.M1中找到:

The second "nested path" issues has been fixed via: https://github.com/spring-projects/spring-integration-aws/issues/45. The fix is available in the 1.1.0.M1: https://spring.io/blog/2017/03/09/spring-integration-extension-for-aws-1-1-0-m1-available

这篇关于实施spring-integration-aws的问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！