Spring Batch-从Aws S3读取文件 [英] Spring Batch - Read files from Aws S3

问题描述

我正在尝试从AWS S3读取文件并使用Spring Batch处理它:

I am trying to read files from AWS S3 and process it with Spring Batch:

Spring Itemreader可以处理此任务吗? 如果是这样，我如何将凭据传递到S3客户端并配置spring xml以读取一个文件或多个文件

Can a Spring Itemreader process this Task? If so, How do I pass the credentials to S3 client and config my spring xml to read a file or multiple files

<bean id="itemReader" class=""org.springframework.batch.item.file.FlatFileItemReader"">
    <property name="resource" value=""${aws.file.name}"" />
    </bean>

推荐答案

更新要使用Spring-cloud-AWS，您仍然可以使用FlatFileItemReader，但现在无需进行自定义扩展资源.

Update To use the Spring-cloud-AWS you would still use the FlatFileItemReader but now you don't need to make a custom extended Resource.

相反，您设置aws上下文并将其提供给您的S3Client bean.

Instead you set up a aws-context and give it your S3Client bean.

    <aws-context:context-resource-loader amazon-s3="amazonS3Client"/>

阅读器将像其他阅读器一样进行设置-唯一的独特之处在于，您现在可以自动连接ResourceLoader

The reader would be set up like any other reader - the only thing that's unique here is that you would now autowire your ResourceLoader

@Autowired
private ResourceLoader resourceLoader;

，然后设置该资源加载器:

and then set that resourceloader:

@Bean
public FlatFileItemReader<Map<String, Object>> AwsItemReader() {
    FlatFileItemReader<Map<String, Object>> reader = new FlatFileItemReader<>();
    reader.setLineMapper(new JsonLineMapper());
    reader.setRecordSeparatorPolicy(new JsonRecordSeparatorPolicy());
    reader.setResource(resourceLoader.getResource("s3://" + amazonS3Bucket + "/" + file));
    return reader;
}

---

我将使用FlatFileItemReader，需要进行的自定义是制作您自己的S3资源对象.扩展Spring的AbstractResource以创建您自己的AWS资源，其中包含AmazonS3客户端，存储桶和文件路径信息等.

---

I would use the FlatFileItemReader and the customization that needs to take place is making your own S3 Resource object. Extend Spring's AbstractResource to create your own AWS resource that contains the AmazonS3 Client, bucket and file path info etc..

对于getInputStream，请使用Java SDK:

For the getInputStream use the Java SDK:

        S3Object object = s3Client.getObject(new GetObjectRequest(bucket, awsFilePath));
        return object.getObjectContent();

然后显示contentLength-

Then for contentLength -

return s3Client.getObjectMetadata(bucket, awsFilePath).getContentLength();

和lastModified使用

and lastModified use

.getLastModified().getTime();

您创建的资源将具有AmazonS3Client，其中包含您的spring-batch应用程序与S3通信所需的所有信息.这就是Java配置的样子.

The Resource you make will have the AmazonS3Client which contains all the info your spring-batch app needs to communicate with S3. Here's what it could look like with Java config.

    reader.setResource(new AmazonS3Resource(amazonS3Client, amazonS3Bucket, inputFile));

这篇关于Spring Batch-从Aws S3读取文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！