将S3增量复制到天蓝色斑点 [英] Incrementally copy S3 to azure blob

问题描述

我正在探索将S3 Blob增量复制到天蓝色Blob的方法.我尝试使用azure数据工厂，但找不到此选项.

I am exploring ways to incrementally copy S3 blobs to azure blobs. I tried azure data factory and I was not able to find this option.

S3具有数百万个对象，并且没有增量选项，则需要数小时才能完成.

S3 has millions of objects and without an incremental option it takes hours to complete.

我愿意探索其他工具/选项.

I am open to explore other tools/options.

推荐答案

如果您需要每天/每小时进行增量复制，请滚动窗口触发器.

if you need to do daily/hourly incrementally copy, the scheduled/tumbling window trigger should be the option. Here is an example to reference. ADFv2 also supports compression in copy, you could specify the type and level of compression for the s3 object, click here for more information.

这篇关于将S3增量复制到天蓝色斑点的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！