使用 S3 作为 AWS DMS 的目标:上传的文件名不会改变 [英] Using S3 as target for AWS DMS: Uploaded File name doesn't change

问题描述

我们正在使用 DMS 从 SQL Server 获取数据并将其加载到 S3 存储桶中，然后使用 Snowpipe 将数据最终加载到 Snowflake DB 中以进行完全加载.

We are using DMS to get data from SQL Server and load it in S3 bucket, after which the data is finally loaded into Snowflake DB using Snowpipe for Full Load.

现在，为了让 Snowpipe 知道 S3 存储桶中有新数据，文件名需要与上一个不同.已尝试所有可用的任务设置选项(DROP_AND_CREATE、DO_NOTHING、TRUNCATE)使文件名不同，但仍然无法正常工作.它将文件名加载为 LOAD00000001.csv

Now, in order for Snowpipe to know there is new data in S3 bucket, the filename needs to be different than the last one. Have tried all the task setting options available (DROP_AND_CREATE, DO_NOTHING, TRUNCATE) to have the file name different, but still not working. It loads the file name as LOAD00000001.csv

在文档中，它显示文件名将是增量的(例如 LOAD00000001.csv、LOAD00000002.csv .. 等等)，但它没有发生.这就是 Snowpipe 无法注册更改的原因.

In documentation it shows that file name will be incremental (eg. LOAD00000001.csv, LOAD00000002.csv .. and so on) but it's not happening. Which is why the Snowpipe is not able to register the changes.

https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Target.S3.html

有人可以帮忙吗?

推荐答案

对于 DMS，每次运行任务时，增量计数器都会从 1 重新开始.它没有不覆盖现有对象"功能.

For DMS the incremental counter is started over from 1 each time the task is run. It does not have a "Don't override existing objects" feature.

最好的办法可能是通过在文件夹中查找更新的对象时间戳或设置 S3 事件通知来自己处理负载.

Your best bet may be to handle the load yourself by looking for updated object timestamps in your folder or setting up S3 event notifications.

这篇关于使用 S3 作为 AWS DMS 的目标:上传的文件名不会改变的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！