SFTP连接器和foreach [英] SFTP connector and foreach

问题描述

我有一个要求，我必须从外部SFTP位置向Azure blob存储区传输大约50,000个文件，每个文件少于2 MB。我最多可以有10个并发连接到SFTP。所以我按照下面的工作流程，

I have a requirement where i have to transfer about 50,000 files each less than 2 MB each from an external SFTP location to Azure blob storage. I can have up to 10 concurrent connections to the SFTP. so i am following the workflow as below,

1）列出文件夹中的文件

1) list files in the folder

2）foreach（Body）---在程度下并行选择10

2) foreach(Body) ---under degree of parallelism selected 10

a）使用路径获取文件内容

a) get file content using path

b）创建blob（blob内容文件内容）

b) create blob(blob content-file content)

面临的问题：

1）转移时间非常长，如9小时

1) The transfer is taking very long like 9 hours

2）有时我得到429（超出限额）

2) sometimes i am getting 429 (Rate limit exceeded)

有人可以帮我解决一下我缺少的东西吗？

can somebody help me here on what am i missing?

推荐答案

由于它们只是2 MB的文件，你实际上可能会对连接器施加限制限制 - 主要是对于

blob 和
sftp 。并不是因为上传本身需要时间。

Since they are just 2 MB files, you might be actually hitting throttling limits on the connectors - mostly for both blob and sftp. Not that its taking time for the upload itself.

另外，我不认为Logic Apps是最好的解决方案。你可以使用
$ b您的SFTP服务器上的$ b AzCopy 实用程序会更快。

Also, I don't think Logic Apps is the best solution here. You could just use the AzCopy utility on your SFTP server which would be much faster.

即使您希望这是定期同步的内容，您最好还是复制首先是所有现有文件，然后设置

SFTP触发 Logic App以复制任何新文件，然后进入。

Even if you want this to be something that you sync in a regular basis, you are better off copying all the existing files first and then setup a SFTP Triggered Logic App to copy any new files as and when then come in.

这篇关于SFTP连接器和foreach的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！