数据工厂查找大小限制 [英] Data Factory Lookup size limitation

问题描述

我有一个问题要问你。我们目前正在创建一个DF管道，它从Azure SQL数据库读取数据并将其推送到azure blob存储。



我们正在为SQL中的每一行创建blob。如果有是100K行，我们必须创建100k blob并且这些blob被策划并用于Azure搜索中的索引。



我们使用查找活动查看来自SQL的记录和通过foreach活动传递它。但是，Lookup有5000行的限制，因此我们只能创建5000个blob。

是否有任何工作来获取所有记录并获得blob的确切数量并且每次从缓存中读取数据而不是查询数据库。

I have one question for you. We are currently creating a DF pipeline which reads data from Azure SQL database and pushing it to azure blob storage.

We are creating blobs for each row in SQL.If there are 100K rows we have to create 100k blobs and this blobs are curated and used for indexing in Azure Search.

We are using lookup activity to look the records from SQL and passing it through foreach activity.However, Lookup has a limitation of 5000 rows so we are able to create only 5000 blobs.
Is there any work around to get all the records and get the exact count of blobs and read data from cache instead of quering the database each time.

推荐答案

嗨Nayanjyoti，

Hi Nayanjyoti,

如果有更多条目，则API返回客户端需要在后续请求中发送的延续令牌。

If there are more entries, the API returns a continuation token which the client needs to send in a subsequent request.

每次调用返回的路径数量有限。如果要返回的路径数超过此限制，则在响应头x-ms-continuation中返回延续令牌。当在响应中返回延续令牌时，必须在后续调用列表操作中指定
以继续列出路径。

The number of paths returned with each invocation is limited. If the number of paths to be returned exceeds this limit, a continuation token is returned in the response header x-ms-continuation. When a continuation token is returned in the response, it must be specified in a subsequent invocation of the list operation to continue listing the paths.

请注意，如果列表操作交叉一个分区边界，然后服务将返回一个继续令牌，用于检索结果的其余部分。因此，服务可能会返回的结果少于最大结果，
或默认值5000指定的结果。 如果参数设置为小于或等于零的值，则服务器返回状态代码400（错误请求）。

Note that if the listing operation crosses a partition boundary, then the service will return a continuation token for retrieving the remainder of the results. For this reason, it is possible that the service will return fewer results than specified by maxresults, or than the default of 5000.  If the parameter is set to a value less than or equal to zero, the server returns status code 400 (Bad Request).

您可以参考
SO 解决类似问题的线程。

You may refer SO thread which addressing similar issue.

希望这会有所帮助。

这篇关于数据工厂查找大小限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！