容错并在Azure Blob存储中记录不兼容的行 [英] Fault tolerance and log the incompatible rows in Azure Blob storage
问题描述
客户要求:
-
使用azure数据工厂将blob存储中的csv文件导入到SQL数据仓库中.
using use azure data factory to import csv file in blob storage to SQL data warehouse.
使用ADF中的策略容错并在Azure Blob存储中记录不兼容的行".
using the strategy "Fault tolerance and log the incompatible rows in Azure Blob storage" in ADF.
并使用Azure Function将处理后的文件存档到Blob存储中的其他位置:成功导入了这些文件的一个位置，而失败存储了一个位置(这些文件的数据不兼容-格式错误，长度错误)
And using Azure Function to archive the processed file to other place in blob storage: one place for those files are imported successfully and one for fail files (the files have incompatible data - wrong format, wrong length)
=> 所以我需要获取活动窗口"的skippedRowCount的值才能知道此活动具有一些不兼容的行吗?有什么方法可以解决我的问题吗?非常感谢.
推荐答案
在ADF V2中，跳过的行数作为复制活动输出的"rowsSkipped"属性返回.请参阅以下两个链接: https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview#monitoring 和
In ADF V2, the number of skipped rows is returned as "rowsSkipped" property of copy activity output. See these two links: https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-overview#monitoring and https://docs.microsoft.com/en-us/azure/data-factory/copy-activity-fault-tolerance#monitor-skipped-rows
ADF V2还允许您使用类似"@activity('MyCopyActivity').output.rowsSkipped)"的表达式在后续活动中使用上一个复制活动的输出.输出Lookup活动，您就可以适应您的特定情况.
ADF V2 also allows you to use the output from a previous copy activity in the subsequent activity, using an expression like "@activity('MyCopyActivity').output.rowsSkipped")". Here is an example of how to use output form Lookup activity, and you can adapt to your particular situation.
对于您的用例，您可以将复制活动与两个Web活动链接在一起，一个活动调用成功的文件档案，另一个活动记录/重新处理在Storage Blob或ADLS中记录的失败行.
For you use case, you can chain the copy activity with two Web activities, one to invoke the file archive for successful files, and another to record/reprocess failed rows logged in Storage blob or ADLS.
这篇关于容错并在Azure Blob存储中记录不兼容的行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
