使用Azure数据工厂将压缩的XML文件从HTTP链接源复制和提取到Azure Blob存储 [英] Copy and Extracting Zipped XML files from HTTP Link Source to Azure Blob Storage using Azure Data Factory

问题描述

我正在尝试建立Azure Data Factory复制数据管道.源是开放的HTTP链接源(网址参考:

请注意，ADF解压缩.xml文件花了很长时间，因为其中有很多文件.我的结果显示在Azure存储资源管理器中:

I am trying to establish an Azure Data Factory copy data pipeline. The source is an open HTTP Linked Source (Url reference: https://clinicaltrials.gov/AllPublicXML.zip). So basically the source contains a zipped folder having many XML files. I want to unzip and save the extracted XML files in Azure Blob Storage using Azure Data Factory. I was trying to follow the configurations mentioned here: How to decompress a zip file in Azure Data Factory v2 but I am getting the following error:

ErrorCode=UserErrorSourceNotSeekable,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Your HttpServer source can't support random read which is requied by current copy activity setting, please create two copy activities to work around it: the first copy activity binary copy your HttpServer source to a staging file store(like Azure Blob, Azure Data Lake, File, etc.), second copy activity copy from the staged file store to your destination with current settings.,Source=Microsoft.DataTransfer.ClientLibrary,'

Not exactly sure what is going wrong, but it would be really helpful if someone can guide me with the procedure.

解决方案

I broke this up in to two Copy data activities in order to separate the donwloading of the zip file (which is quite large) and the unpacking. You could try and do them in one step but I think you're going to run into timeout issues. With my approach you also have a copy of the original zip file which would be good for audit trail and debugging purposes.

I try and document my ADF patterns in a boxes and lines format which shows the key details for each component. So here there are two Copy activities, and the supporting linked services and datasets - try and follow this, let me know how you get on:

NB it took quite a long time for ADF to unpack the .xml files as there are rather a lot of them. My results showing in Azure Storage Explorer:

这篇关于使用Azure数据工厂将压缩的XML文件从HTTP链接源复制和提取到Azure Blob存储的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！