将数据从内部SQL Server复制为Parquet文件到Azure Blog存储 [英] Copy data from on-premise SQL Server to Azure Blog storage as parquet file

问题描述

人们

将本地SQL Server表中的数据复制到Azure Blob存储上的镶木文件中时出现问题，如下所示:

I got a problem when copying data from my on-premise SQL server table into a parquet file on Azure blob storage as follow:

{"errorCode":"2200"，"message":失败发生在接收器"端. ErrorCode = UserErrorJavaInvocationException，'Type = Microsoft.DataTransfer.Common.Shared.HybridDeliveryException，Message = An 调用Java时发生错误，消息:java.lang.UnsatisfiedLinkError:java.library.path \ n总条目中没有snappyjava:18 \ r \ njava.lang.ClassLoader.loadLibrary(未知源)\ r \ njava.lang.Runtime. loadLibrary0(未知源)\ r \ njava.lang.System.loadLibrary(未知 源)\ r \ norg.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:170)\ r \ norg.xerial.snappy.SnappyLoader.load(SnappyLoader.java:145)\ r \ norg.xerial.snappy.Snappy .< clinit>(Snappy.java:47)\ r \ norg.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:67)\ r \ norg.apache.hadoop.io.compress.CompressorStream. compress(CompressorStream.java:81)\ r \ norg.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92)\ r \ norg.apache.parquet.hadoop.CodecFactory $ BytesCompressor.compress(CodecFactory. java:112)\ r \ norg.apache.parquet.hadoop.ColumnChunkPageWriteStore $ ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:89)\ r \ norg.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:152) \ r \ norg.apache.parquet.column.impl.ColumnWriterV1.flush(ColumnWriterV1.java:240)\ r \ norg.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:126)\ r \ norg .apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore (InternalParquetRecordWriter.java:164)\ r \ norg.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:113)\ r \ norg.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:297)\ r \ ncom.microsoft.datatransfer.bridge.parquet.ParquetWriterBridge.close(ParquetWriterBridge.java:29)\ r \ n，Source = Microsoft.DataTransfer.Common，''Type = Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException，Message =，源= Microsoft.DataTransfer.Richfile.HiveOrcBridge，"， "failureType":"UserError"，"target":"Copy Data1" }

{ "errorCode": "2200", "message": "Failure happened on 'Sink' side. ErrorCode=UserErrorJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.lang.UnsatisfiedLinkError:no snappyjava in java.library.path\ntotal entry:18\r\njava.lang.ClassLoader.loadLibrary(Unknown Source)\r\njava.lang.Runtime.loadLibrary0(Unknown Source)\r\njava.lang.System.loadLibrary(Unknown Source)\r\norg.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:170)\r\norg.xerial.snappy.SnappyLoader.load(SnappyLoader.java:145)\r\norg.xerial.snappy.Snappy.<clinit>(Snappy.java:47)\r\norg.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:67)\r\norg.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81)\r\norg.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92)\r\norg.apache.parquet.hadoop.CodecFactory$BytesCompressor.compress(CodecFactory.java:112)\r\norg.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:89)\r\norg.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:152)\r\norg.apache.parquet.column.impl.ColumnWriterV1.flush(ColumnWriterV1.java:240)\r\norg.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:126)\r\norg.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:164)\r\norg.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:113)\r\norg.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:297)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetWriterBridge.close(ParquetWriterBridge.java:29)\r\n,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'", "failureType": "UserError", "target": "Copy Data1" }

我的ADF在东南亚地区，Azure Blob存储也在东南亚地区.  我我不确定是否有人遇到过类似问题.如果您能分享您的想法，我将非常感激 解决方法或解决方案.

My ADF is in Southeast Asia region and so does the Azure Blob storage. I am not sure if anyone have encountered such similar issue. Really appreciate it if you could share your workaround or solution to fix it.

注意事项

Di Truong

推荐答案

您的自托管IR版本是什么?是否将SQL身份验证或Windows身份验证用于SQL Server链接服务?

What is your selfhosted IR version? Are use using SQL auth or Windows auth for your sql server linked service?

也许您可以尝试升级您的IR或尝试使用SQL身份验证.

Maybe you could try to upgrade your IR or try to use SQL auth. 

这篇关于将数据从内部SQL Server复制为Parquet文件到Azure Blog存储的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！