通过microsoft技术将avro文件数据转储到SQL数据库表中 [英] Dump avro file data into SQL database table by microsoft technology

问题描述

我有AVRO文件，其中包含二进制格式的数据。我想从AVRO文件读取数据到SQL服务器数据库。我需要一个Microsoft解决方案，所以请帮助我。

I have AVRO file which contains data in binary format. I want to read that data from AVRO file to SQL server database. I need a Microsoft solution, so please help me.

推荐答案

假设您希望存储任意大小的文件（二进制数据），您需要声明一个字段大到足以容纳数据。一旦你读完文件，你还需要能够将数据块传输到字段中。



声明类型的字段并且您可以存储最多2GB的二进制数据

声明您在过程/函数中使用的类型，用于处理数据 varbinary（MAX）。



如果数据块（文件内容）的大小超过8KB，那么仅将文件中的数据作为varbinary（MAX）类型移动是绝对必要的。如果它通过另一种类型，它将被截断为8KB。



如果您的MS SQL Server不支持使用MAX大小声明传输大块数据，那么你需要使用指针（正在逐步淘汰）。

Assuming you wish to store an arbitrary sized file (of binary data), you'll need to declare a field large enough to hold the data. You'll also need to be able to transfer the block of data into the field once you've read the file.

Declare the field of type IMAGE and you can store up to 2GB of binary data
Declare the type you use in your procedures/function that handle the data as varbinary(MAX).

If the size of your data block (file content) exceeds 8KB then it is absolutely essential that you only move the data from the file as varbinary(MAX) types. Should it pass through another type it will be truncated to 8KB.

If your MS SQL Server doesn't support the transfer of large blocks of data using MAX size declaration then you'll need to use pointer (which are being phased out).

非常感谢Balboos的回复。但我的问题不是存储任意大小的文件。它与大小或二进制数据无关。

实际上AVRO是Apache的Hadoop项目中开发的远程过程调用和数据序列化框架。它使用JSON来定义数据类型和协议，并以紧凑的二进制格式序列化数据。

所以在Microsoft环境中，我有通过Microsoft Azure系统上传的AVRO文件。

所以我的问题陈述是如何将存储在AVRO文件中的数据提取到SQL数据库表中。我期待一些程序或代码可以从AVRO文件中读取数据并将其存储在SQL表中。

Thank you very much Balboos for your reply. But my question was not about storing arbitrary sized file. It doesn't matter with size or binary data.
Actually AVRO is a remote procedure call and data serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format.
So In Microsoft environment, I have AVRO files which are uploaded over Microsoft Azure system.
So my problem statement is that how can I fetch that data which is stored in AVRO files into a SQL database table. I am expecting some procedures or codes which can read the data from AVRO file and store it in SQL table.

这篇关于通过microsoft技术将avro文件数据转储到SQL数据库表中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！