如何将数据从SQL Server流式传输到Azure数据仓库? [英] How do I Stream Data from SQL Server into Azure Data Warehouse?

问题描述

Microsoft过程看起来像是将数据从SQL Server复制到Azure数据仓库的批处理导入方法.是否有一种更简单的方法，可以将近乎实时的数据从MS SQL Server直接流式传输到Datawarehouse.这似乎太过分了 与两个过多的ETL步骤(Azure Data Factory，然后是Polybase)并存.我们是否可以简化数据并将数据从SQL Server连续流传输到数据仓库中? (也许将表1到1尽可能关闭，我们不打算创建Dim或事实)

The Microsoft process looks like a batch import method of copying data from SQL Server into Azure Data Warehouse. Is there a simpler method, conducting close to real time of streaming data from MS SQL Server directly into Datawarehouse. This seems overly complicated with two excessive ETL steps, (Azure Data Factory, and then Polybase) . Can we simplify and continually stream data from SQL Server into Data Warehouse? (maybe map tables 1 to 1 close as possible, we do Not plan on creating Dim or facts)

我们知道AWS允许将数据从SQL Server流式传输到Redshift DW

We know AWS allows streaming of data from SQL server into Redshift DW

https://aws.amazon.com/about-aws/whats-new/2017/05/aws-schema-conversion-tool-exports-from-sql-server-to-amazon-redshift/

https://aws.amazon.com/about-aws/whats-new/2017/05/aws-schema-conversion-tool-exports-from-sql-server-to-amazon-redshift/

推荐答案

您好，

上图是较高级别，没有传达某些提取细节.使用ADF将数据加载到Azure SQL数据仓库时，它将使用Polybase.请参考本文的两个用例(使用Polybase和 使用Polybase进行分阶段复制):https://docs.microsoft.com/zh-cn/azure/data-factory/connector-azure-sql-data-warehouse#use-polybase-to-load-data-into- Azure SQL数据仓库

The graph above is high level and does not convey some of the ingestion details.  When using ADF to load data into Azure SQL Data Warehouse, it will use Polybase.  Please refer to this article on the two use cases (direct copy using Polybase and staged copy using Polybase): https://docs.microsoft.com/en-us/azure/data-factory/connector-azure-sql-data-warehouse#use-polybase-to-load-data-into-azure-sql-data-warehouse

您还有第二个要求，那就是持续流式传输-您能具体说明一下这是什么意思吗?您是在寻找低调度时间间隔(5分钟?1分钟?甚至更低?)还是以推送模式而不是计划拉动模式?

You also have a second requirement which is continuously streaming - can you please be more specific what you mean by that?  Are you looking for low schedule interval (5min?  1min?  Even lower?) or a push mode instead of scheduled pull mode?

这篇关于如何将数据从SQL Server流式传输到Azure数据仓库?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！