使用自动加载器从AWS S3加载到Azure Datalake时的增量表事务保证 [英] Delta Table transactional guarantees when loading using Autoloader from AWS S3 to Azure Datalake

问题描述

尝试使用自动加载器，其中AWS S3是源，而Delta Lake在Azure Datalake Gen中.当我尝试读取文件时，出现以下错误

Trying to use autoloader where AWS S3 is source and Delta lake is in Azure Datalake Gen. When I am trying to read files it gives me following error

从非AWS写入AWS上的Delta表在提供交易担保方面是不安全的.如果可以保证没有其他人会同时修改同一个Delta表，则可以通过设置SparkConf来关闭此检查:启动集群时为false.

尝试在集群级别设置设置，并且工作正常.我的问题是，有什么方法可以确保将交易数据从AWS3加载到Azure Datalake(Datalake是Delta Lake的后端存储).我们不想将"spark.databricks.delta.logStore.crossCloud.fatal"设置为在集群级别.如果这样做会不会有任何问题，这对于生产ETL管道将是一个好的解决方案吗?

Tried setting up settings at cluster level and it works fine. My question is, is there any way we can ensure transactional guarantee wile loading data from AWS3 to Azure Datalake (Datalake is backend storage for our Delta Lake). We don't want to set "spark.databricks.delta.logStore.crossCloud.fatal" at Cluster level. Will there be any issue if we do and will it be a good solution for a production ETL pipeline?

推荐答案

Databricks检测到您正在执行多云工作时，将显示此警告.但是此警告适用于当您使用Delta 使用写入AWS S3时的情况，因为AWS没有原子写入操作(例如，如果没有放置的话)，因此它需要某种协调器仅在AWS上可用的过程.

This warning appears when Databricks detects that you're doing the multicloud work. But this warning is for case when you're writing into AWS S3 using Delta, because AWS doesn't have atomic write operation (like, put-if-absent), so it requires some kind of coordinator process that is available only on AWS.

但是在您的情况下，您可以忽略此消息，因为您只是从AWS S3中读取数据，并写入Azure Datalake上的Delta.

But in your case you can ignore this message because you're just reading from AWS S3, and writing into Delta that is on Azure Datalake.

这篇关于使用自动加载器从AWS S3加载到Azure Datalake时的增量表事务保证的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！