在Hive中逐渐改变尺寸 - SCD1和SCD2的实现 [英] Slowly changing dimensions- SCD1 and SCD2 implementation in Hive

问题描述

我正在寻找Hive（1.2.1）中的SCD1和SCD2实现。我知道在Hive（0.14）之前加载SCD1和SCD2表的解决方法。以下是用解决方法加载SCD1和SCD2的链接 http：/ / /hortonworks.com/blog/four-step-strategy-incremental-updates-hive/

现在Hive支持ACID操作，只是想知道是否存在是一种更好的或直接的加载方式。

解决方案

由于HDFS是不可变的存储，因此可能会认为版本控制数据和历史记录（SCD2）应该是加载尺寸的默认行为。您可以在您的Hadoop SQL查询引擎（Hive，Impala，Drill等）中创建一个视图，使用窗口函数检索当前状态/最新值。你可以找到更多关于 Hadoop上的三维模型在我的博客文章中，例如如何处理大型维度和事实表。

I am looking for SCD1 and SCD2 implementation in Hive (1.2.1). I am aware of the workaround to load SCD1 and SCD2 tables prior to Hive (0.14). Here is the link for loading SCD1 and SCD2 with the workaround approach http://hortonworks.com/blog/four-step-strategy-incremental-updates-hive/

Now that Hive supports ACID operations just want to know if there is a better or direct way of loading it.

解决方案

As HDFS is immutable storage it could be argued that versioning data and keeping history (SCD2) should be the default behaviour for loading dimensions. You can create a View in your Hadoop SQL query engine (Hive, Impala, Drill etc.) that retrieves the current state/latest value using windowing functions. You can find out more about dimensional models on Hadoop in my blog post, e.g. how to handle a large dimension and fact table.

这篇关于在Hive中逐渐改变尺寸 - SCD1和SCD2的实现的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！