一般ETL原则 - ETL从未知数量的表？ [英] General ETL principle - ETL from unknown number of tables?

问题描述

AFAIK，ETL集成来自固定数量的源/表的数据。考虑到这些表可能是不同类型（客户，运输，员工，会计等），ETL是否可以整合未来可能添加的未知数量的表中的数据？

EDIT -

让A，B，C成为表格。我们可以有以下ETL场景 - A + B + C = 1行或A + B + C = 3行

我有第一个。后来，我们可以添加表D，E，F，G ...等，所有这些可能在它们包含的数据方面彼此不同。

在某些自定义数据库中预定义转换效率没有预先定义转换的效率。 SSIS反正。优点是已经构建了SSIS平台！

您需要在某个时间定义转换，并且您无法评估这些转换将会直到您可以看到源，目标和业务规则。

您可能会对复制感到困惑。在这种情况下，没有变换。您只是通过ETL复制相同的表。例如，如果表D，E，F，G简单需要被相同地复制没有变换，那么对于大多数复制平台，它相对简单，只需添加表而不需要做任何变换。

BIML是一种自动生成SSIS包的方法。也许你应该做这方面的一些研究。

AFAIK, ETL integrates data from a fixed number of sources/tables. Is it possible that ETL can integrate data from an unknown number of tables which might be added in the future, given that these tables might be of different types (customer, shipping, employee, accounting etc.) ?

EDIT -

Let A,B,C be tables. We can have the following ETL scenarios - A + B + C = 1 row or A + B + C = 3 rows

I have the first one. Later, we can add tables D,E,F,G...etc all of which might be different from each other in terms of the data they contain.

解决方案

There's no way around defining the transformations.

There is no efficiency in pre-defining transformations in some custom database vs pre defining transformations in SSIS anyway. The advantage is that the SSIS platform has already been built!

You need to define the transformations at some point and you cannot evaluate what these will be until you can see the source, the destination, and the business rules.

You might be getting confused with replication. In this case there are no transformations. You are just replicating an identical table via ETL. For example if tables D,E,F,G simple need to be replicated identically with no transformations then with most replication platforms its relatively simple to just add the table without needing to do any transformations.

BIML is a way of automatically generating SSIS packages. Perhaps you should do some research on this.

这篇关于一般ETL原则 - ETL从未知数量的表？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！