从平面文件或Azure Analysis Services中的文件中增量加载数据 [英] Incremental loading of data from flat file or files in Azure Analysis Services

问题描述

我有一个要求，要在Azure Blob顶部使用Azure Analysis Services.我需要实现一些方案:

I have a requirement where in Azure Analysis services is used on top of Azure blobs. I need to implement few scenarios:

(1)仅从Blob容器中的平面文件增量加载新记录或更改记录.我们怎样才能做到这一点?将处理添加"在平面文件上工作?

(1) Incremental loading of new records or changed recorded only from a flat file in Blob container. How can we achieve this? Will "Process Add" work on flat files?

(2)在Blob容器中存在多个文件的情况下.我只需要处理容器中最新的那些文件.考虑即将到来的文件名称中带有时间戳.我不想从那些文件中读取数据 已经处理.我需要只读的最新文件.无法从Blob中删除旧文件.另外，从旧文件读取的数据必须保留在AAS模型中

(2) There can be a scenario when multiple files are present in Blob container. I just need to process those files which are latest in the container. Consider files coming have timestamp in their name. I don't want to read data from those files which are already processed. I need latest files to be read only. Can't remove the old files from Blob. Also, the data which was read from old file has to stay in AAS model

请让我知道以前是否有人这样做?

Please let me know if anybody has done this before?

推荐答案

(1)增量加载新记录或更改的记录仅来自Blob容器中的平面文件.我们怎样才能做到这一点?将处理 添加"在平面文件上工作?

(1) Incremental loading of new records or changed recorded only from a flat file in Blob container. How can we achieve this? Will "Process Add" work on flat files?

[答案]:如果我没错，如果不支持，您可以将数据附加到平面文件中(顺便说一下，您使用的是哪个etl工具) 开箱即用etl工具，然后编写自定义代码以执行该操作.

[Ans]: If i am not wrong you can append the data into the flat files (btw which etl tool you are using for it), if it is not supported out of the box by the etl tool then write a custom code to perform the operation.

(2) Blob容器中存在多个文件的情况.我只需要处理那些最新的文件 容器.考虑即将到来的文件名称中带有时间戳.我不想从那些已经处理过的文件中读取数据.我需要只读的最新文件.无法从Blob中删除旧文件.另外，从旧文件读取的数据具有 保持AAS模式

(2) There can be a scenario when multiple files are present in Blob container. I just need to process those files which are latest in the container. Consider files coming have timestamp in their name. I don't want to read data from those files which are already processed. I need latest files to be read only. Can't remove the old files from Blob. Also, the data which was read from old file has to stay in AAS model

[Ans]:数据不需要在同一文件夹(在这种情况下为旧文件)中存在正在处理的文件(在MOLAP存储中) 它们放在其他文件夹(称为存档)中，因此您始终可以拥有单个文件，并且可以对多维数据集进行处理添加/更新.

[Ans]: The data which is being processed (In MOLAP storage) need not to be present in the same folder (old file in this case, you can move them in some other folder say archive), so you can have single file always and you can do process add/update to the cube.

这篇关于从平面文件或Azure Analysis Services中的文件中增量加载数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！