使用ADF v2审核/记录ELT / ETL的Famework [英] Auditing/Logging Famework for ELT/ETL using ADF v2
问题描述
尊敬的社区老百姓，
我知道这个问题的范围有点广泛，不能仅限于此。对于ADF，但我认为在集成解决方案论坛而不是个人服务论坛上推销它仍然更有意义。它可能是一个冗长的，请耐心等待我
：）
I Know the question is bit broader in scope and cannot be limited purely to ADF , but i think it still makes more sense to pitch it in integration solutions forum rather than individual service forum.It might be a lengthy one , please bear with me :)
所以我有一个 几百个 ADFv2通过从结构化和非结构化存储中获取文件和数据并放入blob来执行ELT的管道。发布有相同的转换阶段应用于同一个 原始数据和最后
数据复制到ADW和AZ SQL服务器和AAS。
So i have a couple of Hundred ADFv2 Pipelines which do the ELT by grabbing files and data from structured and unstructured stores and putting into blobs. Post that there are phases of transformations being applied on same raw data and finally data copied to ADW and AZ SQL servers and AAS.
现在我想建立一个自定义日志/审计框架，它可以向我展示整个世界的血统执行活动的流程，成功和失败的自定义消息以及每个操作中受影响的数据行的信息 和 定制 
状态和阶段信息。请记住，我的管道正在触发复制活动，数据砖活动，SQL服务器存储过程，并从单个管道中调用其他管道，我想跟踪整个ELT过程通过使用日志信息单个地方
。
Now i want to build a custom logging/auditing framework which can show me the lineage of the whole process with activities performed, custom message on success and failure and the information about data rows affected in each operation and custom status and phase information.Bear in mind, that my pipeline is triggering copy activity , data bricks activity , SQL server stored procs , and call other pipelines from within a single pipeline and i want to track the whole ELT process from a single place by using the logging info.
关于如何实现这一点的任何想法？我一直在考虑通过创建自定义例程将所有这些信息记录到AZsql服务器上。在每个服务中 写 数据到azsql服务器。但是任何更好的方法都会非常有用。
Any idea on how to achieve this? i have been thinking about logging all of this info to a AZsql serer by creating custom routine in each of the service which writes data to azsql server. But any better approach would be really helpful.
Azure OMS Log Analytics可以帮助我吗？但如何做自定义消息中继？
Can Azure OMS Log Analytics help me? but how to do custom message relays?
提前致谢
Viswa
Viswa
推荐答案
Azure Monitor / Log Analytics支持自定义日志记录。
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-custom-logs
Azure Monitor / Log Analytics does support custom logging.
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-sources-custom-logs
这篇关于使用ADF v2审核/记录ELT / ETL的Famework的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
