使用Azure函数和C＃调用REST API并将结果保存在Azure Data Lake gen2中 [英] Using Azure Functions with C# to call REST API and save results in Azure Data Lake gen2

问题描述

嗨！

我想调用rest api并将结果保存为Azure Data Lake Gen2中的csv或json文件。基于我所读到的，Azure函数是要走的路。

webservice返回如下格式的数据：

< pre class ="lang-cs prettyprint"style ="padding：12px 8px; border-width：0px; border-style：initial; border-color：initial; line-height：inherit; font-family：Consolas，Menlo，Monaco ，'Lucida Console'，'Liberation Mono'，'DejaVu Sans Mono'，'Bitstream Vera Sans Mono'，'Courier New'，monospace，sans-serif; font-size：13px; vertical-align：baseline; width：auto ; max-height：600px; background-color：＃eff0f1; color：＃393318"> "ID"，"产品名称"，"公司"，"b"，"1"，"苹果"，"Alfreds futterkiste"，"b $ b"，"2"，" ; Oranges"，"Alfreds futterkiste"
"3"，"Bananas"，"Alfreds futterkiste"
"4"，"Salad"，"Alfreds futterkiste"
...下一行

我在C＃中编写了一个控制台应用程序，目前将数据输出到控制台.web服务使用分页并返回1000行（由& num-parameter确定，最大值为1000）。在第一个请求之后我可以使用& next-parameter到
根据ID获取接下来的1000行。例如url

  http://testWebservice123.com/Example.csv?auth=abc&number=1000&next=1000  

将从ID 1001到2000获取行。（API的调用和实际中的分页有点复杂，因此我不能使用例如Azure Data Factory_v2来执行加载到Azure Data Lake - 这就是为什么我认为我需要Azure功能 - 除非
我忽略了另一个服务器？所以以下只是一个学习如何写入Azure Data Lake的演示。）

我有以下C＃：

  static void Main（string [] args）
 {
 
 
 string startUrl =" http://testWebservice123.com/Example.csv？ auth = abc& number = 1000"; 
 string url ="" ;; 
 string deltaRequestParameter ="" ;; 
 string lastLine; 
 int numberOfLines = 0; 
 
 do 
 {
 url = startUrl + deltaRequestParameter; 
 WebClient myWebClient = new WebClient（）; 
 
 using（Stream myStream = myWebClient。 OpenRead（url））
 {
 
使用（StreamReader sr = new StreamReader（myStream））
 {
 numberOfLines = 0; 
 while（！sr） .EndOfStream）
 {
 var row = sr.ReadLine（）; 
 var values = row.Split（'，'）; 
 
 //做任何事情行数现在 - 即写入控制台
 Console.WriteLine（values [0] +" " +值[1]）; 
 
 lastLine = values [0] .Replace（" \""，""）; //循环中的最后一行 - 获取最后一个ID。 
 numberOfLines ++; 
 deltaRequestParameter ="& next =" + lastLine; 
} 
 
} 
 
} 
} while（numberOfLines == 1001）; //因为每次行数为1001时都会返回标题，直到我们到达最后一个请求为止
 
 
}  

我想以最有效的方式将数据写入csv文件到data-lake。如何重写上述代码以在Azure Function中工作并保存到Azure数据湖gen2中的csv？

解决方案

< blockquote>

以下是实现结果所需的步骤：

1）创建一个azure函数并触发你可以保持HTTPTrigger / TimerTrigger，或根据你的需要。

2）我假设你有代码在循环中调用api直到它给你想要的结果。

3）在内存中存储数据之后，必须编写以下代码才能将其写入Azure数据湖。

使用c＃代码访问ADLS的先决条件：

1）在Azure AD中注册应用程序

数据湖商店的授权许可

以下是创建ADLS客户端的代码

 // ADLS connection 
 var adlCreds = GetCreds_SPI_SecretKey（tenantId，ADL_TOKEN_AUDIENCE ，serviceAppIDADLS，servicePrincipalSecretADLS）; 
 var adlsClient = AdlsClient.CreateClient（adlsName，adlCreds）; 
 
 
 
 private static ServiceClientCredentials GetCreds_SPI_SecretKey（string tenant，Uri tokenAudience，string clientId，string secretKey）
 {
 SynchronizationContext.SetSynchronizationContext（new SynchronizationContext（））; 
 var serviceSettings = ActiveDirectoryServiceSettings.Azure; 
 serviceSettings.TokenAudience = tokenAudience; 
 var creds = ApplicationTokenProvider.LoginSilentAsync（tenant，clientId，secretKey，serviceSettings）.GetAwaiter（）。GetResult（）; 
 return creds; 
} 

最后编写实现以将文件保存在Azure数据湖中

 const string delim ="，" ;; 
 static string adlsInputPath = ConfigurationManager.AppSettings.Get（" AdlsInputPath"）; 
 
 public static void ProcessUserProfile（此SampleProfile，AdlsClient adlsClient，string fileNameExtension =""）
 { 
 using（MemoryStream memStreamProfile = new MemoryStream（））
 {
 using（TextWriter textWriter = new StreamWriter（memStreamProfile））
 {
 string profile; 
 string header = Helper.GetHeader（delim，Entities.FBEnitities.Profile）; 
 string fileName = adlsInputPath + fileNameExtension +" /profile.csv" ;; 
 adlsClient.DataLakeFileHandler（textWriter，header，fileName）; 
 profile = socialProfile .UserID 
 + delim + socialProfile.Profile.First_Name 
 + delim + socialProfile.Profile.Last_Name 
 + delim + socialProfile.Profile.Name 
 + delim + socialProfile.Profile.Age_Range_Min 
 + delim + socialProfile.Profile.Age_Range_Max 
 + delim + socialProfile.Profile.Birthday 
; 
 
 textWriter.WriteLine（profile）; 
 textWriter.Flush（）; 
 memStreamProfile.Flush（）; 
 adlsClient.DataLakeUpdateHandler（fileName，memStreamProfile）; 
} 
} 
} 

希望它有所帮助。

Hi!

I want to call a rest api and save the results as a csv or json file in Azure Data Lake Gen2. Based on what I have read Azure Functions is the way to go.

The webservice returns data like the following format:

"ID","ProductName","Company"
"1","Apples","Alfreds futterkiste"
"2","Oranges","Alfreds futterkiste"
"3","Bananas","Alfreds futterkiste"
"4","Salad","Alfreds futterkiste"
 ...next rows

I have written a console app in C# which at the moment outputs the data to a console. The webservice uses pagination and returns 1000 rows (determined by the &num-parameter with a max of 1000). After the first request i can use the &next-parameter to fetch the next 1000 rows based on ID. For instance the url

http://testWebservice123.com/Example.csv?auth=abc&number=1000&next=1000

will get me rows from ID 1001 to 2000. (the call of the API and the pagination in reality is a bit more complex and thus I cannot use for instance Azure Data Factory_v2 to do the load to Azure Data Lake - this is why I think i need Azure Functions - unless I have overlooked another servic??. So the following is just a demo to learn how to write to Azure Data Lake.)

I have the following C#:

static void Main(string[] args)
    {


        string startUrl = "http://testWebservice123.com/Example.csv?auth=abc&number=1000";
        string url = "";
        string deltaRequestParameter = "";
        string lastLine;
        int numberOfLines = 0;

        do
        {
            url = startUrl + deltaRequestParameter;
            WebClient myWebClient = new WebClient();

            using (Stream myStream = myWebClient.OpenRead(url))
            {

                using (StreamReader sr = new StreamReader(myStream))
                {
                    numberOfLines = 0;
                    while (!sr.EndOfStream)
                    {
                        var row = sr.ReadLine();
                        var values = row.Split(',');

                        //do whatever with the rows by now - i.e. write to console
                        Console.WriteLine(values[0] + " " + values[1]); 

                        lastLine = values[0].Replace("\"", ""); //last line in the loop - get the last ID.
                        numberOfLines++;
                        deltaRequestParameter = "&next=" + lastLine;
                    }

                }

            }
        } while (numberOfLines == 1001); //since the header is returned each time the number of rows will be 1001 until we get to the last request


    }

I want to write the data to a csv-file to the data-lake in the most effective way. How would I rewrite the above code to work in Azure Function and save to a csv in Azure data lake gen2?

解决方案

Here are the steps which you need to do for achieving the result:

1) Create an azure function and trigger you can keep it HTTPTrigger/TimerTrigger, or as per your need.

2) I am assuming you have the code to call api in loop until it gives you desired result.

3) Once you have the Data in memory , you have to write following code to write it in Azure data lake.

Prerequisite for accessing ADLS using your c# code:

1) Register an app in Azure AD

Grant permission in data lake store

Below is the code for creating ADLS client

// ADLS connection 
                var adlCreds = GetCreds_SPI_SecretKey(tenantId, ADL_TOKEN_AUDIENCE, serviceAppIDADLS, servicePrincipalSecretADLS);
                var adlsClient = AdlsClient.CreateClient(adlsName, adlCreds);



private static ServiceClientCredentials GetCreds_SPI_SecretKey(string tenant,Uri tokenAudience,string clientId,string secretKey)
        {
            SynchronizationContext.SetSynchronizationContext(new SynchronizationContext());
            var serviceSettings = ActiveDirectoryServiceSettings.Azure;
            serviceSettings.TokenAudience = tokenAudience;
            var creds = ApplicationTokenProvider.LoginSilentAsync(tenant,clientId,secretKey,serviceSettings).GetAwaiter().GetResult();
            return creds;
        }

Finally write the implementation to save the file in Azure data lake

 const string delim = ",";
        static string adlsInputPath = ConfigurationManager.AppSettings.Get("AdlsInputPath");

public static void ProcessUserProfile(this SampleProfile, AdlsClient adlsClient, string fileNameExtension = "")
        {
            using (MemoryStream memStreamProfile = new MemoryStream())
            {
                using (TextWriter textWriter = new StreamWriter(memStreamProfile))
                {
                    string profile;
                    string header = Helper.GetHeader(delim, Entities.FBEnitities.Profile);
                    string fileName = adlsInputPath + fileNameExtension + "/profile.csv";
                    adlsClient.DataLakeFileHandler(textWriter, header, fileName);
                    profile = socialProfile.UserID                                                
                                    + delim + socialProfile.Profile.First_Name
                                    + delim + socialProfile.Profile.Last_Name
                                    + delim + socialProfile.Profile.Name
                                    + delim + socialProfile.Profile.Age_Range_Min
                                    + delim + socialProfile.Profile.Age_Range_Max
                                    + delim + socialProfile.Profile.Birthday
                                   ;

                    textWriter.WriteLine(profile);
                    textWriter.Flush();
                    memStreamProfile.Flush();
                    adlsClient.DataLakeUpdateHandler(fileName, memStreamProfile);
                }
            }
        }

Hope it helps.

这篇关于使用Azure函数和C＃调用REST API并将结果保存在Azure Data Lake gen2中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！