如何组合/加速多个API调用以提高性能？ [英] How can I combine/speed up multiple API calls to improve performance?

问题描述

更新：我找到了可能有用的东西，但我仍然遇到了一些麻烦，找出如何实施它。如果我尝试像这样映射get_data，我不知道如何将每个调用的结果分配给相应的变量。

  [service，profile_id，'30daysAgo'，'ga：browser'，'sessions :: condition :: ga：deviceCategory == desktop']，
 [service，profile_id， '60daysAgo'，'ga：browser'，'sessions :: condition :: ga：deviceCategory == desktop']，
 ... 
 [service，profile_id，'90daysAgo'，'ga：browser ，使用ThreadPoolExecutor（max_workers = 4）作为执行程序：
 executor，使用ga：browserVersion'，'sessions :: condition :: ga：deviceCategory == mobile'] 
] 
 
。 map（get_data，parameters）
  

---

一个Python应用程序（使用谷歌分析API），它允许用户得到前10个桌面浏览器的报告，根据版本细分的桌面浏览器，移动浏览器和移动操作系统用于访问过去30,60年的给定站点，和90天。截至目前，一切似乎都正常。

然而，表现已经到来。有12个API请求被提出 - 4个数据集中的每一个都有3个。有时应用程序需要大约10秒钟才能运行，有时需要一分钟以上。看起来这完全取决于API如何响应。所以我的问题是：有什么方法可以将这些请求中的一部分组合起来，或者将它们安排成可以同时执行的方式？

我尝试了解整合请求的方法，以便我可能只需要对每组数据执行一次请求，以便返回30,60和90天的信息，但我无法找到任何内容。至于要求同时进行，我只是不太确定如何去做这样的事情。我能找到的最接近的东西是这个问题/答案，但我无法相当

以下是相关的代码：

  def get_data（service，profile_id，days，dimensions，segment）：
 return service.data（）。ga（）。get（
 ids ='ga：'+ profile_id，
 start_date = days，
 end_date ='today'，
 metrics ='ga：sessions'，
 dimensions = dimensions，
 sort =' -  ga：sessions'，
 segment = segment，
 max_results = 10）.execute（）
 
 $ b $ get get_results（service，profile_id）：
全局glob_startdate $ b $全局glob_months 
 
＃获得前10名桌面浏览器
 print（获得前10名桌面浏览器...）
 data_1a = get_data（服务，profile_id，'30daysAgo'，'ga：browser' ，'sessions :: conditi on :: ga：deviceCategory == desktop'）
 data_1b = get_data（service，profile_id，'60daysAgo'，'ga：browser'，'sessions :: condition :: ga：deviceCategory == desktop'）
 data_1c = get_data（service，profile_id，'90daysAgo'，'ga：browser'，'sessions :: condition :: ga：deviceCategory == desktop'）
 data1 = [data_1a，data_1b，data_1c] 
 
＃获得前10个桌面浏览器版本
 print（获得前10个桌面浏览器版本...）
 data_2a = get_data（服务，profile_id，'30daysAgo'，'ga：浏览器，ga：browserVersion'，'sessions :: condition :: ga：deviceCategory ==桌面'）
 data_2b = get_data（服务，profile_id，'60daysAgo'，'ga：browser，ga：browserVersion'，'sessions :: condition :: ga：deviceCategory == desktop'）
 data_2c = get_data（service，profile_id，'90daysAgo'，'ga：browser，ga：browserVersion'，'sessions :: condition :: ga：deviceCategory = = desktop'）
 data2 = [data_2a，data_2b，data_2c] 
 
 ＃获得排名前10的移动操作系统的
 print（获得前10名移动操作系统的...）
 data_3a = get_data（service，profile_id，'30daysAgo'，'ga：operatingSystem，ga：operatingSystemVersion' sessions :: condition :: ga：deviceCategory == mobile'）
 data_3b = get_data（service，profile_id，'60daysAgo'，'ga：operatingSystem，ga：operatingSystemVersion'，'sessions :: condition :: ga：deviceCategory =='mobile'）
 data_3c = get_data（service，profile_id，'90daysAgo'，'ga：operatingSystem，ga：operatingSystemVersion'，'sessions :: condition :: ga：deviceCategory == mobile'）
数据3 = [数据_3a，数据_3b，数据_3c] 
 
＃获得前10名移动浏览器
 print（获得前10名移动浏览器...）
 data_4a = get_data（服务， profile_id，'30daysAgo'，'ga：browser，ga：browserVersion'，'sessions :: condition :: ga：deviceCategory == mobile'）
 data_4b = get_data（service，profile_id，'60daysAgo'，'ga：浏览器，ga：browserVersion'，'se ssions :: condition :: ga：deviceCategory == mobile'）
 data_4c = get_data（service，profile_id，'90daysAgo'，'ga：browser，ga：browserVersion'，'sessions :: condition :: ga：deviceCategory =='mobile'）
 data4 = [data_4a，data_4b，data_4c] 
  

谢谢！ / p>

解决方案

您可以 batch 由于API 配额和限制。

  from apiclient.http import BatchHttpRequest 
 import httplib2 
 
 $ b def call_back（request_id，response，exception）：
对每个调用的响应做一些事情
 pass 
 $ b $ def get_request（service，profile_id，days，dimensions，segment）：
注意我从这个结尾删除了execute （）。get（
 ids ='ga：'+ profile_id，
 start_date = days，
 end_date = 'today'，
 metrics ='ga：sessions'，
 dimensions = dimensions，
 sort =' -  ga：sessions'，
 segment = segment，
 max_results = 10）
 
＃创建一个批量Http请求对象
 batch = BatchHttpRequest（callback = self.call_back）
 
 
＃构建您的查询。 
＃获得前10名桌面浏览器
 print（获得前10名桌面浏览器...）
 request_1a = get_request（服务，profile_id，'30daysAgo'，'ga：browser'，' sessions :: condition :: ga：deviceCategory == desktop'）
 request_1b = get_request（service，profile_id，'60daysAgo'，'ga：browser'，'sessions :: condition :: ga：deviceCategory == desktop' ）
 request_1c = get_request（service，profile_id，'90daysAgo'，'ga：browser'，'sessions :: condition :: ga：deviceCategory == desktop'）
 
 for request in [ request_1a，request_1b，request_1c]：
 batch.add（请求）
 
 batch.execute（http = httplib2.Http（））
  pre> 
Update: I found something that might be useful, but I'm still having a bit of trouble figuring out how to implement it. If I try to map the get_data like so, I'm not sure how I can assign the results of each call to the respective variable.
parameters = [
    [service, profile_id, '30daysAgo', 'ga:browser', 'sessions::condition::ga:deviceCategory==desktop'],
    [service, profile_id, '60daysAgo', 'ga:browser', 'sessions::condition::ga:deviceCategory==desktop'],
    ...
    [service, profile_id, '90daysAgo', 'ga:browser,ga:browserVersion', 'sessions::condition::ga:deviceCategory==mobile']
]

with ThreadPoolExecutor(max_workers=4) as executor:
    executor.map(get_data, parameters)




I'm writing a Python application (using the Google analytics API) that allows a user to get a report of the top 10 desktop browsers, desktop browsers broken down by version, mobile browsers, and mobile OS's used to access a given site over the last 30, 60, and 90 days. As of right now, everything seems to be working fine.

However, performance is all over the place. There are 12 API requests being made - 3 for each of the 4 sets of data. Sometimes the application takes about 10 seconds to run, and sometimes it takes well over a minute. It seems like it's all dependent on how the API is responding. So my question is: are there ways I can either combine some of these requests or arrange them in such a way that they'll get executed concurrently?

I tried looking into ways to consolidate the requests so that maybe I'd only have to do one request per set of data that would return information for 30, 60, and 90 days, but I wasn't able to come across anything. As for getting the requests to concurrently, I'm just not quite sure how to go about doing something like that. The closest thing I could find was this question/answer, but I couldn't quite follow the answer about batch processing.

Here's the relevant code:
def get_data(service, profile_id, days, dimensions, segment):
    return service.data().ga().get(
        ids='ga:' + profile_id,
        start_date=days,
        end_date='today',
        metrics='ga:sessions',
        dimensions=dimensions,
        sort='-ga:sessions',
        segment=segment,
        max_results=10).execute()


def get_results(service, profile_id):
    global glob_startdate
    global glob_months

    # get top 10 desktop browsers
    print("Getting top 10 desktop browsers...")
    data_1a = get_data(service, profile_id, '30daysAgo', 'ga:browser', 'sessions::condition::ga:deviceCategory==desktop')
    data_1b = get_data(service, profile_id, '60daysAgo', 'ga:browser', 'sessions::condition::ga:deviceCategory==desktop')
    data_1c = get_data(service, profile_id, '90daysAgo', 'ga:browser', 'sessions::condition::ga:deviceCategory==desktop')
    data1 = [data_1a, data_1b, data_1c]

    # get top 10 desktop browser versions
    print("Getting top 10 desktop browser versions...")
    data_2a = get_data(service, profile_id, '30daysAgo', 'ga:browser,ga:browserVersion', 'sessions::condition::ga:deviceCategory==desktop')
    data_2b = get_data(service, profile_id, '60daysAgo', 'ga:browser,ga:browserVersion', 'sessions::condition::ga:deviceCategory==desktop')
    data_2c = get_data(service, profile_id, '90daysAgo', 'ga:browser,ga:browserVersion', 'sessions::condition::ga:deviceCategory==desktop')
    data2 = [data_2a, data_2b, data_2c]

    # get top 10 mobile OS's
    print("Getting top 10 mobile OS's...")
    data_3a = get_data(service, profile_id, '30daysAgo', 'ga:operatingSystem,ga:operatingSystemVersion', 'sessions::condition::ga:deviceCategory==mobile')
    data_3b = get_data(service, profile_id, '60daysAgo', 'ga:operatingSystem,ga:operatingSystemVersion', 'sessions::condition::ga:deviceCategory==mobile')
    data_3c = get_data(service, profile_id, '90daysAgo', 'ga:operatingSystem,ga:operatingSystemVersion', 'sessions::condition::ga:deviceCategory==mobile')
    data3 = [data_3a, data_3b, data_3c]

    # get top 10 mobile browsers
    print("Getting top 10 mobile browsers...")
    data_4a = get_data(service, profile_id, '30daysAgo', 'ga:browser,ga:browserVersion', 'sessions::condition::ga:deviceCategory==mobile')
    data_4b = get_data(service, profile_id, '60daysAgo', 'ga:browser,ga:browserVersion', 'sessions::condition::ga:deviceCategory==mobile')
    data_4c = get_data(service, profile_id, '90daysAgo', 'ga:browser,ga:browserVersion', 'sessions::condition::ga:deviceCategory==mobile')
    data4 = [data_4a, data_4b, data_4c]
Thanks!
解决方案
You can batch up to 10 requests at a time because of API quota and limits.
from apiclient.http import BatchHttpRequest
import httplib2


def call_back(request_id, response, exception):
  """Do something with the response of each call"""
  pass

def get_request(service, profile_id, days, dimensions, segment):
   """Note I removed the execute() from the end of this method."""
   return service.data().ga().get(
     ids='ga:' + profile_id,
     start_date=days,
     end_date='today',
     metrics='ga:sessions',
     dimensions=dimensions,
     sort='-ga:sessions',
     segment=segment,
     max_results=10)

# Create a batch Http Request object
batch = BatchHttpRequest(callback=self.call_back)


# Construct your queries.
# get top 10 desktop browsers
print("Getting top 10 desktop browsers...")
request_1a = get_request(service, profile_id, '30daysAgo', 'ga:browser', 'sessions::condition::ga:deviceCategory==desktop')
request_1b = get_request(service, profile_id, '60daysAgo', 'ga:browser', 'sessions::condition::ga:deviceCategory==desktop')
request_1c = get_request(service, profile_id, '90daysAgo', 'ga:browser', 'sessions::condition::ga:deviceCategory==desktop')

for request in [request_1a, request_1b, request_1c]:
    batch.add(request)

batch.execute(http=httplib2.Http())


                        
这篇关于如何组合/加速多个API调用以提高性能？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！