从 Google Analytics 中抓取实时访客 [英] Scraping Real Time Visitors from Google Analytics

问题描述

我有很多网站，我想构建一个仪表板，在单个页面上显示每个网站的实时访问者数量.(其他人会想要这个吗?)目前查看此信息的唯一方法是为每个站点打开一个新标签.

Google 没有实时 API，所以我想知道是否可以抓取这些数据.Eduardo Cereto 发现 Google 通过实时/绑定网络请求传输实时数据.任何更精明的人都知道我应该如何开始?这是我的想法:

1. 弄清楚如何以编程方式进行身份验证

2. 检查所有实时/绑定请求，看看它们是如何变化的.每个请求都有唯一的键吗?这是从哪里来的?以下是我对请求的细分:

   这与 Google Analytics API 非常相似.要开始对此进行开发，https://developers.google.com/analytics/devguides/reporting/realtime/v3/devguide

   I have a lot of sites and want to build a dashboard showing the number of real time visitors on each of them on a single page. (would anyone else want this?) Right now the only way to view this information is to open a new tab for each site.

   Google doesn't have a real-time API, so I'm wondering if it is possible to scrape this data. Eduardo Cereto found out that Google transfers the real-time data over the realtime/bind network request. Anyone more savvy have an idea of how I should start? Here's what I'm thinking:

   1. Figure out how to authenticate programmatically

   2. Inspect all of the realtime/bind requests to see how they change. Does each request have a unique key? Where does that come from? Below is my breakdown of the request:

      https://www.google.com/analytics/realtime/bind?VER=8

      &key= [What is this? Where does it come from? 21 character lowercase alphanumeric, stays the same each request]

      &ds= [What is this? Where does it come from? 21 character lowercase alphanumeric, stays the same each request]

      &pageId=rt-standard%2Frt-overview

      &q=t%3A0%7C%3A1%3A0%3A%2Ct%3A11%7C%3A1%3A5%3A%2Cot%3A0%3A0%3A4%2Cot%3A0%3A0%3A3%2Ct%3A7%7C%3A1%3A10%3A6%3D%3DREFERRAL%3B%2Ct%3A10%7C%3A1%3A10%3A%2Ct%3A18%7C%3A1%3A10%3A%2Ct%3A4%7C5%7C2%7C%3A1%3A10%3A2!%3Dzz%3B%2C&f

      The q variable URI decodes to this (what the?): t:0|:1:0:,t:11|:1:5:,ot:0:0:4,ot:0:0:3,t:7|:1:10:6==REFERRAL;,t:10|:1:10:,t:18|:1:10:,t:4|5|2|:1:10:2!=zz;,&f

      &RID=rpc

      &SID= [What is this? Where does it come from? 16 character uppercase alphanumeric, stays the same each request]

      &CI=0

      &AID= [What is this? Where does it come from? integer, starts at 1, increments weirdly to 150 and then 298]

      &TYPE=xmlhttp

      &zx= [What is this? Where does it come from? 12 character lowercase alphanumeric, changes each request]

      &t=1

   3. Inspect all of the realtime/bind responses to see how they change. How does the data come in? It looks like some altered JSON. How many times do I need to connect to get the data? Where is the active visitors on site number in there? Here is a dump of sample data:

   19
   [[151,["noop"]
   ]
   ]
   388
   [[152,["rt",[{"ot:0:0:4":{"timeUnit":"MINUTES","overTimeData":[{"values":[49,53,52,40,42,55,49,41,51,52,47,42,62,82,76,71,81,66,81,86,71,66,65,65,55,51,53,73,71,81],"name":"Total"}]},"ot:0:0:3":{"timeUnit":"SECONDS","overTimeData":[{"values":[0,1,1,1,1,0,1,0,1,1,1,0,2,0,2,2,1,0,0,0,0,0,2,1,1,2,1,2,0,5,1,0,2,1,1,1,2,0,2,1,0,5,1,1,2,0,0,0,0,0,0,0,0,0,1,1,0,3,2,0],"name":"Total"}]}}]]]
   ]
   388
   [[153,["rt",[{"ot:0:0:4":{"timeUnit":"MINUTES","overTimeData":[{"values":[52,53,52,40,42,55,49,41,51,52,47,42,62,82,76,71,81,66,81,86,71,66,65,65,55,51,53,73,71,81],"name":"Total"}]},"ot:0:0:3":{"timeUnit":"SECONDS","overTimeData":[{"values":[2,1,1,1,1,1,0,1,0,1,1,1,0,2,0,2,2,1,0,0,0,0,0,2,1,1,2,1,2,0,5,1,0,2,1,1,1,2,0,2,1,0,5,1,1,2,0,0,0,0,0,0,0,0,0,1,1,0,3,2],"name":"Total"}]}}]]]
   ]
   388
   [[154,["rt",[{"ot:0:0:4":{"timeUnit":"MINUTES","overTimeData":[{"values":[53,53,52,40,42,55,49,41,51,52,47,42,62,82,76,71,81,66,81,86,71,66,65,65,55,51,53,73,71,81],"name":"Total"}]},"ot:0:0:3":{"timeUnit":"SECONDS","overTimeData":[{"values":[0,3,1,1,1,1,1,0,1,0,1,1,1,0,2,0,2,2,1,0,0,0,0,0,2,1,1,2,1,2,0,5,1,0,2,1,1,1,2,0,2,1,0,5,1,1,2,0,0,0,0,0,0,0,0,0,1,1,0,3],"name":"Total"}]}}]]]
   ]
   

   Let me know if you can help with any of the items above!

   解决方案

   To get the same, Google has launched new Real Time API. With this API you can easily retrieve real time online visitors as well as several Google Analytics with following dimensions and metrics. https://developers.google.com/analytics/devguides/reporting/realtime/dimsmets/

   This is quite similar to Google Analytics API. To start development on this, https://developers.google.com/analytics/devguides/reporting/realtime/v3/devguide

   这篇关于从 Google Analytics 中抓取实时访客的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！