使用curl下载flurry事件日志 [英] Download flurry event log using curl
问题描述
我开始使用Flurry Analytics，发现它的分析工具不够，太慢。 3个步骤的简单漏斗处理了3天，而通常查询3个左连接在表上有100,000行时需要0.001秒。
I started to use Flurry Analytics and have found that it's analysing tools are insufficient and too slow. Simple funnel of 3 steps was processed for 3 days, while normally query with 3 left joins take 0,001 seconds on table with 100,000 rows.
Flurry允许在事件日志页面上以csv格式下载原始事件数据，因此我决定导入所有事件并在家里进行分析。
Flurry allows to download raw event data in csv format on Event Logs page, so I decided to import all the events and analyze them at home.
Flurry只允许下载100,000条记录，他们建议只是经常下载以符合此限制。他们有原始事件下载API，但由于某种原因放弃了它。所以唯一的方法是去事件日志页面并手动下载事件数据。但是你可以想象它是非常恼人。
Flurry allows to download only 100,000 records, and they advise just to download often to fit this limit. They had raw event download API but abandoned it for some reason. So the only way is to go to Event Logs page and download events data manually. But as you can imagine it is very annoying.
所以我决定在php中使用curl获取这些数据。我已经复制GET HTTP请求下载链接与标题，并获得数据。
但是整个魔法是在会话/ cookies，我可以从现有会话复制。所以要使curl查询成功，我必须：
So I decided to get this data using curl in php. I've copied the GET HTTP request to download link with headers and got the data. But the whole magic is in session/cookies which I can just copy from existing session. So to make curl query succeed I have to:
- 在浏览器中浏览网站并登录
- 转到事件日志页面，选择时间范围参数并点击下载
- 在sniffer中复制请求标头
- 将它们粘贴到我的PHP代码
- 从现在起，我可以在php中进行此查询，直到会话Cookie过期。
- go to flurry site in browser and login
- go to Event Logs page, choose time frame parameters and click download
- copy request headers in sniffer
- paste them to my php code
- and from now on I can make this query in php until session cookies expire
肯定但是假设cookie将在第二天到期，因此整个这种努力是无用的。
I'm not sure but suppose cookies will expire on the next day, so the whole this effort is useless.
根据我的理解，我应该使用curl尝试POST登录，并保持此连接执行GET以下载数据。但我不能登录，甚至复制整个POST登录请求正文 - 它回答与同一登录页面，虽然应该302重定向到
https://dev.flurry.com/fullPageTakeover.do?originalTarget=&isFirstPostLogin=true&defaultTarget=%2Fhome.do < a>
As I understand I should try to POST login with curl, and keeping this connection perform GET to download data. Yet I can not login even copying the whole POST login request body - it answers with the same login page although should 302 redirect to https://dev.flurry.com/fullPageTakeover.do?originalTarget=&isFirstPostLogin=true&defaultTarget=%2Fhome.do
看起来flurry是以某种方式防止这种卷曲阅读。
It looks like flurry is somehow protected from such curl reading. Or maybe somebody succeed in it?
这里是代码：
$cookie_file_path = "cookies.txt";
$LOGINURL = "https://dev.flurry.com/secure/login.do";
$MY_EMAIL ="my email";
$MY_PASS="password";
$MY_GAME_ID="gameid";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6");
curl_setopt ($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
curl_setopt ($ch, CURLOPT_REFERER, $LOGINURL);
curl_setopt($ch, CURLOPT_URL, $LOGINURL);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "loginEmail=$MY_EMAIL&loginPassword=$MY_PASS&__checkbox_rememberMe=true&struts.token.name=struts.token&struts.token=7NB9NWLOYZ8SD8TWR8LGS63REVDI8SQS");
$result = curl_exec($ch);
$remotePageUrl = "https://dev.flurry.com/eventsLogCsv.do?projectID=$MY_GAME_ID&versionCut=versionsAll&intervalCut=7Days&stream=true&direction=1&offset=0";
curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_URL, $remotePageUrl);
$result = curl_exec($ch);
echo $result;
也尝试传递cookies（类似于浏览器），但没有任何帮助：
Also tried to pass cookies (like it does from browser), but nothing helped:
Cookie: _ga=GA1.2.100867533.1424333566; S0hZTkM0RFRXRjJNSlg2TVdXSEs_fit=1424333594147; fid=SG1162A8DEFC14B8428E7C2AFC71D3AEA00C1872F5; JSESSIONID=w34~mvbdvftm9x9dez9dg9b2pmhs; _map_zoomLevel=0;
_map_zoneId=0; __utmt=1; __utmt_~1=1; S0hZTkM0RFRXRjJNSlg2TVdXSEs_fs=eyJiYSI6MTQyNDMzNzkzMzU2OCwicGF1c2VUaW1lc3RhbXAiOjAsImJjIjotMSwiZXZlbnRDb3VudGVyIjowLCJwdXJjaGFzZUNvdW50ZXIiOjAsImVycm9yQ291bnRlciI6MCwidGltZWRFdmVudHMiOltdfQ==;
__utma=83277827.100867533.1424333566.1424333594.1424336847.2; __utmb=83277827.8.10.1424336847; __utmc=83277827; __utmz=83277827.1424333594.1.1.utmcsr=flurry.com|utmccn=(referral)|utmcmd=referral|utmcct=/; __utma=34058230.100867533.1424333566.1424333566.1424336847.2; __utmb=34058230.8.10.1424336847; __utmc=34058230; __utmz=34058230.1424333566.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _mkto_trk=id:802-TBR-126&token:_mch-flurry.com-1424333577360-64839; S0hZTkM0RFRXRjJNSlg2TVdXSEs_flp=1424338032448
感谢 silkfire Flurry问题已解决！ >
Thanks to silkfire Flurry problem is solved!
推荐答案
struts.token 是一个CRSF令牌，绑定到您的会话在每个页面加载时重新生成。在你的代码，虽然，它是静态的。您需要在您的第一个cURL请求后提取它，然后将其注入您的POST数组，以用于您的第二个请求。
The struts.token is a CRSF token which is bound to your session and regenerated on each page load. In your code though, it's static. You need to fetch it after your first cURL request and then inject it into your POST array to be used for your second request.
此外，您必须登录的页面是 /loginAction.do ，而不是 /login.do 。
Also the page you have to login to is /loginAction.do and not /login.do.
这是我成功登录Flurry的方式：
This is how I successfully logged in to Flurry:
$post = [
'loginEmail' => 'E-MAIL',
'loginPassword' => 'PASSWORD',
'struts.token.name' => 'struts.token'
];
$ch = curl_init('https://dev.flurry.com/secure/login.do');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_COOKIEFILE, null);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
libxml_use_internal_errors(true);
$dom = new DOMDocument('1.0', 'UTF-8');
$dom->loadHTML(curl_exec($ch));
$xpath = new DOMXPath($dom);
$post['struts.token'] = $xpath->query('//input[@name="struts.token"]')->item(0)->getAttribute('value');
curl_setopt($ch, CURLOPT_URL, 'https://dev.flurry.com/secure/loginAction.do');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post));
$data = curl_exec($ch);
echo $data;
这篇关于使用curl下载flurry事件日志的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
