从网站获取请求并检索响应？ [英] Getting requests from a website and retrieving the response?
问题描述
我正在尝试监控网站（ www.bidcactus.com ）。在网站上我打开Firebug，转到网络选项卡，然后单击XHR选项卡。
I am trying to monitor a website (www.bidcactus.com). While on the website I open up Firebug, go to the net tab, and click the XHR tab.
我想接受请求的响应并将其保存到mySql数据库（我的计算机上运行了本地数据库（XAMPP）。
I want to take the responses of the requests and save it to a mySql database (I have a local one running on my computer(XAMPP).
我被告知主要使用jQuery或JavaScript做各种各样的事情，但我也没有经验，所以我想知道是否有人可以帮助我。
I have been told to do a variety of things mainly using jQuery or JavaScript but I'm not experienced either so I was wondering if anyone can help me out here.
有人建议我这个链接
使用Greasemonkey和jQuery拦截页面中的JSON / AJAX数据并进行处理
它使用Greasemonkey，我也不太了解...
Its using Greasemonkey as well which I don't know much about either...
提前感谢您的帮助
示例/更多细节：
监控发送的请求（通过firebug）我在下面看到
Example/more detail:
While monitoring the requests sent(via firebug) I see below
http://www.bidcactus.com/CactusWeb/ItemUpdates?rnd=1310684278585
The response of this link is the following:
{"s":"uk5c","a":[{"w":"MATADORA","t":944,"p":5,"a":413173,"x":10},
{"w":"1000BidsAintEnough","t":6,"p":863,"a":413198,"x":0},
{"w":"YourBidzWillBeWastedHere","t":4725,"p":21,"a":413200,"x":8},
{"w":"iwillpay2much","t":344,"p":9,"a":413201,"x":9},
{"w":"apcyclops84","t":884,"p":3,"a":413213,"x":14},
{"w":"goin_postal","t":165,"p":5,"a":413215,"x":12},
{"w":"487951","t":825,"p":10,"a":413218,"x":6},
{"w":"mishmash","t":3225,"p":3,"a":413222,"x":7},
{"w":"CrazyKatLady2","t":6464,"p":1,"a":413224,"x":2},
{"w":"BOSS1","t":224,"p":102,"a":413230,"x":4},
{"w":"serbian48","t":62,"p":2,"a":413232,"x":11},
{"w":"Tuffenough","t":1785,"p":1,"a":413234,"x":1},
{"w":"apcyclops84","t":1970,"p":1,"a":413240,"x":13},
{"w":"Tuffenough","t":3524,"p":1,"a":413244,"x":5},
{"w":"Cdm17517","t":1424,"p":1,"a":413252,"x":3}],"tau":"0"}
我明白这些信息和我有什么关系我想我可以自己格式化，但网站随机创建新请求。
示例 http://www.bidcactus.com/CactusWeb/ItemUpdates?rnd=XXXXXXXXXXXX
我不知道它是如何创造它们的。
I understand what this information and I think I could format it myself however the website randomly creates new requests.
Example http://www.bidcactus.com/CactusWeb/ItemUpdates?rnd=XXXXXXXXXXXX
and I'm not sure how it creates them.
所以我需要获取所有项目更新请求的响应，并将信息发送到mysql数据库。
So I'm needing to get the response for all the requests that are for item updates and send the information to a mysql database.
推荐答案
好的，这是工作代码，有点针对该网站（仅限首页，没有帐户）。
OK, here's working code, somewhat tuned for that site (front page, no account, only).
使用说明：
-
安装GM脚本。请注意，目前它仅适用于Firefox。
Install the GM script. Note that it is Firefox only, for now.
观察它在Firebug控制台中运行，并调整过滤器部分（清楚标记），以定位您的数据有兴趣。（也许整个一个数组？）
Observe it running in Firebug's console, and tune the filter section (clearly marked), to target the data you are interested in. (Maybe the whole a array?)
请注意，脚本可能需要几秒钟开始打印，ajax拦截开始。
Note that it can take several seconds after "Script Start" is printed, for the ajax intercepts to start.
设置您的Web应用程序和服务器以接收数据。该脚本发布了JSON，因此PHP会抓取数据，如下所示：
Setup your web application and server to receive the data. The script posts JSON, so PHP, for example, would grab the data, like so:
$jsonData = json_decode ($HTTP_RAW_POST_DATA);
将脚本指向您的服务器。
Point the script to your server.
Voilà。她已经完成了。
Voilà. She is done.
/******************************************************************************
*******************************************************************************
** This script intercepts ajaxed data from the target web pages.
** There are 4 main phases:
** 1) Intercept XMLHttpRequest's made by the target page.
** 2) Filter the data to the items of interest.
** 3) Transfer the data from the page-scope to the GM scope.
** NOTE: This makes it technically possibly for the target page's
** webmaster to hack into GM's slightly elevated scope and
** exploit any XSS or zero-day vulnerabilities, etc. The risk
** is probably zero as long as you don't start any feuds.
** 4) Use GM_xmlhttpRequest () to send the data to our server.
*******************************************************************************
*******************************************************************************
*/
// ==UserScript==
// @name _Record ajax, JSON data.
// @namespace stackoverflow.com/users/331508/
// @description Intercepts Ajax data, filters it and then sends it to our server.
// @include http://www.bidcactus.com/*
// ==/UserScript==
DEBUG = true;
if (DEBUG) console.log ('***** Script Start *****');
/******************************************************************************
*******************************************************************************
** PHASE 1 starts here, this is the XMLHttpRequest intercept code.
** Note that it will not work in GM's scope. We must inject the code to the
** page scope.
*******************************************************************************
*******************************************************************************
*/
funkyFunc = ( (<><![CDATA[
DEBUG = false;
//--- This is where we will put the data we scarf. It will be a FIFO stack.
payloadArray = []; //--- PHASE 3a
(function (open) {
XMLHttpRequest.prototype.open = function (method, url, async, user, pass)
{
this.addEventListener ("readystatechange", function (evt)
{
if (this.readyState == 4 && this.status == 200) //-- Done, & status "OK".
{
var jsonObj = null;
try {
jsonObj = JSON.parse (this.responseText); // FF code. Chrome??
}
catch (err) {
//if (DEBUG) console.log (err);
}
//if (DEBUG) console.log (this.readyState, this.status, this.responseText);
/******************************************************************************
*******************************************************************************
** PHASE 2: Filter as much as possible, at this stage.
** For this site, jsonObj should be an object like so:
** { s="1bjqo", a=[15], tau="0"}
** Where a is an array of objects, like:
** a 417387
** p 1
** t 826
** w "bart69"
** x 7
*******************************************************************************
*******************************************************************************
*/
//if (DEBUG) console.log (jsonObj);
if (jsonObj && jsonObj.a && jsonObj.a.length > 1) {
/*--- For demonstration purposes, we will only get the 2nd row in
the `a` array. (Probably stands for "auction".)
*/
payloadArray.push (jsonObj.a[1]);
if (DEBUG) console.log (jsonObj.a[1]);
}
//--- Done at this stage! Rest is up to the GM scope.
}
}, false);
open.call (this, method, url, async, user, pass);
};
} ) (XMLHttpRequest.prototype.open);
]]></>).toString () );
function addJS_Node (text, s_URL)
{
var scriptNode = document.createElement ('script');
scriptNode.type = "text/javascript";
if (text) scriptNode.textContent = text;
if (s_URL) scriptNode.src = s_URL;
var targ = document.getElementsByTagName('head')[0] || d.body || d.documentElement;
targ.appendChild (scriptNode);
}
addJS_Node (funkyFunc);
/******************************************************************************
*******************************************************************************
** PHASE 3b:
** Set up a timer to check for data from our ajax intercept.
** Probably best to make it slightly faster than the target's
** ajax frequency (about 1 second?).
*******************************************************************************
*******************************************************************************
*/
timerHandle = setInterval (function() { SendAnyResultsToServer (); }, 888);
function SendAnyResultsToServer ()
{
if (unsafeWindow.payloadArray) {
var payload = unsafeWindow.payloadArray;
while (payload.length) {
var dataRow = JSON.stringify (payload[0]);
payload.shift (); //--- pop measurement off the bottom of the stack.
if (DEBUG) console.log ('GM script, pre Ajax: ', dataRow);
/******************************************************************************
*******************************************************************************
** PHASE 4: Send the data, one row at a time, to the our server.
** The server would grab the data with:
** $jsonData = json_decode ($HTTP_RAW_POST_DATA);
*******************************************************************************
*******************************************************************************
*/
GM_xmlhttpRequest ( {
method: "POST",
url: "http://localhost/db_test/ShowJSON_PostedData.php",
data: dataRow,
headers: {"Content-Type": "application/json"},
onload: function (response) {
if (DEBUG) console.log (response.responseText);
}
} );
}
}
}
//--- EOF
其他说明：
-
我在该网站的主页上测试了它，没有登录（我不打算创建一个帐户）。
I tested it on the main page of that site, without logging in (I'm not about to to create an account there).
我使用 AdBlock ， FlashBlock ， NoSCript 和 RequestPolicy 全部生效。 JS为 bidcactus.com 打开了（它必须是）但没有其他人。重新开启所有这些问题不应该导致副作用 - 但如果确实如此，我就不会调试它。
I tested with AdBlock, FlashBlock, NoSCript, and RequestPolicy all in full effect. JS was turned on for bidcactus.com (it has to be) but no others. Turning all that crud back on shouldn't cause side effects -- but if it does, I'm not going to debug it.
这样的代码有调整网站以及如何浏览所述网站 1 。这取决于你。希望代码足够自我记录。
Code like this has to be tuned for the site and for how you browse said site1. It's up to you to do that. Hopefully the code is self-documented enough.
享受！
1 主要是： @include 和 @exclude 指令，JSON数据选择和过滤，以及是否需要阻止iFrame。另外，建议将2 DEBUG 变量（一个用于GM范围，一个用于页面范围）设置为 false 调整完成后。
1 Mainly: the @include and @exclude directives, the JSON data selection and filtering, and whether iFrames need to be blocked. Also it is recommended that the 2 DEBUG variables (one for GM scope and one for the page scope) be set to false when the tuning is done.
这篇关于从网站获取请求并检索响应？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
