从Big5到utf-8编码，同时使用Node-request抓取网站 [英] Big5 to utf-8 encoding while scraping website with Node-request
问题描述
我是Node.js的新手，我正在尝试使用请求模型来抓取网站，我在编码方面遇到了问题:目标网站正在使用big5作为编码，我希望将其转换为utf -8，并显示以下代码:
I am new to Node.js, and I am trying to use the request model to scrap a website, I am having problem with the encoding: the target website is using big5 as encoding, and I wished to turn it to utf-8 with the following code:
var Iconv = require('iconv').Iconv;
var fs = require('fs');
var big5_to_utf8 = new Iconv('big5', 'utf-8');
var buffer = big5_to_utf8.convert(fs.readFileSync('./test'));
console.log(buffer.toString());
我怀疑问题可能是由于剪贴过程中的某些错误引起的，因此，我的剪贴代码供您参考:
I doubt that the problem might be caused due to some wrong in the scrapping process, so for your reference, my code for scrapping:
var fs = require('fs');
var request = require('request');
var j = request.jar()
var cookie = request.cookie('ASPSESSIONIDCSDCTTSR=KDMMMIMDCCIHJIJFDKGEDFOH')
j.add(cookie)
request({
url: 'http://amis.afa.gov.tw/v-asp/v101r.asp',
method: "POST",
"Content-type": "application/x-www-form-urlencoded;",
jar:true,
encoding: 'utf-8',
form: {
mhidden1:false,
myy:101,
mmm:9,
mdd:25,
mpno:"FC",
mpnoname:"%ADJ%A5%CA++++",
B1:"%B6%7D%A9l%ACd%B8%DF",
}
}, function (error, response, body) {
console.log(body);
fs.writeFile("test", body);
});
非常感谢您的帮助.
为更具体地说明该错误，以下是代码返回的内容:
To be more specific to the error, the following are what the code returns:
<p align="center"><font color="#800080">�Шϥ��s�����u���C��</font><em><font
size="4" color="#000080">[�W�@��]</font></em><font color="#800080">�^���e�@���J�����e���~���d��</font></p>
这应该返回:
<p align="center"><font color="#800080">請使用瀏覽器工具列中</font><em><font size="4" color="#000080">[上一頁]</font></em><font color="#800080">回到前一輸入條件畫面繼續查詢</font></p>
我还尝试使用iconv-lite代替iconv，将函数调用替换为以下内容:
I also tried to use iconv-lite instead of iconv, replacing the function call to the following:
function (error, response, body) {
var bufferhelper = new BufferHelper();
bufferhelper.concat(body);
console.log(iconv.decode(bufferhelper.toBuffer(), 'Big5'));
});
仅可获取:
<p align="center"><font color="#800080">�濆詉胬胬譃胬舚胬</font><em><font
size="4" color="#000080">[抝胬]</font></em><font color="#800080">䒷胬蓚胬鸜胬胬蓚胬趦胬胬</font</p>
推荐答案
我使用iconv-lite将big5解码为utf8.
I use iconv-lite to decode big5 to utf8.
您应该将encoding:null设置为request将返回原始编码页面.
And you should set encoding:null that request will return raw encoding page.
这是示例代码.
var iconv = require('iconv-lite');
var request = require('request');
request({ url: 'http://amis.afa.gov.tw/v-asp/v101r.asp',encoding:null}, function(err, response, body) {
if (!err && response.statusCode == 200) {
var str = iconv.decode(new Buffer(body), "big5");
console.log(str);
}
});
返回是
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=big5">
<title>v101r</title>
<meta name="GENERATOR" content="Microsoft FrontPage 4.0">
<meta name="Microsoft Theme" content="none, default">
</head>
<body>
<p align="center">查無結果!</p>
<p align="center"><font color="#800080">請使用瀏覽器工具列中</font><em><font
size="4" color="#000080">[上一頁]</font></em><font color="#800080">回到前一輸入條件畫面繼續查詢</font></p>
</body>
</html>
我在RedHat EL 6.4和iconv-lite 0.2.11，request 2.27.0
这篇关于从Big5到utf-8编码，同时使用Node-request抓取网站的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
