如何修复HTTP错误提取URL。在抓取时java中的状态= 500? [英] how to fix HTTP error fetching URL. Status=500 in java while crawling?
问题描述
我正在尝试从评论页面抓取用户对imdb影院电影的评分:
(我的数据库中的电影数量约为600,000)。我使用jsoup解析页面如下:(对不起,我没有在这里写完整个代码,因为它太长了)
I am trying to crawl the user's ratings of cinema movies of imdb from the review page: (number of movies in my database is about 600,000). I used jsoup to parse pages as below: (sorry, I didn't write the whole code here since it is too long)
try {
//connecting to mysql db
ResultSet res = st
.executeQuery("SELECT id, title, production_year " +
"FROM title " +
"WHERE kind_id =1 " +
"LIMIT 0 , 100000");
while (res.next()){
.......
.......
String baseUrl = "http://www.imdb.com/search/title?release_date=" +
""+year+","+year+"&title="+movieName+"" +
"&title_type=feature,short,documentary,unknown";
Document doc = Jsoup.connect(baseUrl)
.userAgent("Mozilla")
.timeout(0).get();
.....
.....
//insert ratings into database
...
我测试了它的前100个,然后是前500个,还有我的数据库中的前2000个电影,它运行良好。但问题是,当我测试了100,000部电影时,我收到了这个错误:
I tested it for the first 100, then first 500 and also for the first 2000 movies in my db and it worked well. But the problem is that when I tested for 100,000 movies I got this error:
org.jsoup.HttpStatusException: HTTP error fetching URL. Status=500, URL=http://www.imdb.com/search/title?release_date=1899,1899&title='Columbia'%20Close%20to%20the%20Wind&title_type=feature,short,documentary,unknown
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:449)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:424)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:178)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:167)
at imdb.main(imdb.java:47)
我搜索了很多这个错误,我发现它是服务器端错误,5xx错误数字。
I searched a lot for this error and I found it is a server side error with 5xx error number.
然后我决定设置一个条件,当连接失败时,它再尝试2次然后如果仍然无法连接,则不会停止并转到下一个网址。因为我是java新手,所以我试图搜索类似的问题并在stackoverflow中阅读这些答案:
Then I decided to set a condition that when connection fails, it tries 2 more times and then if still couldn't connect, does not stop and goes to the next url. since I am new to java I tried to search for similar questions and read these answers in stackoverflow:
但是,当我按照他们的建议尝试使用Connection.Response时,它告诉我Connection.Response无法解析为某种类型。
but, when I try with "Connection.Response" as they suggest, it tells me that "Connection.Response cannot be resolved to a type".
我很感激有人可以帮助我,因为我只是一个新手,我知道这可能很简单,但我不知道如何解决它。
I appreciate if someone could help me, since I am just a newbie and I know it might be simple but I don't know how to fix it.
好吧,我可以修复http错误状态500只需添加ignoreHttpError(true),如下所示:
Well, I could fix the http error status 500 by just adding "ignoreHttpError(true)" as below:
org.jsoup.Connection con = Jsoup.connect(baseUrl).userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21");
con.timeout(180000).ignoreHttpErrors(true).followRedirects(true);
Response resp = con.execute();
Document doc = null;
if (resp.statusCode() == 200) {
doc = con.get();
......
希望它能帮助那些有同样错误的人。
hope it can help those have the same error.
然而,在抓取22907部电影(约12小时)的评论页面后,我又出现了另一个错误:
READ TIMED OUT。
however, after crawling review pages of 22907 movies (about 12 hours), I got another error:
"READ TIMED OUT".
我感谢任何修复此错误的建议。
I appreciate any suggestion to fix this error.
推荐答案
将我的评论升级为答案:
Upgrading my comments to an answer:
Connection.Response
是 org.jsoup.Connection.Response
只有在有效的http代码(200)允许文档
实例时,才能中断通话分为3部分; 连接
,回复
,文件
To allow document
instance only when there is a valid http code (200), break your call into 3 parts; Connection
, Response
, Document
他nce,您上面代码的一部分被修改为:
Hence, your part of the code above gets modified to:
while (res.next()){
.......
.......
String baseUrl = "http://www.imdb.com/search/title?release_date=" + ""
+ year + "," + year + "&title=" + movieName + ""
+ "&title_type=feature,short,documentary,unknown";
Connection con = Jsoup.connect(baseUrl).userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21").timeout(10000);
Connection.Response resp = con.execute();
Document doc = null;
if (resp.statusCode() == 200) {
doc = con.get();
....
}
这篇关于如何修复HTTP错误提取URL。在抓取时java中的状态= 500?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!