python爬虫ieee论文关键词 [英] python crawler ieee paper keywords

问题描述

我尝试使用爬虫来获取 ieee 论文关键字，但现在出现错误如何修复我的爬虫?我的代码在这里

i trying to use crawler to get ieee paper keywords but now i get a error how can to fix my crawler? my code is here

import requests
import json
from bs4 import BeautifulSoup
ieee_content = requests.get("http://ieeexplore.ieee.org/document/8465981", timeout=180)
soup = BeautifulSoup(ieee_content.text, 'xml')
tag = soup.find_all('script')
for i in tag[9]:
    s = json.loads(re.findall('global.document.metadata=(.*;)', i)[0].replace("'", '"').replace(";", ''))

错误就在这里

Traceback (most recent call last):
  File "G:/github/爬蟲/redigg-leancloud/crawlers/sup_ieee_keywords.py", line 90, in <module>
    a.get_es_data(offset=0, size=1)
  File "G:/github/爬蟲/redigg-leancloud/crawlers/sup_ieee_keywords.py", line 53, in get_es_data
    self.get_data(link=ieee_link, esid=es_id)
  File "G:/github/爬蟲/redigg-leancloud/crawlers/sup_ieee_keywords.py", line 65, in get_data
    s = json.loads(re.findall('global.document.metadata=(.*;)', i)[0].replace(";", '').replace("'", '"'))
IndexError: list index out of range

推荐答案

这是另一个答案.在我的代码中加载(替换)之后，我不知道你在用代码中的 's' 做什么.

Here's another answer. I don't know what you are doing with 's' in your code after the load (replace) in my code.

下面的代码没有抛出错误，但是你又是如何使用's'

The code below doesn't thrown an error, but again how are you using 's'

import requests
import json
from bs4 import BeautifulSoup

ieee_content = requests.get("http://ieeexplore.ieee.org/document/8465981", timeout=180)
soup = BeautifulSoup(ieee_content.text, 'xml')
tag = soup.find_all('script')

# i is a list
for i in tag[9]:
   metadata_format = re.compile(r'global.document.metadata=.*', re.MULTILINE)
   metadata = re.findall(metadata_format, i)
   if len(metadata) != 0:
      # convert the list 
      convert_to_json = json.dumps(metadata)
      x = json.loads(convert_to_json)
      s = x[0].replace("'", '"').replace(";", '')
      ###########################################
      # I don't know what you plan to do with 's'
      ###########################################
      print (s)

这篇关于python爬虫ieee论文关键词的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！