urlparse:ModuleNotFoundError,大概在Python2.7和conda下 [英] urlparse: ModuleNotFoundError, presumably in Python2.7 and under conda
问题描述
我正在尝试运行自己的scrapy项目.该代码基于一本写得很好的书,并且作者提供了一个不错的VM游乐场来运行书中示例的脚本.在VM中,代码可以正常工作.但是,为了独自练习,我收到了以下错误:
I am attempting to run my own scrapy project. The code is based off a well written book and the author provides a great VM playground to run scripts exampled in the book. In the VM the code works fine. However, in an attempt to practice on my own, I received the following error:
File "(frozen importlib._bootstrap)", line 978, in _gcd_import
File "(frozen importlib._bootstrap)", line 961, in _find_and_load
File "(frozen importlib._bootstrap)", line 950, in _find_and_load_unlocked
File "(frozen importlib._bootstrap)", line 655, in _load_unlocked
File "(frozen importlib._bootstrap_external)", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "C:\users\me\dictionary_com\spiders\basic.py", line 3, in <module>
import urlparse
ModuleNotFoundError: No module named 'urlparse'
最初,我的Python 3在我的主要外部计算机(VM外部)上运行,似乎作者使用的是Python 2(仍然不知道Atom编辑器的flake 8如何理解这一点?).在检查urllib的Python 2/3问题后,( python 2和3从url中提取域和 Heroku日志在我使用导入urlparse 和 https://github.com/FriendCode/gittle/issues/49 ),我尝试了这些链接中提供的各种导入解决方案.我安装了Python 2.7(并通过$ python -V-> python2.7.13验证了它已设置为path.我什至尝试创建一个conda环境以确保它正在提取python2.7.13.
I initially had Python 3 running on my main external machine (outside the VM), and it seems as though the author was using Python 2 (still don't know how Atom editor's flake 8 was making sense of this?). Upon review of Python 2/3 issues with urllib, (python 2 and 3 extract domain from url and Heroku logs say "No module named 'urlparse'" when I use import urlparse and https://github.com/FriendCode/gittle/issues/49) I tried the various import solutions provided in these links. I installed Python 2.7 (and verified that it is set to path by $python -V -->python2.7.13. I even tried creating a conda enviroment to make sure it was pulling python2.7.13.
我的spider.py脚本如下:
My spider.py script is as follows:
import datetime
import urlparse
import socket
import scrapy
from Terms.items import TermsItem
# you have to import processors from scrapy.loader to use it
from scrapy.loader.processors import MapCompose, Join
# you have to import Itemloader from scrapy.loader to use it
from scrapy.loader import ItemLoader
class BasicSpider(scrapy.Spider):
name = "basic"
allowed_domains = ["web"]
start_urls = [i.strip() for i in open('lused.urls.txt').readlines()]
def parse(self, response):
l = ItemLoader(item=TermsItem(), response=response)
# Load fields using XPath expressions
l.add_xpath('term', '//h1[@class="head-entry"][1]/text()',
MapCompose(unicode.strip, unicode.title))
l.add_xpath('definition', '//*[@class="def-list"][1]/text()',
MapCompose(unicode.strip, unicode.title))
# Housekeeping fields
l.add_value('url', response.url)
l.add_value('project', self.settings.get('BOT_NAME'))
l.add_value('spider', self.name)
l.add_value('server', socket.gethostname())
l.add_value('date', datetime.datetime.now())
return l.load_item()
我的item.py脚本如下:
My item.py script is as follows:
from scrapy.item import Item, Field
class TermsItem(Item):
# Primary fields
term = Field()
definition = Field()
# Housekeeping fields
url = Field()
project = Field()
spider = Field()
server = Field()
date = Field()
在atom编辑器中,flake8 python检查器标志/下划线(adduced:已导入,但未使用"):
In atom editor, the flake8 python checker flags/underlines (adducing:'imported but not used):
导入urlparse"
'import urlparse'
'从scrapy.loader.processors导入MapCompose,加入'
'from scrapy.loader.processors import MapCompose, Join'
但是,当我在Atom编辑器中打开作者提供的VM中使用的几乎相同的代码时,它没有标记任何内容……并且代码运行!!!
However, when I open the virtually identical code used in the author's provided VM in Atom editor, it doesn't flag anything...and the code runs!!??
不幸的是,在尝试上述解决方案尝试后,我仍然遇到相同的错误结果.我希望其他人遇到此问题,或者可以根据上述详细信息发现我的错误.
Unfortunately, I am left with the same error result after trying the above solution attempts. I was hoping someone else encountered this problem or can spot my error based on the above details.
推荐答案
我(部分)通过 Martijn Pietters 的赞赏指导来解决了该问题. Scrapy正在扩展我的系统安装的Python-Python 3.6--而不是我的conda环境python --Python 2.7.基于 conda. io/docs/user-guide/tasks/manage-environments.htm 和miniconda安装文档,将miniconda添加到PATH可能会导致miniconda环境先于其他环境被调用(如果我理解正确的话).看来我需要使用Anaconda终端来创建conda环境.我最初使用的是powershell终端(认为在路径上添加miniconda就足够了).希望我已经足够熟练地解释了这一点,以使其他人避免我的错误.
I (partially) solved the problem, with the appreciated direction from Martijn Pietters. Scrapy was extending my system installed Python --Python 3.6-- instead of my conda environment python --Python 2.7. Based on conda.io/docs/user-guide/tasks/manage-environments.html and miniconda installation documentation, adding miniconda to the PATH can cause the miniconda environment to be called ahead of other environments (if I understand correctly). It seems that I need to use the Anaconda terminal to create a conda environment. I was initially using the powershell terminal (thinking that adding miniconda onto the path was sufficient). Hopefully I have explained this proficiently enough to have others avoid my mistake.
此致
这篇关于urlparse:ModuleNotFoundError,大概在Python2.7和conda下的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!