杂乱的外壳不起作用 [英] Scrapy shell doesn't work

问题描述

我对scrapy还是陌生的，所以我想尝试scrapy shell进行调试和学习，但是奇怪的是shell命令根本不起作用.

I am new to scrapy, then I want to try scrapy shell to debug and learn, but it's strange the shell command doesn't work at all.

• 似乎该网站已成功爬网，但没有输出更多内容.程序正在等待执行，似乎已死，我必须使用ctrl-c结束它.

您能帮忙找出问题所在吗?

can you help to figure out what's wrong?

我正在使用Anaconda + scrapy 1.0.3

I'm using Anaconda + scrapy 1.0.3

$ ping 135.251.157.2

Pinging 135.251.157.2 with 32 bytes of data:
Reply from 135.251.157.2: bytes=32 time=13ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56
Reply from 135.251.157.2: bytes=32 time=14ms TTL=56

Ping statistics for 135.251.157.2:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 13ms, Maximum = 14ms, Average = 13ms


$ scrapy shell "http://135.251.157.2/"
2016-01-28 21:35:18 [scrapy] INFO: Scrapy 1.0.3 started (bot: demo)
2016-01-28 21:35:18 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-01-28 21:35:18 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'demo.spiders', 'SPIDER_MODULES': ['demo.spiders'], 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'demo'}
2016-01-28 21:35:18 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-01-28 21:35:19 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, HttpProxyMiddleware, ChunkedTransferMiddleware, DownloaderStats
2016-01-28 21:35:19 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-28 21:35:19 [scrapy] INFO: Enabled item pipelines:
2016-01-28 21:35:19 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-28 21:35:19 [scrapy] INFO: Spider opened
2016-01-28 21:35:24 [scrapy] DEBUG: Crawled (200) <GET http://135.251.157.2/> (referer: None)
2016-01-28 21:35:24 [root] DEBUG: Using default logger
2016-01-28 21:35:24 [root] DEBUG: Using default logger
ctrl-c

$ 

推荐答案

我想关闭此线程，因为我发现根本原因与不同的终端有关.当我使用Git Bash时，它不起作用，但是如果我使用Anaconda Prompt，它会很好地工作.

I'd like to close this thread, as I find out the root cause is related with different terminal. When I use Git Bash, it doesn't work, but if I use Anaconda Prompt, it works quite well.

[Anaconda2] D:\SVN\tools\Spider\demo>scrapy shell "http://135.251.157.2/"
2016-01-29 13:40:33 [scrapy] INFO: Scrapy 1.0.3 started (bot: demo)
2016-01-29 13:40:33 [scrapy] INFO: Optional features available: ssl, http11, boto
2016-01-29 13:40:33 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'demo.spiders', 'SPIDER_MODULES': ['demo.spiders'], 'LOGSTATS_INTERVAL': 0, 'BOT_NAME': 'demo'}
2016-01-29 13:40:33 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2016-01-29 13:40:33 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, HttpProxyMiddleware, ChunkedTrans
ferMiddleware, DownloaderStats
2016-01-29 13:40:33 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2016-01-29 13:40:33 [scrapy] INFO: Enabled item pipelines:
2016-01-29 13:40:33 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-01-29 13:40:33 [scrapy] INFO: Spider opened
2016-01-29 13:40:41 [scrapy] DEBUG: Crawled (200) <GET http://135.251.157.2/> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x0136B290>
[s]   item       {}
[s]   request    <GET http://135.251.157.2/>
[s]   response   <200 http://135.251.157.2/>
[s]   settings   <scrapy.settings.Settings object at 0x034204B0>
[s]   spider     <DefaultSpider 'default' at 0x3e3c6d0>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser
2016-01-29 13:40:41 [root] DEBUG: Using default logger
2016-01-29 13:40:41 [root] DEBUG: Using default logger

In [1]:

这篇关于杂乱的外壳不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！