Scrapy/Python 和 SQL Server [英] Scrapy / Python and SQL Server

问题描述

是否可以使用 Scrapy 从网站上抓取数据，并将该数据保存在 Microsoft SQL Server 数据库中?

Is it possible to get the data scraped from websites using Scrapy, and saving that data in an Microsoft SQL Server Database?

如果是，是否有这样做的例子?主要是Python问题吗?即如果我发现一些 Python 代码保存到 SQL Server 数据库，那么 Scrapy 可以做同样的事情吗?

If Yes, are there any examples of this being done? Is it mainly a Python issue? i.e. if I find some code of Python saving to an SQL Server database, then Scrapy can do same?

推荐答案

是的，但是您必须自己编写代码才能完成，因为 scrapy 不提供写入数据库的项目管道.

Yes, but you'd have to write the code to do it yourself since scrapy does not provide an item pipeline that writes to a database.

阅读项目管道页面更详细地描述该过程的scrapy 文档(这里有一个 JSONWriterPipeline 为例).基本上，找到一些写入 SQL Server 数据库的代码(使用 PyODBC 之类的东西)，您应该能够对其进行调整以创建直接将项目输出到 SQL Server 数据库的自定义项目管道.

Have a read of the Item Pipeline page from the scrapy documentation which describes the process in more detail (here's a JSONWriterPipeline as an example). Basically, find some code that writes to a SQL Server database (using something like PyODBC) and you should be able to adapt that to create a custom item pipeline that outputs items directly to a SQL Server database.

这篇关于Scrapy/Python 和 SQL Server的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！