将多个Scrapy数据插入mysql [英] Insert multiple Scrapy data into mysql

问题描述

这是我的python代码，用于从 https://bpbd抓取新闻数据. jatengprov.go.id/category/laporan-bencana/'

This is my python code to crawling news data from https://bpbd.jatengprov.go.id/category/laporan-bencana/'

# -*- coding: utf-8 -*-
import scrapy
class BepeSpider(scrapy.Spider):
name = 'bepe'
allowed_domains = ['bpbd.jatengprov.go.id']
start_urls = ['https://bpbd.jatengprov.go.id/category/laporan-bencana/']
COUNT_MAX = 100
count = 0

def parse(self, response):
    for quote in response.css('div.post'):
        item = {
            'judul': quote.css('h2.post-title > a::text').extract_first(), 
            'teks': quote.css('div.entrytext > p::text').extract_first(), 
            'tag': quote.css('div.up-bottom-border > p.postmetadata > a::text').extract(),
        }
        yield item
        self.count = self.count + 1
        #following pagination link
        next_page_url = response.css('div.alignright > a::attr(href)').extract_first() #dapatkan link untuk selanjutnya

    if (self.count < self.COUNT_MAX):
        next_page_url = response.urljoin(next_page_url)
        yield scrapy.Request(url=next_page_url, callback=self.parse)

有没有办法像这样的数组将我的爬网数据插入到mysql中?

Is there any way to INSERT my crawling data into mysql with such array like this?

item = {
        'judul': quote.css('h2.post-title > a::text').extract_first(), 
        'teks': quote.css('div.entrytext > p::text').extract_first(), 
        'tag': quote.css('div.up-bottom-border > p.postmetadata > a::text').extract(),
    }

我在下面尝试过代码，但是无法插入任何数据

I have tried code below but it couldnt insert any data

        conn = Connection()
        mycursor = conn.cursor()
        sql = "insert into berita(judul, isi, tag) values(%s, %s, %s)"

        item = {
            'judul': quote.css('h2.post-title > a::text').extract_first(), 
            'teks': quote.css('div.entrytext > p::text').extract_first(), 
            'tag': quote.css('div.up-bottom-border > p.postmetadata > a::text').extract(),
        }
        val=(item['judul'], item['teks'], item['tag'])
        mycursor.execute(sql,val)
        conn.commit()

对不起，我的英语不好，我希望python的任何专家都能为我提供帮助

Sorry for my bad english and I hope anybody expert in python could help me

推荐答案

经过一番谷歌搜索和阅读问答，我找到了插入列表的方法(如我之前所说的多个数组).

After some googling and read Q&A I found the way to insert list (multiple array as I said before).

基本上，我遵循youtube上的刮y教程，因此该课程无法以这种方式将数据查询到mysql中.在我能够抓取数据之后，我需要插入数据

Basically I follow scrapy tutorial from youtube and the lesson not come this way to query the data into mysql. And after I able to crawling the data I need to insert the data

首先，我们必须首先了解我们的数据类型，因此我发现我的数据类型是列表而不是数组.这是我在youtube scrapy教程中找不到的代码

The first thing we have to know what our data type first so I found my data type is list not array. Then this is my code that not found in the youtube scrapy tutorial

item1 = quote.css('h2.post-title > a::text').extract_first()
item2 = quote.css('div.entrytext > p::text').extract_first()        
item3 = quote.css('div.up-bottom-border > p.postmetadata > a::text').extract()
items3 = ', '.join(item3)

然后查询在下面

mycursor.execute("INSERT INTO berita (judul, isi, tag) VALUES (%s, %s, %s)", (item1, item2, items3))

希望它能为您提供帮助

这篇关于将多个Scrapy数据插入mysql的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！