如何在python中使用scrapy获取直接父节点? [英] How to get immediate parent node with scrapy in python?

问题描述

我是 scrapy 的新手.我想从网上抓取一些数据.我得到了如下所示的 html 文档.

dom style1:<div class="user-info"><p class="用户名">p 标签中的东西</p>我想要的文本数据

dom样式2:<div class="user-info"><div><p class="user-img">p 标签中的东西</p>div 标签中的东西

<div><p class="用户名">p 标签中的东西</p>我想要的文本数据

我想获取数据我想要的文本数据，现在我可以使用css或xpath 选择器通过检查它是否存在来获取它.但我想知道一些更好的方法.比如我可以先得到css p.user-name，然后得到它的父，然后得到它的div/text()，而且我想要的数据总是 p.user-name 的直接父 div 的 text()，但是问题是，我怎样才能获得直接父 p.user-name?

解决方案

使用 xpath，您可以在 css 不支持的各个方向(父级、兄弟级、子级等)遍历 xml 树.
对于您的情况，您可以使用 xpath .. 父符号获取节点的父节点:

//p[@class='user-name']/../text()

说明:
//p[@class='user-name'] - 查找 <p> 具有类值 user-name 的节点.
/.. - 选择节点的父节点.
/text() - 选择当前节点的文本.

这个 xpath 应该适用于您描述的两种情况.

I am new to scrapy. I want to crawl some data from the web. I got the html document like below.

dom style1:
<div class="user-info">
    <p class="user-name">
        something in p tag
    </p>
    text data I want
</div>

dom style2:
<div class="user-info">
    <div>
        <p class="user-img">
            something in p tag
        </p>
        something in div tag
    </div>
    <div>
        <p class="user-name">
            something in p tag
        </p>
        text data I want
    </div>
</div>

I want to get the data text data I want, now I can use css or xpath selector to get it by check it exists. But I want to know some better ways. For example, I can get css p.user-name first, and then I get it's parent, and then I get it's div/text(), and always the data I want is the text() of the p.user-name's immediate parent div, but the question is, how can I get the immediate parent p.user-name?

解决方案

With xpath you can traverse the xml tree in every direction(parent, sibling, child etc.) where css doesn't support this.
For your case you can get node's parent with xpath .. parent notation:

//p[@class='user-name']/../text()

Explanation:
//p[@class='user-name'] - find <p> nodes with class value user-name.
/.. - select node's parent.
/text() - select text of the current node.

This xpath should work in both of your described cases.

这篇关于如何在python中使用scrapy获取直接父节点?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！