使用PHP和xPath从HTML提取数据 [英] Extracting data from HTML using PHP and xPath

问题描述

我正在尝试从网页中提取数据以将其插入数据库。我感兴趣的数据是在div中有一个class =company。在一个网页上，有15个或更少的div是这样的，有很多页面我试图从中提取这些数据。因此，我试图找到一个数据提取的自动解决方案。

I am trying to extract data from a webpage to insert it to a database. The data I'm interested in is in the div's which have a class="company". On one webpage there are 15 or less div's like that, and there are many pages I am trying to extract this data from. For this reason I am trying to find a automatic solution for data extraction.

具有class =company的div如下（有15个或更少的div这在一个页面上有不同的数据）：

The div with a class="company" is as follows (there are 15 or less divs like this on one page with different data):

<div class="company" id="company-6666"> <!-- EXTRACT 'company-6666' from id="company-6666" -->

  <div class="top clearfix">
    <div class="name clearfix">
      <h2>
        <a href="/company-name">Company Name</a>&nbsp; <!-- EXTRACT 'Company Name' from contents of A element and EXTRACT '/company-name' from href attribute -->
        <a href="/branches-list-link?parent_id=6666" class="branches">Branches <span>(5)</span></a> <!-- EXTRACT '/branches-list-link?parent_id=6666' from href attribute -->               
      </h2>
    </div>
  </div>

  <div class="inner clearfix has-logo">

    <div class="clearfix">          
      <div class="logo">
        <a href="/company-name">
          <img src="/graphics/company/logo/listing/123456.jpg?_ts=1365390237" border="0" alt="" /> <!-- EXTRACT '/graphics/company/logo/listing/123456.jpg?_ts=1365390237' from src attribute -->
        </a>
      </div>
      <div class="info">
        <div class="address">StreetName 500, 7777 City, County</div> <!-- EXTRACT 'StreetName 500, 7777 City, County' from contents of class="address" div -->
        <div class="clearfix">
          <div class="slogan">Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi ac condimentum mi.</div> <!-- EXTRACT 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi ac condimentum mi.' from contents of class="slogan" div -->
        </div>
      </div>
    </div>

    <div class="actions-bar clearfix">
      <ul>              
        <li><span class="phone-number">6666666</span></li> <!-- EXTRACT '6666666' from contents of class="phone-number" div -->
        <li><a href="mailto:[e-mail]" target="_blank" title="[e-mail]" class="email">[e-mail]</a></li> <!-- EXTRACT '[e-mail]' from contents of class="email" div -->
        <li><a href="http://www.webpage.com" target="_blank" title="www.webpage.com" class="redirect url">www.webpage.com</a></li> <!-- EXTRACT 'www.webpage.com' from contents of class="redirect url" div -->
      </ul>
    </div>

  </div>

</div>

到目前为止，我有以下PHP代码（$ output具有网页的HTML代码）：

So far I have the following PHP code (the $output has the webpage's HTML code):

<?php

$doc = new DomDocument();
@$doc->loadHTML($output);
$doc->preserveWhiteSpace = false; 

$xpath = new DomXPath($doc);

$elements = $xpath->query("//*[@class='company']");

if (!is_null($elements)) {
    foreach ($elements as $element) {
        echo $element->nodeValue;
    }
}

?>

似乎它获得了所有的15 div与class =company，但我不知道如何提取以前提到的（在HTML代码的注释中）个别的值。

It seems that it gets all of the 15 div's with class="company" but I have no idea how to extract the previously mentioned (in comments of HTML code) individual values.

每个div（我在谈论div = class =company）不会将所有值写入HTML块。所以不知何故，我必须询问公司div内的特定div，我感兴趣的数据是否存在，如果存在，我必须检查它是否不为空（包含标签之间的文本）。如果它存在并且不为空，我将其添加到变量中。

Every div (I am talking about the div with class="company") doesn't have all the values written in the HTML block. So somehow I have to make a query if a specific div inside the company div, where the data i'm interested in, exists and if it exists I have to check if it is not empty (contains text between tags or not). If it exists and is not empty I add it to a variable.

一旦提取了值，我想将它们分配给PHP变量，让我随后与他们一起工作。如果提取的值被放置在数组中会更好，如下所示：

Once values are extracted I would like to assign them to PHP variables which let's me to work with them afterwards. It would be even better if the values extracted are put in array like so:

$result = array(
    // 1'st div's data
    [0] =>  
        'company name' => 'company name',
        'company link' => 'company link',
        'company id' => 'company id',
        'company branches'  => 'branches link',
        'company logo'  => 'logo',
        'company address'  => 'address',
        'company slogan'  => 'slogan',
        'company webpage'  => 'webpage',
        'company email'  => 'email',
        'company phone'  => 'phone'

    // 2'nd div's data
    [1] =>  
        'company name' => 'company name',
        'company link' => 'company link',
        'company id' => 'company id',
        'company branches'  => 'branches link',
        'company logo'  => 'logo',
        'company address'  => 'address',
        'company slogan'  => 'slogan',
        'company webpage'  => 'webpage',
        'company email'  => 'email',
        'company phone'  => 'phone'
    ...
    )

推荐答案

每个公司可以由上下文节点表示，同时使每个属性由相对于它的xpath-expression表示：

Each Company can be represented by a context-node while having each property represented by an xpath-expression relative to it:

Company company-6666:
 ->id ....... = "company-6666"    --    string(@id)
 ->name ..... = "Company Name"    --    .//a[1]/text()
 ->href ..... = "/company-name"    --    .//a[1]/@href
 ->img ...... = "/graphics/company/logo/listing/123456.jpg?_ts=1365390237"    --    .//img[1]/@src
 ->address .. = "StreetName 500, 7777 City, County"    --    .//*[@class="address"]/text()
 ...

如果你把它包装到对象中，这是非常漂亮的使用：

If you wrap that into objects, this is pretty nifty to use:

$doc = new DOMDocument();
$doc->loadHTML($html);

/* @var $companies DOMValueObject[] */
$companies = new Companies($doc);

foreach ($companies as $company) {
    printf("Company %s:\n", $company->id);
    foreach ($company->getObjectProperties() as $name => $value) {
        $expression = $company->getPropertyExpression($name);
        printf(" ->%'.-10s = \"%s\"    --    %s\n", $name.' ', $value, $expression);
    }
}

这与 DOMObjectCollection 和 DOMValueObject ，定义您自己的类型：

This works with DOMObjectCollection and DOMValueObject, defining your own type:

class Companies extends DOMValueCollection
{
    public function __construct(DOMDocument $doc) {
        parent::__construct($doc, '//*[@class="company"]');
    }

    /**
     * @return DOMValueObject
     */
    public function current() {
        $object = parent::current();
        $object->defineProperty('id', 'string(@id)');
        $object->defineProperty('name', './/a[1]/text()');
        $object->defineProperty('href', './/a[1]/@href');
        $object->defineProperty('img', './/img[1]/@src');
        $object->defineProperty('address', './/*[@class="address"]/text()');
        # ... add your definitions
        return $object;
    }
}

对于你的数组要求，有一个 getArrayCopy（）方法：

And for your array requirements there is a getArrayCopy() method:

echo "\nGet Array Copy:\n\n";

print_r($companies->getArrayCopy());

输出：

Get Array Copy:

Array
(
    [0] => Array
        (
            [id] => company-6666
            [name] => Company Name
            [href] => /company-name
            [img] => /graphics/company/logo/listing/123456.jpg?_ts=1365390237
            [address] => StreetName 500, 7777 City, County
        )

)

这篇关于使用PHP和xPath从HTML提取数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！