Prestashop背景批量产品导入 [英] Prestashop background bulk product import

问题描述

我正在尝试在Prestashop电子商务中定期导入有效的1.5万种产品，解析ASCII文件(3.5MB)，并且仅使用Prestashop的API.

I'm trying to import efficiently 15K+ products periodically in a Prestashop e-commerce parsing an ASCII file (3.5MB) and using only Prestashop's API.

全部在docker下运行，并带有来自docker hub的官方映像.

All run under docker, with official images from docker hub.

如果不存在具有相同参考字段的产品，则必须插入新产品(如果有)以对其进行更新.我开发了一个模块，可以通过单击一个自定义管理选项卡来使其实现此功能，但是它可以正常工作，但是整个系统将冻结，直到该过程完成或终止:(将近77分钟).我还尝试将(不是这样的)大文件分成500、100、50的块，但是处理时间线性减少，没有太大帮助:

If a product with the same reference field does not exists I'll have to insert a new product, if it's present to update it. I developed a module that makes this via a click un a custom admin tab, and it works, but the whole system freezes till the process is finished or terminated: (almost 77 minutes). I also tried to split the (not so) big file in chunks of 500, 100, 50 but the time for the processing decreases linearly, it doesn't help very much:

• 500个元素的平均153秒
• 100个元素的平均31秒
• 50个元素平均15秒

我肯定可以每90秒配置一个cron来处理50个元素，并在每晚7-8小时的时间内完成整个导入，但这似乎是一个非常糟糕的妥协:每90秒离线15秒.

I could surely configure a cron every 90 seconds to process 50 elements, and complete the whole import in 7-8 nightly hours, but it seems a very bad compromise: 15 seconds offline every 90.

我不能使用 pthreads ，因为这将是生产Web服务器.

I cannot use pthreads since this will be a production web server.

我试图调整Apache来增加memory_limit，max_input_vars，max_execution_time的时间，但是没有任何区别:DB使用的450MB至550MB的RAM和服务器几乎相同.

I tried to tune Apache increasing memory_limit, max_input_vars, max_execution_time but without any differences: DB using from 450MB to 550MB of RAM and server almost identical.

Linux#1 SMP Debian 4.9.110-3 + deb9u6(2018-10-08)x86_64

Linux #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08) x86_64

Versione软件删除服务器:Apache/2.4.10(Debian)

Versione software del server: Apache/2.4.10 (Debian)

PHP版本:5.6.35

Versione di PHP: 5.6.35

memory_limit = 2048M

memory_limit=2048M

max_input_vars = 1000000

max_input_vars=1000000

max_execution_time = 600000

max_execution_time=600000

MySQL:5.6.40

MySQL: 5.6.40

我是不是以错误的方式面对问题，还是Prestashop的API无效且不是为批量(和有效)产品导入而设计的?

Am I facing the problem in the wrong way, or Prestashop's API are not performant and made for bulk (and performant) product import?

public function batchImportAllProductsFromFile($productsToBeInserted){
    foreach ($productsToBeInserted as $key => $customProduct ) {

        $productIDs = $this->getProductIDsByReference($customProduct->MS_CODMAG);
        if (sizeof($productIDs) == 0) {
            $product = new Product();
        } else if (sizeof($productIDs) == 1) {
            $product = new Product($productIDs[0]);
        } else {
            continue;
        }

        $product->reference = $customProduct->MS_CODMAG;
        $product->name = trim($customProduct->MS_DESCRIZIONE);
        $product->price = $customProduct->MS_PREZZO_1;
        $product->out_of_stock = ($customProduct ->MS_ESAURITO === "S" ? true : false);

        $category = null;

        $msGruppoConverted = $this->buildSubGroupCode($customProduct->MS_GRUPPO, $customProduct->MS_SGRUPPO);

        if ($customProduct->MS_GRUPPO !== 0 && $msGruppoConverted !== 0) {
            $product->id_category = [$customProduct->MS_GRUPPO, $msGruppoConverted];
        } else if ($customProduct->MS_GRUPPO === 0 && $msGruppoConverted !== 0) {
            $product->id_category = [$msGruppoConverted];
        } else if ($customProduct ->MS_GRUPPO !== 0 && $msGruppoConverted === 0) {
            $product->id_category = [$customProduct->MS_GRUPPO];
        }
        try {
            if (sizeof($productIDs) == 0) {
                if ($product->add()) {
                    $product->updateCategories($product->category);
                    $product->addFeatureProductImport($product->id, 1, $customProduct->MS_FAM);
                    //StockAvailable::setQuantity((int)$product->id, 0, $product->quantity, Context::getContext()->shop->id);
                }
            } else if (sizeof($productIDs) == 1) {
                if ($product->update()) {
                    $product->updateCategories($product->category);
                    $alreadySavedFeatures = $product->getFeaturesStatic($productIDs[0]);
                    if (sizeof($alreadySavedFeatures) != 1 || $alreadySavedFeatures[0] != $customProduct->MS_FAM) {
                        $product->deleteProductFeatures();
                        $product->addFeatureProductImport($product->id, 1, $customProduct->MS_FAM);
                    }
                }
            }
        } catch (Exception $e) {
            var_dump("Errore: ", $e, $product);
        }
    }
}

编辑22/10/2018:

EDIT 22/10/2018:

升级到PHP7.2并使用MariaDB 10.3.10并没有带来任何变化:时机仍然相同.带来的好处是安装了FS(EXT4)，数据库在数据库中使用选项 barrers = 0 在/etc/fstab 中存储了信息:性能从153秒提高到35秒500个元素，总共大约需要18分钟(以前是77分钟).

Upgrading to PHP7.2 and using MariaDB 10.3.10 brought me no changes: timing was still the same. What did brought benefits was to mount the FS (EXT4) where the DB stores info with option barrers=0 in /etc/fstab: performances improved from 153 to 35 seconds for 500 elements, resulting in circa 18mins totally (was 77).

仍然存在的问题是为什么系统在导入时变得无响应.

The problem that remains open is why the system became unresponsive while importing.

推荐答案

在更新产品之前，应检查产品是否需要更新.

You should check that your products needs an update before updating it.

这是在Prestashop上导入实体时的工作(高度简化):

Here is what I do (Highly simplified) when importing entities on Prestashop:

<?php

class myProductImporter {
    protected $products;
    protected $products_checksum;

    public function __construct($products) {
        // Your products from the csv file
        $this->products = $products;
        // Here you get an associative array of products references and checksums
        // ex: array('REF01158' => '489f9ze4f4ze9f49ze8', 'REF15616' => '48949844561233132')
        $this->products_checksum = getProductsChecksum();
    }

    public function run() {
        foreach ($this->products as $product) {
            // If the product ref is present in my checksum list, then its an update
            if (isset($this->products_checksum[$product['reference']])) {
                // If the checksum is different, the product needs an update
                if ($this->products_checksum[$product['reference']] != $this->getChecksum($product)) {
                    $this->updateProduct($product);
                }
            // Else it's a new product
            } else {
                $this->addProduct($product);
            }
        }
    }

    protected function updateProduct($product) {
        $PSProduct = getProductByReferebce($product['reference']);
        // Update your product and save its new checksum
    }

    protected function addProduct($product) {
        $PSProduct = new Product();
        // Create the product and save its checksum
    }

    protected function getChecksum($product) {
        // Create a string containing all your product properties
        $checksum = $product['reference'];
        $checksum .= $product['name'];
        $checksum .= $product['description'];
        $checksum .= $product['id_category'];
        return md5($checksum);
    }
}

在不进行任何修改的情况下，您的导入将立即完成，因为不会处理任何查询.

When there's no modification, your import will finish instantly because there's no queries being processed.

关于安装的冻结，这似乎是Docker问题，并且未链接到Prestashop.

Concerning the freezes on your installation it seems to be a Docker problem and not linked to Prestashop.

如@ bruno-leveque所述，您应该考虑升级到PHP7.

As mentioned by @bruno-leveque you should consider upgrading to PHP7.

这篇关于Prestashop背景批量产品导入的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！