Doctrine 2批量插入/更新内存 [英] Doctrine 2 bulk insert/update memory

问题描述

考虑以下代码(包装在函数中):

Consider this code (wrapped inside a function):

    $manager = $this->manager; // local ref
    $q = $manager->createQuery('select c from VendorFeedBundle:Category c');
    $iterableResult = $q->iterate();
    $i = 0;
    $batchSize = 500;

    foreach($iterableResult as $row) {
        $category = $row[0];

        $struct = explode(' ' . $this->separator . ' ', $category->getPath());
        unset($struct[count($struct) - 1]);
        $path = implode(' ' . $this->separator . ' ', $struct);

        if (!$parent = $this->repo->findOneBy(['path' => $path])) {
            continue;
        }

        $category->setParent($parent);

        // flush every x entities
        if (($i % $batchSize) == 0) {
            echo 'Flushing batch...' . "\n";
            echo 'Memory: ' . $this->getReadableSize(memory_get_usage()) . "\n";

            $manager->flush();
            $manager->clear();

            echo 'After batch...' . "\n";
            echo 'Memory: ' . $this->getReadableSize(memory_get_usage()) . "\n";
        }
        ++$i;
    }

    // flush the remaining
    $manager->flush();
    $manager->clear();

它将在我的终端中记录以下内容:

It logs the following in my terminal:

Creating tree...
Memory: 14.91 MB
Flushing batch...
Memory: 18.46 MB
After batch...
Memory: 18.79 MB
Flushing batch...
Memory: 21.01 MB
After batch...
Memory: 23.29 MB
Flushing batch...
Memory: 25.36 MB
After batch...
Memory: 27.87 MB
Flushing batch...
Memory: 29.88 MB
.... etc

getReadAbleSize方法不会将任何变量泄漏到全局范围或任何其他内容. 我已阅读并遵循有关doctrine2批量插入/更新(批处理)的建议: http://docs.doctrine-project.org/en/latest /reference/batch-processing.html

The getReadAbleSize method is not leaking any variables to a global scope or anything. I've read and followed advice about doctrine2 bulk inserts/updates (batch processing): http://docs.doctrine-project.org/en/latest/reference/batch-processing.html

我做错了什么? 每500个项目增加3〜4 MB内存对我来说似乎是一个(很小的)泄漏.

What am I doing wrong? 3~4 MB memory increase per 500 items seems like a (little) leak to me.

旁注:我需要以这种方式更新项目，因为我的系统分为2个进程.首先，我插入类别，然后更新父级关系.

Sidenote: I need to update the items this way because my system is split up in 2 processes; first I insert the categories and secondly I update the parent relationship.

我的类别类是基本的Doctrin2实体，其中添加了一些Gedmo扩展(树，可转换，可时间戳) 请参阅: http://pastie.org/private/oiiyf54zjuouhiqjsjislg

My category class is a basic Doctrin2 entity with a few Gedmo extensions added to it (tree, translateable, timestampable) See: http://pastie.org/private/oiiyf54zjuouhiqjsjislg

我完整的脚本(正在迭代和更新类别): http://pastie.org/private/k5x240vr4taepczhqa4tva

My complete script (which is iterating and updating categories): http://pastie.org/private/k5x240vr4taepczhqa4tva

推荐答案

1. 将SQL记录器设置为null

$em->getConnection()->getConfiguration()->setSQLLogger(null);

1. 在$em->clear()之后手动调用功能gc_collect_cycles()

1. Manually call function gc_collect_cycles() after $em->clear()

$em->clear(); gc_collect_cycles();

$em->clear(); gc_collect_cycles();

不要忘记设置 zend.enable_gc 设为1，或在使用前手动调用 gc_enable() gc_collect_cycles()

Don't forget to set zend.enable_gc to 1, or manually call gc_enable() before use gc_collect_cycles()

1. 如果从控制台运行命令，请添加--no-debug选项.

这篇关于Doctrine 2批量插入/更新内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！