pyomo 编写模型似乎很慢 [英] pyomo seems very slow to write models

问题描述

我有一个相当大的模型(大约 500 万个变量和约束).

I have a pretty big model (around 5 million variables and constraints).

构建时间是几分钟，求解时间也是几分钟(使用 gurobi)

The building time is a few minutes and the solving time is a few minutes too (with gurobi)

但是写模型需要很长时间(大约2个小时)

But it takes very long to write the model (about 2 hours)

这个时候如果我用model.write('model.lp', io_options={'symbolic_solver_labels': True}) 就可以记录了

This is the time if I use model.write('model.lp', io_options={'symbolic_solver_labels': True}) to be able to record it

如果我使用 SolverFactory 和 solve 直接来自 pyomo 的模型，时间差不多

It's about the same time if I use SolverFactory and solve directly the model from pyomo

这里有一个小例子，我知道这个模型对于 gurobi 来说是微不足道的，所以我没有在这里比较求解时间和构建时间，但我不明白为什么这么长，我认为是问题可能来自磁盘写入速度，但似乎磁盘从不超载，几乎没有使用

here is a little sample, I understand that this model is trivial for gurobi, so I'm not comparing the solving time with the building time here, but I don't understand why it's so long, I though that the problem could come from the disk writing speed, but it seems that the disk is never overloaded and almost not used

import pyomo.environ as pyo
import time

size = 500000

model = pyo.ConcreteModel()
model.set = pyo.RangeSet(0, size)
model.x = pyo.Var(model.set, within=pyo.Reals)
model.constrList = pyo.ConstraintList()
for i in range(size):
    model.constrList.add(expr = model.x[i] >= 1)
model.obj = pyo.Objective(expr=sum(model.x[i] for i in range(size)), sense=pyo.minimize)

opt = pyo.SolverFactory('gurobi')

_time = time.time()
res = opt.solve(model)
print(">>> total time () in {:.2f}s".format(time.time() - _time))

print(res)

结果是整个求解函数的时间为27秒，而gurobi的求解时间仅为4秒.

the results are that the time of the whole solve function is 27 s, but the solving time of gurobi is only 4 s.

推荐答案

根据我在加速 pyomo 模型生成方面的经验，您需要首先确定流程的哪个部分正在减慢它的速度.(这确实是性能调整的一般建议)

From my trails with speeding up pyomo model generation you need to benchmark first what part of the process is slowing it down. (Which is really a general advice for perfomance tuning)

所以我把你的代码放到一个函数中:

so I put you code into a function:

def main():
    size = 500000

    model = pyo.ConcreteModel()
    model.set = pyo.RangeSet(0, size)
    model.x = pyo.Var(model.set, within=pyo.Reals)
    model.constrList = pyo.ConstraintList()
    for i in range(size):
        model.constrList.add(expr = model.x[i] >= 1)
    model.obj = pyo.Objective(expr=sum(model.x[i] for i in range(size)), sense=pyo.minimize)
    return model

所以我可以通过 ipython 中的行分析器运行:

so I can run in through the line profiler in ipython:

In [1]: %load_ext line_profiler                                                                                                                                                                         

In [2]: import test_pyo                                                                                                                                                                                 

In [3]: %lprun -f test_pyo.main test_pyo.main()

这表明大部分时间都花在了model.constrList.add(expr = model.x[i] >= 1).

which shows that most of the time is spent in model.constrList.add(expr = model.x[i] >= 1).

通过将其移动到基于规则的约束中，我没有看到太大的改进，因此我决定尝试手动构建表达式，例如 PyPSA 代码.

I did not see much improvement by moving this into a rule based constraint so I decided to try to construct the expression by hand, like in the PyPSA code.

import pyomo.environ as pyo
import time
from pyomo.core.expr.numeric_expr import LinearExpression
from pyomo.core.base.constraint import _GeneralConstraintData
from pyomo.core.base.numvalue import NumericConstant

def main():
    size = 500000

    model = pyo.ConcreteModel()
    model.set = pyo.RangeSet(0, size)
    model.x = pyo.Var(model.set, within=pyo.Reals)
    setattr(model, "constraint", pyo.Constraint(model.set, noruleinit=True))
    v = getattr(model, "constraint")
    for i in v._index:
        v._data[i] = _GeneralConstraintData(None, v)
        expr = LinearExpression()
        expr.linear_vars = [model.x[i]]
        expr.linear_coefs = [1]
        expr.constant = 0
        v._data[i]._body = expr
        v._data[i]._equality = False
        v._data[i]._lower = NumericConstant(1)
        v._data[i]._upper = None

    model.obj = pyo.Objective(expr=pyo.quicksum(model.x[i] for i in range(size)), sense=pyo.minimize)
    return model

这似乎带来了大约 50% 的性能提升.线分析器显示，现在花费大量时间在创建集合、空的 LinearExpression 对象以及创建目标上.摆弄目标可能会稍微改善一些事情.

which seems to yield about 50% performance improvement. The line profiler shows that a lot of time is now spend in creating the set, the empty LinearExpression object, and also in creating the objective. It might be that fiddling with the objective might improve things a little more.

这篇关于pyomo 编写模型似乎很慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！