scipy.optimize.fmin_l_bfgs_b返回'ABNORMAL_TERMINATION_IN_LNSRCH' [英] scipy.optimize.fmin_l_bfgs_b returns 'ABNORMAL_TERMINATION_IN_LNSRCH'
问题描述
我正在使用scipy.optimize.fmin_l_bfgs_b解决高斯混合问题.混合物分布的均值通过回归建模，其权重必须使用EM算法进行优化.
I am using scipy.optimize.fmin_l_bfgs_b to solve a gaussian mixture problem. The means of mixture distributions are modeled by regressions whose weights have to be optimized using EM algorithm.
sigma_sp_new, func_val, info_dict = fmin_l_bfgs_b(func_to_minimize, self.sigma_vector[si][pj],
args=(self.w_vectors[si][pj], Y, X, E_step_results[si][pj]),
approx_grad=True, bounds=[(1e-8, 0.5)], factr=1e02, pgtol=1e-05, epsilon=1e-08)
但是有时我在信息字典中收到警告"ABNORMAL_TERMINATION_IN_LNSRCH":
But sometimes I got a warning 'ABNORMAL_TERMINATION_IN_LNSRCH' in the information dictionary:
func_to_minimize value = 1.14462324063e-07
information dictionary: {'task': b'ABNORMAL_TERMINATION_IN_LNSRCH', 'funcalls': 147, 'grad': array([ 1.77635684e-05, 2.87769808e-05, 3.51718654e-05,
6.75015599e-06, -4.97379915e-06, -1.06581410e-06]), 'nit': 0, 'warnflag': 2}
RUNNING THE L-BFGS-B CODE
* * *
Machine precision = 2.220D-16
N = 6 M = 10
This problem is unconstrained.
At X0 0 variables are exactly at the bounds
At iterate 0 f= 1.14462D-07 |proj g|= 3.51719D-05
* * *
Tit = total number of iterations
Tnf = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip = number of BFGS updates skipped
Nact = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F = final function value
* * *
N Tit Tnf Tnint Skip Nact Projg F
6 1 21 1 0 0 3.517D-05 1.145D-07
F = 1.144619474757747E-007
ABNORMAL_TERMINATION_IN_LNSRCH
Line search cannot locate an adequate point after 20 function
and gradient evaluations. Previous x, f and g restored.
Possible causes: 1 error in function or gradient evaluation;
2 rounding error dominate computation.
Cauchy time 0.000E+00 seconds.
Subspace minimization time 0.000E+00 seconds.
Line search time 0.000E+00 seconds.
Total User time 0.000E+00 seconds.
我并非每次都会收到此警告，但有时会收到. (大多数获得"CONVERGENCE:NORM_OF_PROJECTED_GRADIENT_< = _ PGTOL"或"CONVERGENCE:REL_REDUCTION_OF_F_< = _ FACTR * EPSMCH").
I do not get this warning every time, but sometimes. (Most get 'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL' or 'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH').
我知道这意味着可以在此迭代中达到最小值.我用谷歌搜索了这个问题.有人说它经常发生是因为目标和梯度函数不匹配.但是这里我不提供梯度函数，因为我正在使用'approx_grad'.
I know that it means the minimum can be be reached in this iteration. I googled this problem. Someone said it occurs often because the objective and gradient functions do not match. But here I do not provide gradient function because I am using 'approx_grad'.
我应该调查的可能原因是什么? 舍入误差主导计算"是什么意思?
What are the possible reasons that I should investigate? What does it mean by "rounding error dominate computation"?
======
我还发现对数似然不会单调增加:
I also find that the log-likelihood does not monotonically increase:
########## Convergence !!! ##########
log_likelihood_history: [-28659.725891322563, 220.49993177669558, 291.3513633060345, 267.47745327823907, 265.31567762171181, 265.07311121000367, 265.04217683341682]
通常不会在发生"ABNORMAL_TERMINATION_IN_LNSRCH"时从第二或第三次迭代开始减少.我不知道这个问题是否与上一个问题有关.
It usually start decrease at the second or the third iteration, even through 'ABNORMAL_TERMINATION_IN_LNSRCH' does not occurs. I do not know whether it this problem is related to the previous one.
推荐答案
Scipy调用原始的L-BFGS-B实现.这是一些fortran77(旧但漂亮且超快速的代码)，而我们的问题是下降方向实际上在上升.问题从第2533行开始(链接到底部的代码)
Scipy calls the original L-BFGS-B implementation. Which is some fortran77 (old but beautiful and superfast code) and our problem is that the descent direction is actually going up. The problem starts on line 2533 (link to the code at the bottom)
gd = ddot(n,g,1,d,1)
if (ifun .eq. 0) then
gdold=gd
if (gd .ge. zero) then
c the directional derivative >=0.
c Line search is impossible.
if (iprint .ge. 0) then
write(0,*)' ascent direction in projection gd = ', gd
endif
info = -4
return
endif
endif
换句话说，您是通过上山告诉它要下山.该代码在您提供的下降方向上尝试了20次称为线搜索"的操作，并意识到您不是在告诉它下坡，而是上坡.全部20次.
In other words, you are telling it to go down the hill by going up the hill. The code tries something called line search a total of 20 times in the descent direction that you provide and realizes that you are NOT telling it to go downhill, but uphill. All 20 times.
写这个的人(Jorge Nocedal，他是一个很聪明的人)投入了20，因为这已经足够了.机器epsilon是10E-16，我认为20实际上有点太多了.因此，对于大多数遇到此问题的人来说，我的钱是您的梯度与您的功能不匹配.
The guy who wrote it (Jorge Nocedal, who by the way is a very smart guy) put 20 because pretty much that's enough. Machine epsilon is 10E-16, I think 20 is actually a little too much. So, my money for most people having this problem is that your gradient does not match your function.
现在，也可能是"2.舍入误差主导计算".这样，他的意思是您的功能是一个非常平坦的表面，其增加量约为机器epsilon(在这种情况下，您可以重新缩放功能)， 现在，我想当您的功能太怪异时，也许应该有第三个选择.振荡?我可以看到类似$ \ sin({\ frac {1} {x}})$这样的问题.但是我不是一个聪明人，所以不要以为还有第三种情况.
Now, it could also be that "2. rounding errors dominate computation". By this, he means that your function is a very flat surface in which increases are of the order of machine epsilon (in which case you could perhaps rescale the function), Now, I was thiking that maybe there should be a third option, when your function is too weird. Oscillations? I could see something like $\sin({\frac{1}{x}})$ causing this kind of problem. But I'm not a smart guy, so don't assume that there's a third case.
因此，我认为OP的解决方案应该是您的功能过于平坦.或查看fortran代码.
So I think the OP's solution should be that your function is too flat. Or look at the fortran code.
https://github.com/scipy/scipy/blob/master/scipy/optimize/lbfgsb/lbfgsb.f
在此处进行行搜索以查找想要查看的人. https://en.wikipedia.org/wiki/Line_search
Here's line search for those who want to see it. https://en.wikipedia.org/wiki/Line_search
注意.这太晚了7个月.我把它放在这里是为了将来.
Note. This is 7 months too late. I put it here for future's sake.
这篇关于scipy.optimize.fmin_l_bfgs_b返回'ABNORMAL_TERMINATION_IN_LNSRCH'的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
