使用基于 Spark 数据集的 ML API 时初始化逻辑回归系数? [英] Initializing logistic regression coefficients when using the Spark dataset-based ML APIs?

问题描述

默认情况下，逻辑回归训练将系数初始化为全零.但是，我想自己初始化系数.这将很有用，例如，如果之前的训练运行在多次迭代后崩溃——我可以简单地使用最后一组已知的系数重新开始训练.

By default, logistic regression training initializes the coefficients to be all-zero. However, I would like to initialize the coefficients myself. This would be useful, for example, if a previous training run crashed after several iterations -- I could simply restart training with the last known set of coefficients.

是否可以使用任何基于数据集/数据帧的 API，最好是 Scala?

Is this possible with any of the dataset/dataframe-based APIs, preferably Scala?

看Spark源码，好像有一个方法setInitialModel来初始化模型及其系数，

Looking at the Spark source code, it seems that there is a method setInitialModel to initialize the model and its coefficients, but it's unfortunately marked as private.

基于 RDD 的 API 似乎允许初始化系数:LogisticRegressionWithSGD.run(...) 的重载之一接受 initialWeights 向量.但是，我想使用基于数据集的 API 而不是基于 RDD 的 API，因为(1)前者支持弹性网络正则化(我不知道如何使用基于 RDD 的逻辑回归来做弹性网络)和(2) 因为基于RDD的API处于维护模式.

The RDD-based API seems to allow initializing coefficients: one of the overloads of LogisticRegressionWithSGD.run(...) accepts an initialWeights vector. However, I would like to use the dataset-based API instead of the RDD-based API because (1) the former supports elastic net regularization (I couldn't figure out how to do elastic net with the RDD-based logistic regression) and (2) because the RDD-based API is in maintenance mode.

我总是可以尝试使用反射来调用私有的 setInitialModel 方法，但如果可能的话我想避免这种情况(也许这甚至行不通......我也无法分辨如果 setInitialModel 有充分的理由被标记为私有).

I could always try using reflection to call that private setInitialModel method, but I would like to avoid this if possible (and maybe that wouldn't even work... I also can't tell if setInitialModel is marked private for a good reason).

推荐答案

随意覆盖该方法.是的，您需要将该类复制到您自己的工作区中.没关系:不要害怕.

Feel free to override the method. Yes you will need to copy that class into your own work area. That's fine: do not fear.

当您构建项目时 - 无论是通过 maven 还是 sbt - 您的类的本地副本将获胜"并在 MLlib.幸运的是，同一个包中的其他类不会被着色.

When you build your project -either via maven or sbt - your local copy of the class will "win" and shade the one in MLlib. Fortunately the other classes in that same package will not be shaded.

我多次使用这种方法来覆盖 Spark 类:实际上您的构建时间也应该很短.

I have used this approach many times with overriding Spark classes: actually your build times should be small as well.

这篇关于使用基于 Spark 数据集的 ML API 时初始化逻辑回归系数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！