scikit-learn和keras多核命令n_jobs = -1 [英] Scikit-learn and keras multicore command n_jobs = -1
问题描述
我刚刚用Keras创建了一个人工神经网络,我想将Scikit-learn函数cross_val_score传递给它,以便在数据集的某些X_train和y_train上对其进行训练.
I just created an artificial neural network with Keras and I want to pass to it the Scikit-learn function cross_val_score to train it on some X_train and y_train of a data set.
import keras
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.model_selection import cross_val_score
def build_classifier():
classifier = Sequential()
classifier.add(Dense(units = 16, kernel_initializer = 'uniform', activation = 'relu', input_dim = 30))
classifier.add(Dense(units = 16, kernel_initializer = 'uniform', activation = 'relu'))
classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
classifier.compile(optimizer = 'rmsprop', loss = 'binary_crossentropy', metrics = ['accuracy'])
return classifier
classifier = KerasClassifier(build_fn = build_classifier, batch_size=25, epochs = 10)
results = cross_val_score(classifier, X_train, y_train, cv=10, n_jobs=-1)
我得到的输出是时代1/1重复了4次(我有4个内核),没有别的,因为在那之后它卡住了,计算永远也不会结束. 我使用任何其他Scikit学习算法测试了n_jobs = -1,并且工作正常.我不使用GPU,仅使用CPU.
The output I get is Epoch 1/1 repeated 4 times (I have 4 cores) and nothing else because after that it stucks and calculation never finishes. I tested n_jobs = -1 with any other Scikit-learn algorithms and it works fine. I'm not using GPU, only CPU.
要测试代码,只需添加以下标准化数据集:
To test the code just add the following normalized data set:
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
df = pd.DataFrame(data['data'])
target = pd.DataFrame(data['target'])
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(df, target, test_size = 0.2, random_state = 0)
# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
玩完n_jobs(设置为1,2,3或-1)后,我得到了一些奇怪的结果,例如时代1/1仅重复了3次而不是4次(即使使用n_jobs = -1)或重复了3次这就是我得到的内核:
After playing around with n_jobs (set to 1,2,3 or -1) I get some weird results, like Epoch 1/1 repeated only 3 times instead of 4 (even with n_jobs = -1) or when I interrupt the kernel here is what I get:
Process ForkPoolWorker-33:
Traceback (most recent call last):
File "/home/myname/anaconda3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/myname/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/myname/anaconda3/lib/python3.6/multiprocessing/pool.py", line 108, in worker
task = get()
File "/home/myname/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/pool.py", line 362, in get
return recv()
File "/home/myname/anaconda3/lib/python3.6/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/home/myname/anaconda3/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/home/myname/anaconda3/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
这可能是多处理中的某件事,但我不知道如何解决.
It could be something in multiprocessing but I don't know how to fix it.
推荐答案
我切换到sklearn版本= 0.20.1
I switched to sklearn version = 0.20.1
由于命令运行和完成的时间比n_jobs = 1更短,因此n_jobs现在可以正常运行.
Now the n_jobs issue "works", since the command runs and finishes in shorter time than n_jobs = 1.
尽管如此:
1)n_jobs = 2或更高时,计算时间没有明显改善
1) There is no sensible improvement in computation time for n_jobs = 2 or higher
2)在某些情况下,我会收到此警告:
2) In some cases I get this warning:
[Parallel(n_jobs=-1)]: Using backend LokyBackend with 2 concurrent workers.
/home/my_name/anaconda3/lib/python3.6/site-packages/sklearn/externals/joblib/externals/loky/process_executor.py:706:
UserWarning: A worker stopped while some jobs were given to the executor.
This can be caused by a too short worker timeout or by a memory leak.
"timeout or by a memory leak.", UserWarning
最后一句话:在Jupyter笔记本中,对于n_jobs!= 1而言,神经网络与历元的交互计算不再显示,而是在终端(!?)中显示
And last remark: the interactive computing of the neural network with the epochs has not been showing anymore for n_jobs != 1 in the Jupyter notebook, but in the terminal (!?)
这篇关于scikit-learn和keras多核命令n_jobs = -1的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!