Spark Netlib-Java BLAS [英] Spark netlib-java BLAS

问题描述

我正在尝试对无法正常工作的Apache Spark和Netlib设置进行故障排除，我不知道下一步该怎么做.

i am trying to troubleshoot my non-working apache spark and netlib setup and i don't know what to do next.

这里有一些信息:

• 火花1.3.1(但也尝试过1.5.1)
• 具有3个节点的Mesos群集

• Ubuntu Trusty在每个节点上并按照BLAS软件包安装

• Spark 1.3.1 (but also tried 1.5.1)
• Mesos Cluster with 3 Nodes

• Ubuntu Trusty on every node and installed following BLAS package

$ dpkg -l | grep 'blas\|atlas\|lapack'
ii  libopenblas-base                    0.2.8-6ubuntu1                   amd64        Optimized BLAS (linear algebra) library based on GotoBLAS2

$ update-alternatives --get-selections | grep 'blas\|lapack'
libblas.so.3                   auto     /usr/lib/openblas-base/libblas.so.3

我已经建立了一个示例jar，用于测试 netlib-java 是否可以检测到该库以下代码:

I have built a sample jar for testing if netlib-java can detect this libraries, with following code:

object Main extends App {
  println(com.github.fommil.netlib.BLAS.getInstance().getClass().getName())
  println(com.github.fommil.netlib.LAPACK.getInstance().getClass().getName())
}

当我执行此代码时，我得到以下响应:

When i execute this code i get following response:

$ java -jar artifacts/BLAStest-assembly-1.0.jar
Mar 29, 2016 3:43:33 PM com.github.fommil.netlib.BLAS <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
Mar 29, 2016 3:43:33 PM com.github.fommil.jni.JniLoader liberalLoad
INFO: successfully loaded /tmp/jniloader6790966128222263615netlib-native_ref-linux-x86_64.so
com.github.fommil.netlib.NativeRefBLAS
Mar 29, 2016 3:43:33 PM com.github.fommil.netlib.LAPACK <clinit>
WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
Mar 29, 2016 3:43:33 PM com.github.fommil.jni.JniLoader load
INFO: already loaded netlib-native_ref-linux-x86_64.so
com.github.fommil.netlib.NativeRefLAPACK

所以在这里看起来很好用. 但是spark无法检测到库.我已经将此Java依赖项添加到我的程序集jar中

So it seems to work just fine here. But spark can't detect the libraries. I have added this java dependency to my assembly jar

com.github.fommil.netlib:all:1.1.2

如果我尝试使用此程序包启动Spark Shell，也将无法正常工作.

also if i try to start a spark shell with this package it doesn't work.

spark-shell --packages com.github.fommil.netlib:all:1.1.2

推荐答案

看起来您的netlib-java实现正在加载NativeRefBLAS，但没有加载NativeSystemBLAS.这意味着您可以将"com.github.fommil.netlib:all"包含在内，因为没有它，您将使用非本机F2J实现.问题是您要使用系统提供的BLAS(OpenBLAS)，而不是netlib-java附带的参考实现.这可能只是在火花执行程序可见的位置获得正确的共享库的问题.

It looks like your netlib-java implementation is loading the NativeRefBLAS, but not the NativeSystemBLAS. This means that you're inclusion of "com.github.fommil.netlib:all" is working ok, since without it you would be using the non-native F2J implementation. The issue is that you want to use the system-provided BLAS (OpenBLAS), instead of the reference implementation that came with netlib-java. This probably just is a matter of getting the right shared libraries in a location that's visible to your spark executors.

您说您链接了libblas.so.3，但是如 netlib-java自述文件所述，还需要配置libblas.so，liblapack.so和liblapack.so.3:

You said that you linked libblas.so.3, but as described in the netlib-java readme, you need to also configure the libblas.so, liblapack.so, and liblapack.so.3:

sudo apt-get install libatlas3-base libopenblas-base
sudo update-alternatives --config libblas.so
sudo update-alternatives --config libblas.so.3
sudo update-alternatives --config liblapack.so
sudo update-alternatives --config liblapack.so.3

这篇关于Spark Netlib-Java BLAS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！