Windows上的Spark - winutils究竟是什么,为什么我们需要它? [英] Spark on Windows - What exactly is winutils and why do we need it?
问题描述
我至少知道一个用法,它用于在Windows OS上运行shell命令。您可以在 org.apache.hadoop.util.Shell
中找到它,其他模块依赖于此类并使用它的方法,例如 getGetPermissionCommand() 方法:
$ b
static final String WINUTILS_EXE =winutils.exe ;
...
static {
IOException ioe = null;
String path = null;
文件file = null;
//不变:要么有一个有效的文件和路径,
//或者存在缓存的IO异常。
if(WINDOWS){
try {
file = getQualifiedBin(WINUTILS_EXE);
path = file.getCanonicalPath();
ioe = null;
} catch(IOException e){
LOG.warn(找不到{}:{},WINUTILS_EXE,e);
//堆栈跟踪出现在调试级别
LOG.debug(找不到+ WINUTILS_EXE,e);
file = null;
path = null;
ioe = e;
}
} else {
//在非windows系统上,通过添加明确的异常,不变量保持
// //。
ioe = new FileNotFoundException(E_NOT_A_WINDOWS_SYSTEM);
}
WINUTILS_PATH =路径;
WINUTILS_FILE =文件;
WINUTILS =路径;
WINUTILS_FAILURE = ioe;
public static String getWinUtilsPath(){
if(WINUTILS_FAILURE == null){
return WINUTILS_PATH;
} else {
抛出新的RuntimeException(WINUTILS_FAILURE.toString(),
WINUTILS_FAILURE);
public static String [] getGetPermissionCommand(){
return(WINDOWS)? new String [] {getWinUtilsPath(),ls,-F}
:new String [] {/ bin / ls,-ld};
}
I'm curious! To my knowledge, HDFS needs datanode processes to run, and this is why it's only working on servers. Spark can run locally though, but needs winutils.exe which is a component of Hadoop. But what exactly does it do? How is it, that I cannot run Hadoop on Windows, but I can run Spark, which is built on Hadoop?
I know of at least one usage, it is for running shell commands on Windows OS. You can find it in org.apache.hadoop.util.Shell
, other modules depends on this class and uses it's methods, for example getGetPermissionCommand()
method:
static final String WINUTILS_EXE = "winutils.exe";
...
static {
IOException ioe = null;
String path = null;
File file = null;
// invariant: either there's a valid file and path,
// or there is a cached IO exception.
if (WINDOWS) {
try {
file = getQualifiedBin(WINUTILS_EXE);
path = file.getCanonicalPath();
ioe = null;
} catch (IOException e) {
LOG.warn("Did not find {}: {}", WINUTILS_EXE, e);
// stack trace comes at debug level
LOG.debug("Failed to find " + WINUTILS_EXE, e);
file = null;
path = null;
ioe = e;
}
} else {
// on a non-windows system, the invariant is kept
// by adding an explicit exception.
ioe = new FileNotFoundException(E_NOT_A_WINDOWS_SYSTEM);
}
WINUTILS_PATH = path;
WINUTILS_FILE = file;
WINUTILS = path;
WINUTILS_FAILURE = ioe;
}
...
public static String getWinUtilsPath() {
if (WINUTILS_FAILURE == null) {
return WINUTILS_PATH;
} else {
throw new RuntimeException(WINUTILS_FAILURE.toString(),
WINUTILS_FAILURE);
}
}
...
public static String[] getGetPermissionCommand() {
return (WINDOWS) ? new String[] { getWinUtilsPath(), "ls", "-F" }
: new String[] { "/bin/ls", "-ld" };
}
这篇关于Windows上的Spark - winutils究竟是什么,为什么我们需要它?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!