Windows上的Spark - winutils究竟是什么，为什么我们需要它？ [英] Spark on Windows - What exactly is winutils and why do we need it?

问题描述

我很好奇！据我所知，HDFS需要datanode进程才能运行，这就是为什么它只能在服务器上运行。 Spark虽然可以在本地运行，但需要winutils.exe，它是Hadoop的一个组件。但它究竟做了什么？如何在Windows上运行Hadoop，但是我可以运行基于Hadoop的Spark？

解决方案

我至少知道一个用法，它用于在Windows OS上运行shell命令。您可以在 org.apache.hadoop.util.Shell 中找到它，其他模块依赖于此类并使用它的方法，例如 getGetPermissionCommand（） 方法：
$ b

  static final String WINUTILS_EXE =winutils.exe ; 
 ... 
 static {
 IOException ioe = null; 
 String path = null; 
文件file = null; 
 //不变：要么有一个有效的文件和路径，
 //或者存在缓存的IO异常。 
 if（WINDOWS）{
 try {
 file = getQualifiedBin（WINUTILS_EXE）; 
 path = file.getCanonicalPath（）; 
 ioe = null; 
} catch（IOException e）{
 LOG.warn（找不到{}：{}，WINUTILS_EXE，e）; 
 //堆栈跟踪出现在调试级别
 LOG.debug（找不到+ WINUTILS_EXE，e）; 
 file = null; 
 path = null; 
 ioe = e; 
} 
} else {
 //在非windows系统上，通过添加明确的异常，不变量保持
 // //。 
 ioe = new FileNotFoundException（E_NOT_A_WINDOWS_SYSTEM）; 
} 
 WINUTILS_PATH =路径; 
 WINUTILS_FILE =文件; 
 
 WINUTILS =路径; 
 WINUTILS_FAILURE = ioe; 
 
 
 public static String getWinUtilsPath（）{
 if（WINUTILS_FAILURE == null）{
 return WINUTILS_PATH; 
} else {
抛出新的RuntimeException（WINUTILS_FAILURE.toString（），
 WINUTILS_FAILURE）; 
 
 
 
 public static String [] getGetPermissionCommand（）{
 return（WINDOWS）？ new String [] {getWinUtilsPath（），ls，-F} 
：new String [] {/ bin / ls，-ld}; 
} 
  




I'm curious! To my knowledge, HDFS needs datanode processes to run, and this is why it's only working on servers. Spark can run locally though, but needs winutils.exe which is a component of Hadoop. But what exactly does it do? How is it, that I cannot run Hadoop on Windows, but I can run Spark, which is built on Hadoop?

解决方案

I know of at least one usage, it is for running shell commands on Windows OS. You can find it in org.apache.hadoop.util.Shell, other modules depends on this class and uses it's methods, for example getGetPermissionCommand() method:

static final String WINUTILS_EXE = "winutils.exe";
...
static {
  IOException ioe = null;
  String path = null;
  File file = null;
  // invariant: either there's a valid file and path,
  // or there is a cached IO exception.
  if (WINDOWS) {
    try {
      file = getQualifiedBin(WINUTILS_EXE);
      path = file.getCanonicalPath();
      ioe = null;
    } catch (IOException e) {
      LOG.warn("Did not find {}: {}", WINUTILS_EXE, e);
      // stack trace comes at debug level
      LOG.debug("Failed to find " + WINUTILS_EXE, e);
      file = null;
      path = null;
      ioe = e;
    }
  } else {
    // on a non-windows system, the invariant is kept
    // by adding an explicit exception.
    ioe = new FileNotFoundException(E_NOT_A_WINDOWS_SYSTEM);
  }
  WINUTILS_PATH = path;
  WINUTILS_FILE = file;

  WINUTILS = path;
  WINUTILS_FAILURE = ioe;
}
...
public static String getWinUtilsPath() {
  if (WINUTILS_FAILURE == null) {
    return WINUTILS_PATH;
  } else {
    throw new RuntimeException(WINUTILS_FAILURE.toString(),
        WINUTILS_FAILURE);
  }
}
...
public static String[] getGetPermissionCommand() {
  return (WINDOWS) ? new String[] { getWinUtilsPath(), "ls", "-F" }
                   : new String[] { "/bin/ls", "-ld" };
}

这篇关于Windows上的Spark - winutils究竟是什么，为什么我们需要它？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！