HDP 2.5：Spark History Server UI不会显示不完整的应用程序 [英] HDP 2.5: Spark History Server UI won't show incomplete applications

问题描述

我使用 Hortonworks Data Platform 2.5 设置了一个新的Hadoop集群。在旧集群中（安装 HDP 2.4 ），我可以通过点击链接来查看有关运行Spark作业的信息。显示不完整的申请：

在新安装中，此链接打开页面，但始终是未找到未完成的应用程序！（当仍有应用程序在运行时）。

我刚才看到， YARN ResourceManager UI 在Tracking UI列中显示了两种不同类型的链接，取决于Spark应用程序的状态：

• 应用程序正在运行：Application Master
  
  
  
  
  • 此链接打开 http：// master_url：8088 / proxy / application_1480327991583_0010 /
    
  
  
  • 应用程序已完成：History
    
    
    
    
    • 此链接打开 http：// master_url：18080 / history / application_1480327991583_0009 / jobs /
    
    
  
  
  
  
  

  通过YARN RM链接，我可以看到运行Spark应用程序的信息，但为什么我不能访问通过Spark历史服务器用户界面？是否有一些事情从HDP 2.4改为2.5？

  解决方案我解决了这个问题，这是一个网络问题：某些集群主机（火花从属）由于错误的开关配置而无法到达对方。发现它，因为我试图ping每台主机。
  
  
  

  因为所有主机都可以ping每台主机，所以问题消失了，我可以看到活动和完成的作业在我的Spark历史记录服务器用户界面中再次显示！

  
  
  

  我没有注意到这个问题，因为ambari-agents在每个主机上工作，ambari服务器也可以从每个群集主机！但是，由于所有主机都可以互相访问，因此问题得以解决！

  
  

  I set-up a new Hadoop Cluster with Hortonworks Data Platform 2.5. In the "old" cluster (installed HDP 2.4) I was able to see the information about running Spark jobs via the History Server UI by clicking the link show incomplete applications:

  Within the new installation this link opens the page, but it always sais No incomplete applications found! (when there's still an application running).

  I just saw, that the YARN ResourceManager UI shows two different kind of links in the "Tracking UI" column, dependent on the status of the Spark application:

  • application running: Application Master
    • this link opens http://master_url:8088/proxy/application_1480327991583_0010/
  • application finished: History
    • this link opens http://master_url:18080/history/application_1480327991583_0009/jobs/

  Via the YARN RM link I can see the running Spark app infos, but why can't I access them via Spark History Server UI? Was there somethings changed from HDP 2.4 to 2.5?

  解决方案

  I solved it, it was a network problem: Some of the cluster hosts (Spark slaves) couldn't reach each other due to a incorrect switch configuration. Found it out, as I tried to ping each host from each other.

  Since all hosts can ping each other hosts the problem is gone and I can see active and finished jobs in my Spark History server UI again!

  I didn't noticed the problem, because the ambari-agents worked on each host, and the ambari-server was also reachable from each cluster host! However, since ALL hosts can reach each other the problem is solved!

  这篇关于HDP 2.5：Spark History Server UI不会显示不完整的应用程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！