纱线：yarn-site.xml更改不起作用 [英] Yarn : yarn-site.xml changes not taking effect

问题描述

我们有一个在HDFS 2.7.3上运行的Spark应用程序，使用Yarn作为资源管理器....在运行应用程序时......这两个文件夹

  / tmp / hadoop / data / nm-local-dir / filecache 
 $ b $ / tmp / hadoop / data / nm-local-dir / filecache 
  

正在填满，因此磁盘.....



所以根据我的研究发现，在yarn-site.xml中配置这两个属性将有助于



 < property> 
< name> yarn.nodemanager.localizer.cache.cleanup.interval-ms< / name> 
<值> 2000< /值> 
< / property> 
 
<属性> 
< name> yarn.nodemanager.localizer.cache.target-size-mb< / name> 
<值> 2048< /值> 
< / property> 
  

我已经在每个namenode和masternode上配置了它们，并重新启动了masternode上的纱线..但仍然似乎没有帮助...请让我知道，如果我失去了什么......在做配置更改...这是yarn-site.xml

 <！ - 网站特定的YARN配置属性 - > 
<属性> 
< name> yarn.nodemanager.aux-services< / name> 
< value> mapreduce_shuffle< /值> 
< / property> 
 
<属性> 
< name> yarn.resourcemanager.hostname< / name> 
<值> hdfs-name-node< /值> 
< / property> 
 
<属性> 
< name> yarn.nodemanager.resource.memory-mb< / name> 
<值> 16384< /值> 
< / property> 
 
<属性> 
< name> yarn.nodemanager.resource.cpu-vcores< / name> 
< value> 6< /值> 
< / property> 
 
<属性> 
< name> yarn.scheduler.maximum-allocation-mb< / name> 
<值> 16384< /值> 
< / property> 
 
<属性> 
< name> yarn.nodemanager.local-dirs< / name> 
< value> file：/// tmp / hadoop / data / nm-local-dir< / value> 
< / property> 
 
<属性> 
< name> yarn.nodemanager.localizer.cache.cleanup.interval-ms< / name> 
<值> 2000< /值> 
< / property> 
 
<属性> 
< name> yarn.nodemanager.localizer.cache.target-size-mb< / name> 
<值> 2048< /值> 
< / property> 
 
< / configuration> 
  

更新：我尝试过使用此属性，但仍未清理并且磁盘已满。



 < property> 
< name> yarn.nodemanager.local-dirs< / name> 
< value> file：/// tmp / hadoop / data / nm-local-dir，file：/// tmp / hadoop / data / nm-local-dir / filecache，file：/// tmp / hadoop的/数据/纳米本地-DIR / usercache< /值GT; 
< / property> 
  




解决方案

你说文件夹/ tmp / hadoop /数据/ nm-local-dir / filecache正在填满，因此应定期清理。



但是您为清理指定的参数，特别是yarn.nodemanager.localizer.cache.cleanup.interval-ms和yarn.nodemanager.localizer.cache。 target / size / mb适用于/ tmp / hadoop / data / nm-local-dir / filecache，但file：/// tmp / hadoop / data / nm-local-dir不包括路径中的filecache。将值file：/// tmp / hadoop / data / nm-local-dir修复为file：/// tmp / hadoop / data / nm-local-dir / filecache。

We have a spark streaming application running on HDFS 2.7.3 with Yarn as the resource manager....while running the application .. these two folders

/tmp/hadoop/data/nm-local-dir/filecache

/tmp/hadoop/data/nm-local-dir/filecache

are filling up and hence the disk .....

so from my research found that configuring these two properties in yarn-site.xml will help

<property>
        <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
        <value>2000</value>
</property>

<property>
         <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
         <value>2048</value>
</property>

i have configured them on each namenode and masternode ..and restarted the yarn on masternode ..but still doesn't seem to help ...please let me know if i'm missing anything ...while doing the configuration changes ...this is the yarn-site.xml

<!-- Site specific YARN configuration properties -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hdfs-name-node</value>
</property>

<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>16384</value>
</property>

<property>
    <name>yarn.nodemanager.resource.cpu-vcores</name>
    <value>6</value>
</property>

<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>16384</value>
</property>

<property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>file:///tmp/hadoop/data/nm-local-dir</value>
</property>

<property>
    <name>yarn.nodemanager.localizer.cache.cleanup.interval-ms</name>
    <value>2000</value>
</property>

<property>
     <name>yarn.nodemanager.localizer.cache.target-size-mb</name>
     <value>2048</value>
</property>

</configuration>

Update: i tried with this property but still doesn't clean up and disk is getting full

<property>
        <name>yarn.nodemanager.local-dirs</name>
        <value>file:///tmp/hadoop/data/nm-local-dir,file:///tmp/hadoop/data/nm-local-dir/filecache,file:///tmp/hadoop/data/nm-local-dir/usercache</value>
   </property>

解决方案

You say that the folder "/tmp/hadoop/data/nm-local-dir/filecache" is filling up so should be cleaned up periodically.

But the parameter you specified for the cleanup, specifically "yarn.nodemanager.localizer.cache.cleanup.interval-ms" and "yarn.nodemanager.localizer.cache.target-size-mb" is applicable to "/tmp/hadoop/data/nm-local-dir/filecache" but the "yarn.nodemanager.local-dirs" value of "file:///tmp/hadoop/data/nm-local-dir" does not include the "filecache" in the path. Fix the value "file:///tmp/hadoop/data/nm-local-dir" to "file:///tmp/hadoop/data/nm-local-dir/filecache".

这篇关于纱线：yarn-site.xml更改不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！