删除hdfs早于N天的分区文件夹 [英] delete partitions folders in hdfs older than N days

问题描述

我想删除N天以前的分区文件夹。

下面的命令给出了正好在50天前的文件夹。我想要所有小于50天的文件夹列表

  hadoop fs -ls / data / publish / DMPD / VMCP / staging / tvmcpr_usr_prof / chgdt =`date --date '50 days ago'+ \％Y-\％m-\％d` 
  

解决方案

您可以尝试使用solr hdfsfindtool：

  hadoop jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-job.jar org.apache.solr.hadoop.HdfsFindTool -find / data / publish / DMPD / VMCP / staging / tvmcpr_usr_prof -mtime +50 | xargs hdfs dfs -rm -r -skipTrash 
  

I want to delete the partition folders which are older than N days.

The below command gives the folders which are exactly 50 days ago. I want the list of all folders which are less than 50 days

hadoop fs -ls /data/publish/DMPD/VMCP/staging/tvmcpr_usr_prof/chgdt=`date --date '50 days ago' +\%Y-\%m-\%d`

解决方案

You can try with solr hdfsfindtool:

hadoop jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-job.jar org.apache.solr.hadoop.HdfsFindTool -find /data/publish/DMPD/VMCP/staging/tvmcpr_usr_prof -mtime +50 | xargs hdfs dfs -rm -r -skipTrash

这篇关于删除hdfs早于N天的分区文件夹的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！