连接文件夹中的许多文件时，参数列表过长 [英] Argument list too long when concatenating lots of files in a folder

问题描述

这是将多个相同特征码文件连接成一个大文件的命令.在该文件夹中，我有77k文件，并且参数列表太长

This is a command to concatenate multiple same pattern files into one big files. In the folder I have 77k files, and I get Argument list too long

示例文件名是每分钟: cartreset-2014-05-08-01-12.log

Example file names are per minute: cartreset-2014-05-08-01-12.log

rm -f /tmp/temp.files
ls -1 /var/log/processing/*.log | \
    xargs -n1 basename > /tmp/temp.files
cat /tmp/temp.files | \
    sed -r "s~(.*)-[0-9]{4}(-[0-9]{2})+\.log~cat /var/log/processing/\1* >> /var/log/processing/\1$(date  +"-%Y-%m-%d-%H-%M").log~" | \
    uniq | \
    sh
cd /var/log/processing
xargs rm -rf < /tmp/temp.files
rm -f /tmp/temp.files

sh: /bin/ls: Argument list too long

我有什么选择?

推荐答案

您的完整代码是:

rm -f /tmp/temp.files
ls -1 /var/log/processing/*.log | xargs -n1 basename > /tmp/temp.files
cat /tmp/temp.files | sed -r "s~(.*)-[0-9]{4}(-[0-9]{2})+\.log~cat /var/log/processing/\1* >> /var/log/processing/\1$(date  +"-%Y-%m-%d-%H-%M").log~" | uniq | sh
cd /var/log/processing
xargs rm -rf < /tmp/temp.files
rm -f /tmp/temp.files

但是问题出在 ls -1/var/log/processing/*.log 部分，所以我跳过其余部分.

But the problem lies on the ls -1 /var/log/processing/*.log part, so I am skipping the rest.

由/var/log/processing/*.log 完成的扩展给出了太多结果，以致 ls 本身无法处理所有这些结果，因此打印了参数列表"太长"消息.

The expansion done by /var/log/processing/*.log gives so many results that ls itself cannot handle all of them and hence prints the "Argument list too long" message.

您可以像这样使用 find 语句:

You can use a find statement like this:

find /var/log/processing -name "*.log" -exec basename {} \; > /tmp/temp.files

请参阅我不使用ls解析(阅读有趣的为什么不应该解析ls的输出).

See I am not using ls parsing (read interesting Why you shouldn't parse the output of ls).

这篇关于连接文件夹中的许多文件时，参数列表过长的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！