需要裁剪+调整大小~300000个文件。运行时间= 4天以上。如何加快我的bash脚本？ [英] Need to crop+resize ~300000 files. Runtime = 4+ days. How can I speed up my bash script?

问题描述

我正在制作视频游戏中时光倒流。我拍的所有照片都是以4：3宽高比拍摄的.jpg图像。 2592x1944分辨率。我希望它们在1920x1080时都是16：9。

I am working on creating a video timelapse. All the photos I took are .jpg images shot at 4:3 aspect ratio. 2592x1944 resolution. I want them all to be 16:9 at 1920x1080.

我写了一个小脚本来做这个，但过程不是很快。我花了大约17分钟来裁剪和调整750张图片。我总共需要处理大约300,000个，并且可能会在大约50,000个批次中进行处理。这是每批18小时45分钟，计算总数超过4.5天。

I have written a little script to do this, but the process is not very fast. It took about 17 minutes for me to crop and resize 750 images. I have a total of about 300,000 to deal with, and will probably be doing then in batches of about 50,000. That is 18 hours 45 minutes per batch, and over 4.5 days of computing total.

所以有人知道我可以加速这个程序吗？

So does anyone know a way I can speed up this program?

这里是我写的bash脚本：

here is the bash script I have written:

#!/bin/bash  

mkdir cropped

for f in *.JPG
do
    convert $f -resize 1920x1440 -set filename:name '%t' cropped/'%[filename:name].JPG' #Resize Photo, maintain aspect ratio
    convert cropped/$f -crop 1920x1080+0+$1 -set filename:name '%t' cropped/'%[filename:name].JPG' #Crop to 16:9 aspect ratio, takes in $1 argument for where to begin crop
done

echo Cropping Complete!

在循环中每行之前和之后放置一些echo命令显示调整大小比裁剪花费更多时间，我想这并不奇怪。我尝试过使用 mogrify -path cropped -resize 1920x1440！ $ f 代替转换$ f -resize 但速度似乎没有太大差异。

Putting some echo commands before and after each line within the loop reveals that resizing takes much more time than cropping, which I guess is not surprising. I have tried using mogrify -path cropped -resize 1920x1440! $f in place of convert $f -resizebut there does not seem to be much of a difference in speed.

那么，我可以用任何方式加快运行时间吗？

So, any way I can speed up the runtime on this?

奖励积分如果你能展示的话我可以轻松地在程序运行时给出一个简单的进度指示（例如750个文件中的421个，完成56.13％）。

BONUS POINTS if you can show me an easy way to give a simple indication of progress as the program runs (something like "421 of 750 files, 56.13% complete").

EXTRA BONUS要点如果您可以添加命令从每个帧输出.mp4文件，可以在SONY Vegas等软件程序中编辑。我已经设法使用这些照片中的mencoder制作视频文件（.avi），但结果视频无法在我尝试过的任何视频编辑器中使用。

EXTRA BONUS POINTS if you can add a command to output a .mp4 file from each frame that can be edited in a software program like SONY Vegas. I have managed to make video files (.avi) using mencoder from these photos, but the resulting video wont work in any video editors I have tried.

推荐答案

我想起了一些事情...

A few things spring to mind...

首先，不要每张图像启动两次ImageMagick，一次调整大小，一次调整它什么时候应该可以一次完成两个操作。所以，而不是你的两个转换命令，我只做一个

Firstly, don't start ImageMagick twice per image, once to resize it and once to crop it when it should be possible to do both operations in one go. So, instead of your two convert commands, I would do just one

convert image.jpg -resize 1920x1440 -crop 1920x1080+0+$1 cropped/image.jpg

其次，我不知道你用 set 命令做了什么，有文件名，但是你可以在shell中做到这一点。

Secondly, I don't see what you are doing with the set command, something with the filename, but you can just do that in the shell.

第三，我建议你使用GNU Parallel（我每天定期处理超过65,000张图像）。它易于安装，并确保您支付的所有可爱CPU核心都保持忙碌状态。使用它的最简单方法是，而不是运行命令，只需回显它们并将它们输入 parallel

Thirdly, I would suggest you use GNU Parallel (I regularly process upwards of 65,000 images per day with it). It is easy to install and will ensure all those lovely CPU cores you paid for are kept busy. The easiest way to use it is, instead of running commands, just echo them and pipe them into parallel

#!/bin/bash  
mkdir cropped

for f in *.jpg
do
   echo convert \"$f\" -resize 1920x1440 -crop 1920x1080+0+$1 cropped/\"$f\"
done  | parallel

echo Cropping Complete!

最后，如果你想要一个进度表，或者表明完成了多少以及剩下什么do，使用 - eta 选项（eta =预计到达时间）到 parallel ，它会告诉你有多少工作和剩余时间。

Finally, if you want a progress meter, or indication of how much is done and what is left to do, use the --eta option (eta=Estimated Time of Arrival) to parallel and it tells you how many jobs and how much time is remaining.

当您对 parallel 充满信心时，您可能会像执行整个流程一样运行这个：

When you get confident with parallel you will maybe run your entire process like this:

parallel --eta convert {} -resize 1920x1440 -crop 1920x1080+0+32 cropped/{} ::: *.jpg

我创建了750张与你相同尺寸的图像并以这种方式运行它需要我的中等规格iMac 55秒调整大小和裁剪 - YMMV。请添加评论并说明你是如何进行的 - 处理时间为 parallel 。

I created 750 images the same size as yours and ran them this way and it takes my medium spec iMac 55 seconds to resize and crop the lot - YMMV. Please add a comment and say how you got on - how long the processing time is with parallel.

这篇关于需要裁剪+调整大小~300000个文件。运行时间= 4天以上。如何加快我的bash脚本？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！