在bash中使用“&"并行处理等一下 [英] Parallel processing in bash with "&" and wait

问题描述

因此，我试图从1台服务器中转储很多表，然后将它们恢复到另一台服务器中.我想使转储以1进行1，并在转储完成时恢复它们.我不想通过同时运行1个以上的还原操作使服务器不堪重负.因此，为了达到这一目的，我正在尝试拥有第三个函数，该函数在调用另一个函数之前先等待"恢复函数"完成.但是我无法正确使用等待".该程序根本没有等待.

So I am trying to dump a lot of tables from 1 server and then restore them back in another server. I want to keep the dump going 1 by 1 and restore them as the dump finishes. I dont want to overwhelm the server by running more than 1 restore operation at the same time. So to achive that i am trying to have a third function which "wait" for a "restore function" to finish before calling another. But I am not able to use the "wait" correctly. The program is not waiting at all.

RESTORE(){

sleep 10 && echo "restore done" &

}


RESTORE_CALLER() {


echo "waiting for any restore with pid ${current_restore_pid}"
wait ${current_restore_pid}


echo "Calling restore"
RESTORE &
current_restore_pid=$!



}


DUMP(){

for ((i=0;i<5;i++));do
echo "dumping "

echo "restore caller"
RESTORE_CALLER &

done

} 

DUMP 

推荐答案

只需用管道传输

seq 1 5 |
while read l; do
    DUMP > "$l".dump
    echo "$l"
done |
while read l; do
    RESTORE < "$l".dump
    echo "$l"
done

但是最好使用另一个描述符在管道之间传输数据，这样日志可以很好地打印出来:

But it's probably better to use another descriptor to transfer the data between the pipes, so that logs print nicely:

seq 1 5 |
while read l; do
   DUMP "$l"
   echo "$l" >&3
done 3> >(
    while read l; do
        RESTORE "$l"
    done
) |
cat

具有两个存根的示例执行:

Example execution with two stubs:

DUMP() {
    sleep 0.$(($RANDOM % 10))
    echo "DUMPING $1"
}
RESTORE() {
    sleep 0.$(($RANDOM % 10))
    echo "RESTORING $1"
}

看起来很酷:

DUMPING 1
RESTORING 1
DUMPING 2
RESTORING 2
DUMPING 3
DUMPING 4
DUMPING 5
RESTORING 3
RESTORING 4
RESTORING 5

最后需要| cat来同步进程替换.

The | cat on the end is needed to synchronize the process substitution.

这很酷，您可以使用xargs之类的工具轻松地并行化DUMP和RESTORE函数，并行运行3个DUMP和并行运行2个RESTORE:

What is cool about it, you can use tools like xargs to easily parallelize the DUMP and RESTORE functions, ex run 3 DUMPs in parallel and 2 RESTORE in parallel:

DUMP() {
    echo "DUMPING $1"
    sleep 0.$(($RANDOM % 10))
    echo "DUMPED $1"
}
RESTORE() {
    echo "RESTORING $1"
    sleep 0.$(($RANDOM % 10))
    echo "RESTORED $1"
}

export -f DUMP RESTORE
seq 1 5 |
xargs -n1 -P3 bash -c 'DUMP "$1"; echo "$1" >&3' -- 3> >(
    xargs -n1 -P2 bash -c 'RESTORE "$1"' --
) | cat

它看起来更酷:

DUMPING 1
DUMPING 2
DUMPING 3
DUMPED 3
RESTORING 3
DUMPING 4
DUMPED 4
RESTORED 3
RESTORING 4
DUMPING 5
DUMPED 1
RESTORING 1
RESTORED 4
DUMPED 2
RESTORING 2
DUMPED 5
RESTORED 2
RESTORING 5
RESTORED 1
RESTORED 5

这篇关于在bash中使用“&"并行处理等一下的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！