Bashshell-for loop-echo命令后显示参数太长的消息 [英] Bashshell - argument too long message shows after for loop-echo command
本文介绍了Bashshell-for loop-echo命令后显示参数太长的消息的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我正在使用bash shell在CentOS上工作。
我在代码中没有看到任何错误，但是它也会保留打印参数列表长错误。
不仅适用于 aws cp ，对于 cat / dev / null> $ FILE_DIR / dailey_member.csv 也会打印相同的消息。
echo ===== dailey由成员投票
cat / dev / null> $ FILE_DIR / dailey_member.csv
QUERY_DAILEY_MEMBER_RS =`mysql -h $ MYSQL_URL -u $ MYSQL_USER -p $ MYSQL_PW $ MYSQL_SCHEMA -e从tbl_vote GROUP BY pollDate，usrId选择SELECT pollDate，usrId，count（usrId）;
IFS = $'\n'LINES =（$ QUERY_DAILEY_MEMBER_RS）
for i in $ {！LINES [@]};做
LINE_IDX = $ i
LINE = $ {LINES [$ LINE_IDX]}
LINE = $ {LINE // $'\t'/，}
echo -e $ LINE>> $ FILE_DIR / dailey_member.csv
完成
echo aws s3 cp $ FILE_DIR / dailey_member.csv $ S3_PATH＃复制输出有效。
aws s3 cp $ FILE_DIR / dailey_member.csv $ S3_PATH
退出0
我该如何解决？
谢谢。
仅供参考，
我正在尝试什么要做的就是...
- 执行MYSQL查询
- 解析结果并写入csv文件
- 发送到s3存储库，以便人们可以下载文件
======= =======已编辑＃1 ===============
我将变量设置为文件...
export FILE_DIR = / home / jyoh / applications18 / cron / file
export S3_PATH = s3：// juneyoung / applications18 / voteLogs /
结果将显示在控制台上。
[jyoh @ ip-XX-XX-X-XXX cron] $ sh ./daileyVoteByMember.sh
=== == dailey成员
的投票aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3：// juneyoung / applications18 / voteLogs /
./daileyVoteByMember.sh：第27行： / usr / bin / aws：参数l ist太长
[jyoh @ ip-XX-XX-X-XXX cron] $ aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3：// juneyoung / applications18 / voteLogs /
上传：file / dailey_member.csv到s3：//juneyoung/applications18/voteLogs/dailey_member.csv
line27是 aws s3 cp $ FILE_DIR / dailey_member.csv $ S3_PATH 所在的地方。
= =============编辑过的＃2 ===============
带有选项的结果 -x 。
...跳过...
+ LINE_IDX = 265782
+ LINE ='20180129 9> 2387463256785462364 1'
+ LINE ='20180129,9> 2387463256785462364,1'
+ echo -e'20180129， 9> 2387463256785462364,1'
+ for'''$ {！LINES [@]}'
+ LINE_IDX = 265783
+ LINE ='20180129 9> 3658743656387567834 1'
+ LINE ='20180129,9> 3658743656387567834,1'
+ echo -e'20180129,9> 3658743656387567834,1'
+ echo'aws s3 cp / home / jyoh / applications18 / cron /文件/dailey_member.csv s3：// jun eyoung / applications18 / voteLogs /'
+ aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3：// juneyoung / applications18 / voteLogs /
./daileyVoteByMember.sh：行27：/ usr / bin / aws：参数列表太长
+退出0
= =============编辑过的＃3 ================
复制 aws 在新的bash文件中并执行
文件内容
＃！ / bin / bash
aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3：// juneyoung / applications18 / voteLogs /
exit 0
控制台输出
[jyoh @ ip-XX-XX-X-XXX cron] $ sh testSh.sh
上传：file / dailey_member.csv到s3：// juneyoung / applications18 / voteLogs /
==============已编辑＃4 ========= =======
这可以正常工作。
$ b $也许
for 和>> 文件重定向有问题？ b 函数sendS3（）{
echo S3 send command：aws s3 cp $ 1 $ S3_PATH
aws s3 cp $ 1 $ S3_PATH
}
echo =====每日投票
mysql -h $ MYSQL_URL -u $ MYSQL_USER -p $ MYSQL_PW $ MYSQL_SCHEMA -e选择查询＃1; | sed $’s / \t /，/ g’> $ FILE_DIR /daily.csv
sendS3 $ FILE_DIR /daily.csv
echo =====戴利成员投票
mysql -h $ MYSQL_URL -u $ MYSQL_USER -p $ MYSQL_PW $ MYSQL_SCHEMA -e选择查询＃2; | sed $’s / \t /，/ g’> $ FILE_DIR /daily_member.csv
sendS3 $ FILE_DIR /dailey_member.csv
echo ===== ip的每日投票
mysql -h $ MYSQL_URL -u $ MYSQL_USER -p $ MYSQL_PW $ MYSQL_SCHEMA -e选择查询＃3 | sed $’s / \t /，/ g’> $ FILE_DIR /daily_ip.csv
sendS3 $ FILE_DIR /daily_ip.csv
echo =====成员，项目的每日投票
mysql -h $ MYSQL_URL -u $ MYSQL_USER -p $ MYSQL_PW $ MYSQL_SCHEMA -e选择查询＃4 | sed $’s / \t /，/ g’> $ FILE_DIR /daily_item.csv
sendS3 $ FILE_DIR /daily_item.csv
exit 0
===============编辑过的＃5 ==============
这是完整的脚本。为了安全起见，我替换了url和一些变量。
＃！ / bin / bash
导出SLACK_URL = SLACK_API_URL
导出MYSQL_URL = MYSQL_CONNECTION_URL
导出MYSQL_USER = MYSQL_USER
导出MYSQL_PW = MYSQL_PW
export MYSQL_SCHEMA = USER
export FILE_DIR = FILE_PATH
export QUERY_DAILY_RS =
export QUERY_DAILY_MEMBER_RS =
export QUERY_DAILY_IP_RS =
export QUERY_DAILY_MEMBER_ITEM_RS =
export S3_PATH = S3_URL_TO_STORE
echo -e发送投票报告给Slack
echo $ MYSQL_URL
echo $ MYSQL_USER :: $ MYSQL_PW
函数sendS3（）{
echo S3 send command：aws s3 cp $ 1 $ S3_PATH
aws s3 cp $ 1 $ S3_PATH
}
echo =====每日投票
cat / dev / null> $ FILE_DIR / daily.csv
QUERY_DAILY_RS =`mysql SELECT QUERY 1;`
IFS = $'\n'LINES =（$ QUERY_DAILY_RS）
for i in $ {！LINES [ @]};做
LINE_IDX = $ i
LINE = $ {LINES [$ LINE_IDX]}
LINE = $ {LINE // $'\t'/，}
echo -e $ LINE>> $ FILE_DIR /daily.csv
完成
sendS3 $ FILE_DIR / daily.csv
echo =====每天由成员投票
cat / dev / null> $ FILE_DIR / daily_member.csv
QUERY_DAILY_MEMBER_RS =`mysql SELECT QUERY 2`
IFS = $'\n'LINES =（$ QUERY_DAILY_MEMBER_RS）
for i in $ {！LINES [@ ]};做
LINE_IDX = $ i
LINE = $ {LINES [$ LINE_IDX]}
LINE = $ {LINE // $'\t'/，}
echo -e $ LINE>> $ FILE_DIR / daily_member.csv
完成
sendS3 $ FILE_DIR / daily_member.csv
echo =====每日投票ip
cat / dev / null> $ FILE_DIR / daily_ip.csv
QUERY_DAILY_IP_RS =`mysql SELECT QUERY 3`
IFS = $'\n'LINES =（$ QUERY_DAILY_IP_RS）
for i in $ {！LINES [@ ]};做
LINE_IDX = $ i
LINE = $ {LINES [$ LINE_IDX]}
LINE = $ {LINE // $'\t'/，}
echo -e $ LINE>> $ FILE_DIR / daily_ip.csv
完成
sendS3 $ FILE_DIR / daily_ip.csv
echo =====每日投票member，item
cat / dev / null> $ FILE_DIR / daily_member_item.csv
QUERY_DAILY_MEMBER_ITEM_RS =`mysql SELECT QUERY 4`
IFS = $'\n'LINES =（$$ QUERY_DAILY_MEMBER_ITEM_RS）
for i in $ {！LINES [@ ]};做
LINE_IDX = $ i
LINE = $ {LINES [$ LINE_IDX]}
LINE = $ {LINE // $'\t'/，}
echo -e $ LINE>> $ FILE_DIR / daily_member_item.csv
完成
sendS3 $ FILE_DIR / daily_member_item.csv
curl -X POST -H'Content-type：application / json'--data '{ text：投票数据收集完成}'$ SLACK_URL
exit 0
解决方案
您的代码设置了太多（或太大）的环境变量。
环境变量与命令行参数在同一进程空间中，因此将变量导出到环境会减少命令行参数的可用空间。 （如果您没有在循环中显式导出内容，请检查脚本中的 echo $-是否包含 a ，一个标志，指示外壳程序自动导出每个变量集；如果存在，请找到打开该选项的位置并禁用它。） p>
要演示这一点（使用GNU xargs ），可以运行：
xargs --show-limits< / dev / null
输出应该类似于：
您的环境变量占用2615个字节
POSIX参数长度的上限（此系统）：257481
POSIX参数长度的最小允许上限（所有系统）：4096
我们可以实际使用的最大命令长度使用：254866
我们实际使用的命令缓冲区的大小：131072
最大并行度（--max-procs不能更大）：2147483647
从第一行的数字中减去第二行的数字，这是您的最大命令行长度（大约）。第一行数字太大？查看您的环境，或者取消设置大变量，或者首先不要导出。
I am working on CentOS with bash shell.
I do not see anything wrong in the code, but it keeps print arguments list too long error.
It is not only apply to aws cp, after for cat /dev/null > $FILE_DIR/dailey_member.csv also prints same message.
echo " ===== dailey vote by member"
cat /dev/null > $FILE_DIR/dailey_member.csv
QUERY_DAILEY_MEMBER_RS=`mysql -h $MYSQL_URL -u$MYSQL_USER -p$MYSQL_PW $MYSQL_SCHEMA -e "SELECT voteDate, usrId, count(usrId) FROM tbl_vote GROUP BY voteDate, usrId;"`
IFS=$'\n' LINES=($QUERY_DAILEY_MEMBER_RS)
for i in "${!LINES[@]}"; do
LINE_IDX=$i
LINE=${LINES[$LINE_IDX]}
LINE=${LINE//$'\t'/,}
echo -e $LINE >> $FILE_DIR/dailey_member.csv
done
echo "aws s3 cp $FILE_DIR/dailey_member.csv " $S3_PATH # copy output works.
aws s3 cp "$FILE_DIR/dailey_member.csv" $S3_PATH
exit 0
How could I fix this?
Thanks.
FYI, What I am trying to do is...
- do MYSQL query
- parse result and write as csv file
- send to s3 repository, so people can download file
============== edited #1 ================
I set my variable like on top of the file ...
export FILE_DIR="/home/jyoh/applications18/cron/file"
export S3_PATH="s3://juneyoung/applications18/voteLogs/"
The result puts on the console.
[jyoh@ip-XX-XX-X-XXX cron]$ sh ./daileyVoteByMember.sh
===== dailey vote by member
aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/
./daileyVoteByMember.sh: line 27: /usr/bin/aws: arguments list too long
[jyoh@ip-XX-XX-X-XXX cron]$ aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/
upload: file/dailey_member.csv to s3://juneyoung/applications18/voteLogs/dailey_member.csv
line27 is where aws s3 cp "$FILE_DIR/dailey_member.csv" $S3_PATH is.
============== edited #2 ================
result with option -x.
...SKIP...
+ LINE_IDX=265782
+ LINE='20180129 9>2387463256785462364 1'
+ LINE='20180129,9>2387463256785462364,1'
+ echo -e '20180129,9>2387463256785462364,1'
+ for i in '"${!LINES[@]}"'
+ LINE_IDX=265783
+ LINE='20180129 9>3658743656387567834 1'
+ LINE='20180129,9>3658743656387567834,1'
+ echo -e '20180129,9>3658743656387567834,1'
+ echo 'aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/'
+ aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/
./daileyVoteByMember.sh: line 27: /usr/bin/aws: arguments list too long
+ exit 0
============== edited #3 ================
Copy aws in new bash file and execute
the file contents
#! /bin/bash
aws s3 cp /home/jyoh/applications18/cron/file/dailey_member.csv s3://juneyoung/applications18/voteLogs/
exit 0
console output
[jyoh@ip-XX-XX-X-XXX cron]$ sh testSh.sh
upload: file/dailey_member.csv to s3://juneyoung/applications18/voteLogs/
============== edited #4 ================
This works fine and clean. Maybe some problem with for and >> file redirection?
function sendS3 () {
echo "S3 send command : aws s3 cp $1 $S3_PATH"
aws s3 cp $1 $S3_PATH
}
echo " ===== daily vote"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #1;" | sed $'s/\t/,/g' > "$FILE_DIR"/daily.csv
sendS3 "$FILE_DIR"/daily.csv
echo " ===== dailey vote by member"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #2;" | sed $'s/\t/,/g' > "$FILE_DIR"/daily_member.csv
sendS3 "$FILE_DIR"/dailey_member.csv
echo " ===== daily vote by ip"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #3" | sed $'s/\t/,/g' > "$FILE_DIR"/daily_ip.csv
sendS3 "$FILE_DIR"/daily_ip.csv
echo " ===== daily vote by member,item"
mysql -h "$MYSQL_URL" -u"$MYSQL_USER" -p"$MYSQL_PW" "$MYSQL_SCHEMA" -e "SELECT QUERY #4" | sed $'s/\t/,/g' > "$FILE_DIR"/daily_item.csv
sendS3 "$FILE_DIR"/daily_item.csv
exit 0
============== edited #5 ================
This is the full script. I replaced url and some variables for security.
#! /bin/bash
export SLACK_URL="SLACK_API_URL"
export MYSQL_URL="MYSQL_CONNECTION_URL"
export MYSQL_USER="MYSQL_USER"
export MYSQL_PW="MYSQL_PW"
export MYSQL_SCHEMA="USER"
export FILE_DIR="FILE_PATH"
export QUERY_DAILY_RS=""
export QUERY_DAILY_MEMBER_RS=""
export QUERY_DAILY_IP_RS=""
export QUERY_DAILY_MEMBER_ITEM_RS=""
export S3_PATH="S3_URL_TO_STORE"
echo -e "Sending vote report to Slack"
echo $MYSQL_URL
echo $MYSQL_USER::$MYSQL_PW
function sendS3 () {
echo "S3 send command : aws s3 cp $1 $S3_PATH"
aws s3 cp $1 $S3_PATH
}
echo " ===== daily vote"
cat /dev/null > $FILE_DIR/daily.csv
QUERY_DAILY_RS=`mysql SELECT QUERY 1;`
IFS=$'\n' LINES=($QUERY_DAILY_RS)
for i in "${!LINES[@]}"; do
LINE_IDX=$i
LINE=${LINES[$LINE_IDX]}
LINE=${LINE//$'\t'/,}
echo -e $LINE >> "$FILE_DIR"/daily.csv
done
sendS3 "$FILE_DIR/daily.csv"
echo " ===== daily vote by member"
cat /dev/null > $FILE_DIR/daily_member.csv
QUERY_DAILY_MEMBER_RS=`mysql SELECT QUERY 2`
IFS=$'\n' LINES=($QUERY_DAILY_MEMBER_RS)
for i in "${!LINES[@]}"; do
LINE_IDX=$i
LINE=${LINES[$LINE_IDX]}
LINE=${LINE//$'\t'/,}
echo -e $LINE >> $FILE_DIR/daily_member.csv
done
sendS3 "$FILE_DIR/daily_member.csv"
echo " ===== daily vote by ip"
cat /dev/null > $FILE_DIR/daily_ip.csv
QUERY_DAILY_IP_RS=`mysql SELECT QUERY 3`
IFS=$'\n' LINES=($QUERY_DAILY_IP_RS)
for i in "${!LINES[@]}"; do
LINE_IDX=$i
LINE=${LINES[$LINE_IDX]}
LINE=${LINE//$'\t'/,}
echo -e $LINE >> $FILE_DIR/daily_ip.csv
done
sendS3 "$FILE_DIR/daily_ip.csv"
echo " ===== daily vote by member,item"
cat /dev/null > $FILE_DIR/daily_member_item.csv
QUERY_DAILY_MEMBER_ITEM_RS=`mysql SELECT QUERY 4`
IFS=$'\n' LINES=($QUERY_DAILY_MEMBER_ITEM_RS)
for i in "${!LINES[@]}"; do
LINE_IDX=$i
LINE=${LINES[$LINE_IDX]}
LINE=${LINE//$'\t'/,}
echo -e $LINE >> $FILE_DIR/daily_member_item.csv
done
sendS3 "$FILE_DIR/daily_member_item.csv"
curl -X POST -H 'Content-type: application/json' --data '{"text":"Vote data collecting done"}' $SLACK_URL
exit 0
解决方案
Your code is setting too many (or too-large) environment variables.
Environment variables live in the same per-process space as command line arguments, so exporting variables to the environment reduces the space available for command-line arguments. (If you aren't explicitly exporting content inside your loop, check whether echo $- inside your script contains a, a flag which tells the shell to automatically export every variable set; if it does, find the place where you're turning that option on, and disable it).
To demonstrate this (with GNU xargs), you can run:
xargs --show-limits </dev/null
Output should look something like:
Your environment variables take up 2615 bytes
POSIX upper limit on argument length (this system): 257481
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 254866
Size of command buffer we are actually using: 131072
Maximum parallelism (--max-procs must be no greater): 2147483647
Subtract the number on the second line from the number on the first line, and that's your maximum command-line length (roughly). Number on the first line too big? Look through your environment, and either unset large variables, or don't export them in the first place.
这篇关于Bashshell-for loop-echo命令后显示参数太长的消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
