使用xargs + curls bash脚本对URI进行压力测试失败，状态为空 [英] Stress testing URI using xargs + curls bash script failing with status empty

查看：73
本文介绍了使用xargs + curls bash脚本对URI进行压力测试失败，状态为空的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正在尝试对一个应用程序进行用户接受度测试，该应用程序对GET请求中包含的特定URL参数没有响应.

步骤

  1. 我有 curl ，并运行GET req(精心制作的)复制的curl语法(适用于Unix)，并进行了一些更改并复制到了ubuntu服务器.

     'https://abc.ai/getMultiDashboard/demouser'-H'Cookie:_ga = GA1.2.561275388.1601468723;_hjid = ecd3d778-b7f5-4f7f-b3ef-6f9f12b13d66;54651cc_an = 4;_gid = GA1.2.1366208807.1601560229;_hjTLDTest = 1;54651cc_data = JTdCJTIyaWQlMjIlM0Ellc3NUb2tlbiUyMiUzQSUyMjA2MTk3NjM3NTgwOGE2N2RmZjlhMmJlOWJmODE5NDQzJTIYJTdE;54651cc_loggedin = 1;54651cc_sound = true;54651cc_read = true;54651cc_popup = true;54651cc_disablelastseen = false;54651cc_usertype = loginuser;_hjIncludedInPageviewSample = 1;_hjAbsoluteSessionInProgress = 0;abc = s％3A8ZGd7Mol31n_Y8OCLq39dHoo3_mIlRhZ.pFQWz5gG9McKsQLzOikcTB​​mmb2Wcrxo％2B9u9iPpqoyxw;pageUrl =/#/dashboard/18;_gat_gtag_UA_97985973_5 = 1'" https://abc.ai/getTagTrends/E1_CPU_PERCENTAGE/2020-9-12%2013:4:0/202 **'23548'** 0-09-15 |％2013:04:00'"" https://abc.ai/getTagTrends/E1_CPU_PERCENTAGE/2020-9-12%2013:4:0/202 **'`23548`'** 0-09-15 |％2013:04:00'"

  2. ** 星号不是实际值的一部分；我用它们来划定我的 injected 值

  3. 使用一个小的bash脚本，我为Curl生成了数千个(唯一)有效载荷组合.

     <代码>#/bin/bash对于((i = 0; i <1000; ++ i));做回声'https://abc.ai/getMultiDashboard/demouser'-H'Cookie:_ga = GA1.2.561275388.1601468723;_hjid = ecd3d778-b7f5-4f7f-b3ef-6f9f12b13d66;54651cc_an = 4;_gid = GA1.2.1366208807.1601560229;_hjTLDTest = 1;54651cc_data = JTdCJTIyaWQlMjIlM0ElMjJkZW1vdXNlciU yMiUyQyUyMm4lMjIlM0ElMjJkZW1vdXNlciUyMiUyQyUyMmZyaWVuZHMlMjIlM0ElMjIlMjIlMkMlMjJhdXRoJTIyJTNBJTIyZWQ0YjVhNDFkMzJlY2U4MzQ3Mzk0ZjlkZT U5YThjMWQlMjIlMkMlMjJyZWZlcmVyJTIyJTNBJTIyaXJpZGl1bS1wcmVwcm9kLmVtcGlyaWMuYWklMjIlMkMlMjJhY2Nlc3NUb2tlbiUyMiUzQSUyMjA2MTk3NjM3NTgwOGE2N2RmZjlhMmJlOWJmODE5NDQzJTIyJTdE;54651cc_loggedin = 1;54651cc_sound = true;54651cc_read = true;54651cc_popup = true;54651cc_disablelastseen = false;54651cc_usertype = loginuser;_hjIncludedInPageviewSample = 1;_hjAbsoluteSessionInProgress = 0;abc = s％3A8ZGd7Mol31n_Y8OCLq39dHoo3_mIlRhZ.pFQWz5gG9McKsQLzOikcTB​​mmb2Wcrxo％2B9u9iPpqoyxw;pageUrl =/#/dashboard/18;_gat_gtag_UA_97985973_5 = 1'\" https://abc.ai/getTagTrends/E1_CPU_PERCENTAGE/2020-9-12%2013:4:0/202'$(((1 + RANDOM％10000000))'0-09-15 |％2013:04:00 \">URL.txt完毕

  4. 最终测试命令(单行)失败，原因是 cat URL.txt |xargs -I {}-curl -O {}

输出:

 总计百分比已接收百分比Xferd平均速度时间时间时间当前Dload上传总花费左速度0 0 0 0 0 0 0 0-:-:-0:00:01-:-:-0

预期输出当我手动运行curl时，从URL文件中复制内容

  [{"dashboard_id":18，"user_id":"demouser"，"dashboard_name":"My_dashboard_1"，"description":"Test description One"，"creation_date";:"2020-09-21 10:13:00"，"dashboard_config":null，"id":5}]]< html>< head< title> 504网关超时</title></head><身体>< center>< h1> 504网关超时</h1</center>< hr>中心nginx/1.18.0</center>

为了进行故障排除，我在外壳cmd-line上使用了 set -x ，我看不到为什么或如何由curl进程处理和处理请求.curl输出显示(上面)在所有字段中全为0的输出，这告诉我它只是一个格式错误的错误请求，这不是实际情况，因为我多次手动测试运行URL.txt中给出的URL有效负载有效.

 空行代码新队代码新队...

我想生成尽可能多的并行请求，而不必等待第一个请求完成.

调试

使用单行代码在 -v 上运行它(仅显示重要行)

 >GET/getMultiDashboard/demouser -H Cookie:_ga = GA1.2.561275388.1601468723;_hjid = ecd3d778-b7f5-4f7f-b3ef-6f9f12b13d66;54651cc_an = 4;_hjTLDTest = 1;54651cc_data = JTdCJTIyaWQlMjIlM0ElMgwOGE2N2RmZjlhMmJlOWJmODE5NDQzJTIyJTdE;54651cc_loggedin = 1;54651cc_sound = true;54651cc_read = true;54651cc_popup = true;54651cc_disablelastseen = false;54651cc_usertype = loginuser;_gid = GA1.2.1722546791.1601890062;_hjIncludedInPageviewSample = 1;_hjAbsoluteSessionInProgress = 0;abc = s％3AKsRWcfNnOkbDHh1e65C3NwiDSZMx4LYg.zxLIymu488Ii5Z2％2Brz0qiwS17BzK2P7A0OoTSCHlMQM;pageUrl =/HTTP/1.1>主持人:abc.ai>用户代理:curl/7.58.0>接受: */*>{[5字节数据]<HTTP/1.1 400 BAD_REQUEST<内容长度:0<连接:关闭

当我单独使用 curl 运行而不使用 xargs 时，我得到了正确的输出，没有 400 错误的请求

 >Cookie:_ga = GA1.2.561275388.1601468723;_hjid = ecd3d778-b7f5-4f7f-b3ef-6f9f12b13d66;54651cc_an = 4;_hjTLDTest = 1;54651cc_data = JTdCJTIyaWQlMjIlM0ElMjJkZW1vdXNlciUyMiUyQyUyMm4lMjIlM0ElMjJkZW1vdXNJlOWJmODE5NDQzJTIYJTdE;54651cc_loggedin = 1;54651cc_sound = true;54651cc_read = true;54651cc_popup = true;54651cc_disablelastseen = false;54651cc_usertype = loginuser;_gid = GA1.2.1722546791.1601890062;_hjIncludedInPageviewSample = 1;_hjAbsoluteSessionInProgress = 0;abc = s％3AKsRWcfNnOkbDHh1e65C3NwiDSZMx4LYg.zxLIymu488Ii5Z2％2Brz0qiwS17BzK2P7A0OoTSCHlMQM;pageUrl =/#/dashboard;_gat_gtag_UA_97985973_5 = 1><HTTP/1.1 200 OK<内容类型:text/html;字符集= utf-8<日期:2020年10月5日星期一09:48:51 GMT<ETag:W/"3b4-gP1vMAXMzUZy + pt7cwyOmQslPT8".<伺服器:nginx/1.18.0<严格的运输安全性:max-age = 15552000；includeSubDomains<有所不同:接受编码<X-Content-Type-Options:nosniff<X-DNS-Prefetch-Control:关闭<X-Download-Options:noopen<X-Frame-Options:SAMEORIGIN<X-XSS-Protection:1；模式=阻止<内容长度:948<连接:保持活动状态<*与主机abc.ai的连接#0保持不变[{"dashboard_id":18，"user_id":"demouser"，"dashboard_name":"My_dashboard_1"，"description":"Test description One"，"creation_date":"2020-09-21 10:13:00"，"2020-08-12 09:08:00"，"dashboard_config":{}，"sort_id":4，"id":2}，{"dashboard_id:5}] *找到主机abc.ai的捆绑软件:0x55836cf75a50 [can管道]*重用现有连接！(#0)与主机abc.ai*连接到abc.ai(52.86.136.249)端口443(#0)>GET/getTagTr/E1_CP/2020-9-12％2013:4:0/202'6368'0-09-15 |％2013:04:00 HTTP/1.1>主持人:abc.ai>用户代理:curl/7.58.0>接受: */*>Cookie:_ga = GA1.2.561275388.1601468723;_hjid = ecd3d778-b7f5-4f7f-b3ef-6f9f12b13d66;54651cc_an = 4;_hjTLDTest = 1;54651cc_data = JTdCJTIyaWQlMjIlM0ElMjJkZW1vdXNlciUyMiUyQyUyMmjM3NTgwOGE2N2RmZjlhMmJlOWJmODE5NDQzJTIYJTdE;54651cc_loggedin = 1;54651cc_sound = true;54651cc_read = true;54651cc_popup = true;54651cc_disablelastseen = false;54651cc_usertype = loginuser;_gid = GA1.2.1722546791.1601890062;_hjIncludedInPageviewSample = 1;_hjAbsoluteSessionInProgress = 0;abc = s％3AKsRWcfNnOkbDHh1e65C3NwiDSZMx4LYg.zxLIymu488Ii5Z2％2Brz0qiwS17BzK2P7A0OoTSCHlMQM;pageUrl =/#/dashboard;_gat_gtag_UA_97985973_5 = 1

解决方案

在同一文件中具有多个 curl 参数和选项会增加复杂性，可能不值得解决.基本上，

  echo" http://example.com -H'X-Hello:Hello'"|xargs curl -O

将整个参数作为单个字符串传递给 echo 到 curl ，后者将其解释为要获取的URL.

我的建议是将URL和其他任何参数放在命令行上，并且仅将 -H 选项的参数存储在文件中.

((i = 0; i< 1000; ++ i))的

 做curl -O http://example.com -H"$(sed" s/％|/％$((1 + RANDOM))|/"xm.cookiefile)"完毕

并行运行400个(或任何其他)作业，也许就像常规的后台进程一样，或者如果您认为它可以增加价值，则可以使用 xargs .(也许还会看一下GNU parallel ，它简化了某些方面.)

我拿出了大模，因为它什么也没做. $ RANDOM 会生成0-32767范围内的整数，因此，如果您需要更大的数字，则可以将多个 $ RANDOM 数字粘贴在一起，或者可以使用其他随机数./p>

I'm trying to do user acceptance testing on an application which becomes unresponsive on a particular URL parameter included in the GET request.

Steps

  1. I have curl and run the GET req (crafted) copied curl syntax for Unix and copied to ubuntu server along with some changes.

    'https://abc.ai/getMultiDashboard/demouser' -H 'Cookie: _ga=GA1.2.561275388.1601468723; _hjid=ecd3d778-b7f5-4f7f-b3ef-6f9f12b13d66; 54651cc_an=4; _gid=GA1.2.1366208807.1601560229; _hjTLDTest=1; 54651cc_data=JTdCJTIyaWQlMjIlM0Ellc3NUb2tlbiUyMiUzQSUyMjA2MTk3NjM3NTgwOGE2N2RmZjlhMmJlOWJmODE5NDQzJTIyJTdE; 54651cc_loggedin=1; 54651cc_sound=true; 54651cc_read=true; 54651cc_popup=true; 54651cc_disablelastseen=false; 54651cc_usertype=loginuser; _hjIncludedInPageviewSample=1; _hjAbsoluteSessionInProgress=0; abc=s%3A8ZGd7Mol31n_Y8OCLq39dHoo3_mIlRhZ.pFQWz5gG9McKsQLzOikcTBmmb2Wcrxo%2B9u9iPpqoyxw; pageUrl=/#/dashboard/18; _gat_gtag_UA_97985973_5=1' 
"https://abc.ai/getTagTrends/E1_CPU_PERCENTAGE/2020-9-12%2013:4:0/202**'23548'**0-09-15|%2013:04:00"
"https://abc.ai/getTagTrends/E1_CPU_PERCENTAGE/2020-9-12%2013:4:0/202**'`23548`'**0-09-15|%2013:04:00"

  2. The ** asterisks are not part of the actual values; I use them to demarcate my injected value

  3. Using a small bash script I have generated 1000s of (unique) payload combinations for Curl.

    #/bin/bash   
for ((i=0; i<1000; ++i)); do
        echo "
'https://abc.ai/getMultiDashboard/demouser' -H 'Cookie: _ga=GA1.2.561275388.1601468723; _hjid=ecd3d778-b7f5-4f7
f-b3ef-6f9f12b13d66; 54651cc_an=4; _gid=GA1.2.1366208807.1601560229; _hjTLDTest=1; 54651cc_data=JTdCJTIyaWQlMjIlM0ElMjJkZW1vdXNlciU yMiUyQyUyMm4lMjIlM0ElMjJkZW1vdXNlciUyMiUyQyUyMmZyaWVuZHMlMjIlM0ElMjIlMjIlMkMlMjJhdXRoJTIyJTNBJTIyZWQ0YjVhNDFkMzJlY2U4MzQ3Mzk0ZjlkZT    U5YThjMWQlMjIlMkMlMjJyZWZlcmVyJTIyJTNBJTIyaXJpZGl1bS1wcmVwcm9kLmVtcGlyaWMuYWklMjIlMkMlMjJhY2Nlc3NUb2tlbiUyMiUzQSUyMjA2MTk3NjM3NTgwO
GE2N2RmZjlhMmJlOWJmODE5NDQzJTIyJTdE; 54651cc_loggedin=1; 54651cc_sound=true; 54651cc_read=true; 54651cc_popup=true; 54651cc_disable
lastseen=false; 54651cc_usertype=loginuser; _hjIncludedInPageviewSample=1; _hjAbsoluteSessionInProgress=0; abc=s%3A8ZGd7Mol31n_
Y8OCLq39dHoo3_mIlRhZ.pFQWz5gG9McKsQLzOikcTBmmb2Wcrxo%2B9u9iPpqoyxw; pageUrl=/#/dashboard/18; _gat_gtag_UA_97985973_5=1' \"https://abc.ai/getTagTrends/E1_CPU_PERCENTAGE/2020-9-12%2013:4:0/202'$((1 + RANDOM % 10000000))'0-09-15|%2013:04:00\"" 
> URL.txt   
done

  4. Final command for testing (one-liner) fails as cat URL.txt | xargs -I{} -- curl -O {}

Output:

     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0

Expected output when I run the curl manually copying the contents from URL file I get

[{"dashboard_id": 18, "user_id": "demouser", "dashboard_name": "My_dashboard_1", "description": "Test description One", "creation_date": "2020-09-21 10:13:00", "dashboard_config": null, "id": 5}]


<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.18.0</center>

In order to troubleshoot, i used set -x on shell cmd-line I can't see why or how the request is crafted and handled by the curl processes. The curl output shows output (above) which has all 0 values in all fields, this tell me its just a bad malformed request, which isn't the actual case since i manually tested running the URL payload given in URL.txt multiple times it works.

EMPTY LINE
CODE
NEW-LINE

CODE
NEWLINE

...

I want to generate as many parallel requests as possible, without waiting for the first one to finish.

Debug

running it with -v using one-liner (showing only importants lines)

> GET /getMultiDashboard/demouser -H Cookie: _ga=GA1.2.561275388.1601468723; _hjid=ecd3d778-b7f5-4f7f-b3ef-6f9f12b13d66; 54651cc_an=4; _hjTLDTest=1; 54651cc_data=JTdCJTIyaWQlMjIlM0ElMgwOGE2N2RmZjlhMmJlOWJmODE5NDQzJTIyJTdE; 54651cc_loggedin=1; 54651cc_sound=true; 54651cc_read=true; 54651cc_popup=true; 54651cc_disablelastseen=false; 54651cc_usertype=loginuser; _gid=GA1.2.1722546791.1601890062; _hjIncludedInPageviewSample=1; _hjAbsoluteSessionInProgress=0; abc=s%3AKsRWcfNnOkbDHh1e65C3NwiDSZMx4LYg.zxLIymu488Ii5Z2%2Brz0qiwS17BzK2P7A0OoTSCHlMQM; pageUrl=/ HTTP/1.1
> Host: abc.ai
> User-Agent: curl/7.58.0
> Accept: */*
>
{ [5 bytes data]
< HTTP/1.1 400 BAD_REQUEST
< Content-Length: 0
< Connection: Close

When I run it with curl alone not using xargs I get the correct output no 400 bad request

> Cookie: _ga=GA1.2.561275388.1601468723; _hjid=ecd3d778-b7f5-4f7f-b3ef-6f9f12b13d66; 54651cc_an=4; _hjTLDTest=1; 54651cc_data=JTdCJTIyaWQlMjIlM0ElMjJkZW1vdXNlciUyMiUyQyUyMm4lMjIlM0ElMjJkZW1vdXNJlOWJmODE5NDQzJTIyJTdE; 54651cc_loggedin=1; 54651cc_sound=true; 54651cc_read=true; 54651cc_popup=true; 54651cc_disablelastseen=false; 54651cc_usertype=loginuser; _gid=GA1.2.1722546791.1601890062; _hjIncludedInPageviewSample=1; _hjAbsoluteSessionInProgress=0; abc=s%3AKsRWcfNnOkbDHh1e65C3NwiDSZMx4LYg.zxLIymu488Ii5Z2%2Brz0qiwS17BzK2P7A0OoTSCHlMQM; pageUrl=/#/dashboard; _gat_gtag_UA_97985973_5=1
>
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Date: Mon, 05 Oct 2020 09:48:51 GMT
< ETag: W/"3b4-gP1vMAXMzUZy+pt7cwyOmQslPT8"
< Server: nginx/1.18.0
< Strict-Transport-Security: max-age=15552000; includeSubDomains
< Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
< X-DNS-Prefetch-Control: off
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< Content-Length: 948
< Connection: keep-alive
<
* Connection #0 to host abc.ai left intact
[{"dashboard_id": 18, "user_id": "demouser", "dashboard_name": "My_dashboard_1", "description": "Test description One", "creation_date": "2020-09-21 10:13:00",  "2020-08-12 09:08:00", "dashboard_config": {}, "sort_id": 4, "id": 2}, {"dashboard_id": 5}]* Found bundle for host abc.ai: 0x55836cf75a50 [can pipeline]
* Re-using existing connection! (#0) with host abc.ai
* Connected to abc.ai (52.86.136.249) port 443 (#0)
> GET /getTagTr/E1_CP/2020-9-12%2013:4:0/202'6368'0-09-15|%2013:04:00 HTTP/1.1
> Host: abc.ai
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: _ga=GA1.2.561275388.1601468723; _hjid=ecd3d778-b7f5-4f7f-b3ef-6f9f12b13d66; 54651cc_an=4; _hjTLDTest=1; 54651cc_data=JTdCJTIyaWQlMjIlM0ElMjJkZW1vdXNlciUyMiUyQyUyMmjM3NTgwOGE2N2RmZjlhMmJlOWJmODE5NDQzJTIyJTdE; 54651cc_loggedin=1; 54651cc_sound=true; 54651cc_read=true; 54651cc_popup=true; 54651cc_disablelastseen=false; 54651cc_usertype=loginuser; _gid=GA1.2.1722546791.1601890062; _hjIncludedInPageviewSample=1; _hjAbsoluteSessionInProgress=0; abc=s%3AKsRWcfNnOkbDHh1e65C3NwiDSZMx4LYg.zxLIymu488Ii5Z2%2Brz0qiwS17BzK2P7A0OoTSCHlMQM; pageUrl=/#/dashboard; _gat_gtag_UA_97985973_5=1

解决方案

Having multiple curl arguments and options in the same file adds a complication which probably isn't worth working around. Basically,

echo "http://example.com -H 'X-Hello: Hello'" | xargs curl -O

passes the entire argument to echo as a single string to curl, which interprets it as the URL to fetch.

My suggestion would be to put the URL and any other arguments on the command line, and only store the -H option's argument in the file.

for ((i=0; i<1000; ++i)); do
    curl -O http://example.com -H "$(sed "s/%|/%$((1 + RANDOM))|/" xm.cookiefile)"
done

and run 400 (or whatever) of these jobs in parallel, perhaps just as regular background processes, or maybe with xargs if you think it adds value. (Maybe also look at GNU parallel which simplifies some aspects of this.)

I took out the big modulo because it's not doing anything; $RANDOM produces integers in the range 0-32767 so if you need a much bigger number, maybe paste together multiple $RANDOM numbers, or maybe use a different random source.

这篇关于使用xargs + curls bash脚本对URI进行压力测试失败，状态为空的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆