使用bash访问Azure blob存储，curl [英] Accessing Azure blob storage using bash, curl

问题描述

我尝试使用REST API从bash脚本中使用Azure blob存储服务。我知道可以使用各种其他工具或语言来完成此操作，但我想将其作为bash脚本。

下面的脚本试图列出Azure存储容器中的blob。

此脚本会导致身份验证错误。根据REST API，签名字符串和标题看起来是否正确（参考）文档。我怀疑问题可能在于玩弄签署过程的各个部分。

有没有人成功地使用bash和curl来存取云端储存资源，例如Azure或其他提供者？

 ＃！/ bin / bash 
 
＃列出Azure存储容器中的blob。 
 
 echousage：$ {0 ## * /}< storage-account-name>< container-name>< access-key> 
 
 storage_account =$ 1
 container_name =$ 2
 access_key =$ 3
 
 blob_store_url =blob.core.windows.net 
 authorization =SharedKey
 
 request_method =GET
 request_date = $（TZ = GMT date+％a，％d％h％Y％H：％M ：％S％Z）
 storage_service_version =2011-08-18
 
＃HTTP请求标头
 x_ms_date_h =x-ms-date：$ request_date
 x_ms_version_h =x-ms-version：$ storage_service_version
 
＃构建签名字符串
 canonicalized_headers =$ {x_ms_date_h} \\\
 $ {x_ms_version_h}
 canonicalized_resource =/ $ {storage_account} / $ {container_name}
 
 string_to_sign =$ {request_method} \\\
\\\
\\\
\\\
\\\
\\\
\\\
 \\\
\\\
\\\
\\\
\\\
 $ {canonicalized_headers} \\\
 $ {canonicalized_resource} \\\
comp：list\\\
restype：container
 
＃解码Base64编码访问密钥，转换为十六进制。 
 decoded_hex_key =$（echo -n $ access_key | base64 -d -w0 | xxd -p -c256）
 
＃为授权头创建HMAC签名
签名= $（echo -n$ string_to_sign| openssl dgst -sha256 -mac HMAC -macopthexkey：$ decoded_hex_key| sed's /^.*= //'| base64 -w0）
 
 authorization_header =授权：$ authorization $ storage_account：$ signature
 
 curl \ 
 -H$ x_ms_date_h\ 
 -H$ x_ms_version_h\ 
 -H$ authorization_header\ 
https：// $ {storage_account}。$ {blob_store_url} / $ {container_name}？restype = container& comp = list
  

更新 - 存储服务错误和脚本生成的相应签名字符串。

以下是存储服务为 AuthenticationFailed 错误返回的内容。

 <？xml version =1.0encoding =utf-8？& 
<错误> 
< Code> AuthenticationFailed< / Code> 
< Message>服务器无法验证请求。确保授权报头的值正确形成，包括签名。 
 RequestId：27e6337e-52f3-4e85-98c7-2fabaacd9ebc 
时间：2013-11-21T22：10：11.7029042Z< / Message> 
< AuthenticationErrorDetail>在HTTP请求
'OGYxYjk1MTFkYmNkMCgzN2YzODQwNzcyNiIyYTQxZDg0OWFjNGJiZDlmNWY5YzM1ZWQzMWViMGFjYTAyZDY4NAo ='
中找到的MAC签名与任何计算的签名不同。服务器使用以下字符串签名：
'GET 
 
 x-ms-date：Thu，21 Nov 2013 22:10:11 GMT 
 x-ms-version：2011- 08-18 
 / storage_account_name / storage_container 
 comp：list 
 restype：container'
< / AuthenticationErrorDetail> 
< / Error> 
  

接下来是脚本生成的 string_to_sign 。

  GET\\\
\\\
\\\
\\\
\\\
\ n\\\
\\\
\\\
\\\
\\\
\\\
x-ms-date：Thu，21 Nov 2013 22:10:11 GMT\\\
x-ms-version：2011-08-18\ n / storage_account_name / storage_container\\\
comp：list\\\
restype：container 
  

解决方案>

我能够得到它的工作。
这个代码有两个问题，第一个，如帕特里克·帕克指出的，用 printf 替换 echo -n code>。第二个是用openssl上的 -binary 选项替换 sed magic。

比较原文：

  signature = $（echo -n$ string_to_sign| openssl dgst -sha256 -mac HMAC -macopthexkey：$ decoded_hex_key-binary | sed's /^.*= //'| base64 -w0）
  

固定：

  signature = $（printf$ string_to_sign| openssl dgst  - sha256 -mac HMAC -macopthexkey：$ decoded_hex_key-binary | base64 -w0）
  

因为 echo -n 不会将 \\\
转换为实际的换行符。

需要改变 -binary ，因为即使你剥离了坏的部分，openssl仍然输出ascii-encoded-十六进制，不是二进制。所以在它被传递给 base64 后，结果是十六进制表示的b64编码版本，而不是原始值。

I am attempting to use the Azure blob storage service from a bash script using the REST API. I know it is possible to accomplish this using various other tools or languages, however I'd like to do it as a bash script.

The script below is an attempt to list the blobs in an Azure storage container.

This script results in an authentication error. The signing string and headers look correct based on the REST API (reference) documentation. I suspect the problem may be in juggling the various parts of the signing process.

Has anyone successfully used bash and curl to access cloud storage resources like Azure or other providers?

#!/bin/bash

# List the blobs in an Azure storage container.

echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key>"

storage_account="$1"
container_name="$2"
access_key="$3"

blob_store_url="blob.core.windows.net"
authorization="SharedKey"

request_method="GET"
request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
storage_service_version="2011-08-18"

# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"

# Build the signature string
canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${storage_account}/${container_name}"

string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"

# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"

# Create the HMAC signature for the Authorization header
signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" | sed 's/^.*= //' | base64 -w0)

authorization_header="Authorization: $authorization $storage_account:$signature"

curl \
  -H "$x_ms_date_h" \
  -H "$x_ms_version_h" \
  -H "$authorization_header" \
  "https://${storage_account}.${blob_store_url}/${container_name}?restype=container&comp=list"

Update - The storage service error and the corresponding signing string that the script generated.

Following is what the storage service returns for the AuthenticationFailed error.

<?xml version="1.0" encoding="utf-8"?>
<Error>
  <Code>AuthenticationFailed</Code>
  <Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:27e6337e-52f3-4e85-98c7-2fabaacd9ebc
Time:2013-11-21T22:10:11.7029042Z</Message>
  <AuthenticationErrorDetail>The MAC signature found in the HTTP request
'OGYxYjk1MTFkYmNkMCgzN2YzODQwNzcyNiIyYTQxZDg0OWFjNGJiZDlmNWY5YzM1ZWQzMWViMGFjYTAyZDY4NAo='
is not the same as any computed signature. Server used following string to sign:
'GET

x-ms-date:Thu, 21 Nov 2013 22:10:11 GMT
x-ms-version:2011-08-18
/storage_account_name/storage_container
comp:list
restype:container'
  </AuthenticationErrorDetail>
</Error>

Next is the string_to_sign that the script generates.

GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 21 Nov 2013 22:10:11 GMT\nx-ms-version:2011-08-18\n/storage_account_name/storage_container\ncomp:list\nrestype:container

解决方案

I was able to get it working. There were two things wrong with this code, the first, as Patrick Park noted, was replacing the echo -n with printf. The second was replacing the sed magic with the -binary option on openssl.

Compare the original:

signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | sed 's/^.*= //' | base64 -w0)

with the fixed:

signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary |  base64 -w0)

The echo change is needed because echo -n will not convert the \n into actual newlines.

The -binary change is needed because even though you are stripping off the bad part, openssl was still outputting the signature in ascii-encoded-hex, not in binary. So after it was passed to base64, the result was the b64 encoded version of the hex representation, instead of the raw value.

这篇关于使用bash访问Azure blob存储，curl的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！