如何从亚马逊的请求者支付桶下载数据? [英] How to download data from Amazon's requester pay buckets?

问题描述

我花了大约一周的时间来下载arXiv文章，如下所述: http://arxiv. org/help/bulk_data_s3#src .

I have been struggling for about a week to download arXiv articles as mentioned here: http://arxiv.org/help/bulk_data_s3#src.

我尝试了很多事情:s3Browser，s3cmd.我能够登录到存储桶，但是无法从arXiv存储桶下载数据.

I have tried lots of things: s3Browser, s3cmd. I am able to login to my buckets but I am unable to download data from arXiv bucket.

我尝试过:

1. s3cmd get s3://arxiv/pdf/arXiv_pdf_1001_001.tar

请参阅:

$ s3cmd get s3://arxiv/pdf/arXiv_pdf_1001_001.tar


s3://arxiv/pdf/arXiv_pdf_1001_001.tar -> ./arXiv_pdf_1001_001.tar  [1 of 1]
s3://arxiv/pdf/arXiv_pdf_1001_001.tar -> ./arXiv_pdf_1001_001.tar  [1 of 1]
ERROR: S3 error: Unknown error

1. s3cmd get与x-amz-request-payer:requester

它又给了我同样的错误:

It gave me same error again:

$ s3cmd get --add-header="x-amz-request-payer:requester" s3://arxiv/pdf/arXiv_pdf_manifest.xml
s3://arxiv/pdf/arXiv_pdf_manifest.xml -> ./arXiv_pdf_manifest.xml  [1 of 1]
s3://arxiv/pdf/arXiv_pdf_manifest.xml -> ./arXiv_pdf_manifest.xml  [1 of 1]
ERROR: S3 error: Unknown error

1. 复制

我也尝试过从该文件夹复制文件.

I have tried copying files from that folder too.

$ aws s3 cp s3://arxiv/pdf/arXiv_pdf_1001_001.tar .

A client error (403) occurred when calling the HeadObject operation: Forbidden
Completed 1 part(s) with ... file(s) remaining

这可能意味着我犯了一个错误.问题是我不知道添加方式和内容将传达我的下载付费许可.

This probably means that I made a mistake. The problem is I don't know how and what to add that will convey my permission to pay for download.

我无法弄清楚应该如何从S3下载数据.我在AWS网站上读了很多书，但是没有地方可以找到我的问题的精确解决方案.

I am unable to figure out what should I do for downloading data from S3. I have been reading a lot on AWS sites, but nowhere I can get pinpoint solution to my problem.

如何批量下载arXiv数据?

How can I bulk download the arXiv data?

推荐答案

尝试下载s3cmd版本1.6.0: http://sourceforge.net/projects/s3tools/files/s3cmd/

$ s3cmd --configure

输入您在Amazon AWS网站界面的账户管理选项卡中找到的凭证.

Enter your credentials found in the account management tab of the Amazon AWS website interface.

$ s3cmd get --recursive --skip-existing s3://arxiv/src/ --requester-pays

这篇关于如何从亚马逊的请求者支付桶下载数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！