Google云存储-从网络下载文件 [英] Google cloud storage - Download file from web

问题描述

我想在下一个项目中使用Google云存储. 我的目的是跟踪各种网站并收集一些照片.作为，我阅读了gsutil的文档；我可以将文件手动下载到我的服务器，然后使用gsutil将其上传到Google云存储中.

I want to use Google cloud storage in my next project. My aim is tracking various web sites and collecting some photos. As, I read the documentation for gsutil; I'm able download the file manually to my server and upload it google cloud storage by using gsutil.

下载和上传文件会在我的服务器中产生大量流量. 有没有办法让Google Cloud直接从http下载文件?

Downloading and uploading files generates so much traffic in my server. Are there a way to let google cloud download file direct from http?

推荐答案

主要答案

使用Google Cloud Shell非常容易做到这一点.似乎适用于所有文件大小:

Main answer

This is very easy to do from the Google Cloud Shell. Seems to work for all file sizes:

 curl http://speedtest.tele2.net/10GB.zip | gsutil cp - gs://YOUR_BUCKET_NAME/10GB.zip

基本上可以将数据直接卷曲到存储桶中.

Basically curl streams the data directly to the bucket.

原始方法:只要您的下载量小于4.6 GB，此方法就起作用.启动Cloud Shell(在GCP中登录项目后，在右上角的第一个图标)，然后使用wget下载所需的文件.例如，要下载7-Zip类型:

Original approach: This works as long as your download is less than ~ 4.6 GB. Launch the Cloud Shell (first icon on your top right after you login to your project in GCP) and use wget to download the file you want. For instance, to download 7-Zip type:

wget https://www.7-zip.org/a/7z1805-x64.exe

现在，有了Cloud Shell用户主目录中的文件，您可以使用gsutil命令将其复制到Google Cloud Storage存储桶中:

Now with the file in your Cloud Shell user home you can copy it to a Google Cloud Storage bucket using the gsutil command:

gsutil cp ./7z1805-x64.exe gs://your_bucket_name/

替代(大于4.6 GB)

如果文件大于4.6 GB，您仍然可以执行此操作，但是您需要使用gcsfuse将存储桶安装到Cloud Shell中:

Alternative (bigger than ~ 4.6 GB)

If the file is bigger than 4.6 GB you can still do it but you need to mount the bucket in your Cloud Shell using gcsfuse:

在Cloud Shell用户主目录中创建目录

Create a directory in your Cloud Shell user home

 mkdir ~/mybucket

现在使用gcsfuse将存储桶安装在该目录中:

Now mount your bucket in that directory using gcsfuse:

 gcsfuse bucket_name ~/mybucket

将当前目录更改为挂载点目录:

Change the current directory to mount point directory:

 cd mybucket

(如果您想获得一些乐趣，请运行"df -h"以查看在该安装点中获得了多少空间)

(if you want to have some fun run "df -h ." to see how much space you got in that mount point)

现在使用wget将文件直接放入您的存储桶(使用网络上的10GB文件作为样本):

Now use wget to get the file directly into your bucket (sample using 10GB file off the web):

 wget https://speed.hetzner.de/10GB.bin

这篇关于Google云存储-从网络下载文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！