如何使用 Google Colab TPU 连接到私有存储桶 [英] How to connect to private storage bucket using the Google Colab TPU

问题描述

我正在使用 google colab pro 和提供的 TPU.我需要将预先训练的模型上传到 TPU.

I am using google colab pro and the provided TPU. I need to upload a pre-trained model into the TPU.

• TPU 只能从谷歌云存储桶加载数据.
• 我创建了一个云存储桶，并提取了桶中预训练的模型文件.

现在我需要授予 TPU 访问我的私有存储桶的权限，但我不知道 TPU 的服务帐户.我如何找到它?

Now I need to give permission to the TPU to access my private bucket, but I don't know the service account of the TPU. How do I find it?

目前我只有 All:R 对存储桶的读取权限，并且 TPU 已成功初始化，但显然这不是最佳解决方案.

For now I just have All:R read permission to the bucket and the TPU initialized successfully but clearly this is not the optimal solution.

推荐答案

我自己一直在为这个场景而苦苦挣扎(尽管使用免费版 Colab)并且刚刚让它工作.这个特定的用例似乎没有很好的文档记录——官方文档似乎主要处理涉及 Compute Engine VM 的案例，而不是自动分配的 TPU.对我有用的过程如下:

I've been struggling with this scenario myself (although with the free version of Colab) and just got it to work. This specific use case doesn't appear to be very well-documented—it seems the official documentation mostly deals with cases involving a Compute Engine VM, rather than an auto-assigned TPU. The process that worked for me went as follows:

1. 运行 Google Cloud SDK 身份验证并设置项目(这两件事可能是多余的——我还没有尝试过只做其中一项)

!gcloud auth login
!gcloud config set project [Project ID of Storage Bucket]

和

from google.colab import auth
auth.authenticate_user()

1. 初始化 TPU(来自 Tensorflow TPU 文档)

resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)
strategy = tf.distribute.experimental.TPUStrategy(resolver)

1. 尝试加载模型

model = tf.keras.models.load_model('gs://[Bucket name and path to saved model]')

这最初失败了，但错误消息包括尝试访问目录的 TPU 服务帐户，这是我授予访问权限的地址，如云存储文档.地址在service-[PROJECT_NUMBER]@cloud-tpu.iam.gserviceaccount.com 格式，但项目编号不是我的存储桶所在项目的项目 ID，也不是我能够获得的值找到其他任何地方.

This initially failed, but the error message included the service account of the TPU trying to access the directory, and this is the address I gave access to as described in the Cloud Storage docs. The address is in the service-[PROJECT_NUMBER]@cloud-tpu.iam.gserviceaccount.com format but the project number isn't the Project ID of the project my bucket is in, nor a value I've been able to find anywhere else.

在我授予该服务帐户的权限后(我只能在错误消息中找到)，我能够从我的私人存储桶加载和保存模型.

After I gave permissions to that service account (which I was only able to find in the error message), I was able to load and save models from my private bucket.

这篇关于如何使用 Google Colab TPU 连接到私有存储桶的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！