如何将 AWS S3 中的存储桶图像读取到 Sagemaker Jupyter 实例中 [英] How to read bucket image from AWS S3 into Sagemaker Jupyter Instance
问题描述
我对 AWS 和云环境非常陌生.我是一名机器学习工程师,我计划在 AWS 环境中构建一个自定义 CNN,以预测给定的图像是否存在 iPhone.
I am very new to AWS and the cloud environment. I am a machine learning engineer, I am planning to build a custom CNN into the AWS environment to predict a given image has an iPhone present or not.
我做了什么:
第 1 步:
我为 iPhone 分类器创建了一个 S3 存储桶,文件夹结构如下:
I have created a S3 bucket for iPhone classifier with the below folder structure :
Iphone_Classifier > Train > Yes_iphone_images > 1000 images
> No_iphone_images > 1000 images
> Dev > Yes_iphone_images > 100 images
> No_iphone_images > 100 images
> Test > 30 random images
权限 - >阻止所有公共访问
第 2 步:
然后我转到 Amazon Sagemaker,并创建一个实例:
Then I go to Amazon Sagemaker, and create an instance:
我选择以下
Name: some-xyz,
Type: ml.t2.medium
IAM : created new IAM role ( root access was enabled.)
others: All others were in default
然后创建并打开笔记本实例.
Then the notebook instance was created and opened.
第 3 步:
一旦我打开了实例,
1. I used to prefer - conda_tensorflow2_p36 as interpreter
2. Created a new Jupyter notebook and stated.
3. I checked image classification examples but was confused, and most others used CSV files, but I want to retrieve images from S3 buckets.
问题:
1. How simply can we access the S3 bucket image dataset from the Jupiter Instances of Sagemaker?
2. I exactly need the reference code to access the S3 bucket images.
3. Is it a good approach to copy the data to the notebook or is it better to work from the S3 bucket.
我尝试过的是:
import boto3
client = boto3.client('s3')
# I tried this one and failed
#path = 's3://iphone/Train/Yes_iphone_images/100.png'
# I tried this one and failed
path = 's3://iphone/Test/10.png'
# I uploaded to the notebook instance an image file and when I try to read it works
#path = 'thiyaga.jpg'
print(path)
import cv2
from matplotlib import pyplot as plt
print(cv2.__version__)
plt.imshow(img)
推荐答案
如果你的图片是二进制编码的,你可以试试这个:
If your image is binary-encoded, you could try this:
import boto3
import matplotlib.pyplot as plt
# Define Bucket and Key
s3_bucket, s3_key = 'YOUR_BUCKET', 'YOUR_IMAGE_KEY'
with BytesIO() as f:
boto3.client("s3").download_fileobj(Bucket=s3_bucket, Key=s3_key, Fileobj=f)
f.seek(0)
img = plt.imread(f, format='png')
在其他情况下,以下代码有效(基于 文档):
in other case, the following code works out (based on the documentation):
s3 = boto3.resource('s3')
img = s3.Bucket(s3_bucket).download_file(s3_key, 'local_image.jpg')
在这两种情况下,您都可以使用 plt.imshow(img)
可视化图像.
In both cases, you can visualize the image with plt.imshow(img)
.
在您的路径示例 path = 's3://iphone/Test/10.png'
中,存储桶和密钥将是 s3_bucket = 'iphone'
和 <代码>s3_key=Test/10.png
In your path example path = 's3://iphone/Test/10.png'
, the bucket and key will be s3_bucket = 'iphone'
and s3_key=Test/10.png
其他资源:https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html
这篇关于如何将 AWS S3 中的存储桶图像读取到 Sagemaker Jupyter 实例中的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!