Hadoop 中 Amazon S3 和 S3n 之间的差异 [英] Differences between Amazon S3 and S3n in Hadoop

问题描述

当我将 Hadoop 集群连接到 Amazon 存储并将文件下载到 HDFS 时，我发现 s3:// 不起作用.在 Internet 上寻找帮助时，我发现我可以使用 S3n.当我使用 S3n 时，它起作用了.我不明白在我的 Hadoop 集群中使用 S3 和 S3n 之间的区别，有人可以解释一下吗?

When I connected my Hadoop cluster to Amazon storage and downloaded files to HDFS, I found s3:// did not work. When looking for some help on the Internet I found I can use S3n. When I used S3n it worked. I do not understand the differences between using S3 and S3n with my Hadoop cluster, can someone explain?

推荐答案

我认为您的主要问题与将 S3 和 S3n 作为 Hadoop 的两个独立连接点有关.s3n:// 表示一个普通文件，可从外部世界读取，位于此 S3 url".s3:// 是指映射到位于 AWS 存储集群上的 S3 存储桶的 HDFS 文件系统.因此，当您使用 Amazon 存储桶中的文件时，您必须使用 S3N，这就是您的问题得到解决的原因.@Steffen 补充的资料也很棒！！

I think your main problem was related with having S3 and S3n as two separate connection points for Hadoop. s3n:// means "A regular file, readable from the outside world, at this S3 url". s3:// refers to an HDFS file system mapped into an S3 bucket which is sitting on AWS storage cluster. So when you were using a file from Amazon storage bucket you must be using S3N and that's why your problem is resolved. The information added by @Steffen is also great!!

这篇关于Hadoop 中 Amazon S3 和 S3n 之间的差异的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！