当上传一个5G的文件到Amazon S3 EntityTooLarge错误 [英] EntityTooLarge error when uploading a 5G file to Amazon S3
问题描述
Amazon S3的文件大小限制应该按照这个的公布，但我上传一个5G的文件时，发现了以下错误
<$p$p><$c$c>'/mahler%2Fparquet%2Fpageview%2Fall-2014-2000%2F_temporary%2F_attempt_201410112050_0009_r_000221_2222%2Fpart-r-222.parquet' XML错误信息： ＆LT; XML版本=1.0编码=UTF-8＆GT？; ＆LT;错误＆GT; ＆LT; code取代; EntityTooLarge＆LT; / code取代; ＆LT;消息＆GT;你提出的上传超过了最大允许大小和LT; /消息＆GT; ＆LT; ProposedSize＆GT; 5374138340＆LT; / ProposedSize＆GT; ... ＆LT; MaxSizeAllowed＆GT; 5368709120＆LT; / MaxSizeAllowed＆GT; ＆LT; /错误＆GT;这使得它看起来像S3只接受5G上传。我使用的Apache星火SQL编写出使用 SchemRDD.saveAsParquetFile 方法的实木复合地板的数据集。
完整的堆栈跟踪
org.apache.hadoop.fs.s3.S3Exception：org.jets3t.service.S3ServiceException：S3 PUT失败'/mahler%2Fparquet%2Fpageview%2Fall-2014-2000%2F_temporary%2F_attempt_201410112050_0009_r_000221_2222%2Fpart-r-222.parquet' XML错误信息：？？＆LT; XML版本=1.0编码=UTF-8＆GT;＆LT;错误＆GT;＆LT; code取代; EntityTooLarge＆LT; / code＆GT;＆LT;消息＆GT;您的建议上传超过允许的最大size</Message><ProposedSize>5374138340</ProposedSize><RequestId>20A38B479FFED879</RequestId><HostId>KxeGs$p$pQ0hO7mm7DTcGLiN7vi7nqT3Z6p2Nbx1aLULSEzp6X5Iu8Kj6qM7Whm56ciJ7uDEeNn4=</HostId><MaxSizeAllowed>5368709120</MaxSizeAllowed></Error>
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:82)
sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）
sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）
sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）
java.lang.reflect.Method.invoke（Method.java:606）
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod（RetryInvocationHandler.java:82）
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke（RetryInvocationHandler.java:59）
org.apache.hadoop.fs.s3native。$ Proxy10.storeFile（来源不明）
org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:174)
org.apache.hadoop.fs.FSDataOutputStream $ PositionCache.close（FSDataOutputStream.java:61）
org.apache.hadoop.fs.FSDataOutputStream.close（FSDataOutputStream.java:86）
parquet.hadoop.ParquetFileWriter.end（ParquetFileWriter.java:321）
parquet.hadoop.InternalParquetRecordWriter.close（InternalParquetRecordWriter.java:111）
parquet.hadoop.ParquetRecordWriter.close（ParquetRecordWriter.java:73）
org.apache.spark.sql.parquet.InsertIntoParquetTable.org$apache$spark$sql$parquet$InsertIntoParquetTable$$writeShard$1(ParquetTableOperations.scala:305)
org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:318)
org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:318)
org.apache.spark.scheduler.ResultTask.runTask（ResultTask.scala：62）
org.apache.spark.scheduler.Task.run（Task.scala：54）
org.apache.spark.executor.Executor $ TaskRunner.run（Executor.scala：177）
java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1145）
java.util.concurrent.ThreadPoolExecutor中的$ Worker.run（ThreadPoolExecutor.java:615）
java.lang.Thread.run（Thread.java:745）
时的上传限制仍然5T？如果是，为什么我会得到这个错误，我该如何解决这个问题？
在对象的大小被限制为5 TB。在上传的大小仍然是5 GB，因为说明书中介绍：
根据您上载的数据的大小，亚马逊S3提供了以下选项：
上传单个对象的操作，使用单个
PUT操作，您可以在大小上载对象多达5 GB。上传的对象部分，使用多部分上传API，您可以上传大的物体，最多5 TB。
<一个href="http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html">http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html
一旦你做了多部分上传，S3验证和重组的部分，然后你有S3单个对象，高达5TB的大小，可以下载作为一个单一的entitity，有一个HTTP GET 的请求......但上载有潜在的更快，即使是在文件大于5GB小，因为你可以上传并行的部分，甚至重试的是没有成功的任何部分上传第一次尝试。
Amazon S3 file size limit is supposed to be 5T according to this announcement, but I am getting the following error when uploading a 5G file
'/mahler%2Fparquet%2Fpageview%2Fall-2014-2000%2F_temporary%2F_attempt_201410112050_0009_r_000221_2222%2Fpart-r-222.parquet' XML Error Message:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>EntityTooLarge</Code>
<Message>Your proposed upload exceeds the maximum allowed size</Message>
<ProposedSize>5374138340</ProposedSize>
...
<MaxSizeAllowed>5368709120</MaxSizeAllowed>
</Error>
This makes it seem like S3 is only accepting 5G uploads. I am using Apache Spark SQL to write out a Parquet data set using SchemRDD.saveAsParquetFile method.
The full stack trace is
org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 PUT failed for '/mahler%2Fparquet%2Fpageview%2Fall-2014-2000%2F_temporary%2F_attempt_201410112050_0009_r_000221_2222%2Fpart-r-222.parquet' XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed size</Message><ProposedSize>5374138340</ProposedSize><RequestId>20A38B479FFED879</RequestId><HostId>KxeGsPreQ0hO7mm7DTcGLiN7vi7nqT3Z6p2Nbx1aLULSEzp6X5Iu8Kj6qM7Whm56ciJ7uDEeNn4=</HostId><MaxSizeAllowed>5368709120</MaxSizeAllowed></Error>
org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:82)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
org.apache.hadoop.fs.s3native.$Proxy10.storeFile(Unknown Source)
org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsOutputStream.close(NativeS3FileSystem.java:174)
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
parquet.hadoop.ParquetFileWriter.end(ParquetFileWriter.java:321)
parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:111)
parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:73)
org.apache.spark.sql.parquet.InsertIntoParquetTable.org$apache$spark$sql$parquet$InsertIntoParquetTable$$writeShard$1(ParquetTableOperations.scala:305)
org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:318)
org.apache.spark.sql.parquet.InsertIntoParquetTable$$anonfun$saveAsHadoopFile$1.apply(ParquetTableOperations.scala:318)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
org.apache.spark.scheduler.Task.run(Task.scala:54)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
Is the upload limit still 5T? If it is why am I getting this error and how do I fix it?
The object size is limited to 5 TB. The upload size is still 5 GB, as explained in the manual:
Depending on the size of the data you are uploading, Amazon S3 offers the following options:
Upload objects in a single operation—With a single
PUToperation you can upload objects up to 5 GB in size.Upload objects in parts—Using the Multipart upload API you can upload large objects, up to 5 TB.
http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html
Once you do a multipart upload, S3 validates and recombines the parts, and you then have a single object in S3, up to 5TB in size, that can be downloaded as a single entitity, with a single HTTP GET request... but uploading is potentially much faster, even on files smaller than 5GB, since you can upload the parts in parallel and even retry the uploads of any parts that didn't succeed on first attempt.
这篇关于当上传一个5G的文件到Amazon S3 EntityTooLarge错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
