如果AWS搜寻器的大小> 1MB，则无法对S3中的文件类型存储进行分类 [英] AWS crawler could not classify the file type stores in S3 if its size >1MB

问题描述

当IAM尝试使用大小> = 1MB的输入Json文件的Crawler来检测文件类型时 它在胶水中创建一个表，其分类类型为未知". 但是，当大小小于1MB时，它将文件类型成功分类为JSON.

When iam trying to detect the file type using Crawler of size >=1MB of input Json file It creates a table in glue with is classification type is "Unknown". But when the size is <1MB it successfully classifies the file type as JSON.

我对文件进行了交叉检查，以确保其有效的json文件.

I crosschecked the file to ensure its a valid json file.

这对AWS爬虫来说是一个限制.

It is something a limitation for aws crawler.

如果是这样，此问题还有其他选择.

If so is there any alternative to this issue.

推荐答案

是的，这是由搜寻器设计的，如果元数据(内部搜寻器创建的)超过了1mb，您将得到上述错误，搜寻器会爬行1mb对于大于1mb的文件或整个文件(如果文件大小小于1Mb).如果元数据本身不适合1Mb，则文件将以Unkowntype结尾.

Yes, that is by design of the crawler, if the meta data ( Internally crawler creates it) exceeds 1mb you'll get the above error, Crawler crawls 1mb for files that are more than 1mb or the entire file if the file size is less than 1Mb. If the metadata itself doesn't fit 1Mb then the file will end up in Unkowntype.

这篇关于如果AWS搜寻器的大小> 1MB，则无法对S3中的文件类型存储进行分类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！