Errno 22当从S3存储桶中的“子文件夹"下载多个文件时，出现错误. [英] Errno 22 When downloading multiple files from S3 bucket "sub-folder"
问题描述
我一直在尝试使用AWS CLI从AWS的子文件夹下载所有文件，但是，在下载了头几个文件后，它无法下载其余文件.我相信这是因为它在文件名中添加了扩展名，然后将其视为无效的文件路径.
我正在使用以下命令；
aws s3 cp s3://my_bucket/sub_folder/tmp/-递归 对于子文件夹中的几乎所有文件，它都会给我以下错误；
[Errno 22]无效的参数:'C:\\ tmp \\ 2019-08-15T16:15:02.tif.deDBF2C2 我认为这是由于.deDBF2C2扩展名所致，尽管我不知道为什么这样做，但似乎在下载时已将其添加到文件中.在实际存储区中，文件名都以.tif结尾.
有人知道是什么原因吗?
更新:从Linux计算机执行该命令后，该命令即可工作.似乎特定于Windows.
这是AWS的监督，它在日志文件名称中使用Windows保留字符！当您执行命令时，它将创建所有目录，但是名称中带有 :: 的所有日志均无法下载.
问题在这里讨论:
然后您将在命令中看到以下几行:aws s3 cp，Bucket文件和本地文件路径.我们需要删除至:
右侧的本地文件路径中的 :: aws s3 cp s3://config-bucket-7XXXXXXXXXXX3/AWSLogs/7XXXXXXXXXXX3/Config/ap-southeast-2/2019/10/1/ConfigHistory/7XXXXXXXXXXX3_Config_ap-southeast-2_ConfigHistory_AWS :: RDS :: DBInstance_20191001T103223.json.gz到AWSLogs \ 7XXXXXXXXXXX3 \ Config \ ap-southeast-2 \ 2019 \ 10 \ 1 \ ConfigHistory \ 7XXXXXXXXXXX3_Config_ap-southeast-2_ConfigHistory_AWS :: RDS :: DBInstance_20191001T103223Z_20191001T103223Z_1.json.gz 我们只能使用正则表达式(.*):: 将 :: 替换为-，而不是在S3存储桶路径的本地路径中删除每行末尾最后出现的字符:
在这里我们可以看到我通过单击两次全部替换"将连字符:::替换为 $ 1-:
下一步，删除至(忽略图像中的光标图标，至不应替换为任何内容)
最后选择复制/粘贴到命令提示符中的所有行，以下载所有带有保留文件字符的文件！
I've been trying to use the AWS CLI to download all files from a sub-folder in AWS however, after the first few files download it fails to download the rest. I believe this is because it adds an extension to the filename and it then sees that as an invalid filepath.
I'm using the following command;
aws s3 cp s3://my_bucket/sub_folder /tmp/ --recursive
It gives me the following error for almost all of the files in the subfolder;
[Errno 22] Invalid argument: 'C:\\tmp\\2019-08-15T16:15:02.tif.deDBF2C2
I think this is because of the .deDBF2C2 extension it seems to be adding to the files when downloading though I don't know why it does. The filenames all end with .tif in the actual bucket.
Does anyone know what causes this?
Update: The command worked once I executed it from a linux machine. Seems to be specific to windows.
This is an oversight by AWS using Windows reserved characters in Log files names! When you execute the command it will create all the directory's however any logs with :: in the name fail to download.
Issue is discussed here: https://github.com/aws/aws-cli/issues/4543
So frustrated I came up with a workaround by executing a "DryRun" which prints the expected log output and porting that to a text file, eg:
>aws s3 cp s3://config-bucket-7XXXXXXXXXXX3 c:\temp --recursive --dryrun > c:\temp\aScriptToDownloadFilesAndReplaceNames.txt
The output file is filled with these aws log entries we can turn into aws script commands:
(dryrun) download: s3://config-bucket-7XXXXXXXXXXX3/AWSLogs/7XXXXXXXXXXX3/Config/ap-southeast-2/2019/10/1/ConfigHistory/7XXXXXXXXXXX3_Config_ap-southeast-2_ConfigHistory_AWS::RDS::DBInstance_20191001T103223Z_20191001T103223Z_1.json.gz to \AWSLogs\7XXXXXXXXXXX3\Config\ap-southeast-2\2019\10\1\ConfigHistory\703014955993_Config_ap-southeast-2_ConfigHistory_AWS::RDS::DBInstance_20191001T103223Z_20191001T103223Z_1.json.gz
In Notepad++ or other text editor you replace the (dryrun) download: with aws s3 cp
Then you will see the following lines with the command: aws s3 cp, the Bucket file and the local file path. We need to remove the :: in the local file path on the right side of the to:
aws s3 cp s3://config-bucket-7XXXXXXXXXXX3/AWSLogs/7XXXXXXXXXXX3/Config/ap-southeast-2/2019/10/1/ConfigHistory/7XXXXXXXXXXX3_Config_ap-southeast-2_ConfigHistory_AWS::RDS::DBInstance_20191001T103223Z_20191001T103223Z_1.json.gz to AWSLogs\7XXXXXXXXXXX3\Config\ap-southeast-2\2019\10\1\ConfigHistory\7XXXXXXXXXXX3_Config_ap-southeast-2_ConfigHistory_AWS::RDS::DBInstance_20191001T103223Z_20191001T103223Z_1.json.gz
We can replace the :: with - only in local paths not S3 Bucket path's using a regex (.*):: that removes the last occurrence of chars at the end of each line:
And here we can see I've replaced the ::'s with hyphens $1- by clicking 'Replacing All' twice:
Next remove the to (ignore the cursor icon in the image, to should be replaced with nothing)
Finally select all the lines copy/paste into a command prompt to download all the files with reserved file characters!
这篇关于Errno 22当从S3存储桶中的“子文件夹"下载多个文件时，出现错误.的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
