"gsutil ls"每次显示不同的列表 [英] "gsutil ls" shows a different list every time
问题描述
我们将GCS用作数据流管道的数据接收器，由于某种原因，每次我在目录上尝试"gsutil ls"时，输出目录都会显示"不同的文件列表. 具体来说，文件数应恰好为4,000(因为指定了管道以将输出分片为4,000个文件). 但是，我看到的列表是这4,000个文件中的 some ($ prefix-?????-04000)和临时文件中的 some ($ prefix- temp- *). 自数据流作业(2016-12-18_19_30_32-7274262445792076535)完成以来已经有10多个小时，而且我仍然看到不同的文件列表(它不仅在增加，而且在减少，这意味着有些文件消失了，然后又出现了). 这会影响我们运行的其他从该目录读取的数据流管道.
We are using GCS as the data sink of a dataflow pipeline, and for some reason the output directory "shows" a different list of files every time I try "gsutil ls" on the directory. Specifically, the number of files should be exactly 4,000 (as the pipeline was specified to shard the output to 4,000 files). However, the list I see is some of those 4,000 files ($prefix-?????-of-04000) and some of the temp files ($prefix-temp-*). It's been 10+ hours since the dataflow job (2016-12-18_19_30_32-7274262445792076535) completed, and I am still seeing different file lists (it's not just increasing, but sometimes decreasing meaning some files disappear and then appear again). This is affect other dataflow pipelines we run which read from this directory.
是数据流问题还是GCS问题，我们该如何解决? 我以前已经看到过GCS的这种行为，但是通常是在数据流管道完成后的头几分钟，但是这次似乎正在进行了一段时间.
Is this Dataflow issue or GCS issue, and how can we resolve this? I've seen this behavior of GCS before, but it was usually for the first few minutes after a dataflow pipeline was completed, but this time it seems to be on-going for a while.
推荐答案
GCS的列表操作最终是一致.这可能意味着列出存储桶只会在一段时间内返回部分数据.
GCS's list operation is eventually consistent. This may mean that listing a bucket only returns partial data for a period of time.
如果您查看4000中的特定文件，是否始终存在?
If you look at a specific file from the 4000, is it consistently there?
更新:GCS存在一个临时问题，导致列表存储区的结果不一致: https ://status.cloud.google.com/incident/storage/16036
Update: There was a temporary issue with GCS causing inconsistent results for list buckets: https://status.cloud.google.com/incident/storage/16036
这篇关于"gsutil ls"每次显示不同的列表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
