如何修复 tensorflow 截断的 tfrecords? [英] How to fix truncated tfrecords for tensorflow?

问题描述

我创建了一个大的 .tfrecords 文件，但没有看到任何错误.但是，在训练期间，当 TfRecordReader 到达 .tfrecord 文件末尾附近的某个位置时，我看到了错误在 XXXX 截断记录".如何快速检查 .tfrecord 文件是否真的损坏?如果是这样，我该如何快速修复文件(丢弃最后几个键值对就可以了)?

I created a large .tfrecords file without seeing any error. However, during training, I saw the error "truncated record at XXXX" when the TfRecordReader reached somewhere near the end of .tfrecord file. How can I quickly check if the .tfrecord file is really corrupted? If so, how can I quickly fix the file (It's OK to discard the last few key-value pair)?

推荐答案

该消息就是它所说的 --- TFRecord 文件似乎在记录中意外结束.

The message means what it says --- the TFRecord file seems to end unexpectedly part way through a record.

如果您想了解幕后发生的事情，文件格式非常简单，并在此处记录:https://www.tensorflow.org/versions/r0.11/api_docs/python/python_io.html#tfrecords-format-details

If you want to understand what's going on under the hood, the file format is quite simple and is documented here: https://www.tensorflow.org/versions/r0.11/api_docs/python/python_io.html#tfrecords-format-details

快速检查一下:您正在读取的文件真的是 TFRecord 文件吗?有把握总是好的.

One quick thing to check: is the file you are reading really a TFRecord file? It's always good to be sure.

很难就 TFRecord 文件的损坏程度"给出一个好的答案——阅读器代码所能做的就是告诉您某些内部不一致.

It's hard to give a good answer on "how corrupted" a TFRecord file is --- all the reader code can do is tell you that something is inconsistent internally.

(您的写入过程是否正确终止并在完成后关闭文件?)

(Did your writing process terminate correctly and close the file when it was done?)

如果您想修复该文件，最好的办法可能是重新生成它.

If you want to fix the file, probably your best bet is to regenerate it.

或者，您可以使用上面链接中记录的读取器函数读入文件的内容，然后将它们写出到一个新的 TFRecord 文件中.您将丢失损坏的记录，但您应该能够复制其他所有内容.

Alternatively you can read in the contents of the file using the reader functions documented at the link above, and write them out to a new TFRecord file. You'll lose the corrupted records, but you should be able to copy everything else over.

这篇关于如何修复 tensorflow 截断的 tfrecords?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！