如何将多个tfrecords文件合并为一个文件? [英] How can I merge multiple tfrecords file into one file?

问题描述

我的问题是，如果要为我的数据创建一个tfrecords文件，大约需要15天才能完成，它有500000对模板，每个模板为32帧(图像).为了节省时间，我有3个GPU，所以我想可以在一个GPU上创建三个tfrocords文件，每个文件一个，然后我可以在5天内完成创建tfrecords的操作.但是后来我搜索了一种将这三个文件合并到一个文件中的方法，却找不到合适的解决方案.

My question is, if I want to create one tfrecords file for my data , it will take approximately 15 days to finish it, it has 500000 pairs of template , and each template is 32 frames( images). In order to save the time, I have 3 GPUs, so I thought I can create three tfrocords file each one file on one GPUs and then I can finish creating the tfrecords in 5 days. But then I searched about a way to merge these three files in one file and couldn't find proper solution.

那么有没有办法将这三个文件合并到一个文件中，或者有没有办法知道我使用的是Dataset API，通过提供从三个tfrecords文件中提取的一批示例来训练我的网络.

So Is there any way to merge these three files in one file, OR is there any way that I can train my network by feeding batch of example extracted form the three tfrecords files, knowing I am using Dataset API.

推荐答案

两个月前提出这个问题时，我认为您已经找到了解决方案.对于以下情况，答案是否定的，您无需创建单个HUGE tfrecord文件.只需使用新的DataSet API:

As the question is asked two months ago, I thought you already find the solution. For the follows, the answer is NO, you do not need to create a single HUGE tfrecord file. Just use the new DataSet API:

dataset = tf.data.TFRecordDataset(filenames_to_read,
    compression_type=None,    # or 'GZIP', 'ZLIB' if compress you data.
    buffer_size=10240,        # any buffer size you want or 0 means no buffering
    num_parallel_reads=os.cpu_count()  # or 0 means sequentially reading
)

# Maybe you want to prefetch some data first.
dataset = dataset.prefetch(buffer_size=batch_size)

# Decode the example
dataset = dataset.map(single_example_parser, num_parallel_calls=os.cpu_count())

dataset = dataset.shuffle(buffer_size=number_larger_than_batch_size)
dataset = dataset.batch(batch_size).repeat(num_epochs)
...

有关详细信息，请查看文档.

For details, check the document.

这篇关于如何将多个tfrecords文件合并为一个文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！