改组tfrecords文件 [英] Shuffling tfrecords files

问题描述

我有5个tfrecords文件，每个对象一个。训练时，我想从所有5个tfrecord中平均读取数据，即如果我的批量大小为50，则我应该从第一个tfrecord文件中获取10个样本，从第二个tfrecord文件中获取10个样本，依此类推。目前，它只是从所有三个文件中顺序读取，即我从同一记录中获得了50个样本。有没有办法从不同的tfrecords文件中采样？

I have 5 tfrecords files, one for each object. While training I want to read data equally from all the 5 tfrecords i.e. if my batch size is 50, I should get 10 samples from 1st tfrecord file, 10 samples from the second tfrecord file and so on. Currently, it just reads sequentially from all the three files i.e. I get 50 samples from the same record. Is there a way to sample from differnt tfrecords files?

推荐答案

我建议您阅读教程，由@mrry在 tf.data 。他在幻灯片42上说明了如何使用 tf.data .dataset.interleave（） 可以同时读取多个tfrecord文件。

I advise you to read the tutorial by @mrry on tf.data. On slide 42 he explains how to use tf.data.Dataset.interleave() to read multiple tfrecord files at the same time.

例如，如果您有5个文件，包含：

For instance if you have 5 files, containing:

file0.tfrecord: [0, 1]
file1.tfrecord: [2, 3]
file2.tfrecord: [4, 5]
file3.tfrecord: [6, 7]
file4.tfrecord: [8, 9]

您可以这样编写数据集：

You can write the dataset like this:

files = ["file{}.tfrecord".format(i) for i in range(5)]
files = tf.data.Dataset.from_tensor_slices(files)
dataset = files.interleave(lambda x: tf.data.TFRecordDataset(x),
                           cycle_length=5, block_length=1)

dataset = dataset.map(_parse_function)  # parse the record

interleave 的参数为：
- cyc le_length ：要同时读取的文件数。如果要读取所有文件以创建批处理，请将其设置为文件数（在这种情况下，这是您应该执行的操作，因为每个文件都包含一种标签类型）
- block_length ：每次我们从文件中读取时，都会从该文件中读取 block_length 个元素

The parameters of interleave are: - cycle_length: number of files to read concurrently. If you want to read from all your files to create a batch, set this to the number of files (in your case this is what you should do since each file contains one type of label) - block_length: each time we read from a file, reads block_length elements from this file

我们可以测试它是否按预期工作：

We can test that it works as expected:

iterator = dataset.make_one_shot_iterator()
x = iterator.get_next()

with tf.Session() as sess:
    for _ in range(num_samples):
        print(sess.run(x))

将打印：

0
2
4
6
8
1
3
5
7
9

这篇关于改组tfrecords文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！