将 .tfrecords 文件拆分为多个 .tfrecords 文件 [英] Split .tfrecords file into many .tfrecords files

问题描述

有没有什么办法可以直接将.tfrecords文件拆分成多个.tfrecords文件，而不用写回每个Dataset示例?

Is there any way to split .tfrecords file into many .tfrecords files directly, without writing back each Dataset example ?

推荐答案

你可以使用这样的函数:

You can use a function like this:

import tensorflow as tf

def split_tfrecord(tfrecord_path, split_size):
    with tf.Graph().as_default(), tf.Session() as sess:
        ds = tf.data.TFRecordDataset(tfrecord_path).batch(split_size)
        batch = ds.make_one_shot_iterator().get_next()
        part_num = 0
        while True:
            try:
                records = sess.run(batch)
                part_path = tfrecord_path + '.{:03d}'.format(part_num)
                with tf.python_io.TFRecordWriter(part_path) as writer:
                    for record in records:
                        writer.write(record)
                part_num += 1
            except tf.errors.OutOfRangeError: break

例如，要将文件 my_records.tfrecord 拆分为每个 100 条记录的部分，您可以这样做:

For example, to split the file my_records.tfrecord into parts of 100 records each, you would do:

split_tfrecord(my_records.tfrecord, 100)

这将创建多个较小的记录文件 my_records.tfrecord.000、my_records.tfrecord.001 等

This would create multiple smaller record files my_records.tfrecord.000, my_records.tfrecord.001, etc.

这篇关于将 .tfrecords 文件拆分为多个 .tfrecords 文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！