TF 数据 API:如何有效地从图像中采样小块 [英] TF data API: how to efficiently sample small patches from images

问题描述

考虑从高分辨率图像目录中创建随机小图像块采样数据集的问题.Tensorflow 数据集 API 提供了一种非常简单的方法，通过构建图像名称的数据集，对它们进行混洗，将其映射到加载的图像，然后映射到随机裁剪的补丁.

Consider the problem of creating a dataset of sampling random small image patches from a directory of high-resolution images. The Tensorflow dataset API allows for a very easy way of doing this, by constructing a dataset of image names, shuffling them, mapping it to loaded images, then to random cropped patches.

然而，这种简单的实现非常低效，因为将加载和裁剪单独的高分辨率图像以生成每个补丁.理想情况下，图像可以加载一次并重复使用以生成许多补丁.

However, this naive implementation is very inefficient as a separate high-resolution image will be loaded and cropped to generate each patch. Ideally an image could be loaded once and reused to generate many patches.

之前讨论过的一种简单方法是从图像生成多个补丁并将它们展平.然而，这有一个不幸的影响，即过度偏向数据.我们希望每个训练批次都来自不同的图像.

One simple way that was discussed previously is to generate multiple patches from an image and flatten them. However this has the unfortunate effect of biasing the data too much. We want each training batch to come from different images.

理想情况下，我想要的是随机缓存过滤器"转换，它采用底层数据集并将其 N 个元素缓存到内存中.它的迭代器将从缓存中返回一个随机元素.此外，以预定义的频率，它将用来自底层数据集的新元素替换缓存中的随机元素.此过滤器将允许更快的数据访问，但会减少随机化和更高的内存消耗.

Ideally what I would like is a "random caching filter" transformation that takes an underlying dataset and caches N elements of it into memory. Its iterator will return a random element from the cache. Also, with pre-defined frequency it will replace a random element from the cache with a new one from the underlying dataset. This filter will allow for faster data access at the expense of less randomization and higher memory consumption.

有这样的功能吗?

如果不是，它应该作为一个新的数据集转换来实现还是只是一个新的迭代器?似乎只需要一个新的迭代器.关于如何创建新的数据集迭代器(最好是在 C++ 中)的任何指示?

If not, should it be implemented as a new dataset transformation or simply a new iterator? It seems a new iterator is all that is needed. Any pointers on how to create a new dataset iterator, ideally in C++?

推荐答案

您应该能够使用 tf.data.Dataset.shuffle 来实现你想要的.以下是目标的快速摘要:

You should be able to use tf.data.Dataset.shuffle to achieve what you want. Here is a quick summary for the objectives:

• 加载非常大的图像，从图像中生成较小的随机裁剪并将它们批处理
• 通过在加载图像后从大图像创建多个补丁来提高管道效率
• 添加足够的 shuffle，使一批补丁多样化(所有补丁来自不同的图像)
• 不要在缓存中加载太多大图片

您可以通过执行以下步骤使用 tf.data API 实现所有这些:

You can achieve all that using the tf.data API by doing the following steps:

1. 打乱大图片的文件名
2. 阅读大图
3. 从此图像生成多个补丁
4. 使用足够大的缓冲区大小再次混洗所有这些补丁(有关缓冲区大小，请参阅此答案).调整缓冲区大小是在良好改组和缓存补丁大小之间的权衡
5. 批量处理
6. 预取一批

1. shuffle the filenames of the big images
2. read the big images
3. generate multiple patches from this image
4. shuffle again all these patches with a big enough buffer size (see this answer on buffer size). Adjusting the buffer size is a tradeoff between good shuffling and size of the cached patches
5. batch them
6. prefetch one batch

---

这是相关的代码:

---

Here is a the relevant code:

filenames = ...  # filenames containing the big images
num_samples = len(filenames)

# Parameters
num_patches = 100               # number of patches to extract from each image
patch_size = 32                 # size of the patches
buffer_size = 50 * num_patches  # shuffle patches from 50 different big images
num_parallel_calls = 4          # number of threads
batch_size = 10                 # size of the batch

get_patches_fn = lambda image: get_patches(image, num_patches=num_patches, patch_size=patch_size)

# Create a Dataset serving batches of random patches in our images
dataset = (tf.data.Dataset.from_tensor_slices(filenames)
    .shuffle(buffer_size=num_samples)  # step 1: all the  filenames into the buffer ensures good shuffling
    .map(parse_fn, num_parallel_calls=num_parallel_calls)  # step 2
    .map(get_patches_fn, num_parallel_calls=num_parallel_calls)  # step 3
    .apply(tf.contrib.data.unbatch())  # unbatch the patches we just produced
    .shuffle(buffer_size=buffer_size)  # step 4
    .batch(batch_size)  # step 5
    .prefetch(1)  # step 6: make sure you always have one batch ready to serve
)

iterator = dataset.make_one_shot_iterator()
patches = iterator.get_next()  # shape [None, patch_size, patch_size, 3]


sess = tf.Session()
res = sess.run(patches)

函数 parse_fn 和 get_patches 定义如下:

The functions parse_fn and get_patches are defined like this:

def parse_fn(filename):
    """Decode the jpeg image from the filename and convert to [0, 1]."""
    image_string = tf.read_file(filename)

    # Don't use tf.image.decode_image, or the output shape will be undefined
    image_decoded = tf.image.decode_jpeg(image_string, channels=3)

    # This will convert to float values in [0, 1]
    image = tf.image.convert_image_dtype(image_decoded, tf.float32)

    return image


def get_patches(image, num_patches=100, patch_size=16):
    """Get `num_patches` random crops from the image"""
    patches = []
    for i in range(num_patches):
        patch = tf.image.random_crop(image, [patch_size, patch_size, 3])
        patches.append(patch)

    patches = tf.stack(patches)
    assert patches.get_shape().dims == [num_patches, patch_size, patch_size, 3]

    return patches

这篇关于TF 数据 API:如何有效地从图像中采样小块的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！