使用 TensorFlow 对图像中的点进行插值采样 [英] Interpolated sampling of points in an image with TensorFlow

问题描述

给定的是灰度图像 I 作为 2D 张量(维度 W，H)和坐标张量 C(Dim. None,2).我想将 C 的行解释为 I 中的坐标，使用某种插值在这些坐标处对 I 进行采样(双线性可能没问题)对于我的用例)，并将结果值存储在一个新的张量 P(维度为 None，即具有与 C 行一样多的条目的一维)中.

Given is a grayscale image I as 2D Tensor (Dimension W,H) and a Tensor of coordinates C (Dim. None,2). I want to interpret the rows of C as coordinates in I, sample I at those coordinates using some kind of interpolation (bilinear would probably be fine for my use case), and store the resulting values in a new Tensor P (of dimension None, i.e. 1-dimensional with as many entries as C has rows).

使用 TensorFlow 是否可以(高效)实现?我能找到的只是调整图像大小(如果你喜欢等距重采样)的函数.但是我在坐标列表中找不到任何开箱即用的样本.

Is this possible (efficiently) with TensorFlow? All I can find are functions for resizing (equidistant resampling if you like) of images. But I can't find anything out-of-the-box to sample at a list of coordinates.

即我本来希望找到类似 tf.interpolate() 函数的东西:

I.e. I would have expected to find something like a tf.interpolate() function:

I = tf.placeholder("float", shape=[128, 128])
C = tf.placeholder("float", shape=[None, 2])
P = tf.interpolate(I, C, axis=[0, 1], method="linear")

理想情况下，我会寻找一种解决方案，允许我使用形状为 (None, M) 的 C 沿 M 维插入 N 维张量 I并产生 N-M+1 维输出，如上面代码中的axis"参数所示.

Ideally I would be looking for a solution that would allow me to interpolate in an N dimensional tensor I along M dimensions using a C with shape (None, M) and produce an N-M+1 dimensional output, as indicated by the "axis" parameter in the code above.

(顺便说一下，我的应用程序中的图像"不是图片，它是从物理模型(用作占位符时)或替代学习模型(用作变量时)中采样的数据.现在这个物理模型有2 个自由度，因此现在在图像"中进行插值就足够了，但我将来可能会研究更高维的模型.)

(The "image" in my application isn't a picture btw., it's sampled data from a physical model (when used as placeholder) or an alternative learned model (when used as variable). Right now this physical model has 2 degrees of freedom, thus interpolating in an "image" is sufficient for now, but I might look into higher dimensional models in the future.)

如果现有 TensorFlow 功能无法实现类似的功能:当我想实现类似 tf.interpolate() 运算符的功能时，我应该从哪里开始?(文档和/或简单的示例代码)

In case something like that is not possible with existing TensorFlow features: Where should I start when I'd like to implement something like this tf.interpolate() operator? (documentation and/or simple example code)

推荐答案

没有执行这种插值的内置操作，但您应该能够使用现有 TensorFlow 操作的组合来完成.对于双线性情况，我建议采用以下策略:

There is no built-in op that performs this kind of interpolation, but you should be able to do it using a composition of existing TensorFlow ops. I'd suggest the following strategy for the bilinear case:

1. 从你的索引张量 C 计算对应于四个角点的整数张量.例如(假设原点在左上角的名称):

1. From your tensor C of indices, compute integer tensors corresponding to the four corner points. For example (with names assuming that the origin is at the top left):

top_left = tf.cast(tf.floor(C), tf.int32)

top_right = tf.cast(
    tf.concat(1, [tf.floor(C[:, 0:1]), tf.ceil(C[:, 1:2])]), tf.int32)

bottom_left = tf.cast(
    tf.concat(1, [tf.ceil(C[:, 0:1]), tf.floor(C[:, 1:2])]), tf.int32)

bottom_right = tf.cast(tf.ceil(C), tf.int32)

从代表特定角点的每个张量中，从这些点处的 I 中提取值向量.例如，对于以下函数，在二维情况下执行此操作:

From each tensor representing a particular corner point, extract a vector of values from I at those points. For example, for the following function does this for the 2-D case:

def get_values_at_coordinates(input, coordinates):
  input_as_vector = tf.reshape(input, [-1])
  coordinates_as_indices = (coordinates[:, 0] * tf.shape(input)[1]) + coordinates[:, 1]
  return tf.gather(input_as_vector, coordinates_as_indices)

values_at_top_left = get_values_at_coordinates(I, top_left)
values_at_top_right = get_values_at_coordinates(I, top_right)
values_at_bottom_left = get_values_at_coordinates(I, bottom_left)
values_at_bottom_right = get_values_at_coordinates(I, bottom_right)

先计算水平方向的插值:

Compute the interpolation in the horizontal direction first:

# Varies between 0.0 and 1.0.
horizontal_offset = C[:, 0] - tf.cast(top_left[:, 0], tf.float32)

horizontal_interpolated_top = (
    ((1.0 - horizontal_offset) * values_at_top_left)
    + (horizontal_offset * values_at_top_right))

horizontal_interpolated_bottom = (
    ((1.0 - horizontal_offset) * values_at_bottom_left)
    + (horizontal_offset * values_at_bottom_right))

现在计算垂直方向的插值:

Now compute the interpolation in the vertical direction:

vertical_offset = C[:, 1] - tf.cast(top_left[:, 1], tf.float32)

interpolated_result = (
    ((1.0 - vertical_offset) * horizontal_interpolated_top)
    + (vertical_offset * horizontal_interpolated_bottom))

这篇关于使用 TensorFlow 对图像中的点进行插值采样的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！