在 tensorflow 数据集中标准化窗口 [英] Normalizing windows in tensorflow dataset

问题描述

我正在尝试从单变量时间序列构建一个窗口数据集.这个想法是如果系列看起来像 [1, 2, 3, 4, 5, 6] 并且窗口长度是 2，那么我会用长度为 3 的窗口来考虑 2 个 X 特征和 Y 目标输出，所以[[1, 2, 3], [2, 3, 4], [3, 4, 5], [4, 5, 6]] 然后我会洗牌以避免偏见从中分离出每个窗口的目标输出的输入特征:[[[1, 2], [3]], [[2, 3], [4]], [[3, 4], [5]], [[4, 5], [6]]]

I am trying to build a windowed dataset from a univariate time series. The idea is if the series looks like [1, 2, 3, 4, 5, 6] and the window length was 2, then I'd take windows of length 3 to account for 2 X features and Y target output, so [[1, 2, 3], [2, 3, 4], [3, 4, 5], [4, 5, 6]] then I'll shuffle them up to avoid bias from that, and split out the input features from target output for each window: [[[1, 2], [3]], [[2, 3], [4]], [[3, 4], [5]], [[4, 5], [6]]]

def windowed_dataset(series):
    # Initially the data is (N,) expand dims to (N, 1)
    series = tf.expand_dims(series, axis=-1)

    # Tensorflow Dataset from the array
    ds = tf.data.Dataset.from_tensor_slices(series)

    # Create the windows that will serve as input features and label (hence +1)
    ds = ds.window(window_len + 1, shift=1, drop_remainder=True)
    ds = ds.flat_map(lambda w: w.batch(window_len + 1))

    # randomize order 
    ds = ds.shuffle(shuffle_buffer)
    # Separate  the inputs and the target output(label)
    ds = ds.map(lambda w: (w[:-1], w[-1]))
    return ds.batch(batch_size).prefetch(1)

但是我想添加一些规范化.例如，如果我的窗口是 w=[1, 2, 3] 那么我想根据 [p/w[0] - 1 for p in w]代码>

However I'd like to add some normalization. For example if my window is w=[1, 2, 3] then I'd like to normalize according to [p/w[0] - 1 for p in w]

我以为我可以用 ds.map 和

I thought I could achieve this with ds.map and

    def normalize_window(w):
      return [((i/w[0]) -1) for i in w]


    ds = ds.map(normalize_window)

因为 map 应该将函数应用于数据集中的每个窗口，但这不起作用.tensorflow 数据集文档中的所有示例都使用 map 和 lambda 函数，但我认为它也适用于常规函数

because map is supposed to apply the function to each window in the dataset, but this didn't work. All the example in tensorflow dataset docs use map with lambda functions but I presume it works with regular functions too

有人知道应该怎么做吗?

Does anyone know how it should be done?

编辑

我得到的回溯是

<ipython-input-39-929295e1b775> in <module>()
----> 1 dataset = model_forecast_datasets(btc_model, np_data[:6])

11 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    263       except Exception as e:  # pylint:disable=broad-except
    264         if hasattr(e, 'ag_error_metadata'):
--> 265           raise e.ag_error_metadata.to_exception(e)
    266         else:
    267           raise

OperatorNotAllowedInGraphError: in user code:

    <ipython-input-38-b3d0f7e17689>:12 normalize_window  *
        return [(i/w[0] -1) for i in w]
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:561 __iter__
        self._disallow_iteration()
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:557 _disallow_iteration
        self._disallow_in_graph_mode("iterating over `tf.Tensor`")
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:537 _disallow_in_graph_mode
        " this function with @tf.function.".format(task))

    OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

推荐答案

你需要一个向量化计算的函数，比如

You would need a function that vectorizes the calculation, something like

def normalize(data):
    mean = tf.math.reduce_mean(data)
    std = tf.math.reduce_std(data)
    data = tf.subtract(data, mean)
    data = tf.divide(data, std)
    return data

ds = ds.map(normalize)

对于您的特定规范化，这可能有效:

For your specific normalization this may work:

def normalize(data):
    data1 = tf.subtract(data, tf.constant(1))
    data1 = tf.divide(data1, data[0])
    return data1

(这必须在批处理之后进行 ds = ds.flat_map(...)

(this would have to go after batching ds = ds.flat_map(...)

这篇关于在 tensorflow 数据集中标准化窗口的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！