使用KERAS的Python生成器线程安全 [英] python generator thread safety using Keras

问题描述

我正在为一些ML使用Kera，并使用以下生成器来生成数据和标签：

def createBatchGenerator(driving_log,batch_size=32):
    batch_images = np.zeros((batch_size, 66, 200, 3))
    batch_steering = np.zeros(batch_size)
    while 1:
        for i in range(batch_size):
            x,y = get_preprocessed_row(driving_log)
            batch_images[i]=x
            batch_steering[i]=y
        yield batch_images, batch_steering

当我在本地使用它时，它运行得很好，但当我在带有GPU的AWS g2.2xLarge上运行它时，我得到了这个错误：ValueError：生成器已经在执行中。有人能帮我解决这个问题吗？

推荐答案

您需要创建一个generator that can support multi-threading以确保生成器同时被两个线程调用：

import threading

class createBatchGenerator:

    def __init__(self, driving_log,batch_size=32):
        self.driving_log = driving_log
        self.batch_size = batch_size
        self.lock = threading.Lock()

    def __iter__(self):
        return self

    def __next__(self):
        with self.lock:
           batch_images = np.zeros((batch_size, 66, 200, 3))
           batch_steering = np.zeros(batch_size)

           for i in range(self.batch_size):
               x,y = get_preprocessed_row(self.driving_log)
               batch_images[i]=x
               batch_steering[i]=y
           return batch_images, batch_steering

这篇关于使用KERAS的Python生成器线程安全的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！