如何使用os.walk在keras中读取文本文件并将其转换为批处理数据集 [英] how to read text files in keras using os.walk and converting to batched dataset

问题描述

我有一些文本文件需要在keras和tensorflow中进行分析，我使用os.walk读取了文件路径并将其添加到列表中，然后尝试读取文本内容并使用keras和tensorflow对其进行批处理，如下所示

I have text files that I need to analyse in keras and tensorflow i read the file paths using the os.walk and added them to list am then trying to read the text content and batch them using keras and tensorflow as follows

classes = []
labs={}
read_files = []
labels = []
path = "./data"
for root,folders,files in os.walk(path):
    itt = 0
    for folder in folders:
        labs[folder] = itt
        itt+=1
        classes.append(folder)
    for file in files:
        file = root+"/"+file
        file_class = file.split("/")[-2]
        labels.append(labs[file_class])
        read_files.append(root+"/"+file)


dataset = tf.data.TextLineDataset(read_files)
dataset = dataset.batch(batch_size)
print(next(dataset))

这些不是确切的代码，因为我正在处理多个文件，但总的来说，直到我得到错误为止，我要做的一切

These is not the exact code since am working on several files but in general that is all I do till I get the error

当我在最后一行上方运行代码时抛出错误

when i run the code above the last line throws an error

File "/home/kim/Desktop/programs/python/text_processing/prog/main.py", line 10, in <module>
    0.2,32)
File "/home/kim/Desktop/programs/python/text_processing/prog/text_process.py", line 44, in load_data
    print(next(train_set))
TypeError: 'BatchDataset' object is not an iterator

我想知道为什么此代码无法正常工作，在conda中使用tf 2.0.0会感激如何解决此问题的任何帮助 甚至是关于执行此操作的更好方法的建议，我仍然想知道如何将数据拆分为验证和训练集

I was am wondering why this code cannot work,am using tf 2.0.0 in conda will appreciate any help in how to solve this or even a suggestion on a better way to do this, am still wondering also how I can sp-lit my data to validation and training set

推荐答案

尝试一下:

print(next(iter(dataset)))

这篇关于如何使用os.walk在keras中读取文本文件并将其转换为批处理数据集的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！