如何将基于自定义图像的数据集加载到 Pytorch 以与 CNN 一起使用? [英] How do I load custom image based datasets into Pytorch for use with a CNN?
问题描述
我在互联网上搜索了几个小时，以找到解决我问题的好方法.以下是一些相关的背景信息，可帮助您回答我的问题.
这是我的第一个深度学习项目，我不知道我在做什么.我知道理论但不知道实践元素.
我使用的数据可以在这个链接的 kaggle 上找到:(https://www.kaggle.com/alxmamaev/flowers-recognition)
我的目标是使用 CNN 根据数据集中提供的图像对花朵进行分类.
这是迄今为止我尝试用来加载数据的一些示例代码，这是我最好的尝试，但正如我所提到的，我一无所知，而且 Pytorch 文档没有提供太多我可以理解的帮助.(https://pastebin.com/fNLVW1UW)
# 加载用于 CNN 的图像.def load_images(image_size=32, batch_size=64, root="../images"):变换 = 变换.撰写([变换.调整大小(32)，transforms.ToTensor(),transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])train_set = datasets.ImageFolder(root=root, train=True, transform=transform)train_loader = torch.utils.data.DataLoader(train_set，batch_size=batch_size，shuffle=True，num_workers=2)返回 train_loader# 定义用于 CNN 的变量.班级 =('雏菊'、'蒲公英'、'玫瑰'、'向日葵'、'郁金香')train_loader_data = load_images()# 训练样本.n_training_samples = 3394train_sampler = SubsetRandomSampler(np.arange(n_training_samples, dtype=np.int64))# 验证样本.n_val_samples = 424val_sampler = SubsetRandomSampler(np.arange(n_training_samples, n_training_samples + n_val_samples, dtype=np.int64))# 测试样本.n_test_samples = 424test_sampler = SubsetRandomSampler(np.arange(n_test_samples, dtype=np.int64))以下是我也需要回答的直接问题:
如何修复我的代码以在 80/10/10 拆分中加载到数据集中以进行训练/测试/验证?
如何为这些图像创建所需的标签/类，这些图像已经被/images 中的文件夹划分了?
查看Kaggle的数据和你的代码，你的数据加载有问题.
数据应该在每个类标签的不同文件夹中，以便 PyTorch ImageFolder 正确加载它.在您的情况下，由于所有训练数据都在同一个文件夹中，PyTorch 将其作为一个训练集加载.您可以使用以下文件夹结构来纠正此问题 - train/daisy、train/dandelion、test/daisy、test/dandelion 然后将训练和测试文件夹分别传递给训练和测试 ImageFolder.只需更改文件夹结构就可以了.看看torchvision.datasets.Imagefolder 有一个类似的例子.
正如你所说，这些图像已经被/images中的文件夹划分了.PyTorch ImageFolder 假设图像按以下方式组织.但是这个文件夹结构只有在你使用训练集的所有图像时才是正确的:
```/images/daisy/100080576_f52e8ee070_n.jpg/图像/雏菊/10140303196_b88d3d6cec.jpg.../图像/蒲公英/10043234166_e6dd915111_n.jpg/图像/蒲公英/10200780773_c6051a7d71_n.jpg``其中雏菊"、蒲公英"等是类别标签.
如果要将数据集拆分为训练集和测试集，请使用正确的文件夹结构(请注意，我知道您想将数据集拆分为训练集、验证集和测试集，但这并不重要只是一个例子来表达这个想法):
```/images/train/daisy/100080576_f52e8ee070_n.jpg/图像/火车/雏菊/10140303196_b88d3d6cec.jpg../图像/火车/蒲公英/10043234166_e6dd915111_n.jpg/图像/火车/蒲公英/10200780773_c6051a7d71_n.jpg../图像/测试/雏菊/300080576_f52e8ee070_n.jpg/images/test/daisy/95140303196_b88d3d6cec.jpg../图像/测试/蒲公英/32143234166_e6dd915111_n.jpg/图像/测试/蒲公英/65200780773_c6051a7d71_n.jpg``然后，您可以参考以下完整代码示例来了解如何编写数据加载器:
导入操作系统将 numpy 导入为 np进口火炬将 torch.nn 导入为 nn导入 torch.nn.functional 作为 F从 torch.autograd 导入变量导入 torch.utils.data 作为数据进口火炬视觉从 torchvision 导入转换纪元 = 2BATCH_SIZE = 10LEARNING_RATE = 0.003TRAIN_DATA_PATH = "./images/train/"TEST_DATA_PATH = "./images/test/"TRANSFORM_IMG = transforms.Compose([变换.调整大小(256)，变换.CenterCrop(256),transforms.ToTensor(),transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225] )])train_data = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH，transform=TRANSFORM_IMG)train_data_loader = data.DataLoader(train_data，batch_size=BATCH_SIZE，shuffle=True，num_workers=4)test_data = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH，transform=TRANSFORM_IMG)test_data_loader = data.DataLoader(test_data，batch_size=BATCH_SIZE，shuffle=True，num_workers=4)类CNN(nn.Module):#省略...如果 __name__ == '__main__':print("训练样本数:", len(train_data))print("测试样本数:", len(test_data))print("Detected Classes are: ", train_data.class_to_idx) # 通过文件夹结构检测类模型 = CNN()优化器 = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)loss_func = nn.CrossEntropyLoss()# 训练和测试对于范围内的纪元(EPOCHS):对于 enumerate(train_data_loader) 中的步骤 (x, y):b_x = Variable(x) # 批量 x(图像)b_y = Variable(y) # 批次 y(目标)输出 = 模型(b_x)[0]损失 = loss_func(输出，b_y)optimizer.zero_grad()损失.向后()优化器.step()如果步骤 % 50 == 0:test_x = 变量(test_data_loader)test_output，last_layer = 模型(test_x)pred_y = torch.max(test_output, 1)[1].data.squeeze()准确度 = sum(pred_y == test_y)/float(test_y.size(0))print('Epoch: ', epoch, '| train loss: %.4f' % loss.data[0], '| 测试准确度:%.2f' %准确度)I have searched for hours on the internet to find a good solution to my issue. Here is some relevant background information to help you answer my question.
This is my first ever deep learning project and I have no idea what I am doing. I know the theory but not the practical elements.
The data that I am using can be found on kaggle at this link: (https://www.kaggle.com/alxmamaev/flowers-recognition)
I am aiming to classify flowers based on the images provided in the dataset using a CNN.
Here is some sample code I have tried to use to load data in so far, this is my best attempt but as I mentioned I am clueless and Pytorch docs didn't offer much help that I could understand at my level. (https://pastebin.com/fNLVW1UW)
# Loads the images for use with the CNN.
def load_images(image_size=32, batch_size=64, root="../images"):
transform = transforms.Compose([
transforms.Resize(32),
transforms.ToTensor(),
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
train_set = datasets.ImageFolder(root=root, train=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=2)
return train_loader
# Defining variables for use with the CNN.
classes = ('daisy', 'dandelion', 'rose', 'sunflower', 'tulip')
train_loader_data = load_images()
# Training samples.
n_training_samples = 3394
train_sampler = SubsetRandomSampler(np.arange(n_training_samples, dtype=np.int64))
# Validation samples.
n_val_samples = 424
val_sampler = SubsetRandomSampler(np.arange(n_training_samples, n_training_samples + n_val_samples, dtype=np.int64))
# Test samples.
n_test_samples = 424
test_sampler = SubsetRandomSampler(np.arange(n_test_samples, dtype=np.int64))
Here are my direct questions that I require answers too:
How do I fix my code to load in the dataset in an 80/10/10 split for training/test/validation?
How do i create the required labels/classes for these images which are already divided by folders in /images ?
Looking at the data from Kaggle and your code, there are problems in your data loading.
The data should be in a different folder per class label for PyTorch ImageFolder to load it correctly. In your case, since all the training data is in the same folder, PyTorch is loading it as one train set. You can correct this by using a folder structure like - train/daisy, train/dandelion, test/daisy, test/dandelion and then passing the train and the test folder to the train and test ImageFolder respectively. Just change the folder structure and you should be good. Take a look at the official documentation of torchvision.datasets.Imagefolder which has a similar example.
As you said, these images which are already divided by folders in /images. PyTorch ImageFolder assumes that images are organized in the following way. But this folder structure is only correct if you are using all the images for train set:
```
/images/daisy/100080576_f52e8ee070_n.jpg
/images/daisy/10140303196_b88d3d6cec.jpg
.
.
.
/images/dandelion/10043234166_e6dd915111_n.jpg
/images/dandelion/10200780773_c6051a7d71_n.jpg
```
where 'daisy', 'dandelion' etc. are class labels.
The correct folder structure if you want to split the dataset into train and test set in your case (note that I know you want to split the dataset into train, validation, and test set, but it doesn't matters as this is just an example to get the idea out):
```
/images/train/daisy/100080576_f52e8ee070_n.jpg
/images/train/daisy/10140303196_b88d3d6cec.jpg
.
.
/images/train/dandelion/10043234166_e6dd915111_n.jpg
/images/train/dandelion/10200780773_c6051a7d71_n.jpg
.
.
/images/test/daisy/300080576_f52e8ee070_n.jpg
/images/test/daisy/95140303196_b88d3d6cec.jpg
.
.
/images/test/dandelion/32143234166_e6dd915111_n.jpg
/images/test/dandelion/65200780773_c6051a7d71_n.jpg
```
Then, you can refer to the following full code example on how to write a dataloader:
import os
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.utils.data as data
import torchvision
from torchvision import transforms
EPOCHS = 2
BATCH_SIZE = 10
LEARNING_RATE = 0.003
TRAIN_DATA_PATH = "./images/train/"
TEST_DATA_PATH = "./images/test/"
TRANSFORM_IMG = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(256),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225] )
])
train_data = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)
train_data_loader = data.DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)
test_data = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)
test_data_loader = data.DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)
class CNN(nn.Module):
# omitted...
if __name__ == '__main__':
print("Number of train samples: ", len(train_data))
print("Number of test samples: ", len(test_data))
print("Detected Classes are: ", train_data.class_to_idx) # classes are detected by folder structure
model = CNN()
optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
loss_func = nn.CrossEntropyLoss()
# Training and Testing
for epoch in range(EPOCHS):
for step, (x, y) in enumerate(train_data_loader):
b_x = Variable(x) # batch x (image)
b_y = Variable(y) # batch y (target)
output = model(b_x)[0]
loss = loss_func(output, b_y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
if step % 50 == 0:
test_x = Variable(test_data_loader)
test_output, last_layer = model(test_x)
pred_y = torch.max(test_output, 1)[1].data.squeeze()
accuracy = sum(pred_y == test_y) / float(test_y.size(0))
print('Epoch: ', epoch, '| train loss: %.4f' % loss.data[0], '| test accuracy: %.2f' % accuracy)
这篇关于如何将基于自定义图像的数据集加载到 Pytorch 以与 CNN 一起使用?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
