如何使用 Torch Vision 在 Google Colab 上加载 CelebA 数据集，而不会耗尽内存? [英] How do I load the CelebA dataset on Google Colab, using torch vision, without running out of memory?

查看：18
本文介绍了如何使用 Torch Vision 在 Google Colab 上加载 CelebA 数据集，而不会耗尽内存?的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正在学习关于 DCGAN 的教程.每当我尝试加载 CelebA 数据集时，torchvision 会耗尽我所有运行时的内存 (12GB) 并且运行时崩溃.我正在寻找如何在不占用运行时资源的情况下加载和应用数据集转换的方法.

复制

这是导致问题的代码部分.

# 数据集的根目录data_root = '数据/塞巴'# 训练图像的空间大小，图像被调整到这个大小.图像大小 = 64celeba_data = datasets.CelebA(data_root,下载=真，变换=变换.撰写([变换.调整大小(图像大小)，transforms.CenterCrop(image_size),transforms.ToTensor(),transforms.Normalize(mean=[0.5, 0.5, 0.5],标准差=[0.5, 0.5, 0.5])]))

可以在此处找到完整的笔记本>

环境

  • PyTorch 版本:1.7.1+cu101

  • 是否为调试版本:False

  • 用于构建 PyTorch 的 CUDA:10.1

  • 用于构建 PyTorch 的 ROCM:不适用

  • 操作系统:Ubuntu 18.04.5 LTS (x86_64)

  • GCC 版本:(Ubuntu 7.5.0-3ubuntu1~18.04)7.5.0

  • Clang 版本:6.0.0-1ubuntu2(标签/RELEASE_600/final)

  • CMake 版本:3.12.0 版

  • Python 版本:3.6(64 位运行时)

  • CUDA 是否可用:正确

  • CUDA 运行时版本:10.1.243

  • GPU 型号和配置:GPU 0:Tesla T4

  • Nvidia 驱动程序版本:418.67

  • cuDNN 版本:/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

  • HIP 运行时版本:不适用

  • MIOpen 运行时版本:不适用

相关库的版本:

  • [pip3] numpy==1.19.4
  • [pip3] 火炬==1.7.1+cu101
  • [pip3] torchaudio==0.7.2
  • pip3] torchsummary==1.5.1
  • [pip3] torchtext==0.3.1
  • [pip3] torchvision==0.8.2+cu101
  • [conda] 无法收集

附加上下文

我尝试过的一些事情是:

  • 在单独的行上下载和加载数据集.例如:

# 只下载数据集datasets.CelebA(data_root，下载=真)# 在这里加载数据集celeba_data = datasets.CelebA(data_root，下载=假，变换=...)

  • 使用 ImageFolder 数据集类而不是 CelebA 类.例如:

# 只下载数据集datasets.CelebA(data_root，下载=真)# 使用 ImageFolder 类加载数据集celeba_data = datasets.ImageFolder(data_root, transforms=...)

在任何一种情况下，内存问题仍然存在.

解决方案

我没有设法找到内存问题的解决方案.但是，我想出了一个解决方法，自定义数据集.这是我的实现:

import os导入压缩文件导入 gdown进口火炬从 natsort 进口 natsorted从 PIL 导入图像从 torch.utils.data 导入数据集从 torchvision 导入转换## 设置#可用的GPU数量ngpu = 1device = torch.device('cuda:0' if (torch.cuda.is_available() 和 ngpu >0) 其他 'cpu')## 从 Google Drive 获取数据# 数据集的根目录data_root = '数据/塞巴'# 数据集所在文件夹的路径dataset_folder = f'{data_root}/img_align_celeba'# CelebA 数据集的 URLurl = 'https://drive.google.com/uc?id=1cNIac61PSA_LqDFYFUeyaQYekYPc75NH'# 数据集下载路径download_path = f'{data_root}/img_align_celeba.zip'# 创建需要的目录如果不是 os.path.exists(data_root):os.makedirs(data_root)os.makedirs(dataset_folder)# 从谷歌驱动器下载数据集gdown.download(网址，下载路径，安静=假)# 解压下载的文件使用 zipfile.ZipFile(download_path, 'r') 作为 ziphandler:ziphandler.extractall(dataset_folder)## 创建自定义数据集类类 CelebADataset(数据集):def __init__(self, root_dir, transform=None):"参数:root_dir(字符串):包含所有图像的目录变换(可调用，可选):要应用于每个图像样本的变换"# 读取根目录下的图片名称image_names = os.listdir(root_dir)self.root_dir = root_dirself.transform = 变换self.image_names = natsorted(image_names)def __len__(self):返回 len(self.image_names)def __getitem__(self, idx):# 获取图片的路径img_path = os.path.join(self.root_dir, self.image_names[idx])# 加载图像并将其转换为 RGBimg = Image.open(img_path).convert('RGB')# 对图像应用变换如果 self.transform:img = self.transform(img)返回图像##加载数据集# 包含所有图像的目录路径img_folder = f'{dataset_folder}/img_align_celeba'# 训练图像的空间大小，图像被调整到这个大小.图像大小 = 64# 应用于每个单独图像样本的变换变换=变换.撰写([变换.调整大小(图像大小)，transforms.CenterCrop(image_size),transforms.ToTensor(),transforms.Normalize(mean=[0.5, 0.5, 0.5],标准差=[0.5, 0.5, 0.5])])# 从文件加载数据集并应用转换celeba_dataset = CelebADataset(img_folder, 变换)## 创建一个数据加载器# 训练期间的批次大小批量大小 = 128# 数据加载器的工人数量num_workers = 0 如果 device.type == 'cuda' else 2# 是否将获取的数据张量放入固定内存pin_memory = True if device.type == 'cuda' else Falseceleba_dataloader = torch.utils.data.DataLoader(celeba_dataset，批量大小=批量大小，num_workers=num_workers,pin_memory=pin_memory,洗牌=真)

此实现具有内存效率，适用于我的用例，即使在训练期间使用的内存平均约为(4GB).但是，对于可能导致内存问题的原因，我希望能有进一步的直觉.

I am following a tutorial on DCGAN. Whenever I try to load the CelebA dataset, torchvision uses up all my run-time's memory(12GB) and the runtime crashes. Am looking for ways on how I can load and apply transformations to the dataset without hogging my run-time's resources.

To Reproduce

Here is the part of the code that is causing issues.

# Root directory for the dataset
data_root = 'data/celeba'
# Spatial size of training images, images are resized to this size.
image_size = 64

celeba_data = datasets.CelebA(data_root,
                              download=True,
                              transform=transforms.Compose([
                                  transforms.Resize(image_size),
                                  transforms.CenterCrop(image_size),
                                  transforms.ToTensor(),
                                  transforms.Normalize(mean=[0.5, 0.5, 0.5],
                                                       std=[0.5, 0.5, 0.5])
                              ]))

The full notebook can be found here

Environment

  • PyTorch version: 1.7.1+cu101

  • Is debug build: False

  • CUDA used to build PyTorch: 10.1

  • ROCM used to build PyTorch: N/A

  • OS: Ubuntu 18.04.5 LTS (x86_64)

  • GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

  • Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)

  • CMake version: version 3.12.0

  • Python version: 3.6 (64-bit runtime)

  • Is CUDA available: True

  • CUDA runtime version: 10.1.243

  • GPU models and configuration: GPU 0: Tesla T4

  • Nvidia driver version: 418.67

  • cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5

  • HIP runtime version: N/A

  • MIOpen runtime version: N/A

Versions of relevant libraries:

  • [pip3] numpy==1.19.4
  • [pip3] torch==1.7.1+cu101
  • [pip3] torchaudio==0.7.2
  • pip3] torchsummary==1.5.1
  • [pip3] torchtext==0.3.1
  • [pip3] torchvision==0.8.2+cu101
  • [conda] Could not collect

Additional Context

Some of the things I have tried are:

  • Downloading and loading the dataset on seperate lines. e.g:

# Download the dataset only
datasets.CelebA(data_root, download=True)
# Load the dataset here
celeba_data = datasets.CelebA(data_root, download=False, transforms=...)

  • Using the ImageFolder dataset class instead of the CelebA class. e.g:

# Download the dataset only
datasets.CelebA(data_root, download=True)
# Load the dataset using the ImageFolder class
celeba_data = datasets.ImageFolder(data_root, transforms=...)

The memory problem is still persistent in either of the cases.

解决方案

I did not manage to find a solution to the memory problem. However, I came up with a workaround, custom dataset. Here is my implementation:

import os
import zipfile 
import gdown
import torch
from natsort import natsorted
from PIL import Image
from torch.utils.data import Dataset
from torchvision import transforms

## Setup
# Number of gpus available
ngpu = 1
device = torch.device('cuda:0' if (
    torch.cuda.is_available() and ngpu > 0) else 'cpu')

## Fetch data from Google Drive 
# Root directory for the dataset
data_root = 'data/celeba'
# Path to folder with the dataset
dataset_folder = f'{data_root}/img_align_celeba'
# URL for the CelebA dataset
url = 'https://drive.google.com/uc?id=1cNIac61PSA_LqDFYFUeyaQYekYPc75NH'
# Path to download the dataset to
download_path = f'{data_root}/img_align_celeba.zip'

# Create required directories 
if not os.path.exists(data_root):
  os.makedirs(data_root)
  os.makedirs(dataset_folder)

# Download the dataset from google drive
gdown.download(url, download_path, quiet=False)

# Unzip the downloaded file 
with zipfile.ZipFile(download_path, 'r') as ziphandler:
  ziphandler.extractall(dataset_folder)

## Create a custom Dataset class
class CelebADataset(Dataset):
  def __init__(self, root_dir, transform=None):
    """
    Args:
      root_dir (string): Directory with all the images
      transform (callable, optional): transform to be applied to each image sample
    """
    # Read names of images in the root directory
    image_names = os.listdir(root_dir)

    self.root_dir = root_dir
    self.transform = transform 
    self.image_names = natsorted(image_names)

  def __len__(self): 
    return len(self.image_names)

  def __getitem__(self, idx):
    # Get the path to the image 
    img_path = os.path.join(self.root_dir, self.image_names[idx])
    # Load image and convert it to RGB
    img = Image.open(img_path).convert('RGB')
    # Apply transformations to the image
    if self.transform:
      img = self.transform(img)

    return img

## Load the dataset 
# Path to directory with all the images
img_folder = f'{dataset_folder}/img_align_celeba'
# Spatial size of training images, images are resized to this size.
image_size = 64
# Transformations to be applied to each individual image sample
transform=transforms.Compose([
    transforms.Resize(image_size),
    transforms.CenterCrop(image_size),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5, 0.5, 0.5],
                          std=[0.5, 0.5, 0.5])
])
# Load the dataset from file and apply transformations
celeba_dataset = CelebADataset(img_folder, transform)

## Create a dataloader 
# Batch size during training
batch_size = 128
# Number of workers for the dataloader
num_workers = 0 if device.type == 'cuda' else 2
# Whether to put fetched data tensors to pinned memory
pin_memory = True if device.type == 'cuda' else False

celeba_dataloader = torch.utils.data.DataLoader(celeba_dataset,
                                                batch_size=batch_size,
                                                num_workers=num_workers,
                                                pin_memory=pin_memory,
                                                shuffle=True)

This implementation is memory efficient and works for my use case, even during training the memory used averages around(4GB). I would however, appreciate further intuition as to what might be causing the memory problems.

这篇关于如何使用 Torch Vision 在 Google Colab 上加载 CelebA 数据集，而不会耗尽内存?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
Python最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆