解析Yann LeCun的MNIST IDX文件格式 [英] Parsing Yann LeCun's MNIST IDX file format

问题描述

我想了解如何打开

I would like to understand how to open this version of the MNIST data set. For example, the training set label file train-labels-idx1-ubyte is defined as:

TRAINING SET LABEL FILE (train-labels-idx1-ubyte):
[offset] [type]          [value]          [description]
0000     32 bit integer  0x00000801(2049) magic number (MSB first)
0004     32 bit integer  60000            number of items
0008     unsigned byte   ??               label
0009     unsigned byte   ??               label
........
xxxx     unsigned byte   ??               label

And I found some code online that seems to work, but do not understand how it works:

with open('train-labels-idx1-ubyte', 'rb') as f:
    bytes = f.read(8)
    magic, size = struct.unpack(">II", bytes)

print(magic) # 2049
print(size)  # 60000

My understanding is that struct.unpack interprets the second argument as a big-endian byte string of two 4-byte integers (See here). When I actually print the value of bytes, though, I get:

b'\x00\x00\x08\x01\x00\x00\xea`'

The first four-byte integer makes sense:

b'\x00\x00\x08\x01'

The first two bytes are 0. The next indicates the data are unsigned bytes. And 0x01 indicates a 1-dimensional vector of labels. Assuming my understanding is correct so far, what is happening with the next three (four?) bytes:

...\x00\x00\xea`

How does this translate to 60,000?

解决方案

I wrote the following code in case anyone needs to parse the whole dataset of images (as it appears in the question's title), and not just the first two bytes.

import numpy as np
import struct

with open('samples/t10k-images-idx3-ubyte','rb') as f:
    magic, size = struct.unpack(">II", f.read(8))
    nrows, ncols = struct.unpack(">II", f.read(8))
    data = np.fromfile(f, dtype=np.dtype(np.uint8).newbyteorder('>'))
    data = data.reshape((size, nrows, ncols))

And just to check, show the first digit. In my case it's a 7.

import matplotlib.pyplot as plt
plt.imshow(data[0,:,:], cmap='gray')
plt.show()

这篇关于解析Yann LeCun的MNIST IDX文件格式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！