在MATLAB中读取MNIST图像数据库二进制文件 [英] Reading MNIST Image Database binary file in MATLAB

问题描述

我有一个来自MNIST图像数据库的二进制文件，重命名为myfile.dat。它由一组4个无符号32位整数组成，后跟一串无符号8位整数。我想读取此文件并将其内容存储为数组。这是我的代码：

I have a binary file from the MNIST image database, renamed as "myfile.dat". This consists of a set of 4 unsigned 32-bit integers, followed by a chain of unsigned 8-bit integers. I want to read this file and store its contents as an array. Here is my code:

file_id = fopen('myfile.dat', 'rb');
data = fread(file_id, 'int');
size(data)
class(data)

输出为：

ans =

    2502           1


ans =

double

（2502,1）的大小与预期一致。但是为什么它告诉我数据是 double ，当我指定它是 int ？

The size of (2502, 1) is as expected. But why is it telling me that the data is double, when I have specified it to be int?

我知道前几个数字应该是什么，输出数据不符合预期。我还试过 int32 ， uint 和 uint32 哪个给出同样的问题。

I know what the first few numbers should be and the output data is not as expected. I have also tried int32, uint and uint32 which give the same problem.

推荐答案

根据我们的评论，MATLAB一次读入4个整数的方式是 little-endian 格式，而MNIST数据库文件的文件格式为 big-endian 。因此，当我们读入文件的前四个字节时，字节是 0x00,0x00,0x08,0x03 ，正如我们所期望的那样。但是，MATLAB将在 0x03,0x08,0x00,0x00 中读取。将此转换为整数时，我们实际上会得到 50855936 这不是我们想要的。

With our comments, the way MATLAB reads in 4 integers at a time is in little-endian format while the file format of the MNIST database file is in big-endian. As such, when we read in the first four bytes of the file, the bytes are 0x00, 0x00, 0x08, 0x03 as we expect. However, MATLAB will read this in as 0x03, 0x08, 0x00, 0x00. When converting this to an integer, we actually will get 50855936 which is not what we want.

一个hack to解决这个问题的时候，我们需要一次读取几个字节，我们需要确保指定 uint8 数据类型。这会将每个字节存储到数组中的单独元素中。然后，我们可以通过将每个字节按位移位指定的量来计算所需的数字，然后对结果求和。我们需要对数据文件中的前几个字节执行此操作。

A hack to solve this would be when we need to read in several bytes at a time, we need to ensure that we specify the uint8 data type. This will store each byte into separate elements in an array. We can then compute the necessary number we need by bit-shifting each byte over by a specified amount, then summing the results. We will need to do this for the first few bytes in the data file.

或者，我们可以使用您在评论中说明的 swapbytes 方法，这样就可以准确地给出我们一样的东西。您只需读取1字节的 uint32 类型，然后交换字节的顺序，以便字节在 big-endian中格式。您需要记住，即使您将数据读入 uint32 ，该数字也将作为 double 因此您需要在进入 swapbytes 之前转换该数字。

Alternatively, we can use the swapbytes method as you have stated in your comments and that'll give us exactly the same thing. You can just read 1 byte that is of a uint32 type, then swap the order of the bytes so that the bytes are in big-endian format. You'll need to bear in mind that even when you read in the data as uint32, the number will be stored in MATLAB as double so you will need to cast the number before going into swapbytes.

一旦我们到达实际的图像数据，我们可以一次读入 numRows x numCols 字节，然后重新整形数组，使其成为图像。我们可以将每个图像存储到单元阵列中。不用多说了，这是代码。

Once we get to the actual image data, we can read in numRows x numCols bytes at a time, then reshape the array so that it becomes an image. We can store each image into a cell array. Without further ado, here's the code.

clear all;
close all;

%//Open file
fid = fopen('t10k-images-idx3-ubyte', 'r');

%//Read in magic number
%//A = fread(fid, 4, 'uint8');
%//magicNumber = sum(bitshift(A', [24 16 8 0]));

%//OR
A = fread(fid, 1, 'uint32');
magicNumber = swapbytes(uint32(A));

%//Read in total number of images
%//A = fread(fid, 4, 'uint8');
%//totalImages = sum(bitshift(A', [24 16 8 0]));

%//OR
A = fread(fid, 1, 'uint32');
totalImages = swapbytes(uint32(A));

%//Read in number of rows
%//A = fread(fid, 4, 'uint8');
%//numRows = sum(bitshift(A', [24 16 8 0]));

%//OR
A = fread(fid, 1, 'uint32');
numRows = swapbytes(uint32(A));

%//Read in number of columns
%//A = fread(fid, 4, 'uint8');
%//numCols = sum(bitshift(A', [24 16 8 0]));

%// OR
A = fread(fid, 1, 'uint32');
numCols = swapbytes(uint32(A));

%//For each image, store into an individual cell
imageCellArray = cell(1, totalImages);
for k = 1 : totalImages
    %//Read in numRows*numCols pixels at a time
    A = fread(fid, numRows*numCols, 'uint8');
    %//Reshape so that it becomes a matrix
    %//We are actually reading this in column major format
    %//so we need to transpose this at the end
    imageCellArray{k} = reshape(uint8(A), numCols, numRows)';
end

%//Close the file
fclose(fid);

如果检查行数和列数（存储为 numRows ， numCols ），幻数（存储为 magicNumber ）和图像总数（存储为 totalImages ），这应该等于 2051,28,28 和 10000 分别。在此代码之后， imageCellArray 的 kth 元素将存储 kth MNIST数据库中的数字。如果您执行 imshow（imageCellArray {k}）; ，其中 k 是之间的任何整数> 1 到 10000 ，你应该可以看到一个数字。

If you check the number of rows and columns (stored as numRows, numCols), the magic number (stored as magicNumber) and the total number of images (stored as totalImages), this should be equal to 2051, 28, 28 and 10000 respectively. After this code, the kth element of imageCellArray will store the kth digit in the MNIST database. If you do imshow(imageCellArray{k});, where k is any integer between 1 to 10000, you should be able to see a digit.

此外，最后一点注意：由于矩阵数据中的读数将在 double 中，我们需要将其转换为 uint8 因为图像是来自数据库的那种类型。

Also, one final note: As reading in the matrix data will be in double, we need to cast this so that it's uint8 as the images are of that type from the database.

祝你好运！

这篇关于在MATLAB中读取MNIST图像数据库二进制文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！