MATLAB编写多页tiff指数变慢 [英] MATLAB write multipage tiff exponentially slow

问题描述

我正在尝试编写一个多页tiff文件，该文件为128像素x 128像素x 122000帧的16位无符号整数.在快速的计算机上，ImageJ或简短的Python脚本可以在不到一分钟的时间内完成此操作.在同一台机器上，使用我尝试过的任何方法，MATLAB可能都需要花费几天的时间.到目前为止，这是我尝试过的:

I am trying to write a single multipage tiff file which is 128 pixels x 128 pixels x 122000 frames of 16-bit unsigned integers. ImageJ or a short Python script can do this in less than one minute on a fast machine. On the same machine, MATLAB would likely take days to do so using any method I have tried. Here is what I have tried so far:

使用imwrite将IMG写入test.tif(由MATLAB文档推荐)

    tic
    numframes=size(IMG,3);
    divider=10^(floor(log10(numframes))-1);
    imwrite(IMG(:,:,1),'test.tif');
    for i=2:numframes;
        imwrite(IMG(:,:,i),'test.tif','WriteMode','append');
        if (round(i/divider)==i/divider)
            fprintf('Frame %d written in %.0f seconds, %2d percent complete, time                 left=%.0f seconds \n', ...
                i, toc, i/numframes*100, (numframes - i)/(i/toc));
        end
    end

这将导致以下输出:

在104秒内写入10000帧，完成了8.196721e + 00％， 剩余时间= 1163秒，以296秒写入帧20000， 1.639344e + 01％完成，剩余时间= 1509秒帧30000写入590秒，2.459016e + 01％完成，剩余时间= 1809 以1035秒写入帧40000，3.278689e + 01％ 完成，剩余时间= 2121秒，在1682秒内写入了帧50000， 完成4.098361e + 01％，剩余时间= 2421秒

Frame 10000 written in 104 seconds, 8.196721e+00 percent complete, time left=1163 seconds Frame 20000 written in 296 seconds, 1.639344e+01 percent complete, time left=1509 seconds Frame 30000 written in 590 seconds, 2.459016e+01 percent complete, time left=1809 seconds Frame 40000 written in 1035 seconds, 3.278689e+01 percent complete, time left=2121 seconds Frame 50000 written in 1682 seconds, 4.098361e+01 percent complete, time left=2421 seconds

请注意，随着写入更多帧，时间呈指数增长.

Notice the exponential increase in time as more frames are written.

直接使用Tiff类

    if bigtiff
        t = Tiff(fname,'w8');
    else
        t = Tiff(fname,'w');
    end
    tagstruct.ImageLength = size(image,1);
    tagstruct.ImageWidth = size(image,2);
    tagstruct.Photometric = Tiff.Photometric.MinIsBlack;
    if bitspersamp==16
        tagstruct.BitsPerSample = 16;
    end
    if bitspersamp==32
        tagstruct.BitsPerSample = 32;
    end
    tagstruct.SamplesPerPixel = 1;
    tagstruct.RowsPerStrip = 256;
    tagstruct.PlanarConfiguration = Tiff.PlanarConfiguration.Chunky;
    tagstruct.Software = 'MATLAB';
    t.setTag(tagstruct);
    t.write(image(:,:,1));
    numframes = size(image,3);
    divider = 10^(floor(log10(numframes))-1);
    tic
    for i=2:numframes
        t.writeDirectory();
        t.setTag(tagstruct);
        t.write(image(:,:,i));
        if (round(i/divider)==i/divider)
            fprintf('Frame %d written in %.0f seconds, %2d percent complete, time left=%.0f seconds \n', ...
                i, toc, i/numframes*100, (numframes - i)/(i/toc));
        end
    end
    t.close();

这将导致以下输出:

Frame 10000 written in 66 seconds, 8.196721e+00 percent complete, time left=743 seconds 
Frame 20000 written in 225 seconds, 1.639344e+01 percent complete, time left=1145 seconds 
Frame 30000 written in 481 seconds, 2.459016e+01 percent complete, time left=1474 seconds 
Frame 40000 written in 915 seconds, 3.278689e+01 percent complete, time left=1877 seconds 
Frame 50000 written in 1512 seconds, 4.098361e+01 percent complete, time left=2177 seconds

尝试使用BigTIFF库不起作用

在这里进行以下讨论: http://blogs.mathworks.com/steve/2013/08/07/tiff-bigtiff-and-blockproc/

Following the discussion here: http://blogs.mathworks.com/steve/2013/08/07/tiff-bigtiff-and-blockproc/

我试图通过将第73行更改为以下代码来转换代码以使用uint16数据:

I attempted to convert the code to work with uint16 data by changing line 73 to:

    obj.TiffObject.setTag('BitsPerSample', 16);

但是用

outFileWriter = bigTiffWriter('test.tif', inFileInfo(1).Height, inFileInfo(1).Width, tileSize(1), tileSize(2));
for i=1:122000
    blockproc(IMG(:,:,i),tileSize,@(b) b.data,'Destination',outFileWriter);
    if rem(i,10000)==0
        fprintf('Frame %d done\n',i)
    end
end

尝试读回时出现以下错误:

I get the following error when trying to read back in:

Unexpected Standard exception from MEX file.
What() is:std::bad_alloc
..

Error in imtifinfo (line 27)
raw_tags = tifftagsread(filename,0,0,0);

Error in imfinfo (line 183)
info = feval(fmt_s.info, filename);

Error in TiffReader (line 11)
InfoImage=imfinfo(fname);

在相关说明中，以正确的大小在磁盘上预分配文件没有区别

我认为这是文件I/O问题的可能性很小，在这种情况下，预分配磁盘上的空间可能是有意义的，所以我尝试了这里提到的内容:

I thought there was a slim chance this was a file I/O problem, in which case preallocating the space on disk might be relevant, so I tried what was mentioned here: http://www.mathworks.co.uk/matlabcentral/newsreader/view_thread/241072, namely:

% Create the file
fh = javaObject('java.io.RandomAccessFile', 'test.dat', 'rw');
% Allocate the right amount of space
fh.setLength(1024);
% Close the file
fh.close();

但这没什么区别.

任何帮助将不胜感激.

推荐答案

我也曾经遇到过这个问题. Matlab 2018b似乎有两个可以处理tiff文字的.mexw64文件.

I also once ran into this problem. It seems Matlab 2018b has two .mexw64 files that can handle tiff writing.

wtifc.mexw64，由imwrite使用

wtifc.mexw64 in \toolbox\matlab\imagesci\private called by writetif.mwhich is used by imwrite

和

tifflib.mexw64，相同的位置，由Tiff对象使用.

tifflib.mexw64, same location, used by the Tiff object.

使用多图像tiff文件时，两者都变慢.使用fwrite写入所有图像数据并以指向该数据的指针结尾要快得多.

Both slow down with multi image tiff files. It is a lot faster to use fwrite to write all image data and end with the pointers to this data.

我使用此脚本测试了三种方法，差异很大. 这是使用matlab 2018b完成的.

I tested three methods using this script and the difference is quite big. This was done using matlab 2018b.

clear all;close all; clc; fclose all;
%generate some data
N=1E3;
IM=imread('landOcean.jpg');
IM = uint16(sum(IM,3));
IM = IM(100:310,960:1170);
IM = IM-min(IM(:));
IM=IM*(2^15/max(IM(:)));
IM = repmat(IM,[1,1,N])+randi((2^15)-1,[size(IM,1),size(IM,2),N],'uint16');
S = (numel(IM)/N*2)/2^20;


%imread writespeed
methods = {'imwrite','tifflib','fTIF'};
for M = 1:length(methods)
    method = methods{M};
    %file
    filename = [method,'.tif'];
    if exist(filename,'file'), delete(filename);end
    switch method
        case 'imwrite'
            %timing vector
            t = zeros(1,100+1);
            tic;
            imwrite(IM(:,:,1),filename);
            t(2)=toc;
            for ct = 2:100
                imwrite(IM(:,:,ct),filename,'WriteMode','append');
                t(ct+1)=toc;
            end
        case 'tifflib'
            %timing vector
            t = zeros(1,200+1);
            tic;
            tf = Tiff(filename,'w');
            for ct = 1:200
                if ct>1,tf.writeDirectory;end
                tf.setTag('Photometric',Tiff.Photometric.MinIsBlack);
                tf.setTag('Compression',Tiff.Compression.None);
                tf.setTag('BitsPerSample',16);
                tf.setTag('SamplesPerPixel',1);
                tf.setTag('SampleFormat',Tiff.SampleFormat.UInt);
                tf.setTag('ExtraSamples',Tiff.ExtraSamples.Unspecified);
                tf.setTag('ImageLength',size(IM,1));
                tf.setTag('ImageWidth',size(IM,2));
                tf.setTag('PlanarConfiguration',Tiff.PlanarConfiguration.Chunky);
                tf.setTag('ImageDescription',sprintf('ImageJ=1.51j\nchannels=%.0f',size(IM,3)));
                tf.write(IM(:,:,ct));
                t(ct)=toc;
            end
            tf.close();
        case 'fTIF'
            %timing vector
            t = zeros(1,size(IM,3)+1);
            tic
            fTIF = Fast_Tiff(filename);
            for ct = 1:size(IM,3)
                fTIF = fTIF.WriteIMG(IM(:,:,ct)');
                t(ct)=toc;
            end
            tic
            fTIF.close;
            toc
        otherwise
            error('unknown method')
    end
    S = (size(IM,1)*size(IM,2)*2)/2^20; %MB/frame
    y = S./diff(t);
    subplot(1,length(methods),M)
    plot([1:length(y)],y);
    title(sprintf('Writing with %s; mean = %.2f MB/s',method,mean(y)))
    ylabel('Writing speed (MB/s)')
    xlabel('Frame');
    drawnow;
end

fTIF方法的代码:

Code for the fTIF method:

https://github.com/rharkes/Fast_Tiff_Write

这篇关于MATLAB编写多页tiff指数变慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！