巨大的广播变量，无需parfor即可优化代码? [英] Huge broadcast variable, optimizing code without parfor?

问题描述

我有一个40000 x 80000矩阵，从该矩阵中，我可以获取簇"(彼此相邻的具有相同值的元素组)的数量，然后计算每个簇的大小.这是代码块.

I have a 40000 by 80000 matrix from which I'm obtaining the number of "clusters" (groups of elements with the same value that are adjacent to one another) and then calculating the size of each of those clusters. Here it is the chunk of code.

FRAGMENTSIZESCLASS = struct([]);  %We store the data in a structure
for class=1:NumberOfClasses
  %-First we create a binary image for each class-%
  BWclass = foto==class;
  %-Second we calculate the number of connected components (fragments)-%
  L = bwlabeln(BWclass);          %returns a label matrix, L, containing labels for the connected components in BWclass
  clear BWclass
  NumberFragments=max(max(L));
  %-Third we calculate the size of each fragment-%
  FragmentSize=zeros(NumberFragments,1);
  for f=1:NumberFragments      % potential improvement: using parfor while saring the memory between workers
    FragmentSize(f,1) = sum(L(:) == f);
  end
  FRAGMENTSIZESCLASS{class}=FragmentSize;
  clear L
end

问题在于矩阵L太大，以至于如果我使用parfor循环，它将变成一个广播变量，然后内存被相乘，而我的内存不足.

The problem is that the matrix L is so large that if I use a parfor loop it turns into a broadcast variable and then the memory gets multiplied and I run out of memory.

有关如何解决此问题的任何想法?我看过以下文件: https://ch.mathworks.com/matlabcentral/fileexchange /28572-sharedmatrix ，但这不是一个简单的解决方案，即使我拥有24个内核仍会花费很多时间.

Any ideas on how to sort this out? I've seen this file: https://ch.mathworks.com/matlabcentral/fileexchange/28572-sharedmatrix but is not an straightforward solution, even though I have 24 cores still will take a lot of time.

干杯！

这是一张图片，显示了使用问题中发布的代码与@bla建议使用bwconncomp时所花费的时间与图像大小的关系:

Here it is a picture showing the time it takes as a function of image size when using the code I posted in the question vs using bwconncomp as suggested by @bla:

推荐答案

而不是bwlabeln使用内置函数bwconncomp，例如:

instead of bwlabeln use the built in function bwconncomp, for example:

...
s=bwconncomp(BWClass);
fragmentsize=sum(cellfun(@numel,s.PixelIdxList));
....

这篇关于巨大的广播变量，无需parfor即可优化代码?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！