Ruby：比较两个哈希数组 [英] Ruby: Comparing two Arrays of Hashes

问题描述

我绝对是新手ruby（和使用1.9.1），所以任何帮助是赞赏。我学过的关于Ruby的一切都是从使用google。我试图比较两个哈希数组，由于大小，它的方式到长和调情运行内存不足。任何帮助将不胜感激。

I'm definitely a newbie to ruby (and using 1.9.1), so any help is appreciated. Everything I've learned about Ruby has been from using google. I'm trying to compare two arrays of hashes and due to the sizes, it's taking way to long and flirts with running out of memory. Any help would be appreciated.

我有一个类（ParseCSV）有多个方法（初始化，打开，比较，剥离，输出）。
我现在的工作方式如下（并且这通过了我写的测试，只是使用更小的数据集）：

I have a Class (ParseCSV) with multiple methods (initialize, open, compare, strip, output). The way I have it working right now is as follows (and this does pass the tests I've written, just using a much smaller data set):

file1 = ParseCSV.new("some_file")
file2 = ParseCSV.new("some_other_file")

file1.open #this reads the file contents into an Array of Hash’s through the CSV library 
file1.strip #This is just removing extra hash’s from each array index.  So normally there are fifty hash’s in each array index, this is just done to help reduce memory consumption.  

file2.open 
file2.compare("file1.storage") #@storage is The array of hash’s from the open method

file2.output

现在我正在努力的是比较方法。使用较小的数据集，这不是一个大问题，工作速度足够快。但是在这种情况下，我比较大约400,000条记录（所有读入哈希数组）与大约450,000条记录。我想加快速度。此外，我不能运行strip2的方法。这是我现在做的：

Now what I’m struggling with is the compare method. Working on smaller data sets it’s not a big deal at all, works fast enough. However in this case I’m comparing about 400,000 records (all read into the array of hashes) against one that has about 450,000 records. I’m trying to speed this up. Also I can’t run the strip method on file2. Here is how I’m doing it now:

def compare(x)
    #obviously just a verbose message
    puts "Comparing and leaving behind non matching entries"

    x.each do |row|
        #@storage is the array of hashes
        @storage.each_index do |y|       
            if row[@opts[:field]] == @storage[y][@opts[:field]]
               @storage.delete_at(y)
            end
       end
    end
end

希望这很有意义。我知道这将是一个缓慢的过程，因为它必须重复400,000行每次440,000次。但是你有什么其他的想法如何加快它，并可能减少内存消耗？

Hopefully that makes sense. I know it’s going to be a slow process just because it has to iterate 400,000 rows 440,000 times each. But do you have any other ideas on how to speed it up and possibly reduce memory consumption?

推荐答案

Yikes O（n）平方运行时。讨厌。

Yikes, that'll be O(n) squared runtime. Nasty.

更好的办法是使用内置的设置类。

A better bet would be to use the built in Set class.

代码如下：

require 'set'

file1_content = load_file_content_into_array_here("some_file")
file2_content = load_file_content_into_array_here("some_other_file")

file1_set = Set[file1_content]

unique_elements = file1_set - file2_content

假定文件本身具有唯一的内容。应该在通用情况下工作，但可能有奇怪，取决于你的数据和你如何解析它，但只要行可以与 == 比较

That assumes that the files themselves have unique content. Should work in the generic case, but may have quirks depending on what your data looks like and how you parse it, but as long as the lines can be compared with == it should help you out.

使用集合比在嵌套循环中遍历文件内容要快很多。

Using a set will be MUCH faster than doing a nested loop to iterate over the file content.

（是的，我已经做到这一点，以处理大约2百万行的文件，所以它应该能够处理你的情况 - 最终如果你做大量的数据调整，Ruby可能不是最好的选择的工具虽然）

(and yes, I have actually done this to process files with ~2 million lines, so it should be able to handle your case - eventually. If you're doing heavy data munging, Ruby may not be the best choice of tool though)

这篇关于Ruby：比较两个哈希数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！