未处理的异常:OutOfMemoryException或如何优化应用程序? [英] Unhandled Exception: OutOfMemoryException or how to optimize the application?

问题描述

大家好！我的程序带有参数.此参数是一个目录，程序将查找具有相同内容的所有文件，对文件进行分组并在屏幕上打印.使用少量文件，程序运行良好且快速，但是，例如，我使用闪存卡进行搜索和显示(或更生动-C:\目录)，程序逐渐增加RAM使用量，直至达到500Mb，并且编写此异常.我发布了代码，并希望听取批评者和帮助.

这里是比较功能:

Hi everybody! My program takes argument. This argument is a directory, and program will find all files with the same content, group that files and print on screen. With small amount of files, program works good and fast, but, for example, i take my flash-card for searching and compearing(or more dramatic - the C:\ directory), program gradually increase RAM using, until to 500Mb, and write this exception. I post my code, and want to listen the critic and help.

Here the comparing function:

static void CompareFilesRec(List<string> array)
{
    List<string> Trash = new List<string>();
    List<string> Output = new List<string>();
    List<KeyValuePair<long, string>> yeah = new List<KeyValuePair<long, string>>();
    List<string> outp = new List<string>();
    //filling the list with sizes of files(as a key) and paths of files
    for (int j = 0; j <= array.Count - 1; j++)
    {
        FileInfo fii = new FileInfo(array[j]);
        yeah.Add(new KeyValuePair<long, string>(fii.Length, array[j]));
    }
    //filtering the previous list. Move files with another sizes to trash(for processing in the end)
    foreach (var el in yeah)
    {
        if (!Output.Contains(el.Value) && !Trash.Contains(el.Value))
        {
            foreach (var ele in yeah)
            {
                if (el.Key == ele.Key && !Output.Contains(ele.Value))
                {
                    Output.Add(ele.Value);
                }
                else
                {
                    if (!Trash.Contains(ele.Value))
                    {
                        Trash.Add(ele.Value);
                    }
                }
            }
        }
    }
    //Comparing the files(i take the first element in list, and compare with other files)
    foreach (string f in Output)
    {
        int valueOne = 0, valueTwo = 0;
        StreamReader objReader = new StreamReader(f);
        valueOne = objReader.ReadToEnd().GetHashCode();

        StreamReader objReader1 = new StreamReader(Output[0]);
        valueTwo = objReader1.ReadToEnd().GetHashCode();

        if (f != Output[0] && valueOne == valueTwo)
        {
            outp.Add(f);
        }
        if (valueOne != valueTwo)
        {
            Trash.Add(f);
        }

    }
    outp.Add(Output[0]);
    //output of the list
    if (outp.Count > 1)
    {
        foreach (string fi in outp)
        {
            Console.WriteLine(fi);
        }
        outp.Clear();
        Console.WriteLine();
    }
    //recursive
    if (Trash.Count >= 1)
        CompareFilesRec(Trash);
    Output.Clear();
    Trash.Clear();
    outp.Clear();
}

代码:

Code:

using System;
using System.Collections.Generic;
using System.IO;

namespace ConsoleApplication3
{
    class Comparer
    {
        static void Main(string[] args)
        {
            if (args.Length != 0)//перевірка чи є хоча б 1 аргумент
            {
                if (args.Length == 1)//перевірка чи є тільки 1 аргумент
                {
                    List<string> ListOfFiles = new List<string>();//список для файлів

                    ListOfFiles.AddRange(LookIn(args[0]));//отримуєм список файлів
                    Console.WriteLine("Found {0} files", ListOfFiles.Count);//вивести скільки файлів знайдено

                    if(ListOfFiles.Count > 0)
                       CompareFilesRec(ListOfFiles);
                }
                else Console.WriteLine("So many arguments... o_O");//повідомлення, якщо аргументів забагато
            }
            else Console.WriteLine("Write some stuff here!");//повідомлення якщо немає аргументів
        }

        static bool CheckFileLessThan2Gb(string file)//функція перевірки чи файл менший за 2Гб.
        {
            FileInfo someFileInfo = new FileInfo(file);//берем інфу про файл в змінну someFileInfo
            if (someFileInfo.Length <= 2147483648)//перевірка
                return true;//якщо підходить
            else return false;//як не підходить 
        }

        static List<string> LookIn(string path)
        {
            /*Ініціалізую і об*являю два списки:для файлів і папок*/
            List<string> files = new List<string>();
            List<string> dirs = new List<string>();
            
            /*Шукаєм всі доступні файли*/
            try
            {
                /*Добавляю знайдені папки і файли в список*/
                files.AddRange(Directory.GetFiles(path));
                dirs.AddRange(Directory.GetDirectories(path));
                
                for (int i = 0; i <= files.Count; i++)
                {
                    if (!CheckFileLessThan2Gb(files[i]))
                        files.RemoveAt(i);
                    try
                    {
                        string FileValue = null;
                        StreamReader Value = new StreamReader(files[i]);
                        FileValue = Value.ReadToEnd();
                        if (FileValue.Length == 0)
                            files.RemoveAt(i);
                        Value.Dispose();
                        Value.Close();
                    }
                    catch (System.IO.IOException) { }
                }
            }
            catch (UnauthorizedAccessException) { }
            catch (DirectoryNotFoundException) { }
            catch (ArgumentOutOfRangeException) { }

            
            /*"Заглядаєм" за файлами в кожну директорію...*/
            foreach (string dir in dirs)
            {
                files.AddRange(LookIn(dir));//...і додаєм до списку
            }
            return files;//повертаєм повний список знайдених файлів
        }

        static void CompareFilesRec(List<string> array)
        {
            List<string> Trash = new List<string>();
            List<string> Output = new List<string>();
            List<KeyValuePair<long, string>> yeah = new List<KeyValuePair<long, string>>();
            List<string> outp = new List<string>();

            for (int j = 0; j <= array.Count - 1; j++)
            {
                FileInfo fii = new FileInfo(array[j]);
                yeah.Add(new KeyValuePair<long, string>(fii.Length, array[j]));
            }

            foreach (var el in yeah)
            {
                if (!Output.Contains(el.Value) && !Trash.Contains(el.Value))
                {
                    foreach (var ele in yeah)
                    {
                        if (el.Key == ele.Key && !Output.Contains(ele.Value))
                        {
                            Output.Add(ele.Value);
                        }
                        else
                        {
                            if (!Trash.Contains(ele.Value))
                            {
                                Trash.Add(ele.Value);
                            }
                        }
                    }
                }
            }

            foreach (string f in Output)
            {
                int valueOne = 0, valueTwo = 0;
                StreamReader objReader = new StreamReader(f);
                valueOne = objReader.ReadToEnd().GetHashCode();

                StreamReader objReader1 = new StreamReader(Output[0]);
                valueTwo = objReader1.ReadToEnd().GetHashCode();

                if (f != Output[0] && valueOne == valueTwo)
                {
                    outp.Add(f);
                }
                if (valueOne != valueTwo)
                {
                    Trash.Add(f);
                }

            }
            outp.Add(Output[0]);
            if (outp.Count > 1)
            {
                foreach (string fi in outp)
                {
                    Console.WriteLine(fi);
                }
                outp.Clear();
                Console.WriteLine();
            }

            if (Trash.Count >= 1)
                CompareFilesRec(Trash);
            Output.Clear();
            Trash.Clear();
            outp.Clear();
        }
    }
}

推荐答案

如果将CompareFilesRec分为两部分，一个用于查找候选对象，另一个用于比较.

创建字典< long，list>< string>>.用于保持文件大小，文件路径.然后，您只需要在第二次运行中比较相同文件大小的文件，因此，除非您有100万个相同文件的文件，否则CompareFilesRec中出现内存异常的可能性很小.

您希望哈希码足以判断两个文件是否相同.出于安全原因，我将比较文件的内容(无论如何，您都将文件带入内存以创建哈希码).我可能还会将它们读取为字节[]而不是字符串.

if you divide CompareFilesRec in two sections, one for finding candidates and one for comparing.

Create a dictionary<long,list><string>> for keeping file size, filepath. Then you only need to compare the files of the same file size in the second run, so unless you have 1 million files of the same file, that chance of memory exception in your CompareFilesRec is small.

You expect the hashcode to be enough to judge if two files are the same. For safety reasons I would compare the contents of the files (you are bringing the files into memory anyway to make a hashcode). I would probably also read them as byte[] instead of strings.

您可以使用另一个

You could use another thread[^] to search and compare files that are of the same file size, and also show a progress.

您可以通过检查文件是否为内容比较之前大小相同.

如果文件大小相等，则:分配两个内存块，然后重用它们，而不是一次读取内存中的整个文件.

如果您重新使用分配的内存，则内存不足"异常应该不会有问题.

You could probably make it faster by checking if files are the same size before comparing contents.

If files are of equal size, then: instead of reading the entire files in memory at once, allocate two blocks of memory, and reuse those.

If you reuse the allocated memory, you should not have problems with "out of memory" exceptions.

这篇关于未处理的异常:OutOfMemoryException或如何优化应用程序?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！