使用线程数组的多个线程... [英] Multiple threads using thread array...

问题描述

大家好，
在我的应用程序中，我需要收集文件夹中所有文件的信息并将其发送到数据库.由于要对整个磁盘进行此操作，因此这是一个非常耗时的过程.我要做的是为文件夹中的每个文件创建一个单独的线程，并将每个文件同时写入db.我不知道如何启动它.如果有人可以帮助我为将同时运行的所有文件创建线程数组.以下是我需要执行此操作的代码段.

hi everyone,
In my application I need to collect the information of all the files in folder and send them to database.Since this is to be done for whole disk, this is a very time taking process. What I want to do is to create a seperate thread for each file in a folder and write each file simultaneously to db. I am not getting any idea of how to start it. If anyone could please help me in creating an array of threads for all files which will run simultaneously. Below is my code snippet where I need to do this.

private static void collectFileInfo(String dir)
        {
            try
            {
                String[] files = Directory.GetFiles(dir);
                DirectoryInfo dir1 = new DirectoryInfo(dir);
                string chkFlPath = Convert.ToString(dir1.FullName);
                Int32 ParentID = GetParentID(dir1.Name.ToString(), chkFlPath);
                if (files.Length > 0)
                {
           
                    for (Int32 i = 0; i < files.Length; i++)
                    {
                        FileInfo File1 = new FileInfo(files[i]);
                        double FileLength = File1.Length;// File size
                        string strExtn = System.IO.Path.GetExtension(File1.Name.ToString());
                        String fileName = File1.Name;
                        String filePath = File1.FullName;
                        string ParentName = dir.ToString();
                        String fileSize = Convert.ToString(Convert.ToInt32(File1.Length) / 1024);
                        String fileExtension = File1.Extension;
                        DateTime fileCreated = Convert.ToDateTime(File1.CreationTime);
                        DateTime FileModified = Convert.ToDateTime(File1.LastWriteTime);
                        string md5hash = GetMD5Hash(filePath.ToString());
                        string sh1hash = GetSHA1Hash(filePath.ToString());
                        DateTime lastAccessedDate = Convert.ToDateTime(GetLastAccessedDate(filePath.ToString()));
                        Thread t = new Thread(delegate()
                            {
                                WriteToTable(ParentID, fileName, filePath, ParentName, true, dir1.FullName, fileExtension, fileSize, fileCreated, FileModified, md5hash, sh1hash, lastAccessedDate);
                            }
                       );
                        t.Start();
                    }
                }
            }
            catch (Exception ex)
            {
            }
        } 

谢谢&问候
Anurag

Thanks & Regards
Anurag

推荐答案

对于如此大规模的操作，我以前使用SQLite的顺序是我在执行类似SQLiteDB.Query(``update table'');
在所有这些更新表之后，我做了Commit();
结果更快了.
我不知道WriteToTable中使用的API是什么，但是如果您的数据库可以做到，我建议您使用所有这些更新构建一个sql脚本，并将其推送到db中.

至于线程我使用:

For such large scale operations i used to use SQLite in an order that i was doing something like SQLiteDB.Query(''update table'');
and after all those update table i did Commit();
And it ended up way faster.
I cannot know what is the API used inside WriteToTable but if your DB can do it i suggest you build a sql script with all those updates and shove it inside the db.

As for threading i use:

private readonly object lockObject = new object();
volatile int jobCount;
private bool Stopped;

private void btnGo_Click(object sender, EventArgs e)
{
    btnGo.Enabled = false;
    ThreadPool.QueueUserWorkItem(new WaitCallback(Start));
}

private void Start(object obj)
{
    jobCount = 0;
    Stopped = false;
    for (int i = 0; i < Environment.ProcessorCount; i++)
    {//this for could be changed to fit your needs ofc
        ThreadPool.QueueUserWorkItem(new WaitCallback(execute),null);//you could pass a FileInfo here for example instead of null
        jobCount++;
    }
    lock (lockObject)
    {
        while (jobCount > 0)
        {
            Monitor.Wait(lockObject);
        }
    }
    ThreadStart finisher = new ThreadStart(Finish);
    finisher.Invoke();
}

private void execute(object obj)
{
    //FileInfo file = obj as FileInfo;
    while (!Stopped)
    {
        Thread.Yield();
        //execute code here
    }
    Decreace();
}

private void Decreace()
{
    lock (lockObject)
    {
        jobCount--;
        Monitor.Pulse(lockObject);
    }
}

private void Finish()
{
    Thread.Yield();
    btnGo.Enabled = true;
    //other code goes here, you can also skip the next command if you like but i prefer it to release ram after large operations even if it slows down my process alot. But in your case it would make your process seem alot faster though so its your decision.
    GC.Collect();
    Application.DoEvents();
}

忘记声明和初始化Stopped变量:P
Edit2:将代码更改为假定不在表单内(例如，finisher.Invoke())

forgot to declare and initialize Stopped variable :P
Changed Code to assume not within a form (i.e finisher.Invoke())

实际上，您的代码段看上去并不那么糟糕.实际的问题是什么?
为了提高速度，您可以将更多的逻辑外包给线程，并在"For"循环之外放置更多代码.
顺便说一句，创建与文件一样多的线程没有意义.您应该将线程数限制为系统中的CPU/核心数.

Actually your code snippet doesn''t look that bad. What is the actual problem with it?
To improve the speed you could outsource more of your logic to the thread and put more code outside the "For" loop.
By the way, it doesn''t make sense to create as much threads as files. You should limit the number of threads to the number of CPU''s/Cores in the system.

这篇关于使用线程数组的多个线程...的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！