如何在大量文件处理的情况下使用任务并行 [英] How to use task parallelism in case of large number of files handling
问题描述
嗨朋友们，
我有一个函数从文件系统读取文件，更新它并再次回写。我必须为大量文件执行此任务。这些文件名条目位于DataTable中，我在此表上创建了一个DataView。
当前顺序函数如下所示，这很费时间。我需要提高它的性能。我可以在文件处理中使用任务并行吗？我想
喜欢用Parallel.For或Parallel.ForEach替换for循环
Hi Friends,
I have a function which reads a file from file system, updates it and writes back again. I have to do this task for large number of files.These file name entries are in the DataTable and i have created a DataView on this table.
The present sequential function is as like below, which is time consuming. I need to improve performance of it. Can i use task parallelism in file handling? I would
like to replace the for loop with Parallel.For or Parallel.ForEach
private void _RegenerateSerialNumber(List<int> i32IndexList)
{
for (int i32RowCount = 0; i32RowCount < SelectedMeasurementData.MeasurementData.Rows.Count; i32RowCount++)
{
SelectedMeasurementData.MeasurementData.Rows[i32RowCount][_strSerialNoCol] = i32RowCount + 1;
//SelectedMeasurementData.MeasurementData is member of the class of type DataTable
}
DataView dvResetOverlayShapeRows = new DataView(_dtOverlayData);
dvResetOverlayShapeRows.RowFilter = _strMeasurementType + " = '" + SelectedMeasurementType.measurementType.ToString() + "'";
//_dtOverlayData is the member of the class of type DataTable
for (int i32RowCount = 0; i32RowCount < dvResetOverlayShapeRows.Count; i32RowCount++) //I want to replace this for loop with Parallel.For or ForEach
{
int i32NewID = i32RowCount + 1;
dvResetOverlayShapeRows[i32RowCount][_strOverlayShapeID] = i32NewID;
///*****************************************
///For Image Overlay
///*****************************************
//If the data is present in the memory, then update it in memory
if (dvResetOverlayShapeRows[i32RowCount][_strImageOverlayData] != null && dvResetOverlayShapeRows[i32RowCount][_strImageOverlayData].GetType() !=
typeof(System.DBNull))
{
((IImageShapeBehaviour)dvResetOverlayShapeRows[i32RowCount][_strImageOverlayData]).UpdateOverlayShapeID(i32NewID);
}
else //else update it on cache file.
{
BinaryFormatter clsBinaryFormatter = new BinaryFormatter();
//Get the cache file path for image overlay
string strImageOverlayFileName = Convert.ToString(dvResetOverlayShapeRows[i32RowCount][_strImageDataFileName],
System.Globalization.CultureInfo.InvariantCulture);
string strImageOverlayFilePath = _strImageOverlayCacheFolderPath + @"\" + strImageOverlayFileName;
if (File.Exists(strImageOverlayFilePath))
{
using (FileStream _clsImageOverlayFs = new FileStream(strImageOverlayFilePath, FileMode.Open))
{
//Get the overlay object from the cache file
IImageShapeBehaviour _clsImageShapeBehaviour = (IImageShapeBehaviour)clsBinaryFormatter.Deserialize(_clsImageOverlayFs);
//Update this object with new overlay ID
_clsImageShapeBehaviour.UpdateOverlayShapeID(i32NewID);
_clsImageOverlayFs.Close();
};
//Serialize the updated object back to file.
using (FileStream _clsImageOverlayFs = new FileStream(strImageOverlayFilePath, FileMode.Create))
{
clsBinaryFormatter.Serialize(_clsImageOverlayFs, _clsImageShapeBehaviour);
_clsImageOverlayFs.Close();
}
}
}
}
}
可以任意一个让我知道如何在文件处理中使用Parallel.Invoke或Parallel.For或Parallel.ForEach而不是顺序。我已经尝试了以下所有三种结构
Can any one let me know how to use Parallel.Invoke or Parallel.For or Parallel.ForEach instead of sequential for in file handling. I have tried all three constructs as follows
try
{
DataView dvResetOverlayShapeRows = new DataView(_dtOverlayData);
dvResetOverlayShapeRows.RowFilter = _strMeasurementType + " = '" + SelectedMeasurementType.measurementType.ToString() + "'";
int count1 = dvResetOverlayShapeRows.Count;
//for (int i32RowCount = 0; i32RowCount < _dtOverlayData.Rows.Count; i32RowCount++)
Parallel.For(0, count1,
(i32RowCount, loopState) =>
{
try
{
lock (lockObject) //dvResetOverlayShapeRows)
{
int i32NewID = i32RowCount + 1;
dvResetOverlayShapeRows[i32RowCount][_strOverlayShapeID] = i32NewID;
//dvResetOverlayShapeRows[i32RowCount][_strOverlayShapeID] = i32NewID;
///*****************************************
///For Image Overlay
///*****************************************
//If the data is present in the memory, then update it on memory, else update it on cache file.
if (dvResetOverlayShapeRows[i32RowCount][_strImageOverlayData] != null && dvResetOverlayShapeRows[i32RowCount]
[_strImageOverlayData].GetType() != typeof(System.DBNull))
//if (conbgdvResetOverlayShapeRows.ElementAt(0)[i32RowCount][_strImageOverlayData] != null && conbgdvResetOverlayShapeRows.ElementAt
(0)[i32RowCount][_strImageOverlayData].GetType() != typeof(System.DBNull))
{
((IImageShapeBehaviour)(dvResetOverlayShapeRows[i32RowCount][_strImageOverlayData])).UpdateOverlayShapeID(i32NewID);
}
else
{
BinaryFormatter clsBinaryFormatter = new BinaryFormatter();
IImageShapeBehaviour _clsImageShapeBehaviour = null;
//Get the cache file path for image overlay
string strImageOverlayFileName = Convert.ToString(dvResetOverlayShapeRows[i32RowCount][_strImageDataFileName],
System.Globalization.CultureInfo.InvariantCulture);
string strImageOverlayFilePath = _strImageOverlayCacheFolderPath + @"\" + strImageOverlayFileName;
if (File.Exists(strImageOverlayFilePath))
{
using (FileStream _clsImageOverlayFs = new FileStream(strImageOverlayFilePath, FileMode.Open))
{
//Get the overlay object from the cache file
_clsImageShapeBehaviour = (IImageShapeBehaviour)clsBinaryFormatter.Deserialize(_clsImageOverlayFs);
//Update this object with new overlay ID
_clsImageShapeBehaviour.UpdateOverlayShapeID(i32NewID);
_clsImageOverlayFs.Close();
};
//Serialize the updated object back to file.
using (FileStream _clsImageOverlayFs = new FileStream(strImageOverlayFilePath, FileMode.Create))
{
clsBinaryFormatter.Serialize(_clsImageOverlayFs, _clsImageShapeBehaviour);
_clsImageOverlayFs.Close();
}
}
}
///*****************************************
///For Profile Or Histogram Overlay
///*****************************************
}
}
catch (Exception ex)
{
exceptions.Enqueue(ex);
}
});
if (exceptions.Count != 0)
throw new AggregateException(exceptions);
}
catch (AggregateException ex)
{
foreach (Exception inner in ex.InnerExceptions)
{
//Console.WriteLine(inner.Message);
}
}
}
我尝试了2到3种变化，有时它会挂锁（如锁定，自旋锁，互斥锁，Parallel.for与不同的委托等），如果锁定评论赛车发生。
以下是与不同的代表（问题是，它是更新localTable而不是相应的DataView（dvResetOverlayShapeRows）和相关的DataTable（_dtOverlayData）。
I tried 2 to 3 variations still some times it hangs at lock(like lock, spinlock, mutex, Parallel.for with different delegates etc ), if lock commented racing happens.
The below is with different delegates(problem with this is, it is updating localTable but not the corresponding DataView(dvResetOverlayShapeRows) and associated DataTable(_dtOverlayData).
try
{
DataView dvResetOverlayShapeRows = new DataView(_dtOverlayData);
dvResetOverlayShapeRows.RowFilter = _strMeasurementType + " = '" + SelectedMeasurementType.measurementType.ToString() + "'";
int count1 = dvResetOverlayShapeRows.Count;
//for (int i32RowCount = 0; i32RowCount < _dtOverlayData.Rows.Count; i32RowCount++)
Parallel.For(0, count1,
() =>
{
//lock (dvResetOverlayShapeRows)
{
//Create a temp table per thread that has the same schema as the main table
return (dvResetOverlayShapeRows.ToTable().Clone());
}
},
(i32RowCount, loopState, localTable) =>
{
try
{
//lock (localTable)
{
int i32NewID = i32RowCount + 1;
localTable.Rows[i32RowCount][_strOverlayShapeID] = i32NewID;
//dvResetOverlayShapeRows[i32RowCount][_strOverlayShapeID] = i32NewID;
///*****************************************
///For Image Overlay
///*****************************************
//If the data is present in the memory, then update it on memory, else update it on cache file.
if (localTable.Rows[i32RowCount][_strImageOverlayData] != null && localTable.Rows[i32RowCount][_strImageOverlayData].GetType() !=
typeof(System.DBNull))
//if (conbgdvResetOverlayShapeRows.ElementAt(0)[i32RowCount][_strImageOverlayData] != null && conbgdvResetOverlayShapeRows.ElementAt
(0)[i32RowCount][_strImageOverlayData].GetType() != typeof(System.DBNull))
{
((IImageShapeBehaviour)(localTable.Rows[i32RowCount][_strImageOverlayData])).UpdateOverlayShapeID(i32NewID);
}
else
{
BinaryFormatter clsBinaryFormatter = new BinaryFormatter();
IImageShapeBehaviour _clsImageShapeBehaviour = null;
//Get the cache file path for image overlay
string strImageOverlayFileName = Convert.ToString(localTable.Rows[i32RowCount][_strImageDataFileName],
System.Globalization.CultureInfo.InvariantCulture);
string strImageOverlayFilePath = _strImageOverlayCacheFolderPath + @"\" + strImageOverlayFileName;
if (File.Exists(strImageOverlayFilePath))
{
using (FileStream _clsImageOverlayFs = new FileStream(strImageOverlayFilePath, FileMode.Open))
{
//Get the overlay object from the cache file
_clsImageShapeBehaviour = (IImageShapeBehaviour)clsBinaryFormatter.Deserialize(_clsImageOverlayFs);
//Update this object with new overlay ID
_clsImageShapeBehaviour.UpdateOverlayShapeID(i32NewID);
_clsImageOverlayFs.Close();
};
//Serialize the updated object back to file.
using (FileStream _clsImageOverlayFs = new FileStream(strImageOverlayFilePath, FileMode.Create))
{
clsBinaryFormatter.Serialize(_clsImageOverlayFs, _clsImageShapeBehaviour);
_clsImageOverlayFs.Close();
}
}
}
///*****************************************
///For Profile Or Histogram Overlay
///*****************************************
}
}
catch (Exception ex)
{
exceptions.Enqueue(ex);
}
return localTable;
},
(localTable) =>
{
//lock (localTable)
{
/* localTable.AcceptChanges();
//_dtOverlayData.DefaultView.RowFilter = _strMeasurementType + " = '" + SelectedMeasurementType.measurementType.ToString() + "'";
for (int i = 0; i < dvResetOverlayShapeRows.Count; i++)
{
dvResetOverlayShapeRows.Delete(i);
}
//_dtOverlayData.AcceptChanges();
//Merge in the thread local table to the master table
_dtOverlayData.Merge(localTable);
_dtOverlayData.AcceptChanges();*/
//dvResetOverlayShapeRows = localTable.AsDataView();
//_dtOverlayData.Dispose();
//_dtOverlayData.Merge(localTable);
/* }
});
if (exceptions.Count != 0)
throw new AggregateException(exceptions);
}
catch (AggregateException ex)
{
foreach (Exception inner in ex.InnerExceptions)
{
//Console.WriteLine(inner.Message);
}
}
}
但我的DataView（dvResetOverlayShapeRows）和相关的DataTable（_dtOverlayData）未正确更新在多任务处理的情况下。在正常情况下，他们正在适当地更新。我也尝试锁定并使用不同的委托和动作构造但不成功。任何人都可以告诉我问题在哪里？是否与锁定有关（我已尝试并发收集（包）而不是锁定，仍然没有用），委托或动作结果（因为我的DataView和DataTable没有得到更新）？如果有人帮我正确的代码，我会很棒的。我正在尝试数据并行。我应该改为任务并行吗？
But my DataView(dvResetOverlayShapeRows) and associated DataTable(_dtOverlayData) are not updated properly in case of multitasking. In normal case they are properly getting updated. I tried locking also and with different delegates and action constructs but not successful.Can any one let me know where the problem is? Is it to do with locking(i have tried concurrent collection(bag) instead of locking, still no use), delegate or action results(because my DataView and DataTable are not getting updated)? I will be great if some one helps me with correct code. I am trying the data parallelism. Should i go for task parallelism instead?
推荐答案
Parallel.For以下用法将解决问题。
Parallel.For usage like following will solve the issue.
ThreadPool.QueueUserWorkItem(new WaitCallback(delegate(object arg)
{
Parallel.For(0, currentScene.Textures.Count, delegate(int i)
{
// The rest of your code .....
}
}));
Enumerable.Range(0, currentScene.Textures.Count).Select(i =>
new Task(() => {
TextureObject texture = (currentScene.Textures[i]);
MainWindow.Instance.Dispatcher.BeginInvoke(new Action(()
=> MainWindow.Instance.StatusBarText.Text = "Loading Texture " + i
+ " - " + texture.Name ), null);
LoadTexture(texture);
});
).Run(x => x.Start());
这篇关于如何在大量文件处理的情况下使用任务并行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
