绕过.NET中2 GB的收集限制 [英] Getting around the 2 GB collection limit in .NET
问题描述
来自该问题 ，我认为我可以通过使用以下模式创建BigList数据类型来绕过2 GB的集合大小限制(顺便说一句，如果您想尝试使用此限制，默认情况下，此限制似乎是x86应用程序所强加的):
using Microsoft.Win32;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace RegistryHawk
{
class Program
{
struct RegistryPath
{
public RegistryView View;
public string Path;
public bool IsKey;
public RegistryValueKind ValueKind;
public string ValueName;
public object Value;
public int HashValue;
}
public class BigList<T>
{
object listLock = new object();
List<List<T>> Items = new List<List<T>>();
int PageSize = 1000000; // Tweak this to be the maximum size you can grow each individual list before reaching the 2 GB size limit of .NET.
public ulong Count = 0;
int listCount = 0;
public BigList()
{
Items.Add(new List<T>());
}
public void Add(T item)
{
lock (listLock)
{
if (Items[listCount].Count == PageSize)
{
Items.Add(new List<T>());
listCount++;
}
Items[listCount].Add(item);
Count++;
}
}
}
static void Main(string[] args)
{
BigList<RegistryPath> snapshotOne = new BigList<RegistryPath>();
WalkTheRegistryAndPopulateTheSnapshot(snapshotOne);
BigList<RegistryPath> snapshotTwo = new BigList<RegistryPath>();
WalkTheRegistryAndPopulateTheSnapshot(snapshotTwo);
}
private static void WalkTheRegistryAndPopulateTheSnapshot(BigList<RegistryPath> snapshot)
{
List<ManualResetEvent> handles = new List<ManualResetEvent>();
foreach (RegistryHive hive in Enum.GetValues(typeof(RegistryHive)))
{
foreach (RegistryView view in Enum.GetValues(typeof(RegistryView)).Cast<RegistryView>().ToList().Where(x => x != RegistryView.Default))
{
ManualResetEvent manualResetEvent = new ManualResetEvent(false);
handles.Add(manualResetEvent);
new Thread(() =>
{
WalkKey(snapshot, view, RegistryKey.OpenBaseKey(hive, view));
manualResetEvent.Set();
}).Start();
}
}
ManualResetEvent.WaitAll(handles.ToArray());
}
private static void WalkKey(BigList<RegistryPath> snapshot, RegistryView view, RegistryKey key)
{
RegistryPath path = new RegistryPath { View = view, Path = key.Name, HashValue = (view.GetHashCode() ^ key.Name.GetHashCode()).GetHashCode() };
snapshot.Add(path);
string[] valueNames = null;
try
{
valueNames = key.GetValueNames();
}
catch { }
if (valueNames != null)
{
foreach (string valueName in valueNames)
{
RegistryValueKind valueKind = RegistryValueKind.Unknown;
try
{
valueKind = key.GetValueKind(valueName);
}
catch { }
object value = key.GetValue(valueName);
RegistryPath pathForValue = new RegistryPath { View = view, Path = key.Name, ValueKind = valueKind, ValueName = valueName, Value = value, HashValue = (view.GetHashCode() ^ key.Name.GetHashCode() ^ valueKind.GetHashCode() ^ valueName.GetHashCode()).GetHashCode() };
snapshot.Add(pathForValue);
}
}
string[] subKeyNames = null;
try
{
subKeyNames = key.GetSubKeyNames();
}
catch { }
if (subKeyNames != null)
{
foreach (string subKeyName in subKeyNames)
{
try
{
WalkKey(snapshot, view, key.OpenSubKey(subKeyName));
}
catch { }
}
}
}
}
}
但是，CLR仍会触发System.OutOfMemory异常.它并没有抛出任何地方，但是我看到程序执行完全在2 GB的RAM处停止，并且当我在Visual Studio中冻结代码时，它表明当我尝试查看其中的变量状态时抛出了内存不足异常应用程序的任何线程.在第一次调用WalkTheRegistryAndPopulateTheSnapshot(snapshotOne);时永远不会发生这种情况，但是当第二次调用WalkTheRegistryAndPopulateTheSnapshot(snapshotTwo);进行时，它最终导致程序执行停止，而我的集合中的RAM总使用量约为2 GB.整个代码都已发布，因此，如果您拥有一个强大的注册表，则可能会看到它是在x86控制台应用程序上生成的.是我在这里无法掌握的东西，还是这种模式不是绕过Stack上另一个问题似乎发挥作用的2 GB收集大小限制的有效手段?
我将扩大我的评论.如果要编写32位应用程序，则在处理大量数据时会遇到一些严重的内存限制.
要记住的最重要的事情是，该32位应用程序的绝对最大内存限制为2 ^ 32字节(4 GB).实际上，通常为2 GB，如果您有那么多内存并且应用程序支持大地址，则可能为3 GB.
.NET还施加2 GB的限制，该限制将任何单个对象的大小限制为不超过2 GB.很少会在32位程序中遇到此限制，这仅仅是因为，即使在内存大于2 GB的计算机上，也不大可能会有连续的2 GB内存块. /p>
.NET的64位版本中也存在2 GB的限制，除非您正在运行.NET 4.5并使用启用大对象的app.config设置.
至于为什么像BigList之类的东西存在于32位版本中，这是解决连续内存块的一种方法.例如，具有2.5亿个项目的List<int>需要一个千兆字节:一个连续的内存块，大小为1 GB.但是，如果使用BigList技巧(如您在代码中所做的那样)，则需要250个单独的内存块，大小为4 MB.您拥有250个4 MB的块的可能性要比单个1 GB的块的可能性大得多.
From this question, I thought I could get around the 2 GB collection size limit by creating a BigList datatype using the following pattern (and by the way, this limit seems to be imposed by default on x86 applications, if you are curious about trying it out):
using Microsoft.Win32;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace RegistryHawk
{
class Program
{
struct RegistryPath
{
public RegistryView View;
public string Path;
public bool IsKey;
public RegistryValueKind ValueKind;
public string ValueName;
public object Value;
public int HashValue;
}
public class BigList<T>
{
object listLock = new object();
List<List<T>> Items = new List<List<T>>();
int PageSize = 1000000; // Tweak this to be the maximum size you can grow each individual list before reaching the 2 GB size limit of .NET.
public ulong Count = 0;
int listCount = 0;
public BigList()
{
Items.Add(new List<T>());
}
public void Add(T item)
{
lock (listLock)
{
if (Items[listCount].Count == PageSize)
{
Items.Add(new List<T>());
listCount++;
}
Items[listCount].Add(item);
Count++;
}
}
}
static void Main(string[] args)
{
BigList<RegistryPath> snapshotOne = new BigList<RegistryPath>();
WalkTheRegistryAndPopulateTheSnapshot(snapshotOne);
BigList<RegistryPath> snapshotTwo = new BigList<RegistryPath>();
WalkTheRegistryAndPopulateTheSnapshot(snapshotTwo);
}
private static void WalkTheRegistryAndPopulateTheSnapshot(BigList<RegistryPath> snapshot)
{
List<ManualResetEvent> handles = new List<ManualResetEvent>();
foreach (RegistryHive hive in Enum.GetValues(typeof(RegistryHive)))
{
foreach (RegistryView view in Enum.GetValues(typeof(RegistryView)).Cast<RegistryView>().ToList().Where(x => x != RegistryView.Default))
{
ManualResetEvent manualResetEvent = new ManualResetEvent(false);
handles.Add(manualResetEvent);
new Thread(() =>
{
WalkKey(snapshot, view, RegistryKey.OpenBaseKey(hive, view));
manualResetEvent.Set();
}).Start();
}
}
ManualResetEvent.WaitAll(handles.ToArray());
}
private static void WalkKey(BigList<RegistryPath> snapshot, RegistryView view, RegistryKey key)
{
RegistryPath path = new RegistryPath { View = view, Path = key.Name, HashValue = (view.GetHashCode() ^ key.Name.GetHashCode()).GetHashCode() };
snapshot.Add(path);
string[] valueNames = null;
try
{
valueNames = key.GetValueNames();
}
catch { }
if (valueNames != null)
{
foreach (string valueName in valueNames)
{
RegistryValueKind valueKind = RegistryValueKind.Unknown;
try
{
valueKind = key.GetValueKind(valueName);
}
catch { }
object value = key.GetValue(valueName);
RegistryPath pathForValue = new RegistryPath { View = view, Path = key.Name, ValueKind = valueKind, ValueName = valueName, Value = value, HashValue = (view.GetHashCode() ^ key.Name.GetHashCode() ^ valueKind.GetHashCode() ^ valueName.GetHashCode()).GetHashCode() };
snapshot.Add(pathForValue);
}
}
string[] subKeyNames = null;
try
{
subKeyNames = key.GetSubKeyNames();
}
catch { }
if (subKeyNames != null)
{
foreach (string subKeyName in subKeyNames)
{
try
{
WalkKey(snapshot, view, key.OpenSubKey(subKeyName));
}
catch { }
}
}
}
}
}
However, CLR still triggers a System.OutOfMemory exception. It is not thrown anywhere, but I see program execution stop entirely at around 2 GB of RAM, and when I freeze my code in Visual Studio, it shows that an out of memory exception was thrown whenever I try to view the state of variables within any thread of the application. It never happens on the first call to WalkTheRegistryAndPopulateTheSnapshot(snapshotOne);, but when the second call to WalkTheRegistryAndPopulateTheSnapshot(snapshotTwo); proceeds, it ends up stopping program execution at around 2 GB of overall RAM usage in my collections. The entire code is posted, so if you have a beefy registry you can probably see it get generated on an x86 console application. Is there something that I failed to grasp here, or is this pattern not a valid means to get around the 2 GB collection size limit that the other question on Stack seems to play up to?
I'm going to expand on my comment. If you're writing a 32-bit app, you have some serious memory constraints when you're working with large amounts of data.
The most important thing to remember is that the 32-bit application is limited to an absolute maximum of 2^32 bytes (4 GB) of memory. In practice, it's usually 2 GB, or perhaps 3 GB if you have that much memory and the application is large address aware.
There's also the .NET imposed 2 GB limit, which limits the size of any single object to no more than 2 GB. It's rare that you'll encounter this limit in a 32-bit program, simply because, even on a machine that has more than 2 GB of memory, it's unlikely that there will be a contiguous chunk of memory that's 2 GB in size.
The 2 GB limit also exists in 64 bit versions of .NET, unless you're running .NET 4.5 and use the app.config setting that enables large objects.
As for why something like BigList exists in 32-bit versions, it's a way to get around requiring a contiguous block of memory. For example, a List<int> with 250 million items requires a gigabyte: a contiguous block of memory that's 1 GB in size. But if you use the BigList trick (as you did in your code), then you need 250 individual blocks of memory that are 4 MB in size. It's a whole lot more likely that you'll have 250 blocks of 4 MB than you will a single 1 GB block.
这篇关于绕过.NET中2 GB的收集限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
