OutOfMemoryException on declaration of Large Array

Since you can't create objects larger than 2GB you can try to use MemoryMappedFile to work with chunk of memory of the required size.


var data = MemoryMappedFile.CreateNew("big data", 12000L * 55000L);
var view = data.CreateViewAccessor();
var rnd = new Random();

for (var i = 0L; i < 12000L; ++i)
{
    for (var j = 0L; j < 55000L; ++j)
    {
        var input = rnd.NextDouble();
        view.Write<double>(i * 55000L + j, ref input);
    }
}


Providing that your total memory is sufficient, you can prevent Out of memory exceptions resulting from LOH fragmentation by creating a bunch of smaller arrays, and wrapping them in a single IList<T>, or some other indexed interface.

Here is a link which describes it:

BigArray<T>, getting around the 2GB array size limit

Credits: this post (C# chunked array).


Each double is 8 bytes, so you're trying to allocate a single array with just over 5GB. The CLR has a per-object limit of around 2GB IIRC, even for a 64-bit CLR. In other words, it's not the total amount of memory available that's the problem (although obviously you'll have issues if you don't have enough memory), but the per-object size.

I suggest you split it into smaller arrays, perhaps behind a facade of some description. I don't believe there's any way to workaround that limit for a single array.

EDIT: You could go for an array of arrays - aka a jagged array:

double[][] array = new double[12000][];
for (int i = 0; i < array.Length; i++)
{
    array[i] = new double[55000];
}

Would that be acceptable to you?

(You can't use a rectangular array (double[,]) as that would have the same per-object size problem.)

Tags:

C#