C 2D array size

GreenGrass · 20 Jul 2006 at 20:48

Im writting a program in Visual Studio 2005 in C and trying to use a 2D array [3072][3240] but it keeps crashing, it only works when smaller arrays are used ie [400][400]..which are no use for what I need. Eventually ill need a [58192][58320] array is there some sort of limit to there size? a setting somewhere?

how would you get around this prob?

Thanks Tour

triggerthat · 20 Jul 2006 at 20:58

I've never done C before - however, does the array require a type? If so, create it as a Long.

scorza · 20 Jul 2006 at 21:19

Its absolutely ages since I did any C, and it would help to see your code, but it sounds to me like when you declare the bigger variable its overlapping with something else causing your program to crash.

You'll probably need to allocate enough memory on the heap for the variable with the malloc command or similar.

xyphic · 20 Jul 2006 at 21:36

What type is it you're trying to declare an array of?

The problem could well be that in C, a multidimensional array is always a contiguous (i.e. unbroken) block of memory. So you need a contiguous block of memory that can store 9953280 of your type in a row. If, for instance, it was a 32-bit integer, you'd need 39813120 bytes or 38 megabytes.

It may be that at compile time this is okay, but at run time your operating system can't allocate that kind of memory. When you allocate an array, you've basically got a pointer to the first element, and if the operating system can't allocate that memory, it might return the null pointer. When you try to access the array, you're dereferencing the null pointer which will cause the program to crash.

So with your really big array you end up with 3393757440 elements, which would be 12946 megabytes, or about 12 gigabytes of contiguous memory with the integer type. You're going to struggle however you choose to represent that.

Now's probably the right time to start thinking about what it is you're trying to achieve, and whether an array is the right way to go about it. Is it possible for you to give some idea of what you're trying to do?

Lithium UK · 20 Jul 2006 at 23:13

Where are you allocating the memory? Stack or Heap?

Given the numbers you have given it sounds like you are using the stack which if I remember correctly is only 1mb.

Instead of declaring like this...

Code:

char example[3072][3240];

Try instead...

Code:

char *example new char[3072*3240];
// do stuff
delete [] example; //must delete when finshed with buffer

GreenGrass · 21 Jul 2006 at 10:17

Basically what im tryin to do is take a 168 generator polynomial from a file not sure how familar you are with them, and generate a generator matrix, which is a shift of the generator polynomial:
i.e:

Code:

        |[gen poly] 0 0 0 0 | 
        |0 [gen poly] 0 0 0 |
3072    |0 0 [gen poly] 0 0 |
        |                   |
        |     etc..         |
        |0 0 0 0 [gen poly] |
               <- 3240 ->

The code so far is:

Code:

/*This produces X by multiplying K*G where G is the generator matrix*/
#include <stdio.h>
#include <stdlib.h>

FILE *genpoly;
FILE *genmatrix;

void main()
{
	/*****************************************************************/
	//Reads generator poly from file
	genpoly = fopen("genpoly.txt","r");
	if (genpoly == 0)
	{ 
		printf("An error occurred while accessing genpoly.txt.\n");
	exit(1);
	}

	int i, gen[169];

	for(i=0; i<169; i++)
	{
		//printf("reading file line %d\n", i);
		fscanf(genpoly, "%d", &gen[i]);
	}
	fscanf(genpoly,"\n");
	fclose(genpoly);

	printf("Generator polynomial:\ng(x) = ");
	for (i = 0; i<169; i++)
	{
	   printf("%d", gen[i]);
	  
	}
	
	/********************************************************************/
	//Creates generator matrix

	int n;
	int k;
	int matrix[3072][3240];		//[k][n] matrix
	int j;
	genmatrix = fopen("genmatrix.txt","w");
	if (genmatrix == 0)
	{ 
		printf("An error occurred while accessing genmatrix.txt.\n");
		exit(1);
	}
	//Calclates matrix
	//Not yet written

	//Prints matrix to file
	for(i=0; i<3072; i++)
	{
		for(j=0; j<3240; j++)
		{
			
			fprintf(genmatrix," %c", &matrix[i][j]);
			fflush(genmatrix);
		}
		
		fscanf(genmatrix, "\n");
	}
	fclose(genmatrix);
	printf("\n\nDone!\n\n");
}

Ive not written how to generate the matrix yet as it doesnt like the array size, eventually when im done i can multiply a 3072 size input againt the matrix and return a 3240 output.

GreenGrass · 21 Jul 2006 at 10:27

xyphic said:
What type is it you're trying to declare an array of?

The problem could well be that in C, a multidimensional array is always a contiguous (i.e. unbroken) block of memory. So you need a contiguous block of memory that can store 9953280 of your type in a row. If, for instance, it was a 32-bit integer, you'd need 39813120 bytes or 38 megabytes.

It may be that at compile time this is okay, but at run time your operating system can't allocate that kind of memory. When you allocate an array, you've basically got a pointer to the first element, and if the operating system can't allocate that memory, it might return the null pointer. When you try to access the array, you're dereferencing the null pointer which will cause the program to crash.

So with your really big array you end up with 3393757440 elements, which would be 12946 megabytes, or about 12 gigabytes of contiguous memory with the integer type. You're going to struggle however you choose to represent that.

Now's probably the right time to start thinking about what it is you're trying to achieve, and whether an array is the right way to go about it. Is it possible for you to give some idea of what you're trying to do?

can I ask how you calculated the numbers? i was curious to the actual size

of these arrays.

Una · 21 Jul 2006 at 10:58

Code:

char **foo;
	foo = (char**)malloc(SIZE1 * sizeof(char*)); // Allocate pointers to rows.
	foo[0] = (char*)malloc((SIZE1 * SIZE2)*sizeof(char)); // Then allocate rows and set pointers to them.

#define SIZE1 3072
#define SIZE2 3240

Of course..

This will dynamically allocate a 2d array on the heap. However as xyphic says it sounds like you are going about your problem the wrong way.

To find the size of a type you use sizeof() suprisingly.

Lithium: thats c++ he's doing C.... so no new/delete operators.

Inquisitor · 21 Jul 2006 at 12:34

Tourmai said:
can I ask how you calculated the numbers? i was curious to the actual size of these arrays.

58,192 * 58,320 = 3,393,757,440 array elements
32b (i.e. a 32-bit int) = 4B
3,393,757,440 * 4 = 13,575,029,760B = ~12.6GB

GreenGrass · 21 Jul 2006 at 12:49

Una said:
#define SIZE1 3072
#define SIZE2 3240

Of course..

This will dynamically allocate a 2d array on the heap. However as xyphic says it sounds like you are going about your problem the wrong way.

To find the size of a type you use sizeof() suprisingly.

Lithium: thats c++ he's doing C.... so no new/delete operators.

ah ok thanks for that, to be honest im not too fussy about using C or C++ which ever one is best with dealing with this, i'd learn a bit more about c++ if it were better at solving me problem.

seems a bit more complicated than i first thought.
How would you go about solving this problem if you had such a large matix / arrays?
The only reason Im using this technique is because i thought i'd see how fast the program would be, and would be a nice implementation.

Inquisitor said:
58,192 * 58,320 = 3,393,757,440 array elements
32b (i.e. a 32-bit int) = 4B
3,393,757,440 * 4 = 13,575,029,760B = ~12.6GB

cheers for that

Lithium UK · 21 Jul 2006 at 15:04

Una said:
Lithium: thats c++ he's doing C.... so no new/delete operators.

I missed that here is the c equivalent.

Code:

int *matrix = malloc(sizeof(int) * 3072*3240);

//do stuff

free(matrix);

ByteJuggler · 22 Jul 2006 at 00:36

Tourmai said:
ah ok thanks for that, to be honest im not too fussy about using C or C++ which ever one is best with dealing with this, i'd learn a bit more about c++ if it were better at solving me problem. seems a bit more complicated than i first thought.
How would you go about solving this problem if you had such a large matix / arrays?
The only reason Im using this technique is because i thought i'd see how fast the program would be, and would be a nice implementation.

cheers for that

Personally, I'd pick C++ almost every time, more ways to abstract things than in C. However if you don't know it at all, this might not be the right time to start learning it... Anyway, you need some sort of sparse array (http://en.wikipedia.org/wiki/Sparse_array and http://en.wikipedia.org/wiki/Sparse_matrix) type structure that behaves like an array but does not have the literal space requirement of the mxn matrix if most entries are blank/zero/null.