Multiplying very large matrices

[PaulM1985]

Hi

I have been trying to complete a project which involves multiplying two very large matrices together. (3000x5000)

Since I am not able to store two matrices of that size and a third of 3000x3000 in my RAM memory, I have created csv files for each matrix.

I have created functions to return a row vector of floats from the first matrix and a column vector from the second to be multiplied together and create a value for the third (solution) matrix.

I admit that this will take along time to do (as there are going to be 5000 multiplications per value), but I seem to be only completing one value a second. Does anybody know of any quicker ways to multiply two matrices?

Paul

[nvteighen]

Hi

I have been trying to complete a project which involves multiplying two very large matrices together. (3000x5000)

Since I am not able to store two matrices of that size and a third of 3000x3000 in my RAM memory, I have created csv files for each matrix.

A question: is this just a part of the project or the project itself? If it's just part, consider seriously to use a library that does this for you... otherwise, you'll be losing efforts in the wrong place...

But anyway, why should a 3000x5000 matrix not fit in your RAM? Unless you're storing it in the stack, which is limited, you could use dynamic memory...

What language are you using for this?

[Npl]

Hi

I have been trying to complete a project which involves multiplying two very large matrices together. (3000x5000)

Since I am not able to store two matrices of that size and a third of 3000x3000 in my RAM memory, I have created csv files for each matrix.

I have created functions to return a row vector of floats from the first matrix and a column vector from the second to be multiplied together and create a value for the third (solution) matrix.

I admit that this will take along time to do (as there are going to be 5000 multiplications per value), but I seem to be only completing one value a second. Does anybody know of any quicker ways to multiply two matrices?

Paul

I`d guess that the File-IO slows everything down. I dont know the csv-Format, but can you read whole rows/columns in chunks, or do you need alot of small reads?

Also, if you have enough memory, then keep the source Matrizes in RAM and write out the result-Matrix.

[PaulM1985]

Hi

I am using Java. Matrix multiplication is just one part of it. I have written functions for matrix transpose, singular value decomposition and pseudo inverse. My project is for mathematical modeling. I have tried storing the info in RAM before and I have had errors in Java saying that there was not enough memory.

I originally carried this out on a smaller scale in Matlab, since functions like pinv were built in. When I tried running the larger matrices through Matlab, I encountered memory errors and that is when I moved to Java.

Once I get the matrix multiplication sorted, I should save a lot of time as this function is heavily used.

I agree that opening and reading the files is probably the cause of the problem. I am currently storing the matrix files on a USB memory stick and running the program on there so that my hard disk doesnt have to do the creating/deleting of large files. Is there a way that I could use the USB stick as additional RAM memory?

Paul

[WW]

Are your matrices dense or sparse (i.e. roughly what percentage of the entries are nonzero)? If only a small percentage are nonzero, you can use some form of sparse matrix representation to save memory.

I don't use Java, so I don't know much about the available libraries, but surely linear algebra libraries already exist that can do this. Can you use any of the software listed at Java Numerics? (Scroll down to the Libraries section.)

[Zugzwang]

Do you know that in the default setting, Java uses only 32MB of ram or so? You can increase the amount by adding "-Xmx512m" (for example: 512MB) to the Java invocation command on the terminal.

[mike_g]

Assuming that you are using 32bit datatype, then you would only require around 120MB of ram for you two matrices plus perhaps another 60 for the output. Like NPL said, file IO is almost definitely your bottlneck. So yeah, try increasing the RAM; that should sort it out.

If theres still issues with speed, one option would be to write a DLL in C, just to handle the calculations. GPUs are also very good at dealing with this kind of stuff, if you can program for them.

[PaulM1985]

Do you know that in the default setting, Java uses only 32MB of ram or so? You can increase the amount by adding "-Xmx512m" (for example: 512MB) to the Java invocation command on the terminal.

Hi Zugzwang,

So, to do this, do I type that in for compiling or running, so is it:

>javac -Xmx512m myfile.java

or

>java -Xmx512m myfile

Paul

[dribeas]

>java -Xmx512m myfile

[Reiger]

But if this is 'the project'; then you're using the wrong tool for the job. You'd be better off using a functional language for this task; because some of those compiles do a pretty good job at optimizing code; not to mention that the languages themselves are geared towards these kinds of problems...