Multiplying very large matrices

[hod139]

I second WW's advice. I have personally used the Colt (http://acs.lbl.gov/~hoschek/colt/) and it is very good. You will hear a lot of people complain about the slower performance of java, but the Colt group has managed to write a BLAS implementation that is 90% as fast as FORTRAN.

[marzshadow]

Hi


I agree that opening and reading the files is probably the cause of the problem. I am currently storing the matrix files on a USB memory stick and running the program on there so that my hard disk doesnt have to do the creating/deleting of large files. Is there a way that I could use the USB stick as additional RAM memory?

Paul

Paul, USB sick memory is very slow. Slower than disk. And it has a limited number of writes (it's flash). If you have to use files, use disk files.

--Mark

[CptPicard]

Also, you should not be using the naive method, there are better ones...

http://en.wikipedia.org/wiki/Matrix_...ion#Algorithms

(But I second the advice of using something from a library...)

[Erdaron]

If you can use Python, then try NumPy, which is made specifically for dealing with matrices and arrays in an efficient manner. I haven't dealt with sets quite as huge as yours, but I've processed data sets with about 200,000 lines and eight columns of data, and while it wasn't super-fast, it certainly faithfully crunched through it without errors. NumPy is also in the Ubuntu repos.

A library is likely to be implementing one of those algorithms for improved efficiency.

[pmasiar]

If you can use Python, then try NumPy, ...A library is likely to be implementing one of those algorithms for improved efficiency.

NumPy is python binding for C libraries, used in scientific calculations. Speed would unlikely be issue there.

[slavik]

haskell can do matrix multiplication faster than C ...

what I would recommend is converting the second matrix so that rows are vertical and columns are horizontal, then you can read the rows you want using readline, multiply them and write them out to a file.

[PaulM1985]

Hi

Thanks with the advice using the -Xmx512m, but unfortunately I ran out of memory again, because I have to have a couple of other large matrices in memory for singular value decomposition.

I am using the USB stick because I didn't want to have any damage to my hard disk because of large scale writing and deleting.

I don't know Python, but know a bit of C. Would you recommend using C instead of Java. Would I be able to read and write to files as I am already doing?

Any other general advice?

Thanks

Paul

[PaulM1985]

haskell can do matrix multiplication faster than C ...

what I would recommend is converting the second matrix so that rows are vertical and columns are horizontal, then you can read the rows you want using readline, multiply them and write them out to a file.

Hi Slavik

Thanks for the idea but that was something I had thought of and tried originally. Unfortunately, it was still taking a very long time to compute.

Paul

[CptPicard]

I am using the USB stick because I didn't want to have any damage to my hard disk because of large scale writing and deleting.

Seriously now... this is the least of your concerns. Reading and writing are what hard drives are built to do, and throughout their lifetime, your matrix ops are a drop in the ocean...

[PaulM1985]

The only reason I am concerned is that I have had to replace 2 hard disks on my laptop in the past and so I don't really want to have to get a 3rd.