Re: Easy Ubuntu Clustering
Hi ajt
Glad you brought this back to the front again!
I was partly involved in the SETI project, quite a number of years ago now though, not in the computer end, just the CPU sharing end is all. UC of Berkely would use my shared computers as a part of their Beowulf Cluster.
More recently I have done a little on-line research into connecting a few of my computers here together into a cluster, hopefully to speed up some of the intense graphics work we do.
I've already retrofitted 3 of my computers with gigabyte LAN cards and played around with trying to figure out how to set them up, but I either kept hitting a brick wall or when I did find some instructions, they were so far over my head it was pitiful.
What led to this in the first place was, I wanted to learn more about how servers work, so I downloaded and installed Edubuntu.
Edubuntu works out of the box so to speak, but all the computers become dumb terminals and use the computing power of the computer the workstations are connected to, as well as burn up all the memory available. Although I liked the idea of a server, I didn't like the fact IT had to provide everything, including the CPU resources.
With so many computers sitting around here, I thought, why not tie them all together. After all, we have dual and quad core computers now, so how hard could it be anyhow......?
Turns out, it's WAY over my head right now, hi hi..........
Nonetheless, I've learned to do different things on different computers using a KVM then taking all the output and stuff it on a file server. That way I can keep working on other things while the crunching is being done on other machines. NOT a Cluster of course, just more efficient use of the machines I do have here.
Since then I've dumbed down to a single Data File Server, which is nothing more than File Sharing, and keep all the working programs on different machines, whichever is best for the project at hand.
But wouldn't it be nice to be able to put all that computing power to use by tying it all together like it was one large computer?
I see it done all over the place. 4, 6, 8, up to 36, 150 and 250 computers all working together. But they are for specific projects designed to use such a system.
Any any case, I'm going to keep monitoring this thread and hope it goes somewhere positive and progressive!
TTUL
Gary
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
Kellemora
Hi ajt
Glad you brought this back to the front again!
Hello, Gary.
Thanks for joining this thread about Ubuntu Clustering :-)
I've been running openMosix for quite a while under Ubuntu 6.06 on our Beowulf cluster:
http://bioinformatics.rri.sari.ac.uk
I've posted a few times about openMosix on the Ubuntu Forums, but the threads are very dispersed so I decided to try and focus a discussion here in Education and Science where, I believe, potential Beowulf users might pick up on the thread!
Quote:
Originally Posted by
Kellemora
I was partly involved in the SETI project, quite a number of years ago now though, not in the computer end, just the CPU sharing end is all. UC of Berkely would use my shared computers as a part of their Beowulf Cluster.
More recently I have done a little on-line research into connecting a few of my computers here together into a cluster, hopefully to speed up some of the intense graphics work we do.
We ran SETI as a test load when I first built our Beowulf, but I found it increasingly difficult to justify using all that electricity and switched to running Folding@home until we got into the top 1,000 ;-)
Quote:
Originally Posted by
Kellemora
I've already retrofitted 3 of my computers with gigabyte LAN cards and played around with trying to figure out how to set them up, but I either kept hitting a brick wall or when I did find some instructions, they were so far over my head it was pitiful.
There are good instructions about how to setup DHCP and PXE boot Kerrighed at:
http://www.kerrighed.org/wiki/index....hed_on_NFSROOT
http://www.kerrighed.org/wiki/index....ROOT_(contrib)
Quote:
Originally Posted by
Kellemora
[...]
But wouldn't it be nice to be able to put all that computing power to use by tying it all together like it was one large computer?
I see it done all over the place. 4, 6, 8, up to 36, 150 and 250 computers all working together. But they are for specific projects designed to use such a system.
It's not that complicated if you are motivated enough to experiment. My summer student Kenny Strouts wrote a brief tutorial about his project installing Kerrighed under Debian Etch:
http://bioinformatics.rri.sari.ac.uk...rial_kerrighed
Quote:
Originally Posted by
Kellemora
Any any case, I'm going to keep monitoring this thread and hope it goes somewhere positive and progressive!
Please do, and see if you can interest any of your friends and colleagues in the thread too.
Bye,
Tony.
Re: Easy Ubuntu Clustering
I am excited about computer clustering as well, but because of no ubuntu availability I have been trying live CD's, and one of the computers I want to use it on doesn't have support for my LAN card. :( and it doesn't know how to do WiFi either, it thinks it is just an ethernet connection
Re: Easy Ubuntu Clustering
The kerrighed project have recently announced a Mandriva live CD running the Kerrighed Kernel:
http://www.kerrighed.org/forum/viewtopic.php?p=607#607
You can use this to see if your computers would run Kerrighed, but beware that it runs a DHCP server, so don't do it on someone else's LAN ;-)
Bye,
Tony.
Re: Easy Ubuntu Clustering
I'm still running openMosix (linux-2.4.26-om1), but I'm planning to upgrade to Kerrighed. I tried out openSSI under Ubuntu 6.06 and 8.04, but had a lot of problems just trying to get it to run. There was some discussion about SSI recently on the beowulf list.
Apparently, kerrighed is used in production by Kerlabs:
http://www.kerlabs.com/-Home-.html
However, I agree Kerrighed is still quite fragile...
My colleague Luca Clivio from the Mario Negri Institute in Milan visited Christine Morin's group and they were extremely helpful. It seems that Kerrighed functionality has been their priority, but the current version is more robust. One thing we could do is help to package Kerrighed for Ubuntu, and get more people interested in using it.
I posted a message to the kerrighed-users list about this dicussion. My objective in starting this thread is to encourage people to try getting Kerrighed running under Ubuntu. It is already quite well supported by Mandriva. However, that is an rpm-based distribution and I want to use Ubuntu because I use bio-Linux (which is based on Ubuntu).
Bye,
Tony.
Re: Easy Ubuntu Clustering
Hi all,
I've been using / compiling kerrighed for some years now. ( 3D, non-academic )
My thoughts / experiences / ideas on kerrighed:
I'll sum up a couple of points that I'll elaborate.
1. It's a bad idea to use the Ubuntu > rather use a stock debian.
Although ubuntu originates from debian, a lot of things changed aside from the kernel version. It's easier to upgrade a debian kernel then to downgrade an ubuntu one. ( And just forget about compiling it the ubuntu way ... )
2. Someone mentioned OpenSSI which is nearly dead > know that Kerrighed re-used most of the OpenSSI + Mosix source.
3. While it's being the most likely candidate to be used by mere mortals, it lacks some elementary functionality ( aside from the outdated howto's & documentation ).
Despite the info on their site, some advertised functionality will not ( ever ) be available
http://www.kerrighed.org/wiki/index.php/Status
Roadmap >november 2008 > thread migration will NOT be implemented any time soon.
This is very misleading, and leaves me with mixed feeling knowing that it is (was) a EU funded operation, that is now run by Kerlabs.
I made some inquiries via a major german corporation in regard to aforementioned feature ( Just to make sure I got an honest reply).
I cannot fully disclose the document but in short it reads that Kerlabs doesn't see any benefit in developing thread migration any further. This because of the small user base that might actually use it ..... (sic)
It must have been a prank if this wasn't a genuine answer. Nothing wrong with out-of-this-world academics, but really how can a company / management be so ignorant for not seeing the huge benefits to the community..
About everyone I know ( in 3D ) would hook up their computers to create a mini cluster(s) ..
I also got an estimate from Kerlabs > the cost to fully implement / support thread migration....
4. Kerrighed live CD > don't bother trying to run it on vmware machines, kernel doesn't support PCNET32 ( NIC drivers vmware uses ).
I'll release my own LiveCD soon ( has proper failover head-node with iptables nat / gateway, webinterface, support for most high-end renderers ( LW / RIB / Maya / Houdini etc .. ) + 3rd party plugin support hooks > to control cluster from within an application ( 3D/video ) etc...
Just need to work a bit on an interface.
( volunteers register on my site )
So in short:
Partially because of for-mentioned reasons there's a big chance I'll fork the project.
Another reason is that my targeted userbase is completely different > I don't need fancy checkpointing on 500 node clusters / hot adding / removing etc...
Most users of this fork will cluster less then 10 computers ( so no need for infiniband / myrinet ).
I've already set up a bug tracker so ....
2 routes to get this done >
-use venture capital ( or other private funding ) > which will make software de-facto proprietary
-use donations ... and keep it open sourced ...
More ideas ?
Re: Easy Ubuntu Clustering
Hello, djamu.
Not sure why you think it's a bad idea to use Ubuntu?
The Kerrighed kernel is based on the 'vanilla' kernel sources, so it doesn't make much difference if you compile it under Debian or Ubuntu.
I think it does make a difference which distro you run: I used Debian from Potato to Etch, but I'm now using Ubuntu because I prefer it. I know that Ubuntu is Debian really, but I think it's much better presented and easier to use both as a server and a desktop. In fact, I use our Beowulf as a terminal server and we run 'embarrasingly' parallel HPC tasks from NX sessions using openMosix or use MPI. I've not looked at Debian Lenny, but Sarge and Etch were crude on the Desktop compared to Ubuntu 6.06...
openSSI looks very interesting but, despite several attempts, I've now abandoned any thought of using it. We must be careful not to accuse anyone of using the 'MOSIX' source, because that's proprietary! I guess you are talking about the GPL version when openMosix was forked ;-)
The three main contenders for FLOSS (Free/Libre Open Source Software) versions of SSI (Single System Image) Linux kernels seem to be openMosix, openSSI and Kerrighed. It's now unlikely the openMosix will continue (but I'm watching development of PMI:
http://linuxpmi.org/trac/
I agree that the openSSI project seems inactive (but don't dismiss it, because it has a lot of good points too). That leaves Kerrighed, which is still an active EU FP6 (Framework 6) funded project until 2010.
Why does venture capital mean something has to be proprietary?
Canonical have invested a lot in Ubuntu...
Thanks for joining in :-)
Bye,
Tony.
Re: Easy Ubuntu Clustering
This is wonderful information- thanks for posting it. I've been interested for a while now in constructing a small cluster (8-16 nodes) for running ecological models, but both my hardware abilities and programming need a bit more development. This will be great stuff to read through!