-
Easy Ubuntu Clustering
Discussions about Ubuntu Clustering seem to be scattered over several forums, and discussion about the Easy Ubuntu Clustering blueprint have been dormant so long that the threads are archived and marked read-only!
I want to re-start discussion about Ubuntu clustering here, where Ubuntu users in Education and Science can contribute: This is related to my proposed blueprint for a "Bioinformatics workstation/server for Ubuntu" https://blueprints.launchpad.net/ubuntu/+spec/biobuntu, which I've marked as obsolete now that NEBC have released Bio-Linux5 based on Ubuntu 8.04 LTS http://nebc.nox.ac.uk/biolinux.html
The Beowulf 'clustering' and grid aspects of my blueprint overlap with those of the "Easy Ubuntu Clustering" blueprint. In particular, creation of Kerrighed packages for Ubuntu: Some (very) old Debian packages have been removed from the Kerrighed website because they are badly out of date with the upstream sources. I've compiled the latest Kerrighed sources under Ubuntu 8.04, and booted it successfully. I'll post details on the "Easy Ubuntu Clustering" wiki https://wiki.ubuntu.com/EasyUbuntuClustering
Tony.
-
Re: Easy Ubuntu Clustering
Hi ajt
Glad you brought this back to the front again!
I was partly involved in the SETI project, quite a number of years ago now though, not in the computer end, just the CPU sharing end is all. UC of Berkely would use my shared computers as a part of their Beowulf Cluster.
More recently I have done a little on-line research into connecting a few of my computers here together into a cluster, hopefully to speed up some of the intense graphics work we do.
I've already retrofitted 3 of my computers with gigabyte LAN cards and played around with trying to figure out how to set them up, but I either kept hitting a brick wall or when I did find some instructions, they were so far over my head it was pitiful.
What led to this in the first place was, I wanted to learn more about how servers work, so I downloaded and installed Edubuntu.
Edubuntu works out of the box so to speak, but all the computers become dumb terminals and use the computing power of the computer the workstations are connected to, as well as burn up all the memory available. Although I liked the idea of a server, I didn't like the fact IT had to provide everything, including the CPU resources.
With so many computers sitting around here, I thought, why not tie them all together. After all, we have dual and quad core computers now, so how hard could it be anyhow......?
Turns out, it's WAY over my head right now, hi hi..........
Nonetheless, I've learned to do different things on different computers using a KVM then taking all the output and stuff it on a file server. That way I can keep working on other things while the crunching is being done on other machines. NOT a Cluster of course, just more efficient use of the machines I do have here.
Since then I've dumbed down to a single Data File Server, which is nothing more than File Sharing, and keep all the working programs on different machines, whichever is best for the project at hand.
But wouldn't it be nice to be able to put all that computing power to use by tying it all together like it was one large computer?
I see it done all over the place. 4, 6, 8, up to 36, 150 and 250 computers all working together. But they are for specific projects designed to use such a system.
Any any case, I'm going to keep monitoring this thread and hope it goes somewhere positive and progressive!
TTUL
Gary
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
Kellemora
Hi ajt
Glad you brought this back to the front again!
Hello, Gary.
Thanks for joining this thread about Ubuntu Clustering :-)
I've been running openMosix for quite a while under Ubuntu 6.06 on our Beowulf cluster:
http://bioinformatics.rri.sari.ac.uk
I've posted a few times about openMosix on the Ubuntu Forums, but the threads are very dispersed so I decided to try and focus a discussion here in Education and Science where, I believe, potential Beowulf users might pick up on the thread!
Quote:
Originally Posted by
Kellemora
I was partly involved in the SETI project, quite a number of years ago now though, not in the computer end, just the CPU sharing end is all. UC of Berkely would use my shared computers as a part of their Beowulf Cluster.
More recently I have done a little on-line research into connecting a few of my computers here together into a cluster, hopefully to speed up some of the intense graphics work we do.
We ran SETI as a test load when I first built our Beowulf, but I found it increasingly difficult to justify using all that electricity and switched to running Folding@home until we got into the top 1,000 ;-)
Quote:
Originally Posted by
Kellemora
I've already retrofitted 3 of my computers with gigabyte LAN cards and played around with trying to figure out how to set them up, but I either kept hitting a brick wall or when I did find some instructions, they were so far over my head it was pitiful.
There are good instructions about how to setup DHCP and PXE boot Kerrighed at:
http://www.kerrighed.org/wiki/index....hed_on_NFSROOT
http://www.kerrighed.org/wiki/index....ROOT_(contrib)
Quote:
Originally Posted by
Kellemora
[...]
But wouldn't it be nice to be able to put all that computing power to use by tying it all together like it was one large computer?
I see it done all over the place. 4, 6, 8, up to 36, 150 and 250 computers all working together. But they are for specific projects designed to use such a system.
It's not that complicated if you are motivated enough to experiment. My summer student Kenny Strouts wrote a brief tutorial about his project installing Kerrighed under Debian Etch:
http://bioinformatics.rri.sari.ac.uk...rial_kerrighed
Quote:
Originally Posted by
Kellemora
Any any case, I'm going to keep monitoring this thread and hope it goes somewhere positive and progressive!
Please do, and see if you can interest any of your friends and colleagues in the thread too.
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
I am excited about computer clustering as well, but because of no ubuntu availability I have been trying live CD's, and one of the computers I want to use it on doesn't have support for my LAN card. :( and it doesn't know how to do WiFi either, it thinks it is just an ethernet connection
-
Re: Easy Ubuntu Clustering
The kerrighed project have recently announced a Mandriva live CD running the Kerrighed Kernel:
http://www.kerrighed.org/forum/viewtopic.php?p=607#607
You can use this to see if your computers would run Kerrighed, but beware that it runs a DHCP server, so don't do it on someone else's LAN ;-)
Bye,
Tony.
-
Kerrighed
Good paper (few years old though) -
"OpenMosix, OpenSSI and Kerrighed: A Comparative Study"
http://hal.inria.fr/docs/00/07/06/04/PDF/RR-5399.pdf
OpenMosix is no longer maintained. Supported distributions for openSSI are Fedora Core 3 and Debian Sarge (with work in progress to port to Etch & Lenny) - so I've no idea as to the status of OpenSSI with regards to Ubuntu. (Maybe someone could shed some light on this?)
Kerrighed is exciting, but still a research project - not yet production ready. At the moment, node failure under Kerrighed causes the whole cluster to freeze. Work is being done to solve this problem; see the following;
http://article.gmane.org/gmane.linux...righed.user/69
At present the best place to seek information concerning Kerrighed is the mailing list, see http://news.gmane.org/gmane.linux.cl...kerrighed.user
-
Re: Easy Ubuntu Clustering
I'm still running openMosix (linux-2.4.26-om1), but I'm planning to upgrade to Kerrighed. I tried out openSSI under Ubuntu 6.06 and 8.04, but had a lot of problems just trying to get it to run. There was some discussion about SSI recently on the beowulf list.
Apparently, kerrighed is used in production by Kerlabs:
http://www.kerlabs.com/-Home-.html
However, I agree Kerrighed is still quite fragile...
My colleague Luca Clivio from the Mario Negri Institute in Milan visited Christine Morin's group and they were extremely helpful. It seems that Kerrighed functionality has been their priority, but the current version is more robust. One thing we could do is help to package Kerrighed for Ubuntu, and get more people interested in using it.
I posted a message to the kerrighed-users list about this dicussion. My objective in starting this thread is to encourage people to try getting Kerrighed running under Ubuntu. It is already quite well supported by Mandriva. However, that is an rpm-based distribution and I want to use Ubuntu because I use bio-Linux (which is based on Ubuntu).
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Hi all,
I've been using / compiling kerrighed for some years now. ( 3D, non-academic )
My thoughts / experiences / ideas on kerrighed:
I'll sum up a couple of points that I'll elaborate.
1. It's a bad idea to use the Ubuntu > rather use a stock debian.
Although ubuntu originates from debian, a lot of things changed aside from the kernel version. It's easier to upgrade a debian kernel then to downgrade an ubuntu one. ( And just forget about compiling it the ubuntu way ... )
2. Someone mentioned OpenSSI which is nearly dead > know that Kerrighed re-used most of the OpenSSI + Mosix source.
3. While it's being the most likely candidate to be used by mere mortals, it lacks some elementary functionality ( aside from the outdated howto's & documentation ).
Despite the info on their site, some advertised functionality will not ( ever ) be available
http://www.kerrighed.org/wiki/index.php/Status
Roadmap >november 2008 > thread migration will NOT be implemented any time soon.
This is very misleading, and leaves me with mixed feeling knowing that it is (was) a EU funded operation, that is now run by Kerlabs.
I made some inquiries via a major german corporation in regard to aforementioned feature ( Just to make sure I got an honest reply).
I cannot fully disclose the document but in short it reads that Kerlabs doesn't see any benefit in developing thread migration any further. This because of the small user base that might actually use it ..... (sic)
It must have been a prank if this wasn't a genuine answer. Nothing wrong with out-of-this-world academics, but really how can a company / management be so ignorant for not seeing the huge benefits to the community..
About everyone I know ( in 3D ) would hook up their computers to create a mini cluster(s) ..
I also got an estimate from Kerlabs > the cost to fully implement / support thread migration....
4. Kerrighed live CD > don't bother trying to run it on vmware machines, kernel doesn't support PCNET32 ( NIC drivers vmware uses ).
I'll release my own LiveCD soon ( has proper failover head-node with iptables nat / gateway, webinterface, support for most high-end renderers ( LW / RIB / Maya / Houdini etc .. ) + 3rd party plugin support hooks > to control cluster from within an application ( 3D/video ) etc...
Just need to work a bit on an interface.
( volunteers register on my site )
So in short:
Partially because of for-mentioned reasons there's a big chance I'll fork the project.
Another reason is that my targeted userbase is completely different > I don't need fancy checkpointing on 500 node clusters / hot adding / removing etc...
Most users of this fork will cluster less then 10 computers ( so no need for infiniband / myrinet ).
I've already set up a bug tracker so ....
2 routes to get this done >
-use venture capital ( or other private funding ) > which will make software de-facto proprietary
-use donations ... and keep it open sourced ...
More ideas ?
-
Re: Easy Ubuntu Clustering
Hello, djamu.
Not sure why you think it's a bad idea to use Ubuntu?
The Kerrighed kernel is based on the 'vanilla' kernel sources, so it doesn't make much difference if you compile it under Debian or Ubuntu.
I think it does make a difference which distro you run: I used Debian from Potato to Etch, but I'm now using Ubuntu because I prefer it. I know that Ubuntu is Debian really, but I think it's much better presented and easier to use both as a server and a desktop. In fact, I use our Beowulf as a terminal server and we run 'embarrasingly' parallel HPC tasks from NX sessions using openMosix or use MPI. I've not looked at Debian Lenny, but Sarge and Etch were crude on the Desktop compared to Ubuntu 6.06...
openSSI looks very interesting but, despite several attempts, I've now abandoned any thought of using it. We must be careful not to accuse anyone of using the 'MOSIX' source, because that's proprietary! I guess you are talking about the GPL version when openMosix was forked ;-)
The three main contenders for FLOSS (Free/Libre Open Source Software) versions of SSI (Single System Image) Linux kernels seem to be openMosix, openSSI and Kerrighed. It's now unlikely the openMosix will continue (but I'm watching development of PMI:
http://linuxpmi.org/trac/
I agree that the openSSI project seems inactive (but don't dismiss it, because it has a lot of good points too). That leaves Kerrighed, which is still an active EU FP6 (Framework 6) funded project until 2010.
Why does venture capital mean something has to be proprietary?
Canonical have invested a lot in Ubuntu...
Thanks for joining in :-)
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
This is wonderful information- thanks for posting it. I've been interested for a while now in constructing a small cluster (8-16 nodes) for running ecological models, but both my hardware abilities and programming need a bit more development. This will be great stuff to read through!
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
ajt
Hello, djamu.
Not sure why you think it's a bad idea to use Ubuntu?
The Kerrighed kernel is based on the 'vanilla' kernel sources, so it doesn't make much difference if you compile it under Debian or Ubuntu.
True, aside from the upstream implementations on ubuntu that might not work on a downgraded kernel... can't think of any right now but I had my share of those before switching my servers back to debian ( hopeless outdated IDS packages for example, ).
Also a base debian install installs much less "bloat" then ubuntu server does.
For example > there's a new service in intrepid (forgot the name) , that let's you sign up for server monitoring...
Now I understand the need for Canonical to generate income.. but not letting me choose whether I want to install it is of a complete different magnitude... ( I try avoid using windows for the very same reason )....
In a server environment this is of less importance as the performance penalty is negligible...
But for tuned HPC cluster kernels ( to reduce interrupt jitter > whether NUMA / SMP ) every unnecessary service interrupt is 1 to many ...
So if I have the option to install a really minimal system (netinstall) or I have to search thru a stock Ubuntu server install to manually delete all unnecessary "helpful" services ... the choice should be obvious, at least common sense dictates the obvious
Quote:
....
but Sarge and Etch were crude on the Desktop compared to Ubuntu 6.06...
....
That's a long time ago, they still don't install sudo in the base install :smile:
Another point in me using debian is that a lot more packages (not desktop related) are available and rigorously tested > the very reason of it's slow cycle...
( A good example is the fail2ban package, stable on debian, typo bug in the 8.04-LTS, preventing it from restart on boot ... submitted to launchpad with fix...., intrepid has proper version ) > BTW as a matter of fact, this kind of package regression is by LTS policy forbidden. The tight release cycle prevents thorough testing...
But let's not get into any kind of flaming, IMO I really think ubuntu is a great desktop distro..., just my personal opinion..
Quote:
openSSI looks very interesting but, despite several attempts, I've now abandoned any thought of using it. We must be careful not to accuse anyone of using the 'MOSIX' source, because that's proprietary! I guess you are talking about the GPL version when openMosix was forked ;-)
Oops sorry my bad.. yes I did mean OpenMosix...
As for OpenSSI, I do think it's dead for 2 reasons.
1. Kerrighed uses a a substantial part of it's source > making OpenSSI practically obsolete
2. The XtreemOS project, which has (about) the same goal as OpenSSI ( where kerrighed is going to be part of )
http://www.xtreemos.eu/
Quote:
...
It's now unlikely the openMosix will continue (but I'm watching development of PMI:
isn't PMI a rebranded OpenMOSIX ?
Quote:
I agree that the openSSI project seems inactive (but don't dismiss it, because it has a lot of good points too). That leaves Kerrighed, which is still an active EU FP6 (Framework 6) funded project until 2010.
True, but since OpenSSI lacks the funding of the XtreemOS consortium ... chances are little it will emerge again..
http://www.xtreemos.eu/overview/plon...43452/partners
System Architecture Overview:
http://www.xtreemos.eu/science-and-r...s-architecture
Quote:
Why does venture capital mean something has to be proprietary?
It doesn't, it's just easier then providing just a service...
And I wouldn't if a couple of kernel devs chip in..
Canonical have invested a lot in Ubuntu... & Mark Shuttleworth flew in space
:lolflag:
On a more serious note.
Tony (AJT) you seem like a knowledgeable guy, I'd like to discuss in depth a couple of things ( compare my trunk compile kernel configs with yours / discuss different types of distributed FS's / tuning & jitter etc ... ) ..
But I'm a little afraid the title of the thread is not well chosen > I'll elaborate .. quite a lot of people reading this have will have a hard time grasping the type of cluster were discussing. To me it's obvious were talking (headless) HPC clustering...
In other words this thread is prone to pollution as soon as people start suggesting for example RHEL or SAN failover mechanisms as an alternative...
It's getting really late now, let's see where this is going..
Jan
-
Re: Easy Ubuntu Clustering
Here's an amazing article on how to turn Xboxs into a Beofwulf cluster. Looks inherently more simple than I thought.
http://www.anandtech.com/linux/showdoc.aspx?i=2271&p=8
-
Re: Easy Ubuntu Clustering
I also had these resources in another thread...
-
Re: Easy Ubuntu Clustering
I would simply be interested in using 3 old computers running as a cluster to slightly speed things up and add some elbow room. I have tried some live CD's but my old laptop's LAN card isn't supported(odd), and I would need to run a fan control program on one of the computers because it's fan control is not automatic from the bios(I find that stupid)
the last one is a pentium MMX machine.
-
Re: Easy Ubuntu Clustering
altonbr,
You could add this to you list o' links:
http://www.calvin.edu/~adams/research/microwulf/
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
machoo02
It's already in there ;)
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
kloplop321
I would simply be interested in using 3 old computers running as a cluster to slightly speed things up and add some elbow room. I have tried some live CD's but my old laptop's LAN card isn't supported(odd), and I would need to run a fan control program on one of the computers because it's fan control is not automatic from the bios(I find that stupid)
the last one is a pentium MMX machine.
Don't want to discourage you but.
If it's a desktop OS which load you want to automatically share between computers..... come back in a couple of years... this is not possible (yet)... Kerrighed is closest for what you want to do & can migrate processes of standard applications to other computer(s) BUT can't split an application using a single process over 2 computers and doesn't distribute threads...
Brief recap of types:
The term supercomputer might be misleading, as in most cases it's not a single OS that runs over a group of CPU's but rather independent computers that run a common application(s) over a shared network(s) > in other words it would be more correct to talk about an HPC cluster then a supercomputer... ( not 1 computer )
Depending on the type of application / dataset(s) more or less speed is required for the interconnects.
An example:
Meet the most powerfull supercomputer on the planet... in everyones :) home.
Folding@home / Seti@home ( BOINC ) > http://boinc.berkeley.edu/
This is an extreme case of an Embarrassingly Parallel ( the correct term for this kind of HPC clustering ) very little data is send over the network by a scheduler to each compute node..
This kind of number crunching doesn't require intermediate data from other processes and doesn't need distributed RAM.
A classical renderfarm is another example ( with a bit more network traffic )
http://en.wikipedia.org/wiki/Embarrassingly_parallel
On the other end you have those applications / dedicated clusters that require massive amounts of intermediate data / huge databases / shared memory > The big boys from the pictures...
These monsters usually have 3D torus shaped high speed networks ( infiniband / myrinet / 10Gb ) with multiple connections / node and are usually build for a single purpose ( you can't run any app on them because they would run inefficient )
These type of clusters usually run a kind of MPI flavor and can't run stock applications
http://en.wikipedia.org/wiki/Message_Passing_Interface
http://en.wikipedia.org/wiki/OpenMP
Closest to a real supercomputer ( in the correct sense of the word ) and what you most likely want to achieve, unrelated to the previous mentioned MPI systems ( which are individual machines connected thru a network ) are the SSI systems. > Kerrighed / (Open)Mosix ( now PMI ) / OpenSSI
http://en.wikipedia.org/wiki/Single-system_image
One more thing worth noting:
The reason why more and more clusters are built from COTS ( commodity of the shelf > normal desktop computers ) parts instead of custom hardware is that before you had 2 distinct types of CPU's
scalar > fast for single instruction single out
http://en.wikipedia.org/wiki/Scalar_processor
http://en.wikipedia.org/wiki/SISD
vector > fast for single instruction multiple out
http://en.wikipedia.org/wiki/Vector_processor
http://en.wikipedia.org/wiki/SIMD
more: http://en.wikipedia.org/wiki/MIMD
While the x86 family originally was a pure scalar CPU it was unsuited ( very slow ) to do any vector processing.. The addition of the MMX and later 3Dnow / SSE etc.. extensions ( which are vector capabilities ) made the x86 hybrid.
So every modern x86 currently has 1 scalar + 1 vector + floating point + .... / core ( the P4 with hyperthreading had 2 scalar cores with 1 vector and 1 floating point core )
The CELL CPU has 1 scalar + 8 vector cores
Roadrunner ( currently the fastest supercomp ) uses hybrid nodes ( 1 AMD Opteron coupled to 1 CELL )
wanted to keep it brief ... pfff look at all that text again :lolflag:
Jan
-
Re: Easy Ubuntu Clustering
The scope of EasyUbuntuClustering is very wide, which might be why the discussions on this thread have been so broad.
Is it realistic for an 'easy' clustering solution to cater to inexperienced developers, offer portable environments via VM's, provide automated load-balancing, and still maintain high QoS?
These are great goals to shoot for, but IMO you will likely have more success if you just pick one or two of the major features.
I can't comment much on the process migration (automated load balancing), since it's been more than 4 years since I've used OpenMosix. These features typically require savvy system admins, and configuration problems can be difficult to diagnose.
Automated deployment of VM's on a cluster or grid (not going to get into a definition war here. I'm referring to a grid as a federated assembly of heterogeneous clusters) is a hot topic that industry giants such as VMware (EMC), IBM, Microsoft, and Amazon are investigating. There are numerous technical subtleties relating this topic, and a logical solution might be to create a specific flavor of debian or ubuntu server that is adapted to hosting the popular VM's. This is especially relevant considering the CISC vendors have product roadmap plans that address common VM bottlenecks.
I admire your efforts to approach this complicated technical issue, and my only suggestion is to consider narrowing your scope a bit.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
ajt
Hello, djamu.
Not sure why you think it's a bad idea to use Ubuntu?
The Kerrighed kernel is based on the 'vanilla' kernel sources, so it doesn't make much difference if you compile it under Debian or Ubuntu.
Its not just a question of building, its how the kernel is built (i.e. using the Debian / Ubuntu patching and build system). The current Kerrighed 2.6.20 kernel (the way its patched and built) would make this problematic.
However, the jump to 2.6.29 would make it much easier.
Quote:
I agree that the openSSI project seems inactive (but don't dismiss it, because it has a lot of good points too). That leaves Kerrighed, which is still an active EU FP6 (Framework 6) funded project until 2010.
I remain subscribed to the OpenSSI lists. Once in a while you'll see a flurry of activity, then radio silence for a few months. I'm still a big fan of the project, I think it will continue to advance for years to come. Its one of those projects where you need a solid block of time to actually do anything useful, so I'd imagine the devs devote what they can to it.
Quote:
Why does venture capital mean something has to be proprietary?
Canonical have invested a lot in Ubuntu...
For a long time many people used the word 'commercial' to refer to non-free software which now causes a lot of confusion and mistrust.
Kerrighed has a very good, very honest and very transparent business model. Use / share / modify the software any way you like .. if you get stuck, paid support and development is available.
You will always have those who are a little over-sensitive to this kind of arrangement. For instance, someone might ask for help to implement thread migration and get a reply that they'll be better off hiring Kerlabs (which they would be). Then, the person will start screaming 'baitware' or worse.
I don't know if you'd be able to find a MOTU that would be interested in keeping up with Kerrighed, but surely nothing is preventing someone from making it available via PPA.
-
Re: Easy Ubuntu Clustering
In the next month I'll try to setup a small cluster (using kerrighed or mosix) to research purpose (http://afni.nimh.nih.gov/afni). I have two dell poweredge 2900 monster (2cpu x 4 core) and several "old" dual CPU Xeon 3Ghz. The first idea is to use ubuntu hardy with a custom kernel (2.6.20 for kerrighed or 2.6.28 for mosix (it's free for researchers)). I think that it'will be a tricky job patching the kernel (with 64 bit support and all the modules to load onto pe2900 and others pcs)...but I'll try to do it, because ubuntu/debian is the most easy and powerful distro to use, configure and administrate.
-
Re: Easy Ubuntu Clustering
rereading previous posts it seems I overlooked this argument I've failed to notice since I thought we're talking headless compute nodes.
Quote:
Originally Posted by
ajt
....
but Sarge and Etch were crude on the Desktop compared to Ubuntu 6.06...
....
So just out of curiosity.
Tony are you really implying an X server / client in the SSI nodes ? Aside from the unnecessary cost this would cause major problems as all GPU's should be of the same brand. Considering you'd only install the client X libraries on the nodes and use a remote X > XDMCP ( the machine which renders the image on screen / has display is the X server ) then still it's nearly impossible since kerrighed doesn't support LVS ( common cluster IP address ) that the X client could use to communicate with a standalone X server > could be handy for a multiseat system :).
I have some experience with OpenGL clusters, and that is a completely different topic in all aspects ( would be nice though ).....
In any case, the easiest way to implement a GUI is a web-based one tied to a job scheduler.
Quote:
Originally Posted by
tinkertim
.
...
However, the jump to 2.6.29 would make it much easier.
in all aspects ... kerrighed being part of the kernel instead of a module, changes things drastically....
Quote:
Originally Posted by
tinkertim
.
For a long time many people used the word 'commercial' to refer to non-free software which now causes a lot of confusion and mistrust.
Kerrighed has a very good, very honest and very transparent business model. Use / share / modify the software any way you like .. if you get stuck, paid support and development is available.
You will always have those who are a little over-sensitive to this kind of arrangement. For instance, someone might ask for help to implement thread migration and get a reply that they'll be better off hiring Kerlabs (which they would be). Then, the person will start screaming 'baitware' or worse.
I feel I need to clarify my statement on this.
The amount of money involved to develop thread migration ( the estimate I got from Kerlabs ) is considerable ( for a person ) and enough to buy myself a nice house.....
Knowing that its very hard to turn OSS into something commercial / proprietary, one could only try to commercialize a web-based front-end / scheduler....
Quote:
Originally Posted by
tinkertim
I don't know if you'd be able to find a MOTU that would be interested in keeping up with Kerrighed, but surely nothing is preventing someone from making it available via PPA.
I doubt an ubuntu MOTU will be interested/has time so yes PPA is a valid option...
Jan
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
kvk
This is wonderful information- thanks for posting it. I've been interested for a while now in constructing a small cluster (8-16 nodes) for running ecological models, but both my hardware abilities and programming need a bit more development. This will be great stuff to read through!
Hello, kvk.
I'm a biologist, turned bioinformatician, and there are lots of people just like me and you building our own DIY Beowulf clusters :-)
The real essence of this thread is to try and make it as 'easy' as possible for Ubuntu users to build this type of Beowulf cluster. I'm sure you already know that it can be quite a steep learning curve if you've never done anything like this before. However, there is nothing magic or particularly difficult about putting the hardware together. To give you a flavour of this approach, there is a chemical modeling cluster called 'COBALT' (Computers On Benches All Linked Together):
http://www.cobalt.chem.ucalgary.ca/
We do some of this type of quantum chemistry modeling under Ubuntu ;-)
The Kerrighed links I posted recently give examples of setting up DHCP and PXE booting of cluster nodes. I'm using this to PXE boot eight of our compute nodes to experiment with Kerrighed. Anyway, thanks for joining the discussion.
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
altonbr
Hello, altonbr.
Connecting the boxes together is easy - Getting them to do something useful is the tricky bit :-)
Actually, I'm quite interested in clustering PS3's...
Not this one:
http://www.engadget.com/2007/08/11/s...arhawk-mayhem/
More like this:
http://gravity.phy.umassd.edu/ps3.html
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
djamu
[...]
But let's not get into any kind of flaming, IMO I really think ubuntu is a great desktop distro..., just my personal opinion..
Hello, Jan.
Good to know you like Ubuntu ;-)
Quote:
Originally Posted by
djamu
[...]
isn't PMI a rebranded OpenMOSIX ?
It's a continuation of the project to port openMosix to the 2.6 kernel.
Quote:
Originally Posted by
djamu
In my (now obsolete) 'biobuntu' blue-print, you will see that I'm also interested in XtreemOS as well as Kerrighed:
https://blueprints.launchpad.net/ubuntu/+spec/biobuntu
Quote:
Originally Posted by
djamu
[...]
[...]
But I'm a little afraid the title of the thread is not well chosen > I'll elaborate .. quite a lot of people reading this have will have a hard time grasping the type of cluster were discussing. To me it's obvious were talking (headless) HPC clustering...
In other words this thread is prone to pollution as soon as people start suggesting for example RHEL or SAN failover mechanisms as an alternative...
I disagree - I've got more than 26 years experience of 'parallel' computing but I obtained most of my computing (and image analysis) experience working as a biologist. I now work as a bioinformatician.
There are lots of people like me working in bioinformatics: A common thread in discussions like this is that it is MUCH more difficult to grasp the scientific questions being asked than it is to build a Beowulf. The technology is a means to an end. The draconian 'lock-down' policies of some highly centralised IT departments lack the flexibility to allow R&D approaches to be used by scientists who often know more about the technology than people trained in provision of IT for the 'enterprise'.
BTW, That's not a criticism of people providing IT for the 'enterprise'. it's an observation, borne of my own experience, that IT for science and IT provided for the 'enterprise' have completely different objectives. My intention in starting a thread in this forum is to try to draw together opinions and help from other people with similar objectives to my own. I agree that it's important to keep the topic clear: It is how we make it 'easy' to cluster Ubuntu systems in order to use the aggregate resources of many computers to solve one problem. This is the essence of Beowulf:
http://www.beowulf.org/
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
kloplop321
I would simply be interested in using 3 old computers running as a cluster to slightly speed things up and add some elbow room.
Hello, kloplop321.
I think that using a few old computers to learn about Beowulf clusters is fine, but at the end of the day they are still old computers and there is an overhead involved in communication between the compute nodes in a cluster. The first rule of 'parallel' computing is to determine what can be done independently by the nodes: That's where you gain an advantage.
Quote:
Originally Posted by
kloplop321
I have tried some live CD's but my old laptop's LAN card isn't supported(odd), and I would need to run a fan control program on one of the computers because it's fan control is not automatic from the bios(I find that stupid)
the last one is a pentium MMX machine.
Maybe this would be a good time to learn how to compile the Linux kernel?
It's not that difficult the second time you do it, but the first time takes a bit longer ;-)
Check the Kerrighed links I posted earlier in this thread for more info.
Thanks for joining in.
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
altonbr
It's already in there ;)
Hello, altonbr.
Why not add things like this to the Wiki:
https://wiki.ubuntu.com/EasyUbuntuClustering
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
djamu
Hello, Jan.
The BIG difference is using SIMD/MIMD instead of Von Neumann Arhitecture:
http://en.wikipedia.org/wiki/Von_Neumann_architecture
SIMD (Single Instruction Multiple Data) is fast, but extremely limited. I used a 9216 SIMD processor array for image analysis some time ago (CLIP4R). There is a brief summary of this type of processor here if anyone is interested:
http://homepages.inf.ed.ac.uk/rbf/CV...ERATOS/iii.htm
However, we concluded that the advantages of using 'exotic' processor architectures like this does not outweigh the difficulty of programming them, or the 'corner-turning' problem of actually getting images into the SIMD processor array. Our CLIP4R was a very interesting machine, but its CPU boards ended up being made into wall clocks. Sad, but well worth the lessons learned. The host controller for the CLIP4R was a pdp11/34, so you can tell how long about this was (1986). I replaced the pdp11 with a Vax 750, which had less than 1MIP integer performance (a MIP is defined as an integer perfomance of one million VAX780 instructions per second).
Enough nostalgia for now - This has little to do with Ubuntu clustering, but might rectify some misunderstandings about what 'parallel' computing involves. What we are talking about in the Easy Ubuntu Clustering blue-print, and this thread, is using a collection of Von Neumann architecture processors in different computers as if they were one large SMP (Symmetric Multi-Processing) computer. This is the SSI (Single System Image) approach to parallel computing used by openMosix, openSSI and Kerrighed. It's also the approach taken by Donald Becker, who invented the Beowulf cluster at NASA and commercialised it:
http://www.scyld.com/home.html
[Scyld, was Beowulf's companion in the Beowulf poem]
http://en.wikipedia.org/wiki/Beowulf
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
meatpan
The scope of EasyUbuntuClustering is very wide, which might be why the discussions on this thread have been so broad.
Is it realistic for an 'easy' clustering solution to cater to inexperienced developers, offer portable environments via VM's, provide automated load-balancing, and still maintain high QoS?
These are great goals to shoot for, but IMO you will likely have more success if you just pick one or two of the major features.
Hello, meatpan.
I found the previous discussions dispersed over several different forums difficult to follow, and repeating many of the same questions. Yes, the discussion so far has been quite broad but, as we realise what people are actually interested in, I think the discussion will become more focussed.
Quote:
Originally Posted by
meatpan
I can't comment much on the process migration (automated load balancing), since it's been more than 4 years since I've used OpenMosix. These features typically require savvy system admins, and configuration problems can be difficult to diagnose.
Not really - I run a 90-node openMosix system and I'm a biologist ;-)
Quote:
Originally Posted by
meatpan
Automated deployment of VM's on a cluster or grid (not going to get into a definition war here. I'm referring to a grid as a federated assembly of heterogeneous clusters) is a hot topic that industry giants such as VMware (EMC), IBM, Microsoft, and Amazon are investigating. There are numerous technical subtleties relating this topic, and a logical solution might be to create a specific flavor of debian or ubuntu server that is adapted to hosting the popular VM's. This is especially relevant considering the CISC vendors have product roadmap plans that address common VM bottlenecks.
Hmm... 'cloudy' today?
The BIG problem with VM's is that they are designed to be using for HAC (High Availablility Computing) not HPC (High Performance Computing). It's not HPC to virtualise instances of e.g. Ubuntu then merge them again into a 'virtual' SSI. It's a way of making your IT provisioning resilient and platform-independant. Nothing wrong with that, if you're an ISP...
Quote:
Originally Posted by
meatpan
I admire your efforts to approach this complicated technical issue, and my only suggestion is to consider narrowing your scope a bit.
Thanks - The technical aspects are really not that difficult, but the advantage of packaging up Kerrighed for Ubuntu would make it a lot easier!
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
tinkertim
Its not just a question of building, its how the kernel is built (i.e. using the Debian / Ubuntu patching and build system). The current Kerrighed 2.6.20 kernel (the way its patched and built) would make this problematic.
However, the jump to 2.6.29 would make it much easier.
Hello, tinkertim.
I'm a bit confused by what you say about patching: The 'patch' program is the same under Debian and Ubuntu, and the Kerrighed patches are relative to the 'vanilla' kernel sources for 2.6.20. The kernel Makefile includes a 'deb-pkg' target, which has nothing to do with the Debian or Ubuntu patches to their kernels. I built and installed Kerrighed kernel debs under Ubuntu 6.06 using only the 2.6.20 sources and kerrighed patches.
Quote:
Originally Posted by
tinkertim
I remain subscribed to the OpenSSI lists. Once in a while you'll see a flurry of activity, then radio silence for a few months. I'm still a big fan of the project, I think it will continue to advance for years to come. Its one of those projects where you need a solid block of time to actually do anything useful, so I'd imagine the devs devote what they can to it.
I've followed openSSI developmenmts for years, and I tried quite hard to get openSSI running, to use as a replacement for openMosix, but gave up in the end.
Quote:
Originally Posted by
tinkertim
[...]
I don't know if you'd be able to find a MOTU that would be interested in keeping up with Kerrighed, but surely nothing is preventing someone from making it available via PPA.
OK, I'll take that as you volunteering to help me do it :-)
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
crazy___cow
In the next month I'll try to setup a small cluster (using kerrighed or mosix) to research purpose (
http://afni.nimh.nih.gov/afni). I have two dell poweredge 2900 monster (2cpu x 4 core) and several "old" dual CPU Xeon 3Ghz. The first idea is to use ubuntu hardy with a custom kernel (2.6.20 for kerrighed or 2.6.28 for mosix (it's free for researchers)). I think that it'will be a tricky job patching the kernel (with 64 bit support and all the modules to load onto pe2900 and others pcs)...but I'll try to do it, because ubuntu/debian is the most easy and powerful distro to use, configure and administrate.
Hello, crazy___cow.
Be very careful how you interpret "free for researchers" - I looked into this too, and the deal is you are given one 'free' snapshot of MOSIX2 kernel patches plus binaries of the MOSIX2 utilities. The snapshot is the current release, which is good, but you have to pay $1,000 per year to get updates and you don't get any sources other than the kernel patches. It depends what your objectives are. MOSIX2 looks very good, but it's not FLOSS. That might not matter to everyone, but it matters to me.
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
djamu
[...]
Tony are you really implying an X server / client in the SSI nodes ? Aside from the unnecessary cost this would cause major problems as all GPU's should be of the same brand. Considering you'd only install the client X libraries on the nodes and use a remote X > XDMCP ( the machine which renders the image on screen / has display is the X server ) then still it's nearly impossible since kerrighed doesn't support LVS ( common cluster IP address ) that the X client could use to communicate with a standalone X server > could be handy for a multiseat system :).
Hello, Jan.
The Xserver runs on the client PC, where it displays the output of the Xclient running on the SSI. My present openMosix system has two servers and 88 compute nodes. One server is the 'head' node of the cluster, providing PXE + DHCP + NFSROOT and a web server:
http://bioinformatics.rri.sari.ac.uk
The other server is for interactive logins via SSH/NX/VNC and is where the user's home directories are stored. They are exported via NFS to the nodes. Only the servers are allowed to migrate processes away to other nodes. The Beowulf becomes unstable if all nodes are allowed to migrate processes. The compute nodes accept migrated processes from the servers, but do NOT migrate their own local processes. This allows us to run MPI and SGE as local processes without them being migrated off nodes...
Quote:
Originally Posted by
djamu
I have some experience with OpenGL clusters, and that is a completely different topic in all aspects ( would be nice though ).....
In any case, the easiest way to implement a GUI is a web-based one tied to a job scheduler.
I've been looking at eyeOS:
http://eyeos.org/
I've installed it, but not done much with it yet.
Most of the time, we just login using NX. If we want to run 'batch' jobs, we use SGE "qmon", or command-line tools to submit jobs. The webapps we run provide a web-based interface via the Beowulf web server using the Apache JK connector. Some, but not all, of the work accepted by these webapps is migrated automatically to compute nodes by openMosix. Any jobs using shared memory, in particular, are not migrated. This is due to the limitations of openMosix, as you know. Interrestingly, I did experiment with the openMosix 'MIGSHM' patch: My processes migrated away fine, but the didn't come back!
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Hi,
How about porting Perceus to ubuntu/debian?
http://www.linux-mag.com/id/6386
To be frank I am a cluster newbie, I have just built my own cluster to learn the technology, and after lot of reading and digging I have started to use Perceus, more precisely Caoslinux, for the same reason you started this wiki, as it makes cluster management easy, and it realy does the job. On the other hand, I am an Ubuntu fun, I am familiar with it, so I would love to do the same with Ubuntu. Unfortunately I am not a linux/ubuntu admin guru, otherwise I would have started the porting already.
Probably you do not know, that Perceus is the provisioning part of Warewulf, Caoslinux is the distribution to manage them, and the whole lot is maintained by Infiscale.
http://www.perceus.org/portal/
http://www.caoslinux.org/index.html
http://www.infiscale.com/
Cheers,
Zsolt
-
Re: Easy Ubuntu Clustering
Hi all!!
Can anyone post a simple and exaustive guide on how to install a cluster of pc using Kerrighed 2.3.0 and Ubuntu 8.10 or 8.04 ?
Thanks
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
masulli
Hi all!!
Can anyone post a simple and exaustive guide on how to install a cluster of pc using Kerrighed 2.3.0 and Ubuntu 8.10 or 8.04 ?
Thanks
Hello, masulli.
Not yet, but I hope we can create a guide:
https://blueprints.launchpad.net/ubu...ntu-clustering
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
zkovax
Hello, Zsolt.
Actually, I do know about Warewulf, but 'cAos' is an rpm-based distro and I want to use a deb-based distro - preferably Ubuntu ;-)
There are similar systems like DRBL:
http://drbl.sourceforge.net/
These systems are very useful for 'jumpstarting' or administering Beowulf clusters - DRBL has been used to deploy Kerrighed:
http://trac.nchc.org.tw/grid/wiki/krg_DRBL
However, systems like this do not address the issue of how to actually use a cluster for HPC (High performance Computing). A Beowulf is not just a COW (Collection Of Workstations), it's more than that...
What is interesting about openMosix, openSSI and Kerrighed is that they do kernel-level load balancing in a similar way that an SMP (Symmetric Multi-Processing) computer does, to make use of multiple CPU's. There is a HUGE difference in cost between e.g. building a Beowulf cluster from 16 ordinary COTS (Commodity Off The Shelf) PC's and buying in a commercial 16p SMP computer system, which is why this is a popular approach.
The BIG difference is that, without non-COTS cluster node interconnects, node latency can be extremely large in comparison to local SMP processor interconnects. Kerrighed uses an optimised 'kernel' TCP/IP communication protocol to minimise the latency in communication between Linux kernels running on processors in different Kerrighed nodes. That's one reason why I think Kerrighed is well worth looking at.
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
hi,
i have tried to cover all the posts here (at least one time) so far. there has been allot of lofty discussion of architecture and so forth that drifts from (and imo confuses) the issue.
i have only read joel adams paper on how he configures his microwulf using ubuntu one time. but it appears that he has got what the original thread post requested.
except for the fact that he gives the instructions where the assumption is that the slave nodes will be diskless the description of how to configure a cluster using ubuntu where the nodes are connected by a cots switch is as easy as it will get without some kind of cluster installer like that for the .iso distro of ubuntu.
once again that link is...
http://www.calvin.edu/~adams/research/microwulf/
...that's it folks!
that is the answer to this thread.
i intend to use it as a guide to try and establish my first cluster of two identical ibm t60 laptops run by ubuntu by simply connecting them with a cat-5 patch chord (no switch) and configuring them using his configuration instructions as a guide. i don't think there will be any problem doing this (without the switch) to get started.
perhaps the best way for those interested in actually building a cluster is to follow my lead with whatever hardware they have available. that is to say just use two boxes that run ubuntu, connect them with a cat-5 patch chord and use joel's configuration instructions as a guide.
once you have two machines that are an actual cluster just scale up.
peace!
-zander
-
Re: Easy Ubuntu Clustering
What I am curious about is how the computers are connected... Certainly it won't work by just connecting two computers with a LAN cable, so it seems that a router or sth more advanced is needed...
All I have is a modem which is provided by my ISP. I guess it is something that you connect it with a LAN cable and a telephone line so that you can access the web.
This is what I have done a month before:
==================================================
Hardware:
1. a win95 box with 133MHz CPU & 32Mb RAM
2. an xp box with 2GHz CPU,256Mb RAM
3. an ADSL modem with 4 slots for LANs
4. two LAN cables
Procedures:
I connected the two boxes to the modem with the LAN cables. Then, I tried to boot both boxes using liveCDs like CHAOS and ClusterKnoppix. Afterthat, I tried some floppies like GoMF and openMosixLOAF. Finally, I tried to start the following shell-script in background:
Quote:
s=0
for (( i=1; i>0; i=i+1 ))
do
s=$(($s+$i))
done
Results.
liveCDs: cannot boot on the old box. success to boot on the new box but cannot configure itself as a node - sth called "DHCP server" was misssing.
floppies:
a. GoMF can boot on both boxes, but the same problem occured - "DHCP server" was absent.
b. openMosixLOAF can boot on both boxes, success to configure as nodes by manually entering private IPs and Gateway. can view cpu loads, ram utilization in real time from any boxes. cannot migrate the shell script because the floppy is not shipped with commands like "mosrun".
================================================== =======
To sum up, any attempt to construct openmosix-cluster using a modem failed (without compiling, of course). Could anyone tell me what is a "DHCP server"?
It seems that the so called "DHCP" is related to auto-detection & auto-configuration, and cannot be turned off. Is it possible to solve this without having to buy new hardware(s)?
-
Re: Easy Ubuntu Clustering
Well, after surfing Wikipedia, I found that my ADSL modem is actually a router... :P
But that changes nothing... my ISP provided me this "modem" as a part of the internet service, but they didn't provide any further support on "advanced" usage, even a not a manual...
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
zander1013
hi,
i have tried to cover all the posts here (at least one time) so far. there has been allot of lofty discussion of architecture and so forth that drifts from (and imo confuses) the issue.
Hello, zander.
I think it's important to place what we are discussing in context: In fact, SIMD instructions are now used in the FPU's of modern processors and I think it's quite useful to have a little background information :-)
Quote:
Originally Posted by
zander1013
i have only read joel adams paper on how he configures his microwulf using ubuntu one time. but it appears that he has got what the original thread post requested.
except for the fact that he gives the instructions where the assumption is that the slave nodes will be diskless the description of how to configure a cluster using ubuntu where the nodes are connected by a cots switch is as easy as it will get without some kind of cluster installer like that for the .iso distro of ubuntu.
I don't think that building a Microwulf could be described as 'easy', and what I've read about Microwulf focusses more on the hardware aspects of how to construct a DIY Beowulf. Segregating 'system' and 'application' network traffic is a good idea, but it's not an original one:
http://bioinformatics.rri.sari.ac.uk/bobcat/
Quote:
Originally Posted by
zander1013
I think the Microwulf project is great, in terms of its hardware, but it doesn't bring anything new to the software requirements of building Ubuntu clusters. In relation to 'diskless' booting, for example, there has been an Ubuntu 'diskless' wiki for a long time:
https://help.ubuntu.com/community/DisklessUbuntuHowto
The type of Beowulf cluster installation/management packages that are discussed by the Microwulf project are well known. What I suggested is that we try to package Kerrighed for Ubuntu, because Kerrighed is a state of the art SSI (Single System Image) Linux kernel load-balancing system that has already been packaged for Mandriva Linux. I think it would be useful to package it for Ubuntu too.
Quote:
Originally Posted by
zander1013
i intend to use it as a guide to try and establish my first cluster of two identical ibm t60 laptops run by ubuntu by simply connecting them with a cat-5 patch chord (no switch) and configuring them using his configuration instructions as a guide. i don't think there will be any problem doing this (without the switch) to get started.
Make sure you use a crossover cable if you do (or check that your laptop has autosensing NIC's). It's quite common for ISP's to provide a combined ADSL modem + router + WiFi these days, but a passive ethernet hub would be another solution for a low-cost experimental setup.
Quote:
Originally Posted by
zander1013
perhaps the best way for those interested in actually building a cluster is to follow my lead with whatever hardware they have available. that is to say just use two boxes that run ubuntu, connect them with a cat-5 patch chord and use joel's configuration instructions as a guide.
once you have two machines that are an actual cluster just scale up.
As Bart says: "Don't have a COW, man...", but that is exactly what you will create if you just connect together a Collection Of Workstations. There's a lot more to building a cluster than simply connecting computers together. However, connecting them up *is* quite important ;-)
Quote:
Originally Posted by
zander1013
peace!
A friend of mine saw graffiti in Florence recently that read:
Make tea, not war
I like that :-)
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
antony_css
[...]
To sum up, any attempt to construct openmosix-cluster using a modem failed (without compiling, of course). Could anyone tell me what is a "DHCP server"?
It seems that the so called "DHCP" is related to auto-detection & auto-configuration, and cannot be turned off. Is it possible to solve this without having to buy new hardware(s)?
Hello, antony_css.
A DHCP (Dynamic Host Configuration Protocol) server listens for computers on the network broadcasting DHCPDISCOVER requests and supplies them with a dynamic network configuration. This normally consists of an IP address, netmask and default router addess. Often, the DHCP server also supplies DNS name server addresses and DNS domains to search. In addition, modern DHCP servers support network booting and supply information about TFTP (Trivial File Transfer Protocol) where a boot image can be downloaded.
Although this might sound complicated, many routers run DHCP servers so all you have to do is connect your devices to the router, and find out what their IP addresses are by running:
ipconfig -a
This is necessary if you want to connect your computers together, because they need to know each other's IP address. This is a common private IP network:
192.168.0.254 router
192.168.0.1 node1
192.168.0.2 node2
...
For a small network, you can put this information in the /etc/hosts file.
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Hi All!
I'm researching for a paper for our company (sadly, the paper is proprietary) on how to use existing open source solutions for cluster computing and balancing the loads between computers that merely do spreadsheets or emails with computers that do intensive compiles/builds. Looks like I found the right source of info!:D
I'll try to catch up on all the info you've guys have written. Interesting stuff.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
punong_bisyonaryo
Hi All!
I'm researching for a paper for our company (sadly, the paper is proprietary) on how to use existing open source solutions for cluster computing and balancing the loads between computers that merely do spreadsheets or emails with computers that do intensive compiles/builds. Looks like I found the right source of info!:D
I'll try to catch up on all the info you've guys have written. Interesting stuff.
Hello, punong_bisyonaryo.
It's a popular wish to harness all the 'idle' CPU cycles on under-used office desktops, but the people using them will not be very happy when your intensive compile/build makes their web browser go catatonic!
What works best under these circumstances is 'volunteer' computing:
http://www.volunteerathome.com/
The point about volunteer computing is that the donor computer decides when to volunteer its resources (e.g. when a screen saver starts up).
In contrast, MOSIX/openMosix was actually designed to work the way that you want to by using the idle CPU's on workstations but the computers all have to run Linux. In practice, though, it's difficult to avoid someone switching a computer off, or rebooting it in the middle of your job that has been migrated onto it. When that happens your job is lost...
Clustering systems are only as reliable as the nodes they contain: If you don't control the nodes, then you don't control the cluster. For that reason, most people build a private cluster LAN with dedicated compute nodes connected to it. Even if you do control your company's desktops, you will still have problems running cluster software on the public (company) LAN, because you will be using up a lot of the available network bandwidth migrating programs and data to and from 'idle' desktops.
I think there are circumstances where it might work if e.g. you take over the company LAN and reboot Linux on the desktop PC's during out-of-hours and run your compile farm then. This is common practice in University PC teaching labs that run Micro$oft Windows by day, but Linux HPC software by night :-)
Good luck + please do post your conclusions here!
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Well, somehow i found that dhcp can be set up as a daemon...
A floppy distribution "eucaristOS" actually use this technique to set up cluster on old computers...
However it does not support my network card... so sad...
-
Re: Easy Ubuntu Clustering
Maybe we can take this information and start collecting it under https://help.ubuntu.com/community/Clustering ???
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
antony_css
Well, somehow i found that dhcp can be set up as a daemon...
A floppy distribution "
eucaristOS" actually use this technique to set up cluster on old computers...
However it does not support my network card... so sad...
Hello, antony_css.
Beware, eucaristOS is based on an even older version of openMosix than I'm using!
You could compile in support for your network card, if you download the openMosix kernel sources, However, I think you would be better off trying out the Kerrighed 'live' CD, which also runs DHCP for you:
http://www.kerlabs.com/dl/kerrighed-live.iso
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
altonbr
Hello, altonbr.
OK, I've created a 'Clustering' page...
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Hi all!
I am trying to create a little home cluster with Ubunto 8.04 and Kerrighed, but I am facing some problems.
I run the build.sh script located in wiki page, but at some point he run the command vi Makefile and stop waiting for something, what i have to do at this point? The comment say "# edit Extraversion" what i need to chage/edit?
Thanks and cherrs ;)
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
swisspipe
Hi all!
I am trying to create a little home cluster with Ubunto 8.04 and Kerrighed, but I am facing some problems.
I run the build.sh script located in wiki page, but at some point he run the command vi Makefile and stop waiting for something, what i have to do at this point? The comment say "# edit Extraversion" what i need to chage/edit?
Thanks and cherrs ;)
Hello, swisspipe.
That build.sh describes the commands I used to compile the Kerrighed kernel under Ubuntu 8.04. You have to edit the 'Extraversion' in the Makefile if you are building a 'deb' because the kernel name must obey the Debian package name conventions. The build failed with the default Kerrighed 'Extraversion', so I tried changing it to:
EXTRAVERSION = -krg-2.3.0
That resulted in a valid Debian package name :-)
The commands in "build.sh" only compile and package the kernel. I've not packaged the Kerrighed user-land tools yet.
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
I really wish to have a workable eucaristOS disk for my computers...
Is it possible to merge the latest kernel in ajt's deb to the eucaristOS source?
Looking into the source directory of eucaristOS, I didn't find any related folders like "/boot" for me to replace the kernel... Isn't that because the disk does not use grub as bootloader?
BTW, I keep making stupid mistakes even when I tried to make the floppy image from the source. It crashes due to an unknown error...like the one below:
Quote:
$ sudo make all
Building floppy image.../bin/sh: Syntax error: Bad fd number
make: *** [floppy] Error 2
What should I do?
-
Re: Easy Ubuntu Clustering
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
zkovax
Hi,
How about porting Perceus to ubuntu/debian?
http://www.linux-mag.com/id/6386
To be frank I am a cluster newbie, I have just built my own cluster to learn the technology, and after lot of reading and digging I have started to use Perceus, more precisely Caoslinux, for the same reason you started this wiki, as it makes cluster management easy, and it realy does the job. On the other hand, I am an Ubuntu fun, I am familiar with it, so I would love to do the same with Ubuntu. Unfortunately I am not a linux/ubuntu admin guru, otherwise I would have started the porting already.
Probably you do not know, that Perceus is the provisioning part of Warewulf, Caoslinux is the distribution to manage them, and the whole lot is maintained by Infiscale.
http://www.perceus.org/portal/
http://www.caoslinux.org/index.html
http://www.infiscale.com/
Cheers,
Zsolt
We have already "ported" perceus (and warewulf) to debian.. It needs
a lot of polishing but it works for i386 and amd64 on our cluster
of ~80 machines.
deb http://biodev.ece.ucsb.edu/debian/ main contrib
On you server:
aptitude install perceus-server
There are scripts for creating debian VNFS there.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
kgkv
We have already "ported" perceus (and warewulf) to debian.
Hi kgkv,
This looks really good. I will have a look.
Zsolt
-
Re: Easy Ubuntu Clustering
It sounds interesting:
http://code.google.com/p/distcc/
Anyway, what is that?
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
masulli
Hi all!!
Can anyone post a simple and exaustive guide on how to install a cluster of pc using Kerrighed 2.3.0 and Ubuntu 8.10 or 8.04 ?
Thanks
Hi Masulli, are you or anyone else still looking for a guide to do this? I've recently managed to get a small Kerrighed 2.3.0 cluster up and running under Ubuntu 8.04 with the 2.6 kernel. I'm currently in the process of writing a guide covering the setup process so other people at work know how to maintain it. If anyone is interested I can post a link on here.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
bigjimjams
Hi Masulli, are you or anyone else still looking for a guide to do this? I've recently managed to get a small Kerrighed 2.3.0 cluster up and running under Ubuntu 8.04 with the 2.6 kernel. I'm currently in the process of writing a guide covering the setup process so other people at work know how to maintain it. If anyone is interested I can post a link on here.
Yes please! Definitely interested.:D
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
punong_bisyonaryo
Yes please! Definitely interested.:D
Seeing as there is a demand for it, I'll get it finished quickly and post it here. :D
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
antony_css
What I am curious about is how the computers are connected... Certainly it won't work by just connecting two computers with a LAN cable, so it seems that a router or sth more advanced is needed...
For two PC's all that is required is a cross-over LAN cable & a configured hosts file. Multiple PCs will require a LAN Switch. A router is not required.
-
Re: Easy Ubuntu Clustering
bigjimjams,
I am another person who would love to see a tutorial for this. I have been following tutorial at http://trac.nchc.org.tw/grid/wiki/krg_DRBL. But since I am using a more recent version of Kerrighed, it seems there are some problems towards the end with actually starting the cluster. I am also very frustrated that this and other tutorials (e.g. http://www.in2dwok.com/indawok/?page_id=16) don't *explain* what is being done but just tell you to do it.
If you can at least get a draft of your tutorial, I would offer to help extend it based on my experiences. I think the owner of this thread started a page for EasyUbuntuClustering on the wiki. We should do the same (either add to that page or start a dedicated one). In my view, before anyone can help make clustering "easy", we need to have more people actually able to start a cluster. :}
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
xingmu
bigjimjams,
I am another person who would love to see a tutorial for this. I have been following tutorial at
http://trac.nchc.org.tw/grid/wiki/krg_DRBL. But since I am using a more recent version of Kerrighed, it seems there are some problems towards the end with actually starting the cluster. I am also very frustrated that this and other tutorials (e.g.
http://www.in2dwok.com/indawok/?page_id=16) don't *explain* what is being done but just tell you to do it.
If you can at least get a draft of your tutorial, I would offer to help extend it based on my experiences. I think the owner of this thread started a page for EasyUbuntuClustering on the wiki. We should do the same (either add to that page or start a dedicated one). In my view, before anyone can help make clustering "easy", we need to have more people actually able to start a cluster. :}
Hi Xingmu, sounds like a good plan, as I'd be interested in other peoples opinions on what I did. If you are using kerrighed 2.3.0, there is a known issue with the krgadm tool, which always states that the cluster is not running even when it is! I'll put a draft of the guide up on the "Easy" Clustering Wiki in the next day or so, as I'm currently swamped with a couple of tasks.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
bigjimjams
Hi Xingmu, sounds like a good plan, as I'd be interested in other peoples opinions on what I did. If you are using kerrighed 2.3.0, there is a known issue with the krgadm tool, which always states that the cluster is not running even when it is! I'll put a draft of the guide up on the "Easy" Clustering Wiki in the next day or so, as I'm currently swamped with a couple of tasks.
Hi Xingmu and anybody else interested, as promised in my earlier post I've added a draft guide for setting up a kerrighed 2.3.0 cluster in Ubuntu 8.04 on the Easy Ubuntu Clustering Wiki. Here is the link:
https://wiki.ubuntu.com/EasyUbuntuCl...edClusterGuide
If you have any questions, comments, suggestions or improvements let me know.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
bigjimjams
Hi Xingmu and anybody else interested, as promised in my earlier post I've added a draft guide for setting up a kerrighed 2.3.0 cluster in Ubuntu 8.04 on the Easy Ubuntu Clustering Wiki. Here is the link:
https://wiki.ubuntu.com/EasyUbuntuCl...edClusterGuide
If you have any questions, comments, suggestions or improvements let me know.
Hello, bigjimjams.
Thanks for your excellent installation guide!
I've done more or less the same installation, but I've not documented it quite so well :-)
One difference is that I use UNFS3 with 'ClusterNFS' extensions enabled for the NFSROOT. This allows all the nodes to share the same NFSROOT filesystem read-only, with 'tagged' links into a writable area for node-specific files.
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
bigjimjams
Hi Xingmu and anybody else interested, as promised in my earlier post I've added a draft guide for setting up a kerrighed 2.3.0 cluster in Ubuntu 8.04 on the Easy Ubuntu Clustering Wiki. Here is the link:
https://wiki.ubuntu.com/EasyUbuntuCl...edClusterGuide
If you have any questions, comments, suggestions or improvements let me know.
Thanks bigjimjams! I actually got the cluster up and running right after I saw your note about the "no running cluster" bug in krgadm. I ended using a package DRBL which configures the dhcp-server, tftp, etc. automatically. I am thinking it might be useful to add this to your tutorial as an alternative installation method.
On another note and somewhat off-topic, has anyone tried installing Kerrighed straight from the SVN? I see that above-mentioned bug has already been fixed in the SVN. I am also having troubles with processes not migrating to multi-core CPUs. I am hoping newer revisions might have fixed these problems. If so, are there any suggested revision numbers to use? Or is every revision supposed to be a working version?
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
ajt
Hello, bigjimjams.
Thanks for your excellent installation guide!
I've done more or less the same installation, but I've not documented it quite so well :-)
One difference is that I use
UNFS3 with 'ClusterNFS' extensions enabled for the NFSROOT. This allows all the nodes to share the same NFSROOT filesystem read-only, with 'tagged' links into a writable area for node-specific files.
Bye,
Tony.
No problem Tony, happy to help out! I was creating the guide for documenting it at work anyway, so thought I'd share what I'd done to save other people having to search around the web like I did. Thanks for the tip about UNFS. I'll have a look at it, as it may be useful when we build our larger Kerrighed cluster in a couple of weeks. :D
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
xingmu
Thanks bigjimjams! I actually got the cluster up and running right after I saw your note about the "no running cluster" bug in krgadm. I ended using a package DRBL which configures the dhcp-server, tftp, etc. automatically. I am thinking it might be useful to add this to your tutorial as an alternative installation method.
On another note and somewhat off-topic, has anyone tried installing Kerrighed straight from the SVN? I see that above-mentioned bug has already been fixed in the SVN. I am also having troubles with processes not migrating to multi-core CPUs. I am hoping newer revisions might have fixed these problems. If so, are there any suggested revision numbers to use? Or is every revision supposed to be a working version?
Hi xingmu, I did see the DRBL option on the Kerrighed documentation webpage, but preferred to do it by hand on a first attempt. I agree that it might be useful to maybe add this as an alternative option. Are there any other benefits or drawbacks?
I'm planning on using the SVN version of kerrighed in a couple of weeks, as mentioned in my previous post. So I can let you know how it goes then if you haven't beaten me to it! ;) I think they've fixed a number bugs since 2.3, from looking at the responses on the mailing list.
As for you multi-core cpu problem, that seems quite strange, as I did plug an intel quad into my kerrighed test cluster and it worked fine with process migration. Was it using a 32-bit or 64-bit kernel?
-
Re: Easy Ubuntu Clustering
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
bigjimjams
Hi xingmu, I did see the DRBL option on the Kerrighed documentation webpage, but preferred to do it by hand on a first attempt. I agree that it might be useful to maybe add this as an alternative option. Are there any other benefits or drawbacks?
I'm planning on using the SVN version of kerrighed in a couple of weeks, as mentioned in my previous post. So I can let you know how it goes then if you haven't beaten me to it! ;) I think they've fixed a number bugs since 2.3, from looking at the responses on the mailing list.
As for you multi-core cpu problem, that seems quite strange, as I did plug an intel quad into my kerrighed test cluster and it worked fine with process migration. Was it using a 32-bit or 64-bit kernel?
Ok, I've tried some of the SVN revisions now. I can confirm that r4762 and r5069 (latest) are working versions. They have "sort of" fixed the cluster status bug. I say "sort of" because krgadm cluster status outputs "0:1". My session_id is 1, but I can't understand what the "0" means (nb_min?). My node_id's are from 1 to 3, so 0 is not a node_id.
Also, auto-migration in the newer revisions doesn't work without setting up a scheduler (http://www.kerrighed.org/wiki/index.php/SchedConfig). I got that part figured out, but I *still* cannot get processes to properly use my dual-core CPUs. I am wondering if there are options I need to change when building the kernel?
BTW, I am using a 32-bit kernel, although the chips are 64-bit Xeon's.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
bigjimjams
No problem Tony, happy to help out! I was creating the guide for documenting it at work anyway, so thought I'd share what I'd done to save other people having to search around the web like I did. Thanks for the tip about UNFS. I'll have a look at it, as it may be useful when we build our larger Kerrighed cluster in a couple of weeks. :D
Hello, bigjimjams.
The '3' in UNFS3 means NFS version 3, so you have to be tell the Kerrighed kernel to look for an NFSROOT on using the NFSv3 protocol because the default is NFSv2. This is how I do it:
Code:
default linux
label linux
kernel vmlinuz-kerrighed
append root=/dev/nfs ip=dhcp nfsroot=192.168.0.254:/NFSROOT,v3 node_id=65 session_id=1
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
xingmu
Ok, I've tried some of the SVN revisions now. I can confirm that r4762 and r5069 (latest) are working versions. They have "sort of" fixed the cluster status bug. I say "sort of" because krgadm cluster status outputs "0:1". My session_id is 1, but I can't understand what the "0" means (nb_min?). My node_id's are from 1 to 3, so 0 is not a node_id.
Also, auto-migration in the newer revisions doesn't work without setting up a scheduler (
http://www.kerrighed.org/wiki/index.php/SchedConfig). I got that part figured out, but I *still* cannot get processes to properly use my dual-core CPUs. I am wondering if there are options I need to change when building the kernel?
BTW, I am using a 32-bit kernel, although the chips are 64-bit Xeon's.
Hi xingmu, the "0" in "0:1" is normally the node id, as far as I know the manual assignment of node id's in 2.3 was broken and the node id's were automatically assigned from the ip address of each node, so for example a node with ip address 192.168.1.0 would automatically be assigned node id 0.
As I mentioned earlier, I used the 32-bit kernel on a intel Q6600 with the other nodes being AthlonXPs and it seemed to work fine with the process migration stuff, as all cores hit 100% usage. Therefore, I'm guessing its something to do with the kernel settings.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
ajt
Hello, bigjimjams.
The '3' in UNFS3 means NFS version 3, so you have to be tell the Kerrighed kernel to look for an NFSROOT on using the NFSv3 protocol because the default is NFSv2. This is how I do it:
Code:
default linux
label linux
kernel vmlinuz-kerrighed
append root=/dev/nfs ip=dhcp nfsroot=192.168.0.254:/NFSROOT,v3 node_id=65 session_id=1
Bye,
Tony.
Thanks Tony, I guessed it was to do with NFS version 3, but the code snippet helps. :D
-
Re: Easy Ubuntu Clustering
Hi BigJimJams,
Thanks for the great guide!
I did notice a few problems with the guide however...:
- The guide doesn't show how to set up /etc/network/interfaces on the server
- In /etc/default/dhcp3-server your missing a semi-colon at the end of the third to last line
- In /etc/default/tftpd-hpa RUN_DAEMON should equal "yes" (all lowercase), mine was yelling at me without this (I was using debian so maybe that's why).
Just as a side note, I was having trouble at the tftp stage. DHCP assigned an IP address ok, but I recieved the "PXE-E32" error. This only happened on two of the three computers I was using.
So as this source (http://www.mail-archive.com/ltsp-dis.../msg32044.html) suggests I tried booting from a pxe cd. I used GPXE.
I went to their website (http://kernel.org/pub/software/utils/boot/gpxe/) and downloaded the latest version. Then I untarred it (tar -xzf gpxe-*), then I changed directories into the new folder (cd gpxe-*).
Then I changed directory into src (cd src) then ran "make bin/gpxe.iso" Then the iso was in "bin/gpxe.iso". I was able to boot from this which resolved the problem.
-
Re: Easy Ubuntu Clustering
Hello everyone, My name is Rob and I am new to clustering and I am looking to find some people to chat with that can help me learn about clustering. Thanks to the help of ajt I am in the right place.
I have got a cluster put together using the idea from Joel Adams and their project Microwulf.
I thought this Idea would be a good way to start.
First off Let me just say that I am not an expert Linux guru but I do know the basics and I have been playing around for quite some time using various Linux OS systems; however, I am new to Ubuntu.
I am also very serious about getting this project done. I have already spent a good amount of money building this cluster and now I want to get it up and running.
One other thing I also liked the tutorial on kerrighed. I would be interested in learning more if anyone would care to help.
Thanks everyone!
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
robasc
Hello everyone, My name is Rob and I am new to clustering and I am looking to find some people to chat with that can help me learn about clustering. Thanks to the help of ajt I am in the right place.
I have got a cluster put together using the idea from Joel Adams and their project Microwulf.
I thought this Idea would be a good way to start.
First off Let me just say that I am not an expert Linux guru but I do know the basics and I have been playing around for quite some time using various Linux OS systems; however, I am new to Ubuntu.
I am also very serious about getting this project done. I have already spent a good amount of money building this cluster and now I want to get it up and running.
One other thing I also liked the tutorial on kerrighed. I would be interested in learning more if anyone would care to help.
Thanks everyone!
Hi robasc, I'm happy to help (if I can). What sort of things do you need help with? It might be worth posting your questions on here.
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
robasc
[...]
I am also very serious about getting this project done. I have already spent a good amount of money building this cluster and now I want to get it up and running.
One other thing I also liked the tutorial on kerrighed. I would be interested in learning more if anyone would care to help.
Thanks everyone!
Hello, robasc.
Thanks for joining this thread :-)
I'm interested in SSI (Single System Image). This is like SMP (Symmetric Multi Processing), but the interconnect between the CPU's spans different computer systems rather than just the memory bus or hyper transport used in multi CPU systems. In COTS (Commodity Off The Shelf) Beowulf clusters the interconnect is usually 100-BaseT or Gigabit ethernet.
In the recent past, three SSI projects were popular: OpenSSI, openMosix and Kerrighed. The OpenSSI project looks good, but seems to be dormant because the code releases are quite old, and the openMosix project is now closed down. That leaves Kerrighed, which is being actively developed by an enthusiastic team:
http://www.kerrighed.org
I run a 90-node openMosix Beowulf, which we are hoping to upgrade to Kerrighed as soon as we can get it to run long enough to use ;-)
There are other ways of using a cluster, but the idea of a 'type 2' Beowulf class machine is that it has a distributed process space and automatically balances the load on different nodes. Older 'type 1' Beowulf clusters use a DRM (Distributed Resource Manager) like SGE (Sun Grid Engine) to balance the load and schedule jobs. Both types of Beowulf are used to run MPI (Message Passing Interface) programs to split the load of a single job over multiple nodes in order to use the aggregate performance of the cluster. SSI is best for 'embarrasingly' parallel computation. A good starting point to learn is ClusterMonkey:
http://www.clustermonkey.net/
Bye,
Tony.
-
Re: Easy Ubuntu Clustering
Thanks for the info ajt. I will be checking it out for sure.
Meanwhile, I have got something here that I need some of you to look over. I decided to go with a kerrighed setup using BigJimJams instructions; however, I was unsuccessful an getting the diskless nodes to boot up. I think you guys could probably help me to find my problems.
Ok, here is every thing wrapped up in a webpage that I have done from beginning to end which includes where I am with the project and what problems I am having. I know the page is lengthy, but I did not want to leave out anything on my setup.Please let me know what I am doing wrong and where I need to start.
Ok, here is the website to got to:
KerrighedCluster.htm
-
Re: Easy Ubuntu Clustering
Quote:
Originally Posted by
robasc
Thanks for the info ajt. I will be checking it out for sure.
[...]
Ok, here is every thing wrapped up in a webpage that I have done from beginning to end which includes where I am with the project and what problems I am having. I know the page is lengthy, but I did not want to leave out anything on my setup.Please let me know what I am doing wrong and where I need to start.
Ok, here is the website to got to:
KerrighedCluster.htm
Hello, robasc.
That's a bit of a long read!
The server 192.168.1.128 is excluded from your network:
192.168.1.128/25 = 192.168.1.0/255.255.255.128
The lowest address in this network is 192.168.1.129
Why do you not just use:
192.168.1.0/24 = 192.168.1.0/255.255.255.0
Bye,
Tony.