Ok, in Ubuntu (and many other distros as well) installation and update of programs are managed centrally in software repositories so that you don't have to go on random sites to download installers like in Windows.
In Ubuntu, you can access these repositories through several means : the Software center, Synaptic package manager (you have to install this yourself if you are using Ubuntu 11.10) and the good old command line. So the repos are like Apple's app store, except all the software is free.
Besides the official repositories maintained by Ubuntu, you can add many ppas (personal program archives, something like that. ) if you want programs not available from the official repos, or want versions which are more up to date. Since you appear to be quite new I will skip this, you will find out later as you learn more.
Other than installing through repositories (official or ppas) You can also download installers for some programs to do right click install like in Windows, but they have to have the .deb extension ( or .rpm sometimes, you can convert some rpms to debs with a tool called alien and then install the debs, it may not work sometimes though, but I will skip that), they are kind of like the .exe files for installation in windows. I would suggest ppas over downloading .debs if ppas are available because once you add a ppa it is managed by the software managing system like programs in the official repos and you will get regular upgrades, whereas sofwtare installed with .deb are static.
Finally you can compile software from source, which is what you are trying to do here. It is a good learning experience and for bleeding edge features but not necessary for most things.
Ok, now to install from the repositories, you can always go to the Software center and search for the package (say Octave) and then just click. The SC is very nice looking but it is slow and buggy and it doesn't have many options so I would recommend synaptic instead. Like I said, in 11.10 synaptic is not installed by default so you have to install it first. You can do it with the software center, or simply with command line. Open the terminal and type
Now you have installed synaptic you can find it in the dash. Every time when you start synaptic you should click the "Reload" button to update repo information. The advantage of synaptic over the command line is you can browse applications by categories and install programs even when you don't know the exact names. In synaptic you can find the base octave package and other tool boxes, just click to install whatever you want (see screenshot) If you decide that you don't want some tool boxes later you can just click to uninstall them. It is that simple, no need to compile anything.
sudo apt-get install synaptic
Hope that helps.