Making Ubuntu Fast using RAM (updated and simplified)

[terminator14]

Sorry to say, but the original "solution" is overcomplicated to simply give the appearance of a faster system. The system is not really any faster since reading the applications and libraries must be read from disk at some point. And it is prone to too many potential problems.

I don't think that we are going to outsmart the linux file caching.

I actually liked the "preloading" the file cache idea. That one is much more natural, simple and does not have any side-effects.

You are right - one of the biggest problems with the script is its complexity and its potential for problems. I tried to overcome most of the problems I came across (like when I first boot a RAM Session, the internet wouldn't work) but there may be others I am not aware of that users of the script may run into (some problems more serious than others). Also, since I'm not an actual linux developer, I am not entirely sure how this script effects the OS under the hood. That means all I can do is fix problems that arise and have limited ability to predict which problems may show themselves in the future.

I still think that the solution might be useful to people with extremely slow hard drives however, since the OS would load everything in one go during boot, and work very quickly afterwards, instead of being slow throughout (until enough data got cached to speed up the OS that is). And despite the potential for problems, I haven't run into too many since the initial batch of problems I ran into (which the script already addresses).

I like the preloading idea too, specifically for its simplicity, and hopefully someone can help us try it out.

[Slim Odds]

Don't get me wrong. I appreciate your effort. But as you've learned, there are several ways to approach a problem and in this case I think that just preloading is much simpler and less error prone. Especially since it takes advantage of the existing file caching (which we all know works very well).

[v5ar]

Don't get me wrong. I appreciate your effort. But as you've learned, there are several ways to approach a problem and in this case I think that just preloading is much simpler and less error prone. Especially since it takes advantage of the existing file caching (which we all know works very well).

yes, but those ways solve different problems

preloading idea works well for whats intended, you preload your application to memory and they are opened faster, but when that application reads/writes data to hard drive it doesnt have any effect on speed of that reading and writing, your hard drive has,
application loading is faster, because your application is already loaded but actually the biggest speed difference you gain or loose is your hard drive speed, because every millisecond some application is writing something to it, hence terminator14 mention having SSD makes your system really fast (and it should)

thats where implementation terminator14 posted comes in, ( and its concept that puppy uses)
everything is in memory, your application, your system, your partition (script gives you choice)
whatever your do, its done in memory, and no preloading can done that, unless you preload everything, and thats how we come to this solution, and still loading squash should be faster since its smaller than preloading everything as it is from your media

boot will be slower, since you have to load everything in memory, and no matter what solution you choose, youll always have to load everything if you want to load everthing, so time spent reading from hard drive is always the same, and the conclusion is that the only simple implementation of that can be made with terminator14s script

ive been thinking about this ide for some time, and since im programmer, thought i could approach this by making utility rather than script but terminator14s idea and implementation is really great and since i have no time maybe i could rather point what would suit better for some of us

in live mode puppy loads everthing to memory and then when you end your work, you can save everything by making squash from that sytem and then that is loaded next time, and again, when you end your work, its saved again, etc so you actully have normal system which runs very fast, no need for separation of /home or anything else obviosly

so your load everything to ram (which takes time) and you save everything to media (which takes time) but your system works really fast when your running it, obviously this is for those who want THAT!

so what i would like is just that, load your system, ok, then u make squash from running system (script does that? will try it after work), and then you run it from memory, and, again, when you want to shutdown, you run that script to make squash from currently running filesytem, which is in memory, but you save squash to your media, and at the next boot, that new squash is loaded to ram and so on

ill try it today, and if im right chroot and sudo rupdate do just that?

anyway, that solution should end up being very simple and error prone(you make squash, mount and load it to memory, use it and then recreate squash from that fileystem once again and use it next)
i think ill try to do that if i find time, just need to see more about squashfs and unionfs

or just to ask, what would happen if i run that script in already running system from memory, where would that squash end up, it seems to me that your script is almost able to do what i want and before i can try for myself why not asking

PS
i use word media rather than hard drive since im doing all this on usb, and its painfully slow having your system on usb (unless its really fast usb, which it isnt)

[terminator14]

so what i would like is just that, load your system, ok, then u make squash from running system (script does that? will try it after work), and then you run it from memory, and, again, when you want to shutdown, you run that script to make squash from currently running filesytem, which is in memory, but you save squash to your media, and at the next boot, that new squash is loaded to ram and so on

ill try it today, and if im right chroot and sudo rupdate do just that?

That's kind of how it works, except the script is NOT run on every shutdown from within the RAM Session. This is because creating a new squashfs image takes a while, and unless you actually changed something (either downloaded updates, or change some settings in /etc), there's no need to recreate the squashfs image every time.

When you do need to update, rupdate does that. It updates the /var/squashfs on your main OS (/var/squashfs is the folder that stores a copy of your main OS) and then rupdate recreates the filesystem.squashfs image so that next time you reboot, and pick the RAM Session from the grub menu, you'll boot into an updated OS.

chroot lets you change settings in /var/squashfs (like /etc/hosts or something), and after running "sudo rupdate -f", a new filesystem.squashfs file will be created with those settings saved.

what would happen if i run that script in already running system from memory

The script has a safeguard to make sure it doesn't run from within the RAM session. If you remove a few lines in the script (remove the safeguard) and ran it from the RAM Session anyway, I imagine it would treat the RAM Session as a regular system and try to copy it to /var/squashfs. If it was successful, it would create a /live/filesystem.squashfs and change your grub to point to it. The problem with that is:

1. Since your /var and /live (which would be created) would be in your RAM, you would run out of RAM very quickly.

2. Even if you had like 12GB of RAM and it completed, as soon as you reboot, the /var/squashfs and /live inside the RAM Session would disappear, which would make grub have an entry pointing to a non-existent squashfs file - that is assuming it would not get any errors trying to create a grub entry for a file located in RAM in the first place.

If you were to modify the script however so that if it is run from the RAM Session, it would use rsync to sync up the RAM Session with /var/squashfs, leave grub alone, and recreate the filesystem.squashfs file, placing it where the old file was, I suppose it could work the way you want.

[Stason]

You could accomplish the same result by simply creating a list of applications that you want to be "first time fast" and just read them from disk in rc.local during startup.

Could you please elaborate on how to list needed programs in rc.local?

[Slim Odds]

Could you please elaborate on how to list needed programs in rc.local?

I'm not really sure. You can make your own list of applications that you use most. Then you can use ldd to determine what libraries it uses. That would be a start.

terminator had a script earlier in this thread that seems to be a good start.

[davek06]

Hi, I have maybe a dumb question, but, if my root partition has 7 GB of data and I have separated /home, does it mean, that I need to have at least 7 GB of RAM ?? And that every time I would like to boot to RAM it would copy 7 GB of data? thx for answer

D.

[terminator14]

Hi, I have maybe a dumb question, but, if my root partition has 7 GB of data and I have separated /home, does it mean, that I need to have at least 7 GB of RAM ?? And that every time I would like to boot to RAM it would copy 7 GB of data? thx for answer

D.

Wow 7GB on the root partition? Really? That sounds unusually large to me. Are you sure you aren't including /home in that? Try:

df -h /

if you are unsure. But if you are sure, then ya, I suppose in that case, you would need 7GB of RAM + another few hundred MB for the OS to work with (at least).

If you know where most of that 7GB is located (like if it's in /root or something), you could try modifying the script to mount that particular folder somewhere else at boot, or modify the RAM Session's /etc/fstab to do so after the script has finished running (modify /var/squashfs/etc/fstab to point to a device for that folder, then reconstruct the squashfs image), exactly like the script does with /home, but as it stands now, the script isn't written to handle something like that.

If you were to run the script, you could see just how large your /live/filesystem.squashfs image had become. I haven't used the script since I've written it (have been running my OS from an SSD instead), so I don't remember if squashfs images do any kind of compression, and what the compression ratio might be. The best way to check is just to run it. If it's too large, and you can't use it, run the uninstall script and it will be deleted.

[TheeMahn2003]

Wow 7GB on the root partition? Really? That sounds unusually large to me. Are you sure you aren't including /home in that? Try:

df -h /

if you are unsure. But if you are sure, then ya, I suppose in that case, you would need 7GB of RAM + another few hundred MB for the OS to work with (at least).

If you know where most of that 7GB is located (like if it's in /root or something), you could try modifying the script to mount that particular folder somewhere else at boot, or modify the RAM Session's /etc/fstab to do so after the script has finished running (modify /var/squashfs/etc/fstab to point to a device for that folder, then reconstruct the squashfs image), exactly like the script does with /home, but as it stands now, the script isn't written to handle something like that.

If you were to run the script, you could see just how large your /live/filesystem.squashfs image had become. I haven't used the script since I've written it (have been running my OS from an SSD instead), so I don't remember if squashfs images do any kind of compression, and what the compression ratio might be. The best way to check is just to run it. If it's too large, and you can't use it, run the uninstall script and it will be deleted.

The squash file system is substantially reduced in size by compression using Ultimate Edition 2.6.1 as an example. The filesystem.squashfs is 2.4 GB (2,620,391,424 bytes) the true filesystem size is: 7,699,632,128 bytes. I do not know if you have seen my screenshots comparing a fairly fast ssd and "toram". I imagine the minimum amount of ram to do as I am doing to be 4GB, maybe 3GB (loading 2.4 GB squash to ram and leaving enough ram to use it effectively to load applications into memory)

I have been giving this real thought in my case having 8GB of ram. To have it fully boot up create another ram-drive say a 1/2 GB and copying my home folder to it then un-mounting the old hard disk based home folder pointing it to the newly apointed ram-disk, on shutdown reversing the process.

I notice alot of load time being from say firefox and cache being pulled from hard disk. It was really noticeable when I set my home folder to a USB stick.

I went back changed the fstab to what was the original boot drive, re-built squash and sudo cp -a /home/* / I now I have 2 separate home folders depending which kernel I select a regular boot uses my regular home folder and boot to ram using /theemahn/ I noticed a considerable speed gain over the 33MB/s of the USB stick, but 100MB/s v/s 2.1GB/s would be the bomb. It would slow down the boot-up and shutdown process, but I typically only boot every few days.

If I have time to look into / write a script to automate this. I will be more then happy to share it with you fellas.

If you would like to see what I refer to in compression this is a few of the command I use to build Ultimate Edition:

#calculate disksize or ubiquity will be inaccurate
printf $(sudo du -sx --block-size=1 edit | cut -f1) > extract-cd/casper/filesystem.size

In your scripts case would be:
printf $(sudo du -sx --block-size=1 /var/squashfs/ | cut -f1) > /tmp/squashfs.size
cat /tmp/squashfs.size #this will display the size of the contents of what will be the virtual disk this we honesty do not care about.
rm /tmp/squashfs.size #cleanup

To obtain what will be the size of the ramdisk required would be a formula size of "results" + say 512mb (maybe a little less)
printf $(sudo du -sx --block-size=1 /live/filesystem.squashfs | cut -f1) > /tmp/ram.size
cat /tmp/ram.size
rm /tmp/ram.size

It would be quite simple instead of cat to assign as variables and calculate.

I just thought I would try and help out. I am sorry to say this, but I am rarely on this board. If you have any questions or need assistance it is probably more likely that you will get a response on our forum see my sig.

I have started testing speed, I have introduced a 500mb ramdisk, I eventually want to populate with my home to test all scenarios. I am afraid this may involve me re-writing sections of casper the friendly ghost.

theemahn@SledgeHammer:~$ df -h
Filesystem Size Used Avail Use% Mounted on
aufs 4.0G 20M 3.9G 1% /
none 4.0G 320K 4.0G 1% /dev
/dev/shm 2.5G 2.5G 56K 100% /live/image
tmpfs 4.0G 20M 3.9G 1% /live/cow
tmpfs 4.0G 0 4.0G 0% /live
none 4.0G 20M 3.9G 1% /var/lib/ureadahead/debugfs
none 4.0G 268K 4.0G 1% /dev/shm
tmpfs 4.0G 12K 4.0G 1% /tmp
none 4.0G 320K 4.0G 1% /var/run
none 4.0G 0 4.0G 0% /var/lock
none 4.0G 0 4.0G 0% /lib/init/rw
/dev/sda2 75G 68G 7.0G 91% /media/Winderz
/dev/sdb1 100G 23G 72G 24% /home
/dev/sdb2 831G 816G 15G 99% /media/TeraByte
/ramdisk 500M 0 500M 0% /ramdisk
/dev/sdc1 466G 443G 24G 95% /media/Tuneage
/dev/sdd1 3.8G 8.0K 3.8G 1% /media/008F-0EEB
/dev/sde1 466G 316G 151G 68% /media/Storage
theemahn@SledgeHammer:~$ nano /etc/fstab


I opened firefox and 20 tabs (huge difference in load as apposed to 1 or blank) with various webpages & began.

Cold start not using ramdrive:
#clear the cache
theemahn@SledgeHammer:~$ sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
theemahn@SledgeHammer:~$/usr/bin/time firefox -P default
5.47user 0.44system 0:09.22elapsed 64%CPU (0avgtext+0avgdata 745648maxresident)k
24inputs+25904outputs (0major+113499minor)pagefaults 0swaps

Cached not using ramdrive:
5.27user 1.33system 0:10.70elapsed 61%CPU (0avgtext+0avgdata 705824maxresident)k
95886inputs+27472outputs (399major+110080minor)pagefaults 0swaps

Very little gain, barely over 1 sec.

If I have time will come back and post results when the entire O/S including home is stored in ram. After watching the thumb drive light flash for things as simple as key-presses I believe the home folder to be a key element & noticeable difference. Bootchart even after loading the ISO to ram had some large spikes as it was accessing the home folder on USB, further leading me to believe this theory.

[terminator14]

The squash file system is substantially reduced in size by compression using Ultimate Edition 2.6.1 as an example. The filesystem.squashfs is 2.4 GB (2,620,391,424 bytes) the true filesystem size is: 7,699,632,128 bytes. I do not know if you have seen my screenshots comparing a fairly fast ssd and "toram". I imagine the minimum amount of ram to do as I am doing to be 4GB, maybe 3GB (loading 2.4 GB squash to ram and leaving enough ram to use it effectively to load applications into memory)

So squashfs does do compression, and a considerable amount apparently. Interesting - thanks for the info. And yes, I have seen your screenshots - and they're almost unbelievable, thanks for those as well.

I have been giving this real thought in my case having 8GB of ram. To have it fully boot up create another ram-drive say a 1/2 GB and copying my home folder to it then un-mounting the old hard disk based home folder pointing it to the newly apointed ram-disk, on shutdown reversing the process.

I notice alot of load time being from say firefox and cache being pulled from hard disk. It was really noticeable when I set my home folder to a USB stick.

I went back changed the fstab to what was the original boot drive, re-built squash and sudo cp -a /home/* / I now I have 2 separate home folders depending which kernel I select a regular boot uses my regular home folder and boot to ram using /theemahn/ I noticed a considerable speed gain over the 33MB/s of the USB stick, but 100MB/s v/s 2.1GB/s would be the bomb. It would slow down the boot-up and shutdown process, but I typically only boot every few days.

If I have time to look into / write a script to automate this. I will be more then happy to share it with you fellas.

A slightly more dangerous approach to keeping your /home past a reboot, as a power outage would revert you back to the last version of your /home, but it should speed up the OS even more, and by the sound of it, considerably so. Interesting - especially since the use of a UPS and the stability of Linux makes this approach not all that much more "dangerous". If you make a modification like that (I'm assuming you'll start with my script rather than from scratch?) I'll be glad to either upload the file on this thread and give you credit, or link to your thread if you chose to make one. Even if you choose to start from scratch (which I don't see a reason for unless you want to do a lot more than just this feature), the projects sound similar enough that I'll link to your thread or wherever you upload it (if you permit me to do so).

If you would like to see what I refer to in compression this is a few of the command I use to build Ultimate Edition:

#calculate disksize or ubiquity will be inaccurate
printf $(sudo du -sx --block-size=1 edit | cut -f1) > extract-cd/casper/filesystem.size

In your scripts case would be:
printf $(sudo du -sx --block-size=1 /var/squashfs/ | cut -f1) > /tmp/squashfs.size
cat /tmp/squashfs.size #this will display the size of the contents of what will be the virtual disk this we honesty do not care about.
rm /tmp/squashfs.size #cleanup

To obtain what will be the size of the ramdisk required would be a formula size of "results" + say 512mb (maybe a little less)
printf $(sudo du -sx --block-size=1 /live/filesystem.squashfs | cut -f1) > /tmp/ram.size
cat /tmp/ram.size
rm /tmp/ram.size

It would be quite simple instead of cat to assign as variables and calculate.

I just thought I would try and help out.

It would be nice if people could run a few, or one line in bash to determine if they have enough RAM for the ram session before ever running the script.

The problem with:

sudo du -sx --block-size=1 /live/filesystem.squashfs  | cut -f1

is that the file /live/filesystem.squashfs is created during the very last stage of the script, so it can't be accessed until the script is done running.

sudo du -sx --block-size=1 /var/squashfs/ | cut -f1

on the other hand can be used before the script runs with a slight modification (since /var/squashfs isn't created until about half way through the script):

sudo du -hcs --one-file-system / --exclude=/dev/* --exclude=/tmp/* --exclude=/sys/* --exclude=/proc/* --exclude=/home/*

which should, in theory, show how much space you need for your RAM session, though I'm not 100% sure about its accuracy. You would also need to add the extra 512MB or w/e for the system to use for regular operations.

I am sorry to say this, but I am rarely on this board. If you have any questions or need assistance it is probably more likely that you will get a response on our forum see my sig.

I have started testing speed, I have introduced a 500mb ramdisk, I eventually want to populate with my home to test all scenarios. I am afraid this may involve me re-writing sections of casper the friendly ghost.

theemahn@SledgeHammer:~$ df -h
Filesystem Size Used Avail Use% Mounted on
aufs 4.0G 20M 3.9G 1% /
none 4.0G 320K 4.0G 1% /dev
/dev/shm 2.5G 2.5G 56K 100% /live/image
tmpfs 4.0G 20M 3.9G 1% /live/cow
tmpfs 4.0G 0 4.0G 0% /live
none 4.0G 20M 3.9G 1% /var/lib/ureadahead/debugfs
none 4.0G 268K 4.0G 1% /dev/shm
tmpfs 4.0G 12K 4.0G 1% /tmp
none 4.0G 320K 4.0G 1% /var/run
none 4.0G 0 4.0G 0% /var/lock
none 4.0G 0 4.0G 0% /lib/init/rw
/dev/sda2 75G 68G 7.0G 91% /media/Winderz
/dev/sdb1 100G 23G 72G 24% /home
/dev/sdb2 831G 816G 15G 99% /media/TeraByte
/ramdisk 500M 0 500M 0% /ramdisk
/dev/sdc1 466G 443G 24G 95% /media/Tuneage
/dev/sdd1 3.8G 8.0K 3.8G 1% /media/008F-0EEB
/dev/sde1 466G 316G 151G 68% /media/Storage
theemahn@SledgeHammer:~$ nano /etc/fstab


I opened firefox and 20 tabs (huge difference in load as apposed to 1 or blank) with various webpages & began.

Cold start not using ramdrive:
#clear the cache
theemahn@SledgeHammer:~$ sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
theemahn@SledgeHammer:~$/usr/bin/time firefox -P default
5.47user 0.44system 0:09.22elapsed 64%CPU (0avgtext+0avgdata 745648maxresident)k
24inputs+25904outputs (0major+113499minor)pagefaults 0swaps

Cached not using ramdrive:
5.27user 1.33system 0:10.70elapsed 61%CPU (0avgtext+0avgdata 705824maxresident)k
95886inputs+27472outputs (399major+110080minor)pagefaults 0swaps

Very little gain, barely over 1 sec.

If I have time will come back and post results when the entire O/S including home is stored in ram. After watching the thumb drive light flash for things as simple as key-presses I believe the home folder to be a key element & noticeable difference. Bootchart even after loading the ISO to ram had some large spikes as it was accessing the home folder on USB, further leading me to believe this theory.

I would have thought caching would have had a bigger impact. I would also be curious to know how long the same test would take in a ram session on the same machine.