JohnLM_the_Ghost
May 9th, 2009, 09:53 AM
Hello!
On first note I should say that this ain't exactly actual programming questions and ideas... not yet at least. These are rather abstract ideas and concepts for potentially upcoming software.
So after buying my new rig and using it for a quite while, I started to wonder that it doesn't fully use it's resources (RAM specifically)... and it sometimes feels it doesn't use cache the way I wanted to...
And then I came with an idea for a cache manager software.
In short it would
Assist user and system (kernel) to better utilize cache.
Allow user to manually manage cache (with front-end)
Relieve some stress from storage devices (by using more aggresive cache rules)
Now what actual manual operations it would provide...
View cache contents (What files are cached)
On-demand file caching (for quick access later)
On-demand dumping of write cache
Edit caching rules for file transfer operations (like copying)
What good would it do?
Imagine you could make your box cache your favourite apps on startup for rapid access (like firefox or OpenOffice maybe?). The Manager should probably recursively discover associated files along with the binary.
Also you could cache very often read (but not written) files, let's say your homepage on your webserver, and likes. Of course Manager would also provide tools to free or at least demote cache priority.
Also dumping write cache is useful. For example there is quite aggressive caching rules for memory cards (and USB flash drives?) already. Often files you copy on it are not physically written on it until you try to unmount the drive and it makes unmounting quite time expensive (and dangerous for long-time Windows users, who like to remove the cards without unmounting). So with Manager you could manually make it dump the cache on device (and you could watch the actual progress!) and then have it unmounted swiftly whenever you feel like it.
Then I also thought something similar could be applied for hard drive. Not the unmounting part really, but more aggressive caching and queueing on file transfers. Let's say when you try to copy more than one big file at the same time, it starts to race for drive access, what definitely doesn't make it any faster, and it also leads to file fragmentation.
The Manager would queue the transfer sequentially, to make most of device's writing capacity (and reducing fragmentation on-the-way).
If the transferred files are fully cached, they also could be accessed by other programs, even before actual transfer is complete (meaning transfer should be transparent for other programs), with all operations done on cache. (or cache/actual mixture when all part of file is not cached anymore and application has it's own reading buffer). I also imagine writing operations to be available on cache, so already edited data is written without overwriting it once again (this would be a special case, cause to use most of the cache it's priority should be promoted, and transfer priority demoted, to minimize required disk access)
Anyway hard drive caching will be especially important for same-device operations. You can imagine how bigger cache buffer would benefit move and copy operations between different partitions on same disk.
Here also the rules configuration would come in. By specifying caching global rules, per-device and maybe even per-application rules you could get the most of your cache and performance.
That all said, I know the target audience will NOT be general public, but advanced computer users, cause the program will not be simple, well at least it won't be usual. Casual users hardly care about caching, but I believe for advanced users (like myself) it only benefits for cache to be revealed instead of being abstracted (this meant in visual sense not in implementation).
Now there are few thing I wanted to know.
Firstly I know my proposed program would need some interaction with kernel... is it possible to do this with current kernel, or would it require to write a module specifically for this purpose?
Secondly I want to know what is actual caching policy for current kernel, and whenever tools exist to alter them?
And also good insights, ideas, comments... maybe some critique too (I don't mind critique, just be constructive)!
I'm not the best programmer ever, but I consider start developing such program.
Also if anyone is up to take initiative for developing this (or if something like this exist) I want to be in it! :)
p.s. I might do a little mockup of front-end when I feel like it.
I might also post this idea in Ubuntu brainstorm site and/or open new Lauchpad project later on.
EDIT:
So on-demand buffer/cache dumping is already implemented by sync command. No need to reinvent that at least!
On first note I should say that this ain't exactly actual programming questions and ideas... not yet at least. These are rather abstract ideas and concepts for potentially upcoming software.
So after buying my new rig and using it for a quite while, I started to wonder that it doesn't fully use it's resources (RAM specifically)... and it sometimes feels it doesn't use cache the way I wanted to...
And then I came with an idea for a cache manager software.
In short it would
Assist user and system (kernel) to better utilize cache.
Allow user to manually manage cache (with front-end)
Relieve some stress from storage devices (by using more aggresive cache rules)
Now what actual manual operations it would provide...
View cache contents (What files are cached)
On-demand file caching (for quick access later)
On-demand dumping of write cache
Edit caching rules for file transfer operations (like copying)
What good would it do?
Imagine you could make your box cache your favourite apps on startup for rapid access (like firefox or OpenOffice maybe?). The Manager should probably recursively discover associated files along with the binary.
Also you could cache very often read (but not written) files, let's say your homepage on your webserver, and likes. Of course Manager would also provide tools to free or at least demote cache priority.
Also dumping write cache is useful. For example there is quite aggressive caching rules for memory cards (and USB flash drives?) already. Often files you copy on it are not physically written on it until you try to unmount the drive and it makes unmounting quite time expensive (and dangerous for long-time Windows users, who like to remove the cards without unmounting). So with Manager you could manually make it dump the cache on device (and you could watch the actual progress!) and then have it unmounted swiftly whenever you feel like it.
Then I also thought something similar could be applied for hard drive. Not the unmounting part really, but more aggressive caching and queueing on file transfers. Let's say when you try to copy more than one big file at the same time, it starts to race for drive access, what definitely doesn't make it any faster, and it also leads to file fragmentation.
The Manager would queue the transfer sequentially, to make most of device's writing capacity (and reducing fragmentation on-the-way).
If the transferred files are fully cached, they also could be accessed by other programs, even before actual transfer is complete (meaning transfer should be transparent for other programs), with all operations done on cache. (or cache/actual mixture when all part of file is not cached anymore and application has it's own reading buffer). I also imagine writing operations to be available on cache, so already edited data is written without overwriting it once again (this would be a special case, cause to use most of the cache it's priority should be promoted, and transfer priority demoted, to minimize required disk access)
Anyway hard drive caching will be especially important for same-device operations. You can imagine how bigger cache buffer would benefit move and copy operations between different partitions on same disk.
Here also the rules configuration would come in. By specifying caching global rules, per-device and maybe even per-application rules you could get the most of your cache and performance.
That all said, I know the target audience will NOT be general public, but advanced computer users, cause the program will not be simple, well at least it won't be usual. Casual users hardly care about caching, but I believe for advanced users (like myself) it only benefits for cache to be revealed instead of being abstracted (this meant in visual sense not in implementation).
Now there are few thing I wanted to know.
Firstly I know my proposed program would need some interaction with kernel... is it possible to do this with current kernel, or would it require to write a module specifically for this purpose?
Secondly I want to know what is actual caching policy for current kernel, and whenever tools exist to alter them?
And also good insights, ideas, comments... maybe some critique too (I don't mind critique, just be constructive)!
I'm not the best programmer ever, but I consider start developing such program.
Also if anyone is up to take initiative for developing this (or if something like this exist) I want to be in it! :)
p.s. I might do a little mockup of front-end when I feel like it.
I might also post this idea in Ubuntu brainstorm site and/or open new Lauchpad project later on.
EDIT:
So on-demand buffer/cache dumping is already implemented by sync command. No need to reinvent that at least!